I use Pingdom's free service to monitor slaptijack.com. Apparently, late Friday night, oom-killer decided that the server needed more memory and took out the MySQL server. To make matters worse, I missed the alarm from Pingdom, and slaptijack.com was down for pretty much all of Saturday. The fact that oom-killer was invoked is annoying, but more on that later.

The beauty of using Pelican rather than Wordpress (or any other database-driven content engine) is one less point of failure for the site. Obviously, I don't have all of slaptijack.com converted to Pelican yet (and perhaps never will), but at least parts of the site were up and working despite MySQL being down. If nothingelse, this incident is enough to convince me that moving to Pelican was a good idea.

OOM'ed

Honestly, this is just plain annoying. The slaptijack.com server is has 1.75 GB of RAM and runs about half a dozen web sites -- all of which are much, much smaller than slaptijack.com. The server really doesn't do anything other than run Apache and MySQL. In fairness, I haven't spent a lot of time tuning either application, but there really shouldn't be a need to.

I thought perhaps one of the sites was under attack. MySQL was OOM'ed at 23:44. I had a look to see if there was a big spike in hits. As the data below shows, there really wasn't anything interesting going on during the 23:00 hour.

$ grep 12/Feb */logs/*.1 | awk '{ print $4 }' >   | cut -d: -f2 | sort | uniq -c
    953 00
    556 01
    626 02
    981 03
    1364 04
    1928 05
    2799 06
    1391 07
    1586 08
    1300 09
    2074 10
    1173 11
    2220 12
    909 13
    982 14
    769 15
    1546 16
    757 17
    480 18
    592 19
    579 20
    625 21
    581 22
    747 23

Anyway, in the hopes of avoiding this in the future, I tweaked the Apache and MySQL server settings to reduce their memory footprint. I cut way back on the number of Apache processes running and reduced the number of MySQL connections allowed. According to mysqltuner, MySQL's max memory usage is now 165.1 MB. Of course, that doesn't take into consideration the footprint of just running MySQL server appears to be a bit over 300 MB:

$ ps auwx | grep [m]ysql
mysql     6780  0.1 27.6 1209700 480656 ?      Ssl  11:04   0:08 /usr/sbin/mysqld

And each Apache process is 70 MB to 100+ MB at the moment:

$ ps auwx | grep [a]pache
root      7717  0.0  1.5 386420 26744 ?        Ss   11:20   0:00 /usr/sbin/apache2 -k start
www-data  7721  0.0  4.3 391612 75532 ?        S    11:20   0:04 /usr/sbin/apache2 -k start
www-data  7723  0.0  4.1 390028 71968 ?        S    11:20   0:04 /usr/sbin/apache2 -k start
www-data  7724  0.0  5.2 406620 91332 ?        S    11:20   0:05 /usr/sbin/apache2 -k start
www-data  7731  0.0  4.7 389644 82796 ?        S    11:21   0:04 /usr/sbin/apache2 -k start
www-data  7732  0.1  5.8 406784 101736 ?       S    11:21   0:06 /usr/sbin/apache2 -k start
www-data  7733  0.0  4.1 390624 72436 ?        S    11:21   0:04 /usr/sbin/apache2 -k start
www-data  7806  0.0  6.2 414232 108988 ?       S    11:26   0:04 /usr/sbin/apache2 -k start
www-data  7854  0.0  4.1 390804 72556 ?        S    11:29   0:05 /usr/sbin/apache2 -k start
www-data  8679  0.0  5.1 408492 89404 ?        S    12:06   0:01 /usr/sbin/apache2 -k start
www-data  9277  0.1  4.1 391384 71948 ?        S    12:43   0:01 /usr/sbin/apache2 -k start