Monday I woke up to a denial of service attack on my blog. Once rebooted, the server would stay alive for five minutes before crashing so bad, only a hard reboot would help. At least, it felt like a DoS, but turns out it was just Proggit readers hammering the blog (thanks, Reg). Hammering might be too strong of a word, the most busy it got was one/two requests a second (counting WP pages, not static content).

Two pages a second is not nearly enough traffic to kill a site, so what gives? Turns out Apache was spawning too many processes, gradually eating up memory, then dipping into swap space. After several minutes of gradual creep Apache would take over 40% of the swap space. Then it got bad. In under a minute it would claim over 80% of swap space and the server would stop responding. It wouldn’t even reboot if asked to.

So I pulled the plug (figuratively, SliceHost calls it hard reboot), waited for the server to come back, quickly logged into the admin panel and disabled the WP-OpenID plugin. Problem solved.

Now it would peak with about 50 processes and still dip into the swap space, but no more than 10%. As it quiets down, memory usage falls under 0.3GB (the slice has 0.5GB), cruising along at 0.3 load with 10~15 processes.

I have no idea why WP-OpenID would make such a difference, after all, I only got four comments the entire day. So why it happened is still a mystery. But I do remember an older version of WP-OpenID messing with the blog – it spewed out error messages faster than the cron task would rotate the logs, and eventually eat up all available disk space. It just made sense to disable it first.

Commenting is still open, but for now, no OpenID on Labnotes. Meanwhile, check what your favorite toys are doing in their spare time.