Surviving A Slashdotting With a Celeron 466: My Slashdot Experience

06 Jul 2005

On Sunday, March 6th of 2005 I was messing around on my workstation retagging some Mp3’s so I could add them into my iPod playlists when I got a GMail notification of a new comment on my blog. I went to check it out and it was fairly basic comment “prepare to be slashdotted. I went to check-out Slashdot and didn’t see anything there, so I assumed it was likely just a spam comment. It wasn’t long after that when a link to an entry I’d wrote a couple of months previous appeared at the top of their front page.

I was actually quite shocked—being linked to from Slashdot is a very big deal in the geek community. For those not familiar with Slashdot, the best comparison I can make is if you were an amateur author and had your work published in Newsweek or something like that. Slashdot does not have nearly the readership of Newsweek, but it’s certainly one of the most trafficked international geek news sites that I know of offhand-- I’m a regular reader myself.

Once I knew for sure I was getting Slashdotted, I actually decided to get-up from the computer and go do something else while I thought about how I could prepare for a load like that rather than freaking out and breaking things. My server at the time was a temporary server built out of spare parts (I host my site at home) which was over 6 years old, so I had no idea what I'd be able to do with it. After remembering articles several years ago about something to the effect of “A Pentium 90 can easily put out enough bandwidth to saturate a T1�?, I kept that in mind and started to think of ways to eliminate bottlenecks. I eventually came back, saw that my server was non-responsive via HTTP. I then blocked port 80 at the firewall, and went to work.

I was not a bit surprised the server was non-responsive; my site at the time was running on a Celeron 466 with 192 MB of RAM. I also of course used Apache HTTPD Server, as well as PHP5 and MySQL to power my Wordpress blog in which the entry was in that was being linked to.

My first course of action was to figure-out how to make the page static so that it wouldn’t have to hit MySQL each time someone requested that page. I assumed there was probably a Wordpress plug-in to do this with, so I googled for one and found one which Matt Mullenweg himself was using, Staticize Reloaded. (Note: I've since switched to an upgraded version of the same plugin.) This was a major help, but not quite enough given that I hadn’t really touched this particular FreeBSD install much as it was temporary until my new server came in.

My second major course of action was to change the settings on Apache to keep it from dying. It’s been quite some time since all of this happened so I don’t remember the exact settings I used, but basically I upped the number of min and max servers, set a server timeout of 15 seconds, and made sure new server processes were started every 500 requests or so to keep runaway processes from killing the server.

One last thing I did was kill all unnecessary processes such as SMTP (I knew my mail would queue on external SMTP servers trying to reach mine, and I get more spam than legitimate mail anyhow), made sure nothing in cron was intensive, etc. Had bandwidth also been a problem after these changes, I planned on moving my site to the default Wordpress template which did not have any graphics—just XHTML & CSS.

From there I decided to reboot the box just to clean it up and open port 80 back-up again to accept HTTP traffic again. Surprisingly FreeBSD never once crashed the whole time even after the initial Slashdotting before I started blocking traffic.. it was just very bogged down with requests (mostly MySQL was dying). I rebooted it after changing the configuration as a precautionary measure. This was a very stock install of FreeBSD-- I didn't even make a custom kernel for it or anything.

Once I put the box back in, it took a bit more Apache tweaking to get it perfect, but eventually my spare parts Celeron 466 was handling a Slashdotting at only 30% CPU utilization! Here is an example of the number of connections it was getting (which is crazy for me to see on a personal machine) as the Slashdot effect was winding down:

Active Internet connections
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0    791  grimmace.http          puddle189.drizzl.3055  ESTABLISHED
tcp4       0  33580  grimmace.http          c-67-166-235-247.4364  ESTABLISHED
tcp4       0      0  grimmace.http          213.58.2.244.1227      ESTABLISHED
tcp4       0  32120  grimmace.http          puddle189.drizzl.3054  ESTABLISHED
tcp4       0  26018  grimmace.http          c-67-166-235-247.4363  ESTABLISHED
tcp4       0      0  grimmace.http          planetlab2.georg.55584 TIME_WAIT
tcp4       0      0  grimmace.http          c-67-170-48-141..61898 ESTABLISHED
tcp4       0      0  grimmace.http          tide505.microsof.39548 ESTABLISHED
tcp4       0      0  grimmace.http          tide532.microsof.54754 ESTABLISHED
tcp4       0      0  grimmace.http          unassigned.purec.1361  ESTABLISHED
tcp4       0      0  grimmace.http          c-67-170-48-141..61897 ESTABLISHED
tcp4       0      0  grimmace.http          tide508.microsof.30208 ESTABLISHED
tcp4       0      0  grimmace.http          tide70.microsoft.53537 ESTABLISHED
tcp4       0      0  grimmace.http          c-24-19-42-108.c.4436  ESTABLISHED
tcp4       0      0  grimmace.http          tide85.microsoft.40545 ESTABLISHED
tcp4       0      0  grimmace.http          S01060050baa51fa.33235 ESTABLISHED
tcp4       0      0  grimmace.http          c-24-19-42-108.c.4434  ESTABLISHED
tcp4       0      0  grimmace.http          S01060050baa51fa.33234 ESTABLISHED
tcp4       0      0  grimmace.http          tide85.microsoft.40500 ESTABLISHED
tcp4       0      0  grimmace.http          tide531.microsof.36377 FIN_WAIT_2
tcp4       0      0  grimmace.http          tide517.microsof.12997 FIN_WAIT_2
tcp4       0      0  grimmace.http          tide518.microsof.18692 FIN_WAIT_2
tcp4       0      0  grimmace.http          tide510.microsof.52133 ESTABLISHED
tcp4       0      0  grimmace.http          c-67-182-8-198.c.60914 FIN_WAIT_2
tcp4       0      0  grimmace.http          68-113-20-232.wa.37925 ESTABLISHED
tcp4       0      0  grimmace.http          c-67-182-8-198.c.60913 FIN_WAIT_2
tcp4       0      0  grimmace.http          68-113-20-232.wa.37924 ESTABLISHED
tcp4       0      0  grimmace.http          msnbot.msn.com.37805   TIME_WAIT
tcp4       0  12427  grimmace.http          dial-148-240-184.3124  ESTABLISHED
tcp4       0      0  grimmace.http          pool-68-163-42-6.4368  TIME_WAIT
tcp4       0      0  grimmace.http          unassigned.purec.1359  ESTABLISHED
tcp4       0  32327  grimmace.http          dial-148-240-184.3122  ESTABLISHED
tcp4       0      0  grimmace.http          relax15.idiap.ch.40978 TIME_WAIT
tcp4       0      0  grimmace.http          relax15.idiap.ch.40977 TIME_WAIT
tcp4       0      0  grimmace.http          pool-68-163-42-6.4366  TIME_WAIT
tcp4       0      0  grimmace.http          relax15.idiap.ch.40975 TIME_WAIT
tcp4       0      0  grimmace.http          24.125.148.80.1176     TIME_WAIT
tcp4       0      0  grimmace.http          213.58.2.244.1208      TIME_WAIT
tcp4       0      0  grimmace.http          CPE0050da6ab5da-.3371  TIME_WAIT
tcp4       0      0  grimmace.http          24.125.148.80.1175     TIME_WAIT
tcp4       0      0  grimmace.http          adsl-70-240-237-.1266  FIN_WAIT_2
tcp4       0      0  grimmace.http          CPE0050da6ab5da-.3370  TIME_WAIT
tcp4       0      0  grimmace.http          bbox.ne.client2..47283 TIME_WAIT
tcp4       0      0  grimmace.http          msnbot.msn.com.36671   TIME_WAIT
tcp4       0      0  grimmace.http          host-148-244-150.46757 TIME_WAIT
tcp4       0      0  grimmace.http          hannibal.lr-s.tu.50584 TIME_WAIT
tcp4       0      0  grimmace.http          rdu25-17-055.nc..54974 TIME_WAIT
tcp4       0      0  grimmace.http          163.80-202-31.ne.57984 TIME_WAIT
tcp4       0      0  grimmace.http          bbox.ne.client2..47282 TIME_WAIT
tcp4       0      0  grimmace.http          rdu25-17-055.nc..54973 TIME_WAIT
tcp4       0      0  grimmace.http          hannibal.lr-s.tu.50578 TIME_WAIT
tcp4       0      0  grimmace.http          c-67-174-111-123.3682  TIME_WAIT
tcp4       0      0  grimmace.http          c-67-174-111-123.3681  TIME_WAIT
tcp4       0      0  grimmace.http          c-67-174-111-123.3680  TIME_WAIT
tcp4       0      0  grimmace.http          relax15.idiap.ch.40952 TIME_WAIT
tcp4       0      0  grimmace.http          relax15.idiap.ch.40953 TIME_WAIT
tcp4       0      0  grimmace.http          c-67-174-111-123.3679  TIME_WAIT
tcp4       0  33304  grimmace.http          relax15.idiap.ch.40943 LAST_ACK
tcp4       0      0  grimmace.http          163.80-202-31.ne.57983 TIME_WAIT
tcp4       0  33304  grimmace.http          relax15.idiap.ch.40931 CLOSE_WAIT
tcp4       0      0  grimmace.http          ool-4570bffb.dyn.54562 TIME_WAIT
tcp4       0      0  grimmace.http          208-59-174-198.c.1771  TIME_WAIT
tcp4       0      0  grimmace.http          xdsl-81-173-248-.3833  TIME_WAIT
tcp4       0      0  grimmace.http          24-117-147-243.c.60104 TIME_WAIT
tcp4       0      0  grimmace.http          pcp658486pcs.cap.29623 TIME_WAIT
tcp4       0      0  grimmace.http          pcp658486pcs.cap.29622 TIME_WAIT
tcp4       0      0  grimmace.http          ool-4570bffb.dyn.54558 TIME_WAIT
tcp4       0      0  grimmace.http          208-59-174-198.c.1770  TIME_WAIT
tcp4       0      0  grimmace.http          24-117-147-243.c.60103 TIME_WAIT
tcp4       0      0  grimmace.http          xdsl-81-173-248-.3825  TIME_WAIT
tcp4       0      0  grimmace.http          pool-68-238-49-1.50253 TIME_WAIT
tcp4       0      0  grimmace.http          pool-68-238-49-1.50252 TIME_WAIT
tcp4       0      0  grimmace.http          pool-68-238-49-1.50251 TIME_WAIT
tcp4       0      0  grimmace.http          pool-68-238-49-1.50250 TIME_WAIT
tcp4       0  24300  grimmace.http          ip160-108.om.apa.14426 CLOSING
tcp4       0  30640  grimmace.http          ip160-108.om.apa.14403 FIN_WAIT_1
tcp4       0  33304  grimmace.http          209.237.230.104.21469  ESTABLISHED
tcp4       0  33580  grimmace.http          S010600067b078af.1423  ESTABLISHED
tcp4       0  29326  grimmace.http          bidwell.textdriv.55107 FIN_WAIT_1
tcp4       0      0  grimmace.http          c68.115.108.235..3517  FIN_WAIT_1
tcp4       0      0  grimmace.http          c68.115.108.235..3516  FIN_WAIT_1
tcp4       0      0  grimmace.http          cust1392.vic01.d.20836 FIN_WAIT_1
tcp4       0      0  grimmace.http          node-423a2a62.df.62359 FIN_WAIT_1
tcp4       0      0  grimmace.http          cust1392.vic01.d.20835 FIN_WAIT_1
tcp4       0      0  grimmace.http          node-423a2a62.df.62358 FIN_WAIT_1
tcp4       0      0  grimmace.http          66.89.66.69.ptr..57168 FIN_WAIT_1
tcp4       0      0  grimmace.http          81.104.95.80.ip..13863 FIN_WAIT_2
tcp4       0      0  grimmace.http          66.89.66.69.ptr..57166 FIN_WAIT_1
tcp4       0      0  grimmace.http          81.104.95.80.ip..13854 FIN_WAIT_1
tcp4       0      0  grimmace.http          lata228-05-c136..29431 FIN_WAIT_2
tcp4       0      0  grimmace.http          lata228-05-c136..29430 FIN_WAIT_2
tcp4       0  24300  grimmace.http          ip160-108.om.apa.14400 CLOSING
tcp4       0      0  grimmace.http          207-171-180-101..44370 FIN_WAIT_1
tcp4       0      0  grimmace.http          207-171-180-101..44302 FIN_WAIT_1
tcp4       0      0  grimmace.http          lata228-05-c136..29429 FIN_WAIT_2
tcp4       0      0  grimmace.http          lata228-05-c136..29428 FIN_WAIT_2
tcp4       0      0  grimmace.http          ausisapc102-dmz..22690 FIN_WAIT_2
tcp4       0      0  grimmace.http          h000625d8511c.ne.4303  FIN_WAIT_2
tcp4       0      0  grimmace.http          209.237.230.104.28587  FIN_WAIT_2
tcp4       0      0  grimmace.http          66.193.93.3.47306      FIN_WAIT_2
tcp4       0      0  grimmace.http          66.193.93.3.47302      FIN_WAIT_2
tcp4       0      0  grimmace.http          adsl-6-96-227.ms.46767 FIN_WAIT_2
tcp4       0  33580  grimmace.http          c-24-6-68-25.cli.1107  ESTABLISHED

Unfortunately I’d estimate I was offline for around 2 hours while I fixed the machine, etc, so there is no telling how much traffic I would of seen while my site was the top link on Slashdot, but here are a few graphical examples of what type of traffic I saw once I got the server working (I think it would have been significantly more had it not been on a Sunday afternoon.):

Usually I do around 2-2.5 GB of web traffic per month, but in one day I did over 4 GB! Granted this isn’t impressive compared to a real production environment, but not bad for an old Celeron on a DSL line. I’ve also had one of my entries end up on del.ici.ous/popular which upped my traffic to about 4 GB+ of bandwidth in a month, but it was much more steady rather than one big rush of requests all at once.

I had other stuff to do on Sunday other than sit around on the computer, but I did spend a considerable amount of time responding to comments both on Slashdot as well as my own site. Surprisingly I didn’t get any comments which I was forced to delete. I did however have some other strange experiences which I recently wrote about.

I'd say my main highlights to pass on from this experience are:

  • Content based sites are easy to scale (having previous work experience with this on a platform that ran about 200 mall websites using a single software platform on just a handful of servers, it was just another data point for said observation)

  • FreeBSD is amazing, and should get more hype though it's unfortunately shadowed by Linux (but I already knew this as well)

  • Wordpress is a great, extensible blogging platform, and can stand-up to load without breaking a sweat

  • Though it's not my language of choice to work in when doing web scripting, the combination of PHP and MySQL is amazingly fast while requiring the least amount of money and resources of other web scripting languages

  • You may not always be able to do the "academically sound" process when faced with a conflict, but you can certainly hack your way through an unexpected situation to get you by in a pinch (though hacking is toxic in the long term)

All in all it was a very interesting experience, and hopefully it will happen again sometime. The server which was on order at the time is currently serving the site, and should be able to handle much more load without me needing to touch it if it were to happen again, but we’ll see…