The Attack of the Killer Robot
By Lincoln D. Stein
It began at about 8 AM on June 6, 1996. A student doing Web development for the MIT Genome Center came into my office to complain that the server machine was running very slowly. I tried to log into it from my office, and discovered she was indeed right. It took nearly a minute for a login prompt to appear, and nearly two minutes to validate my password. Once logged onto the system, each command took so long to complete that it was difficult to completely assess the situation. However, I was able to determine that system load had jumped from a normal value of less than 1.0 to an astronomical 150. When I finally obtained a list of running processes ten minutes later, I discovered that all CPU time and memory was being consumed by a large number of CGI scripts running simultaneously.
I didn't wait around to ask questions. I immediately killed the server and all its CGI processes, and system load dropped back to normal. When things had settled down again, I restarted the server and all seemed to be well.
For a while, that is. About 20 minutes later, the same thing happened: high load on the system and many CGI scripts running out of control. Once again, I brought down the server; this time when I brought it up again, I changed its process limits to put a ceiling on the number of CGI scripts it could spawn. I also increased my shell's priority, so that when the problem happened again I'd be able to get commands through.
At first glance, the server-access log showed an unusual amount of activity for 8:30 AM.
