Beware the Microbots!
By Lincoln D. Stein
In the October 1996 issue of Web Techniques, I wrote about my Web site being attacked by a "killer robot." A large keyword-indexing robot run by the OpenText site had become trapped in the Genome Center's cgi-bin directory and was furiously thrashing about among the database-interface scripts that provide access to hundreds of megabytes of biological data. The robot's automated pounding on the server slowed things to a crawl, and finally became so bad that I had to kill and restart it several times before regaining control.
In those days, the big problem was gargantuan Web crawlers, behemoth programs from sites well-endowed with CPU power, disk-storage space, and wide network pipes. With a combination of convention, courtesy, and common sense, the united Webmasters of the world brought the situation under control with a simple system called "robot rules," whereby robots can learn a site's robot policy by downloading a text file. Robot rules have curbed the robot threat, and turned them into benevolent and even essential Net citizens.
However, a new robotic threat has appeared on the horizon: the Microbots. No towering colossi these. Individually, the Microbots are puny, insect-like creatures, but combined they form an organized army of unsurpassed strength. A platoon of Microbots can quickly overwhelm a site, topple it, and strip it to the bone. Traditional weapons won't stop them, and they're on the march; I can almost hear the pinging of their metallic legs now.<>
