A Home Brewed Logging Appliance
By Jim Jagielski
As founder of an ISP and Web hosting company, as well as someone who's done a lot of consulting, I've witnessed firsthand how companies have begun to embrace network appliances. I can't think of any enterprise-level setup that doesn't contain a load balancer (something that I consider an appliance), a network cache, and at least a data storage appliance or two.
The reasons are obvious. By segmenting different requirements to dedicated equipment, you avoid interdependencies that can adversely affect performance and reliability. There is one appliance that makes logical sense and yet (as far as I know), doesn't exist: the log analyzer appliance.
Businesses often have an outside company perform their log analysis for them. To be fair, specialized firms that do detailed data mining of Web log files are important, and provide valuable insight. But with the log analysis programs available today, you can perform the more mundane analysis in-house, saving the expense of hiring outside firms for the really detailed and involved mining.
I have issues with running log analysis tools on the Web server itself. As frequent readers may know, I'm a proponent of dedicating as many server resources to the Web server as possible, for performance and security reasons. Log analysis programs, by their very nature, tend to consume lots of memory and CPU horsepower—it's hard work.
It makes no sense to tune the hardware and operating system for peak Web performance if you want it to perform the intensive analysis work as well. This is especially true if you want on-demand analysis.
