Knee-Deep in Log Files
By Phil Glatz
Old server logs never die, they just slowly swallow up your disk space. Rolling them over and compressing them at regular intervals can help shrink them, but you'll eventually have to do something with them. Here's a way to offload server-access data to a SQL database, so it can be analyzed later.
Tools used include the mySQL database program, Perl, the Apache Web server, the PHP server-side scripting language, and the Analog analysis program, all of which are Open-Source programs. The scope will be to use general techniques that can be applied to other databases and Web servers. You can expand upon these basics to teach your site more about your visitors.
Summarizing Data
Apache has the ability to create multiple server-access logs. In addition to writing flat text files, you can alternatively run log output through a pipe to process this information with a filter. Lincoln Stein wrote a great article on this subject, "The Joy of SQL," in his Webmaster's Domain column in the October 1998 issue of Web Techniques, showing how to automatically insert the data into a mySQL database on the fly.
This works fine for small-to-medium size databases, but what happens if you try it on a large site that gets hundreds of thousands of page requests per day? It eats up disk space, maybe even more than a flat file would. It also puts a heavy load on your database. I've dealt with this in the past by setting up mySQL on a separate Linux box just for log recording. This is a fairly inexpensive hardware solution to the problem, but it adds complexity to your network.
