Tracking New Arrivals
By Randal L. Schwartz
Today, much of the Web's content is found not by users clicking on interesting URLs posted on other sites, but by users typing search queries in to the big indexing engines like AltaVista, Lycos, and Infoseek. If you're maintaining a referer log (spelled this way for historical reasons), you may have noticed that the query strings typed in by the user sometimes show up when that user follows a search-results link to your page. This happens because the indexer's search page is often a GET form, and the parameters of the search are therefore encoded into the URL of the search-results page.
With this in mind, I decided to write a program that would go through my referer log and extract just the search strings. This is more than idle curiosity; it tells me exactly what people were looking for that brought them to my page, and what I should be providing more of if I want my site to be popular -- especially if I sell ads or want to be famous.
The referer log (available for most popular Web servers) is merely a record of the HTTP Referer header, which will frequently point to the URL from which the URL request is being made. The referer is not necessarily supported on all browsers, and will be messed up on a bookmarked entry. But for the majority of hits, the referer can give valuable information (as you can see by looking at the results of this program on your site).
Extracting Strings
The program to extract the search strings from the referer log is presented in
