Step-by-Step Link Verification
By Randal L. Schwartz
Link verification -- what a fun job. You've got a site with dozens, hundreds, or perhaps even thousands of links. After all, that's the purpose of the Web -- to be able to link from one page to another. But the resources referenced by those links (both internal links to your Web site, and external links to other Web sites) frequently move (either accidentally or deliberately). And then you've got that ugly, evil condition -- a bad link.
In fact, it's such a common problem that I've been using various tools that I'd written about in this column to perform link verification. It started with a simple link checker in October 1996, then advanced to a full cross-referencing tool in June 1997, and then a parallel link checker in July 1998. Well, it's time to write another tool.
The problem with those earlier link verifiers is that they needed to run over the entire tree in one run. This is no big deal if you have 20 or 30 pages at most, but if you have a more complex site, the system might crash some time in the middle of the run. And now that I've been adding more external links to my Web pages, there are more and more things that have to be checked. So a link verifier that has to scan the entire tree in one run just doesn't scale well.
But there was another problem with doing it in one run. Sometimes, a perfectly valid URL triggers a Web site that is temporarily down, or maybe just overloaded or otherwise unresponsive. So, since we're checking that URL only once in a run, we'll get a false alarm, or miss other links.
