I was considering that, but web crawling via shell definitely crosses the line of things I would not want to attempt. Although, I guess you could outsource that complexity with `wget --mirror`, bringing you right back to file-crawling.
There is a very simple way to write a breadth-first webcrawler that works in stages and saves the frontier in an ordinary text file, this kind of webcrawler can be implemented in almost any programming language, particularly shell. That is, you write a script that loops over the urls in the frontier, fetches them, then writes new urls it find in the next frontier, you can use sort, uniq and such to manage the frontier.
That simple breadth first webcrawler has the big advantage that it does not get stuck in web traps (say a calendar that has a button to go the next year which could easily scan up to the year 9595, if man is still alive). Or rather, after a certain number of passes (maybe N=10-15] all the real URLs have been crawled and the remaining URLs are web traps.
In a language like Java it is tempting to write a much more complex crawl manager that has the potential to crawl faster with threads but takes more care to not crash target sites, get caught in web traps, etc. The first web crawler I wrote (1999) had a highly complex crawl control system, future web crawlers I wrote got simpler and simpler because the simple breadth first crawler works very well for small projects.
