I am about to start a project to build a web crawler that will index all of the pages on each site that is in the links DB. I'll then add to the search script so that it returns the url of the reviewed pages, and then the spidered pages.

Has anyone else done this yet? Time is a little tight on this project, I'd appreciate any help you guys can give.