Yes, that's right it's an update. You may want to hold off, however, it's basically a small patch. The entire spider will be rewritten. It was written for links versions ago and it desperately needs an upgrade to incorporate the new features of Links.
Yep, that version 2 is finally starting to come about. Things I know for sure about it is that I'm probably gonig to separate the spidering and indexing phases into two different scripts. Rolling them into one script was my bad idea and the spider occasionally ended up biting off more than it could chew.
I'm already looking into how to index pdf and ps documents, maybe the spider could send any arbitrary format though a commandline program (so then handy utilities from freshmeat could be used)
But I'm just starting on it, I'm open for any suggestions.
