Hmm, got you now (re locking up server).

I currently use DMOZ extractor, LINKS III, and have played around with Innerprise URO Spider Pro.

All have served their purposes, but lack built in conveinence. I run a Canadian Only directory so-

My particular requirement for a Spider is thus:

• Easily add small to large amounts of links from any page with the ability to seperate non-Canadian Links.... this sounds difficult, but a lot of Canadian Links these days have the .ca extension. Or if I am grabbing links from a yahoo.ca subdirectory for example... I know that the links are already all Canadian. At the moment , I have to extract the links, then perform a file converision, then import... three steps... i'd like this to be one step! I do not blanket spider entire sites... I find a page which has say 20-500 links which are relevant to one particular category... ie Canada->Arts->Music->Artisits & Bands->Jazz... and extract from here...
• Fill in and refresh meta fields on existing links i.e. update or add missing description, title etc

LSQL 2.1 already has the ability to check links... so I am assuming this would stay the same with the addition of the spider?

Thanks (I hope this actually answered your question).