Does anyone know of a program that performs a similar function to the Innerprise URL spider pro which collects targeted urls and puts them in a Links formatted database?

Is there another option, spider or program available that is less expensive or freeware that will collect urls from the yahoo or netscape directories on a specified topic, even if the resultant database is not in Links format?

Gossamer-Threads Links is an awesomely great script and could be further enhanced by the addition of a targeted url gatherer utilility. Is this a possibility for an additional feature in the future?

Thanks for any information.

http://www.grohol.com/downloads/

This site may or may not be what you are looking for, but it is somewhat close I believe. The program queries the netscape directory and prints it out on screen. You can add url's to it, but it is totally dependent on netscape. This is basically links on a remote server. It is based off the Open Directory Project but looks a lot more visually appealing. The code inside the program could possibly be modified to work with Links the way you want it to, but that may take alot of code hacking. Again this may not be what you are looking for

to see the script implemented:
http://www.copyright-web.com/pod.cgi

Hope this helps

Actually, I don't think a general spider would be added to Links in the future. Links just cannot handle the amount of links a normal spidering search engine has. See this thread for information on how many links Links SQL can handle: http://www.gossamer-threads.com/...um9/HTML/000003.html

What is wrong with URL Spider Pro (I've never tried it)? It seems to do exactly what you want.

[This message has been edited by t_dog (edited July 28, 1999).]

That isn't exactly what I was looking for. I am searching for a spider program who's results can be put into links but that pod script is very interesting.

Thanks for the info.

My topical links directory probably would not be more than 3 to 5 thousand links but even with that size of a database it would take a lot of time to search, cut and paste and put in the links individually. You are correct .. URl spider does exactly what I want. I've checked out the demo and it will gather the urls and put them in a links formatted database for indivual entry and saves a lot of time but it is pricey. Usually for every kind of program there are at least a few alternatives. Maybe url spider pro has this market niche covered for now.

How do many of the links directories get thousands of urls? Are they all entered in by a manual search or do the directories grow over time by site visitors adding their links?

Thanks for the info.

If you add 20 links a day, in a year, you'll end up with 7120 links! Many people work very hard on their sites (that is, more that 20 new links a day).

Persistence, persistence, persistence...