The new search engine I am developing is going to have WELL MORE than 10,000 links (possibly 100,000+). While Alex's script is excellent, I am considering switching to a simplified search engine like Webcrawler.
Please let me know what you think, and if having a simplified search engine will impact visitors versus a directory search.

Later,

Nate Caminata - Path Rat.com

P.S. Links SQL is out of the question (2 much $$$)

>> The new search engine I am developing is
>> going to have WELL MORE than 10,000 link

Are you developing a search "ENGINE" or are you putting up a links site? Links is, more or less, an _ENGINE_. It organizes and searched the entered links on demand. A links site uses an _ENGINE_ like Links 2.0, LinkSQL or Webcrawler to index/organize the data.

If you are going to have 10,000 links, you should consider something like Link SQL, and for a project that size, the cost is in line with what it would cost for you to try to install and set up some of the lesser programs. It took me less than 6 hours to move my Links 2.0 to LinkSQL, less time than most things I've done -- including deploying the current version of Apache.

If you have a database of 10,000 links, and are spending the effort and time on it, as well as the web hosting resources, you should have the means of re-generating the cost of the Links program in short order.

If you can't, then what is the purpose of your site, vs any of the other multitude of links sites out there?

Forgive me, but this is something I don't understand. What is the purpose of putting the time and effort into something of this size if it's not going to return *at least* the value/cost of the LinkSQL ??

My only problem with purchasing LinkSQL was would it do what I needed, site unseen, since my site is serving up only local pages, not off-site pages. Alex offered me a money back guarantee, so I tried it. It does, and is worth the money. I couldn't have modified Links 2.0 for less time or money.

If you are building this sort of site, then you want to protect your data, so you want it all on your site, not hosted by someone else who can pirate or siphon it off. To do effective searches at 10,000+ links, you need an RDBMS of some sort, and MySQL is pretty good for the range you are looking at.

I don't know what webcrawler is offering, or what some of the other search companies offer, but at last look, LinkSQL was _THE_BEST_ deal out there.



------------------
Robert S. Pataki, MD
President, PUGDOG Enterprises, Inc.
http://www.postcards.com

Perhaps you don't understand, I don't have the time to index several directories. I would do it, but I tried using URL Spider Pro, and have 20 hours of spidering the Dmoz.org Arts directory (for my arts directory), it indexed a total of 200 URL's...that is not going to cut it. If you know a quicker way to get the URL's properly placed in the directories (which I have already created), Please let me know and I will not give up on this endeavor.

Thank you,

Nate

Links SQL does include an RDF_Parser.pl script which you can use to get going. For instance you can set it to "Top/Arts" and it will start you off with the whole Arts section of DMoz.

However if you are looking for just a search engine, and not a DIRECTORY, this might not be the best tool. A directory implies you are actively maintaining it.

Cheers,

Alex

Nate-

Links is a great program, but it honestly doesn't sound like you'll be happy with it. I haven't used links sql, but even that is intended to be a directory like yahoo, not a spider/search engine like excite.

If what your looking for is an instant directory, you may want to look into one of the scripts that lets you incorporate the DMOZ directory on your site without actually hosting it yourself. I haven't used any of them myself, but others have commented on them here before, and I know one person used it on the same home page as their own links powered directory (sorry, I don't remember which site did this--maybe they'll speak up).

One of these DMOZ scripts is at http://grohol.com/downloads/

I just know its there...I can't speak for what it can do. There are other similar around, maybe someone else can give another URL.