I am currently using Innerprise's spider, and searching the URL: http://dir.lycos.com/arts/index.html for keywords I specified for my Arts & Entertainment directory. I have it properly configured to work with Links, but I am still unsure if I am doing this the correct way. Am I?

What's the problem? Did urls come up?

------------------
------------------------------------------
Lavon Russell
LookHard! Search
<url>http://www.lh.yi.org</url>

Well, the thing is, it says that it has *Done* 2,000 URL's, with about 20,000 to go, but when I look at the LOGS, it doesn't show anything....but I think I got it working.

BTW, What is a good limit to stop at for a number of links in the database for Link 2.0. 15,000, 20,000?

Thanks,

Nate

Here is what Alex has to say:

Yeah, that's the thing. My site is going to have over 50,000 links (possibly more), but there is no way I can afford Links SQL. What happens if you go over the 10,000? Does it slow way down?

Nate

I do know that search will become quite unbearable, I'm not quite sure about the other effects.

Question,

I can add links to my web site by using a spider to collect them and add them the links database? Whatever Keyword I choose?

I have been adding them manually.

Is this the wrong way. I was always curious how people got 50,000 links.

------------------
James L. Murray
VirtueTech, Inc.
www.virtuetech.com

Path Rat

I had the same problem and as a last resort contacted Innerprise.
Here is there response -

"The reason it might not be saving anything yet is if you're using a lot of
Targeted keywords/phrases. URL Spider Pro will look for them within the web page
it's trying to index and if it can't find ALL of them, the web page won't be
saved. When using Targeted Indexing is usually best to use one or several of the
major search engines to find web pages similar to what you want indexed and then
add a few of those links to URL Spider Pro. Otherwise it could take sometime to
build your database."

Also I have found that using target url filters such as yahoo.com etc helps.

I hope this helps.

Cheers

 
You can try using the Bulkload utility in the Resource Center.

www.gossamer-threads.com/scripts/resources/jump.cgi?ID=387

[This message has been edited by t_dog (edited July 27, 1999).]