Gossamer Forum
Home : Products : Links 2.0 : Discussions :

Spidering search engines

Quote Reply
Spidering search engines
I am currently using Innerprise's spider, and searching the URL: http://dir.lycos.com/arts/index.html for keywords I specified for my Arts & Entertainment directory. I have it properly configured to work with Links, but I am still unsure if I am doing this the correct way. Am I?
Quote Reply
Re: Spidering search engines In reply to
What's the problem? Did urls come up?

------------------
------------------------------------------
Lavon Russell
LookHard! Search
<url>http://www.lh.yi.org</url>
Quote Reply
Re: Spidering search engines In reply to
Well, the thing is, it says that it has *Done* 2,000 URL's, with about 20,000 to go, but when I look at the LOGS, it doesn't show anything....but I think I got it working.

BTW, What is a good limit to stop at for a number of links in the database for Link 2.0. 15,000, 20,000?

Thanks,

Nate
Quote Reply
Re: Spidering search engines In reply to
Here is what Alex has to say:

Quote:
If your site is going to hold more then 10,000 links, then you should strongly consider using Links SQL.
Quote Reply
Re: Spidering search engines In reply to
Yeah, that's the thing. My site is going to have over 50,000 links (possibly more), but there is no way I can afford Links SQL. What happens if you go over the 10,000? Does it slow way down?

Nate
Quote Reply
Re: Spidering search engines In reply to
I do know that search will become quite unbearable, I'm not quite sure about the other effects.
Quote Reply
Re: Spidering search engines In reply to
Question,

I can add links to my web site by using a spider to collect them and add them the links database? Whatever Keyword I choose?

I have been adding them manually. Smile

Is this the wrong way. I was always curious how people got 50,000 links.

------------------
James L. Murray
VirtueTech, Inc.
www.virtuetech.com
Quote Reply
Re: Spidering search engines In reply to
Path Rat

I had the same problem and as a last resort contacted Innerprise.
Here is there response -

"The reason it might not be saving anything yet is if you're using a lot of
Targeted keywords/phrases. URL Spider Pro will look for them within the web page
it's trying to index and if it can't find ALL of them, the web page won't be
saved. When using Targeted Indexing is usually best to use one or several of the
major search engines to find web pages similar to what you want indexed and then
add a few of those links to URL Spider Pro. Otherwise it could take sometime to
build your database."

Also I have found that using target url filters such as yahoo.com etc helps.

I hope this helps.

Cheers
Quote Reply
Re: Spidering search engines In reply to
 
Quote:
I can add links to my web site by using a spider to collect them and add them the links database? Whatever Keyword I choose?

I have been adding them manually.

You can try using the Bulkload utility in the Resource Center.

www.gossamer-threads.com/scripts/resources/jump.cgi?ID=387

[This message has been edited by t_dog (edited July 27, 1999).]