Gossamer Forum
Quote Reply
SPIDER
Will the spider create duplicate links, if it picks up a link to enter in the data base that is a duplicate will it detect it as a duplicat or will it enter duplicate links that I would have to delete later?

How can I prevent someone from spidering mysite and copying all my links?

Thanks

Quote Reply
Re: SPIDER In reply to
Hello?

Quote Reply
Re: SPIDER In reply to
Can a staff member answer this?

Quote Reply
Re: SPIDER In reply to
Hi Ken,

The spider tries its best to not duplicate data in its database. Before any new link is added to the spider's database it is checked. Duplicates seem to sneak in occasionally, despite this, but the ratio is around 1000 to 1.

To prevent someone from spidering your site, take a look at the following url. It is a guide on how to protect your pages.

http://www.robotstxt.org/wc/norobots.html

Unfortunately, this is all based on trusting the spider, there is no fool proof way of keeping someone off your site. It also makes search engines like google, webcrawler and altavista from listing your pages in their directory.

It will not keep out the dedicated. It is still quite possible for a spider to masquerade as a common browser in every way. However, the links spider does follow the above protocol, so if you follow the instructions no other links spider owner can spider your site (without modifying some code).

Hope this helps