Gossamer Forum
(Advanced Search)
for
Home : General : Chit Chat :

Search Engine Bots

Sep 17, 2003, 9:20 AM
dan
Enthusiast (760 posts)
dan's avatar
Sep 17, 2003, 9:20 AM
Post #1 of 3
Views: 4128
Shortcut
Quote Reply
Search Engine Bots
Man, my site is being crawled and probed at an incredible rate by the search engine robots (inktomisearch.com, looksmart.com, googlebot.com, fastsearch.net, av.com, etc.). Not that I'm complaining ;=) Interesting enough, one read the robots.txt file, and then still crawled a forbidden directory - however, it is password protected. Being going on like this for a few weeks now =)

----
Cheers,

Dan
Founder and CEO

LionsGate Creative
GoodPassRobot
Magelln
Sep 17, 2003, 11:22 AM
Stealth
Veteran (17240 posts)
Stealth's avatar
Sep 17, 2003, 11:22 AM
Post #2 of 3
Views: 4013
Shortcut
Quote Reply
Re: [dan] Search Engine Bots In reply to
How are you identifying the bots to not crawl your "forbidden" directory?

Are you using the following?

User-agent: *
Disallow: /folder/

Even so, some of the bots, including Inktomi and Looksmart) you've listed are notorious for disregarding robots.txt and also the ROBOTS meta tag in web pages.
========================================
Buh Bye!

Cheers,
Me
Sep 17, 2003, 11:52 AM
dan
Enthusiast (760 posts)
dan's avatar
Sep 17, 2003, 11:52 AM
Post #3 of 3
Views: 3986
Shortcut
Quote Reply
Re: [Stealth] Search Engine Bots In reply to
That's what I'm using. Cannot recall which bot ignored the disallow. Oh well, no harm done. They seem to be ignoring the CGI-BIN, and that's more important.

----
Cheers,

Dan
Founder and CEO

LionsGate Creative
GoodPassRobot
Magelln