Gossamer Forum
(Advanced Search)
for
Home : Products : Gossamer Links : Discussions :

SPIDER ACCESS

Sep 5, 2001, 1:43 AM
bachir
Novice (22 posts)
Sep 5, 2001, 1:43 AM
Post #1 of 5
Views: 2761
Shortcut
Quote Reply
SPIDER ACCESS
Hello

When i check my stats log i can't found the trace of spider access!
What the signature of spider access?

Thank you

Bachir
http://www.wilayates.com


Sep 5, 2001, 11:52 AM
kitsune
Staff (53 posts)
Sep 5, 2001, 11:52 AM
Post #2 of 5
Views: 2673
Shortcut
Quote Reply
Re: SPIDER ACCESS In reply to
Hi,

Please take a look at this write up.

http://www.robotstxt.org/wc/faq.html#metoo

Here's a snippet from the site:

In Reply To:
How do I know if I've been visited by a robot?

You can check your server logs for sites that retrieve many documents, especially in a short time.

If your server supports User-agent logging you can check for retrievals with unusual User-agent heder values.

Finally, if you notice a site repeatedly checking for the file '/robots.txt' chances are that is a robot too.
Hope this helps :)
Aki

Sep 6, 2001, 1:17 AM
bachir
Novice (22 posts)
Sep 6, 2001, 1:17 AM
Post #3 of 5
Views: 2670
Shortcut
Quote Reply
Re: SPIDER ACCESS In reply to
Hello

my question is not about the robot?
When i check my access log i don't found any trace of links spider, as googlebot...

Thank you

Bachir

http://www.wilayates.com


Sep 6, 2001, 11:57 AM
pugdog
Veteran / Moderator (6956 posts)
Sep 6, 2001, 11:57 AM
Post #4 of 5
Views: 2663
Shortcut
Quote Reply
Re: SPIDER ACCESS In reply to
Is the spider spidering your site?






PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Sep 7, 2001, 1:15 AM
bachir
Novice (22 posts)
Sep 7, 2001, 1:15 AM
Post #5 of 5
Views: 2658
Shortcut
Quote Reply
Re: SPIDER ACCESS In reply to
Yes