Gossamer Forum
(Advanced Search)
for
Jul 30, 2001, 2:02 PM
Paul
Veteran (19537 posts)
Jul 30, 2001, 2:02 PM
Views: 3940
Shortcut
Quote Reply
Re: SPIDER robots.txt In reply to
WHOAH

I thought it was a long shot but it worked...

Here's Yahoo's robots.txt file...

Code:
User-agent: *
Disallow: /gnn
Disallow: /msn
Disallow: /pacbell
Disallow: /pb

# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /

Installations:http://www.wiredon.net/gt/

Subject Author Views Date
Thread SPIDER robots.txt HEMEN 4056 Jul 30, 2001, 1:37 PM
Thread Re: SPIDER robots.txt 		ryel01 3927 Jul 30, 2001, 1:58 PM
Thread Re: SPIDER robots.txt 		Paul 3940 Jul 30, 2001, 2:02 PM
Thread Re: SPIDER robots.txt 		HEMEN 3931 Jul 30, 2001, 2:13 PM
Post Re: SPIDER robots.txt 		ryel01 3921 Jul 30, 2001, 2:37 PM
Thread Re: SPIDER robots.txt 		kitsune 3916 Jul 30, 2001, 2:53 PM
Post Re: SPIDER robots.txt 		HEMEN 3909 Aug 1, 2001, 12:25 AM