Gossamer Forum
Quote Reply
SPIDER robots.txt
most of my spidered links are http://domain.com/robots.txt as URL

can i hide alle this robot.txt urls?

Subject Author Views Date
Thread SPIDER robots.txt HEMEN 4017 Jul 30, 2001, 1:37 PM
Thread Re: SPIDER robots.txt
ryel01 3890 Jul 30, 2001, 1:58 PM
Thread Re: SPIDER robots.txt
Paul 3902 Jul 30, 2001, 2:02 PM
Thread Re: SPIDER robots.txt
HEMEN 3893 Jul 30, 2001, 2:13 PM
Post Re: SPIDER robots.txt
ryel01 3884 Jul 30, 2001, 2:37 PM
Thread Re: SPIDER robots.txt
kitsune 3879 Jul 30, 2001, 2:53 PM
Post Re: SPIDER robots.txt
HEMEN 3871 Aug 1, 2001, 12:25 AM