most of my spidered links are http://domain.com/robots.txt as URL
can i hide alle this robot.txt urls?
can i hide alle this robot.txt urls?
Subject | Author | Views | Date |
---|---|---|---|
SPIDER robots.txt | HEMEN | 4017 | Jul 30, 2001, 1:37 PM |
Re: SPIDER robots.txt | ryel01 | 3890 | Jul 30, 2001, 1:58 PM |
Re: SPIDER robots.txt | Paul | 3902 | Jul 30, 2001, 2:02 PM |
Re: SPIDER robots.txt | HEMEN | 3893 | Jul 30, 2001, 2:13 PM |
Re: SPIDER robots.txt | ryel01 | 3884 | Jul 30, 2001, 2:37 PM |
Re: SPIDER robots.txt | kitsune | 3879 | Jul 30, 2001, 2:53 PM |
Re: SPIDER robots.txt | HEMEN | 3871 | Aug 1, 2001, 12:25 AM |