Gossamer Forum
(Advanced Search)
for
Jul 30, 2001, 2:13 PM
HEMEN
Novice (20 posts)
Jul 30, 2001, 2:13 PM
Views: 3928
Shortcut
Quote Reply
Re: SPIDER robots.txt In reply to
i know robots.txt files, but i donīt know, why this files where spidered in my search engine.

the robots.txt says, i thought which documents where allow to be spidered. why i have to validate a robots.txt file???

Subject Author Views Date
Thread SPIDER robots.txt HEMEN 4053 Jul 30, 2001, 1:37 PM
Thread Re: SPIDER robots.txt 		ryel01 3924 Jul 30, 2001, 1:58 PM
Thread Re: SPIDER robots.txt 		Paul 3936 Jul 30, 2001, 2:02 PM
Thread Re: SPIDER robots.txt 		HEMEN 3928 Jul 30, 2001, 2:13 PM
Post Re: SPIDER robots.txt 		ryel01 3918 Jul 30, 2001, 2:37 PM
Thread Re: SPIDER robots.txt 		kitsune 3913 Jul 30, 2001, 2:53 PM
Post Re: SPIDER robots.txt 		HEMEN 3905 Aug 1, 2001, 12:25 AM