most of my spidered links are http://domain.com/robots.txt as URL
can i hide alle this robot.txt urls?
can i hide alle this robot.txt urls?
Jul 30, 2001, 1:58 PM
Veteran (1264 posts)
Jul 30, 2001, 1:58 PM
Post #2 of 7
Views: 3919
I was wondering about the same thing.
You're supposed to put any of your 'private' directories in there to stop them being spidered, but then anyone can just bring the robots.txt file up in their browser and see what you're trying hide.
Would also like to hear if anyone has an answer that they use?
Cheers,
R.
You're supposed to put any of your 'private' directories in there to stop them being spidered, but then anyone can just bring the robots.txt file up in their browser and see what you're trying hide.
Would also like to hear if anyone has an answer that they use?
Cheers,
R.
Jul 30, 2001, 2:02 PM
Veteran (19537 posts)
Jul 30, 2001, 2:02 PM
Post #3 of 7
Views: 3931
WHOAH
I thought it was a long shot but it worked...
Here's Yahoo's robots.txt file...
Disallow: /gnn
Disallow: /msn
Disallow: /pacbell
Disallow: /pb
# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /
Installations:http://www.wiredon.net/gt/
I thought it was a long shot but it worked...
Here's Yahoo's robots.txt file...
Code:
User-agent: * Disallow: /gnn
Disallow: /msn
Disallow: /pacbell
Disallow: /pb
# Rover is a bad dog <http://www.roverbot.com>
User-agent: Roverbot
Disallow: /
Installations:http://www.wiredon.net/gt/
Jul 30, 2001, 2:13 PM
Novice (20 posts)
Jul 30, 2001, 2:13 PM
Post #4 of 7
Views: 3922
i know robots.txt files, but i donīt know, why this files where spidered in my search engine.
the robots.txt says, i thought which documents where allow to be spidered. why i have to validate a robots.txt file???
the robots.txt says, i thought which documents where allow to be spidered. why i have to validate a robots.txt file???
Jul 30, 2001, 2:37 PM
Veteran (1264 posts)
Jul 30, 2001, 2:37 PM
Post #5 of 7
Views: 3913
Ahh sorry, I thought you were talking generally about Robots.txt files. I didn't read it properly that you were having problems with the SPIDER plugin - I don't know the answer to that one.
Cheers,
R.
Cheers,
R.
Jul 30, 2001, 2:53 PM
Staff (53 posts)
Jul 30, 2001, 2:53 PM
Post #6 of 7
Views: 3908
Hi Hemen,
The easiest way of dealing with this is to go into Plugins->Database-> Bulk Operations, choose Links and click on Go.
Look for any URL containing robots.txt.
When you have a collection of all such links, delete them from the database.
Hope this helps
The easiest way of dealing with this is to go into Plugins->Database-> Bulk Operations, choose Links and click on Go.
Look for any URL containing robots.txt.
When you have a collection of all such links, delete them from the database.
Hope this helps