Gossamer Forum
Home : Products : DBMan : Discussions :

.htaccess - tip and question | possible solution

Quote Reply
.htaccess - tip and question | possible solution
what I've just now done is created a sub-folder under the dbman folder called databases and I've put the db file(s) there and put .htaccess / .passwd there to! (any other solutions?)
Quote Reply
Re: [rayramon] .htaccess - tip and question | possible solution In reply to
Have you also added those directories to you robots.txt file.

Most search engines will check that file for pages and directories they shouldn't index. Although I have in the past had a problem with google not reading the robots.txt file. They told me the would look into the matter and until then I excluded them from using their robot on my sites.

Unoffical DBMan FAQ

http://creativecomputingweb.com/dbman/index.shtml/
Quote Reply
Re: [LoisC] .htaccess - tip and question | possible solution In reply to
Hi, can you tell me what robots.txt is and how to use it? I also would like to know how to exclude engines like you did with google.

Jan Peter
Quote Reply
Re: [JPWiese] .htaccess - tip and question | possible solution In reply to
There is information on my Web Magic site under the topic of META TAGS for using a robot.txt file. I'm not sure anymore the exact exclusion statement I used for google. The information is on their site most likely in the FAQ or help section.

Another good resource is:

Search Engine Watch by Danny Sullivan
http://searchenginewatch.com/

What I use to exclude linkwalker with a robots.txt file is:

User-agent: linkwalker
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
.... the list of files or directories to exclude starts here

Hope this gets you started.

Unoffical DBMan FAQ

http://creativecomputingweb.com/dbman/index.shtml/
Quote Reply
Re: [JPWiese] .htaccess - tip and question | possible solution In reply to
http://www.searchengineworld.com/.../robots_tutorial.htm