There is information on my Web Magic site under the topic of META TAGS for using a robot.txt file. I'm not sure anymore the exact exclusion statement I used for google. The information is on their site most likely in the FAQ or help section.
Another good resource is:
Search Engine Watch by Danny Sullivan
http://searchenginewatch.com/
What I use to exclude linkwalker with a robots.txt file is:
User-agent: linkwalker
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
.... the list of files or directories to exclude starts here
Hope this gets you started.
Unoffical DBMan FAQ
http://creativecomputingweb.com/dbman/index.shtml/
Another good resource is:
Search Engine Watch by Danny Sullivan
http://searchenginewatch.com/
What I use to exclude linkwalker with a robots.txt file is:
User-agent: linkwalker
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
.... the list of files or directories to exclude starts here
Hope this gets you started.
Unoffical DBMan FAQ
http://creativecomputingweb.com/dbman/index.shtml/
