Besides robots.txt are there options to block spiders from hitting my site, esp. those that ignore robots.txt?

Didn't you just create a Thread about this?



I don't know any other method than editing the robots.txt file. May be you can give us more information on what problems you are experiencing, we could assist you identifying methods of "blocking" spiders.

Regards,

------------------
Eliot Lee
Founder and Editor
Anthro TECH, L.L.C
http://www.anthrotech.com/
info@anthrotech.com
==========================
Coconino Community College
http://www.coco.cc.az.us/
Web Technology
Coordinator
elee@coco.cc.az.us

[This message has been edited by Eliot (edited October 04, 1999).]

You can use mod_rewrite in your servers config or .htaccess file to rewrite requests from identified clients. However, if it's spammers you're after, they're usually way ahead of the pack, so if they ignore robots.txt, they're probably going to spoof their client ID too. And if it's a search engine causing hassle you'd be better off reporting the problem to the people running it than just blocking up access to your site. They're still gonna be using your bandwidth...

adam