Hi,

1. Yes, as long as you have shell access you can use it. To start spidering sites, you must run spider.pl from shell as it runs as a daemon. Once it's running, you can check on its status and stop it from the admin panel.

2. By default the spider will not hit the same site more then once per second. So it's quite friendly on load. We had it running for two days at around 5% CPU resources. The sites do not get indexed right away, the data is just inserted into MySQL, so when you do index the sites, it can cause some server load, but that's for a short time.

3. Pages that get spidered are not validated by default and won't be searchable until they've been validated (which yes, can easily be done in bulk).

a. This is possible, if you pass in spider=1, then the spider database will be searched as well as the user database and be available as spider_hits and spider_results.

b. Not sure about this one. =)

Cheers,

Alex

--
Gossamer Threads Inc.