Gossamer Forum
Quote Reply
spider plugin
Ok this is what I want to do, can I get confirmation that the spider does it.

When someone submits a site, after the link has been validated, I want to be able to but the link in a queue to be spidered.

I only want links that I specify to be spidered AND I want to be able to specify the number of pages per link that is spidered AND I want the spider to be restricted to the domain that the link belongs to.

So I can offer people this deal

Your index page added to a category, and X pages from your site automatically added to the search.

I want to add value to my 'add a site' offer, without adding greatly to disk space, processor usage etc.