Ok this is what I want to do, can I get confirmation that the spider does it.
When someone submits a site, after the link has been validated, I want to be able to but the link in a queue to be spidered.
I only want links that I specify to be spidered AND I want to be able to specify the number of pages per link that is spidered AND I want the spider to be restricted to the domain that the link belongs to.
So I can offer people this deal
Your index page added to a category, and X pages from your site automatically added to the search.
I want to add value to my 'add a site' offer, without adding greatly to disk space, processor usage etc.
When someone submits a site, after the link has been validated, I want to be able to but the link in a queue to be spidered.
I only want links that I specify to be spidered AND I want to be able to specify the number of pages per link that is spidered AND I want the spider to be restricted to the domain that the link belongs to.
So I can offer people this deal
Your index page added to a category, and X pages from your site automatically added to the search.
I want to add value to my 'add a site' offer, without adding greatly to disk space, processor usage etc.