Hi,
I want to know how much Diskspace I need to run the Spider Plugin.
How can I calculate it? Can you give us an example?

Lets say we have 10.000 Links in LinksSQL and we only want
to spider these 10.000 hosts. Lets assume every host has 60 Pages (Links which are to spider)

Then we have to storage 600.000 pages, it is right? So how much diskspace we will need for this example?

Hi,

If we assume the average page is 10k, then 600,000 pages would be 6 gigs, and then double that for the search index, so you'd be looking at 12 gigs of diskspace.

Cheers,

Alex

--
Gossamer Threads Inc.

Is there a possibility to say to the spider: " Hey, limit the diskspace and don´t use more than 10 Gigs!" or can you tell him to only spider 30 pages per URL and Domain?

If not - I think this would be a nice feature, which you should include in the next release.

Hi,

No not built in, but it's something we've added to our todo list.

Cheers,

Alex

--
Gossamer Threads Inc.