Say, on a 1ghz Pentium III + 512 MB ram server - how long will take to build the whole directory (static pages with templates) of about 200,000 links imported from dmoz? How much server resources will this consume during the entire build process?

thanks

It depends on the CPU allowance for for the user that is doing the rebuild. My guess would be about 2-3 hours. It all depends on how many categories you are having, as that is what takes the most time. Also, if you set it to run via cron, then it will run a heck of a lot faster, as it is not passing anything back to your browser.

Andy (mod)
andy@ultranerds.co.uk

---

Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!

Hi,

The main time is in number of categories (as one category = one html page). You'd probably have about 30,000 categories to build, and if you have the dedicated server completely for Links SQL, you could probably build the pages in about 45 minutes I'd imagine.

Cheers,

Alex
--
Gossamer Threads Inc.

While this is going on (the build process) can the end users still do searches - will there be any server resources left?

thanks

Hi,

Yes it does not impact the search ability at all, and since it's just overwriting pages, users can still browse the site while it's building.

There will still be enough resources to run the site.

Cheers,

Alex
--
Gossamer Threads Inc.

Could be possible to make the directory build much faster?
When I profiled the Links 2.0, I found that the most time is spent in the template module.

I know in Links SQL the templates are precompiled, so their parsing, executing are much faster.
How much faster are they?

Could be possible to increase the speed of template parsing by rewriting the template parser in C language?

Best regards,
Webmaster33

---

Paid Support from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...

My server (shared) has less resources than yours, and it typically takes 40mins-60mins per 50,000 links as I guide. So Alex's estimate would be a good one for you system I would think.


http://www.iuni.com/...tware/web/index.html
Links Plugins

Two problems with writing the parsing module in C.

Not all computers have a C/C++ compiler to compile the module (or provide users access), and this impacts on the "pure perl" portable nature of Links.

Writing text parsing/processing options in C/C++ is a lot more complex than in a language like Perl. Things you can take for granted have to be spelled out in the tiniest detail.

A third problem would be Alex is not as proficient in C/C++ as he is in Perl :) He's admitted that on several occasions. You wouldn't want to distract him from the main goals of continued development ;)

Also, while there would probably be an improvement for some sites, and sites not running mod_perl, the speed benefits over mod_perl probably would not make much of a difference (Alex has hinted at that before). The reason the scripts spend the most time in the parsing module, is that is where 80% of the work is done (and it's increasing!) :) No matter what language you use, that work will still be done there.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

Well, I have got an old pentium 800mhz, 512 MB RAM server. If this is a ok server to start with (less than 5,000 searches a day but all dmoz: business, computing and shopping categories + some links).

Any comments if it is ok to start with or if a better server is needed?

thanks