ok.. i wanted to carry on a discussion about this so we could actually get SOMEWHERE..

first of all i already figured out how to do build changed for detailed pages.. (it is pretty easy actually)..

but it depends on what kind of information you keep on your detailed pages.. for instance if you put your root categories on the side and you add a new root category.. it won't get updated.. it only updates if the link has been changed since the time it last built detailed pages..

i really don't think this can be done without a whole bunch of code.. is it really that neccessary?

jerry

It is on very large sites when you have 500,000 links in 50,000 categories and you are forced to rebuild every time a link is added.

A lot of things need to be considered:

1. Adding a link will affect the category page it's on, plus all category pages above it (as counters will need to be updated). It might also affect the New, Cool, and Ratings pages.

2. Same for deleting a link.

3. Modifying a link means affecting the current category page, possibly New, Cool and Ratings pages.

4. Adding a new category means adding a new category page, plus modifiying the category page above it.

5. Deleting a category means redoing all category pages beneath it, as well as modifying the category page above it.

6. Same for modifying a category page.

7. Adding/Deleting/Modifying Relations means redoing the category page the relation is on.

So there is a lot of bookeeping that needs to be done. What I'm looking at doing is just making it so the CategoryHiearchy is automatically maintained, much like the Search Index, so that build_stats is not so intensive for each call.

Any ideas are always welcome. =)

Alex

Alex, what about adding a "dynamic" option to the whole program?

If stats are maintained all along, counters updated, and such then calls to page.cgi will return valid data all the time, and the only thing that happens during a rebuild is that the static category and detailed pages are re-built.

It would proably be much less intensive to calculate stats in real-time, rather than try to brute-force it all at once.

I've been working on the mod for my site (actually waiting for widgetz to come out before trying to commit mine to code) for ratings, and maintaining them in real-time for dynamic "cool" and "top 10" pages is not a real problem....

Could a re-thinking of the design from the ground up shed new light on it? Envision the site as "dynamic" but using static pages to cut down on the load, rather than static, with dynamic pages added on.

I'm finding only the browsers are really using the static pages anyway -- and most access is through search.cgi -- Hence my request for pre-compiled or cached searches.

"browsers" are not really looking for anything, so if the static pages lag a little I'm not concerned -- those looking for something are going to 'search' for it.

Then you have to consider that when counters in the search results might not match counters on static html pages? Also, New and Popular tags, plus any flags might not match up. I'm not sure if this is worse/better. What do you think?

Alex

I think if the counter tags don't match, for
most sites that's not as big a problem as the
server load.

People tend to either use dynamic or static, so consistency is the only real problem.

Some sites might find that a problem -- if
they are using it for voting or something like that, but for just general links, a small margin of error is fine -- what happens on systems that cache pages -- the values are going to be slightly different anyway.

My feeling is that dynmic updates, with regular build-all/build-changed would be the best.

What about a more complex cron job that builds parts of the site in segmental order, so it never puts too much of a load on the system. Say job that fires every 15 or 30 minutes, and checks to see if the last instance ended. If it did, it works on the next chunk.

That way, sites that want to compile once, just do so with a regular cron job or manually, and those that want the incremental updates can use the sequential build.

Rather than make links do it, and worry about html and connection time outs, especially with larger sites, use a sequential cron-triggered build process.

This would pull the 'build' out into a module even more segmented from the functioning.

If so, then different build modules could be designed to let people pick how they want to build -- rather than trying to make one module do everything.

People are not going to change the way the build after the novelty wears off-- they will do what works best on their site.

This is a little complex in the re thinking, but should make updates and upgrades and additions easier in the long run, than trying to code some sort of complex build routine into one all encompassing module.