david at kineticode
Jan 26, 2010, 3:20 PM
Post #20 of 25
(2806 views)
Permalink
|
On Jan 26, 2010, at 12:53 PM, Matt Rolf wrote:
> To me, the database activity is the least of these issues, particularly if postgresql is on a large, multi-core box with oodles of ram, a competent OS, and a nice disk array. Tune it up, let it fly.
Well, yes, except that the slowness of publishing is not due to the speed of the database, but to the amount of data loaded from the database. So if you republish your home page, and the template loads the 25 most recent stories from the database, and then loads any of the elements from those 25 stories (such as a teaser field), then that's a lot of data that gets queried.
> Unfortunately, the other two parts fall to bric_queued, which 1) has to be run on the same box as Bricolage (although I suppose you could put it on another box and then network share the needed Bric libraries)
It's not the libraries, which can be copied, but the templates, which cannot.
> and 2) is confined to two processes, one for publishing and one for distribution for the *entire* bricolage install. In an app that can see both postgres and apache spawn as many processes as are needed and ram allows to keep up with increased use, it seems like this is a pretty glaring scalability weak point.
Agreed. I've seen bric_queued choke on a lot of stories before.
> I was stunned to read in the bric_queued script that "distribution is normally an order of magnitude faster than publishing". Now, I can believe that in regards to file system moves, but FTP and SFTP? Using Net::SSH2, Bricolage could transfer about a file a second, but for 10,000 files, that's a mighty long time to wait. I'm assuming FTP connections have similar latency.
It should not take a second to do the transfer. That's way too slow.
> Regardless of which is faster, bric_queued is probably going to be a choke point for large installs long before db activity. Thousands of stories or media flying out at once are simply going to over-whelm it.
Anyone from WHO want to chime in on this?
> Is there any way bric_queued could be used or easily altered to improve scalability? Running the script more than once to spawn multiple pub and dist thread pairs doesn't seem to help - one pair will grab the jobs queue and the others will just sit there. It seems to me that the possibility of it running on it's own box and using multiple processes to deal with a large file load would be a good start, but maybe I'm thinking about the problem wrong.
It could absolutely be made to scale better. It could spawn separate jobs for each story it publishes, for example, up to n at a time. Other stuff too, I'm sure. Are you offering to take on such a task?
Best,
David
|
