Mailing List Archive: Bricolage: devel

setting up bric_queued

Index | Next | Previous | View Threaded

dawn at dawnthots

May 1, 2011, 5:32 PM

Post #1 of 16 (2448 views)
Permalink

setting up bric_queued

Hi

I've read through the messages I could find on the topic and it seems running /bin/bric_queued/ as a daemon is the best way to set up queued publishing in Bricolage 2.0.
I used the bric_queuedctl script that David wrote in 2009 - to start the script (shouldn't that be included with bric_queued - in a folder perhaps?).

I also set in the bricolage.conf:

queue_publish_job = on

unfortunately it's not publishing anything in the job queue. I check the error messages log file (bric_queued.log) and this is repeated:

[/home/canadianart/bricolage/lib/Bric.pm:931]
[/home/canadianart/bricolage/lib/Bric/Biz/Category.pm:1230]
[/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:97]
[/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:123]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279]
[/home/canadianart/bricolage/data/burn/comp/oc_1025/util/meta_keywords.mc:48]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1284]
[/home/canadianart/bricolage/data/burn/comp/oc_1/autohandler:31]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:473]
[/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Interp.pm:342]
[/home/canadianart/bricolage/lib/Bric/Util/Burner/Mason.pm:261]
[/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1600]
[/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1340]
[/home/canadianart/bricolage/lib/Bric/Util/Job/Pub.pm:187]
[/home/canadianart/bricolage/lib/Bric/Util/Job.pm:1886]
[/home/canadianart/bricolage/bin/bric_queued:246]
[/home/canadianart/bricolage/bin/bric_queued:212]
Bric::Biz::Category->keywords has been deprecated.
Use Bric::Biz::Category->get_keywords instead at /home/canadianart/bricolage/lib/Bric/Biz/Category.pm line 1230.

I am trying to publish a story which had been successfully published before.

Any thoughts on what is wrong with my settings?

thank you
Dawn

dawn at dawnthots

May 1, 2011, 5:42 PM

Post #2 of 16 (2378 views)
Permalink

Re: setting up bric_queued [In reply to]

Update:

OK I logged into the bricolage server and typed this command

/home/sitename/bricolage/bin/bric_queuedctl start

voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now)

However I did the same thing on Friday and fully expected bric_queued to keep working.

Is there a better way for me to set it up to make sure it keeps running?

thanks
Dawn

On 2011-05-01, at 8:32 PM, Dawn Buie wrote:

> Hi
>
> I've read through the messages I could find on the topic and it seems running /bin/bric_queued/ as a daemon is the best way to set up queued publishing in Bricolage 2.0.
> I used the bric_queuedctl script that David wrote in 2009 - to start the script (shouldn't that be included with bric_queued - in a folder perhaps?).
>
> I also set in the bricolage.conf:
>
> queue_publish_job = on
>
> unfortunately it's not publishing anything in the job queue. I check the error messages log file (bric_queued.log) and this is repeated:
>
> [/home/canadianart/bricolage/lib/Bric.pm:931]
> [/home/canadianart/bricolage/lib/Bric/Biz/Category.pm:1230]
> [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:97]
> [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/keyword_list.mc:123]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279]
> [/home/canadianart/bricolage/data/burn/comp/oc_1025/util/meta_keywords.mc:48]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1284]
> [/home/canadianart/bricolage/data/burn/comp/oc_1/autohandler:31]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Component.pm:135]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:1279]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Request.pm:473]
> [/usr/lib/perl5/vendor_perl/5.8.4/HTML/Mason/Interp.pm:342]
> [/home/canadianart/bricolage/lib/Bric/Util/Burner/Mason.pm:261]
> [/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1600]
> [/home/canadianart/bricolage/lib/Bric/Util/Burner.pm:1340]
> [/home/canadianart/bricolage/lib/Bric/Util/Job/Pub.pm:187]
> [/home/canadianart/bricolage/lib/Bric/Util/Job.pm:1886]
> [/home/canadianart/bricolage/bin/bric_queued:246]
> [/home/canadianart/bricolage/bin/bric_queued:212]
> Bric::Biz::Category->keywords has been deprecated.
> Use Bric::Biz::Category->get_keywords instead at /home/canadianart/bricolage/lib/Bric/Biz/Category.pm line 1230.
>
> I am trying to publish a story which had been successfully published before.
>
> Any thoughts on what is wrong with my settings?
>
> thank you
> Dawn
>

leggn at denison

May 2, 2011, 4:46 AM

Post #3 of 16 (2365 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi Dawn,

It sounds like bric_queued had not been started, or it crashed. This
happens to us sometimes. It leaks memory until a certain point, then
dies. I set up a cron job which restarts bric_queued every 12 hours to
avoid this problem.

Thanks,
Nick

On 5/1/2011 8:42 PM, Dawn Buie wrote:
> Update:
>
> OK I logged into the bricolage server and typed this command
>
> /home/sitename/bricolage/bin/bric_queuedctl start
>
> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now)
>
> However I did the same thing on Friday and fully expected bric_queued to keep working.
>
> Is there a better way for me to set it up to make sure it keeps running?
>
> thanks
> Dawn

uchuha at gmail

May 2, 2011, 6:29 AM

Post #4 of 16 (2362 views)
Permalink

Re: setting up bric_queued [In reply to]

We use Monit:

http://mmonit.com/monit/

There should be two instances of bric_queued running at any time. One of
them is for publishing and the other one is for distribution. The publishing
one should be checked for memory use from time to time and sent a term
signal then restarted if it goes over. Allow some time for the restart.

On May 2, 2011 1:46 PM, "Nick Legg" <leggn [at] denison> wrote:
> Hi Dawn,
>
> It sounds like bric_queued had not been started, or it crashed. This
> happens to us sometimes. It leaks memory until a certain point, then
> dies. I set up a cron job which restarts bric_queued every 12 hours to
> avoid this problem.
>
> Thanks,
> Nick
>
> On 5/1/2011 8:42 PM, Dawn Buie wrote:
>> Update:
>>
>> OK I logged into the bricolage server and typed this command
>>
>> /home/sitename/bricolage/bin/bric_queuedctl start
>>
>> voila it cleared the job queue. (I did recall it working on Friday so I
was surprised it wasn't working now)
>>
>> However I did the same thing on Friday and fully expected bric_queued to
keep working.
>>
>> Is there a better way for me to set it up to make sure it keeps running?
>>
>> thanks
>> Dawn

dawn at dawnthots

May 2, 2011, 8:07 AM

Post #5 of 16 (2364 views)
Permalink

Re: setting up bric_queued [In reply to]

On 2011-05-02, at 7:46 AM, Nick Legg wrote:

> Hi Dawn,
>
> It sounds like bric_queued had not been started, or it crashed. This happens to us sometimes. It leaks memory until a certain point, then dies. I set up a cron job which restarts bric_queued every 12 hours to avoid this problem.

Oh that's interesting. And what does memory leak really mean?

I've asked Gossamer Hosts to make sure it's always running. But if they don't have a way to do that I'lll move to the cronjob -

thanks
Dawn

>
> Thanks,
> Nick
>
> On 5/1/2011 8:42 PM, Dawn Buie wrote:
>> Update:
>>
>> OK I logged into the bricolage server and typed this command
>>
>> /home/sitename/bricolage/bin/bric_queuedctl start
>>
>> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now)
>>
>> However I did the same thing on Friday and fully expected bric_queued to keep working.
>>
>> Is there a better way for me to set it up to make sure it keeps running?
>>
>> thanks
>> Dawn
>

jaroskim at who

May 2, 2011, 8:10 AM

Post #6 of 16 (2362 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi Dawn,

A memory leak is when a program requests memory from the system, and then
fails to give it back when it's done. As I understand there is some
controversy over whether the behaviour of Perl is considered a memory leak
or not, since apparently it fails to give the memory back by design.

At least that's my recollection.

In practice what it means is that you can't have a long-running perl
daemon that does anything very interesting. You have to kill it off from
time to time and restart it.

-mark

____________________________________
From: Dawn Buie
Sent: Mon, May 02, 2011 at 11:07:07AM -0400
To: To devel [at] lists
Subject: Re: setting up bric_queued

>
> On 2011-05-02, at 7:46 AM, Nick Legg wrote:
>
> > Hi Dawn,
> >
> > It sounds like bric_queued had not been started, or it crashed. This happens to us sometimes. It leaks memory until a certain point, then dies. I set up a cron job which restarts bric_queued every 12 hours to avoid this problem.
>
> Oh that's interesting. And what does memory leak really mean?
>
> I've asked Gossamer Hosts to make sure it's always running. But if they don't have a way to do that I'lll move to the cronjob -
>
> thanks
> Dawn
>
> >
> > Thanks,
> > Nick
> >
> > On 5/1/2011 8:42 PM, Dawn Buie wrote:
> >> Update:
> >>
> >> OK I logged into the bricolage server and typed this command
> >>
> >> /home/sitename/bricolage/bin/bric_queuedctl start
> >>
> >> voila it cleared the job queue. (I did recall it working on Friday so I was surprised it wasn't working now)
> >>
> >> However I did the same thing on Friday and fully expected bric_queued to keep working.
> >>
> >> Is there a better way for me to set it up to make sure it keeps running?
> >>
> >> thanks
> >> Dawn
> >
>

--
..........................................................................
: Mark Jaroski
: Room 9016
: World Health Organization
: +41 22 791 16 65
:
..........................................................................
More people are killed every year by pigs than by sharks, which shows you
how good we are at evaluating risk.
-- Bruce Schneier
..........................................................................

alex at gt

May 2, 2011, 8:47 AM

Post #7 of 16 (2359 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi Mark,

> A memory leak is when a program requests memory from the system, and then
> fails to give it back when it's done. As I understand there is some
> controversy over whether the behaviour of Perl is considered a memory leak
> or not, since apparently it fails to give the memory back by design.
>
> At least that's my recollection.
>
> In practice what it means is that you can't have a long-running perl
> daemon that does anything very interesting. You have to kill it off from
> time to time and restart it.

Perl doesn't release the memory allocated, but does reuse it. So you can
have Perl daemon's running for a long period of time and doing
interesting things, just can't have them keep growing in memory (much
like any language). =)

Cheers,

Alex

--
Alex Krohn <alex [at] gt>

mark at geekhive

May 2, 2011, 11:23 AM

Post #8 of 16 (2362 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi Alex,

That's a very good point. Maybe we should consider hunting the leak down
with

On Mon, May 2, 2011 at 17:47, Alex Krohn <alex [at] gt> wrote:

> Hi Mark,
>
> > A memory leak is when a program requests memory from the system, and then
> > fails to give it back when it's done. As I understand there is some
> > controversy over whether the behaviour of Perl is considered a memory
> leak
> > or not, since apparently it fails to give the memory back by design.
> >
> > At least that's my recollection.
> >
> > In practice what it means is that you can't have a long-running perl
> > daemon that does anything very interesting. You have to kill it off from
> > time to time and restart it.
>
> Perl doesn't release the memory allocated, but does reuse it. So you can
> have Perl daemon's running for a long period of time and doing
> interesting things, just can't have them keep growing in memory (much
> like any language). =)
>
> Cheers,
>
> Alex
>
> --
> Alex Krohn <alex [at] gt>
>
>

mark at geekhive

May 2, 2011, 11:25 AM

Post #9 of 16 (2360 views)
Permalink

Re: setting up bric_queued [In reply to]

oops... Anyhow, to continue:

http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm

If I had to guess I'd say bric_queued would be an easier debugging
environment than mod_perl, and there's a pretty good chance that if we can
fix the leak under the one it will cover the other as well, since all
bric_queued does is call a burner.

I wonder if it's worth giving up a few nights or a weekend for this.

-mark

On Mon, May 2, 2011 at 20:23, Mark Jaroski <mark [at] geekhive> wrote:

> Hi Alex,
>
> That's a very good point. Maybe we should consider hunting the leak down
> with
>
>
> On Mon, May 2, 2011 at 17:47, Alex Krohn <alex [at] gt> wrote:
>
>> Hi Mark,
>>
>> > A memory leak is when a program requests memory from the system, and
>> then
>> > fails to give it back when it's done. As I understand there is some
>> > controversy over whether the behaviour of Perl is considered a memory
>> leak
>> > or not, since apparently it fails to give the memory back by design.
>> >
>> > At least that's my recollection.
>> >
>> > In practice what it means is that you can't have a long-running perl
>> > daemon that does anything very interesting. You have to kill it off from
>> > time to time and restart it.
>>
>> Perl doesn't release the memory allocated, but does reuse it. So you can
>> have Perl daemon's running for a long period of time and doing
>> interesting things, just can't have them keep growing in memory (much
>> like any language). =)
>>
>> Cheers,
>>
>> Alex
>>
>> --
>> Alex Krohn <alex [at] gt>
>>
>>
>

david at kineticode

May 2, 2011, 12:04 PM

Post #10 of 16 (2360 views)
Permalink

Re: setting up bric_queued [In reply to]

On May 2, 2011, at 11:25 AM, Mark Jaroski wrote:

> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm
>
> If I had to guess I'd say bric_queued would be an easier debugging
> environment than mod_perl, and there's a pretty good chance that if we can
> fix the leak under the one it will cover the other as well, since all
> bric_queued does is call a burner.
>
> I wonder if it's worth giving up a few nights or a weekend for this.

+1

David

dawn at dawnthots

May 2, 2011, 12:14 PM

Post #11 of 16 (2363 views)
Permalink

Re: setting up bric_queued [In reply to]

On 2011-05-02, at 9:29 AM, Mark Jaroski wrote:

> We use Monit:
>
> http://mmonit.com/monit/

Alex is this similar to what Gossamer uses?

>
> There should be two instances of bric_queued running at any time. One of
> them is for publishing and the other one is for distribution. The publishing
> one should be checked for memory use from time to time and sent a term
> signal then restarted if it goes over. Allow some time for the restart.

Interesting. I think it would be good to hunt down and kill the 'memory leak' and then to better document how to use bric_queued. I could help with the later at bric hack hack day.

Dawn

>
>
> On May 2, 2011 1:46 PM, "Nick Legg" <leggn [at] denison> wrote:
>> Hi Dawn,
>>
>> It sounds like bric_queued had not been started, or it crashed. This
>> happens to us sometimes. It leaks memory until a certain point, then
>> dies. I set up a cron job which restarts bric_queued every 12 hours to
>> avoid this problem.
>>
>> Thanks,
>> Nick
>>
>> On 5/1/2011 8:42 PM, Dawn Buie wrote:
>>> Update:
>>>
>>> OK I logged into the bricolage server and typed this command
>>>
>>> /home/sitename/bricolage/bin/bric_queuedctl start
>>>
>>> voila it cleared the job queue. (I did recall it working on Friday so I
> was surprised it wasn't working now)
>>>
>>> However I did the same thing on Friday and fully expected bric_queued to
> keep working.
>>>
>>> Is there a better way for me to set it up to make sure it keeps running?
>>>
>>> thanks
>>> Dawn

alex at gt

May 2, 2011, 12:20 PM

Post #12 of 16 (2367 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi Mark,

> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm
>
> If I had to guess I'd say bric_queued would be an easier debugging
> environment than mod_perl, and there's a pretty good chance that if we can
> fix the leak under the one it will cover the other as well, since all
> bric_queued does is call a burner.
>
> I wonder if it's worth giving up a few nights or a weekend for this.

Definitely. CMS Hack Day is May 9th. =)

The problems we've seen have always been in the parent process (which
does the Publishing queue). The child proc handling the dist queue never
grows.

One workaround would be to re-archictecht it as a parent process that
forks off a child process for each publish. That way the memory is
released (similar to setting a MaxRequestsPerchild in Apache). If people
are interested in that, it's probably not too much work to do and
isolated to just changed bric_queued.

Not nearly as nice as a finding the leak though. =)

Cheers,

Alex

--
Alex Krohn <alex [at] gt>

alex at gt

May 2, 2011, 12:23 PM

Post #13 of 16 (2362 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi,

> > We use Monit:
> >
> > http://mmonit.com/monit/
>
> Alex is this similar to what Gossamer uses?

Yup! =)

> > There should be two instances of bric_queued running at any time. One of
> > them is for publishing and the other one is for distribution. The publishing
> > one should be checked for memory use from time to time and sent a term
> > signal then restarted if it goes over. Allow some time for the restart.
>
> Interesting. I think it would be good to hunt down and kill the 'memory
> leak' and then to better document how to use bric_queued. I could help
> with the later at bric hack hack day.

It can be very difficult to track down, especially if there are circular
references around.

We've got some nice mod_perl handlers for doing memory audits (i.e. logs
memory status before and after each request, differences in %INC, all
the request details, etc).

Cheers,

Alex

--
Alex Krohn <alex [at] gt>

phillip at communitybandwidth

May 2, 2011, 12:42 PM

Post #14 of 16 (2371 views)
Permalink

Re: setting up bric_queued [In reply to]

On 2011-05-02, at 3:04 PM, David E. Wheeler wrote:

> On May 2, 2011, at 11:25 AM, Mark Jaroski wrote:
>
>> http://search.cpan.org/dist/Devel-LeakTrace-Fast/lib/Devel/LeakTrace/Fast.pm
>>
>> If I had to guess I'd say bric_queued would be an easier debugging
>> environment than mod_perl, and there's a pretty good chance that if we can
>> fix the leak under the one it will cover the other as well, since all
>> bric_queued does is call a burner.
>>
>> I wonder if it's worth giving up a few nights or a weekend for this.
>
> +1

Hack Day?
https://github.com/bricoleurs/bricolage/wiki/Bricolage-CMS-Hack-Days/

--
Phillip Smith
http://phillipadsmith.com
http://twitter.com/phillipadsmith
http://linkedin.com/in/phillipadsmith

mark at geekhive

May 2, 2011, 12:49 PM

Post #15 of 16 (2366 views)
Permalink

Re: setting up bric_queued [In reply to]

Well, running child processes in their own environment isn't
*that*different from just killing bric_queued when it gets too big. I
guess it's a
bit cleaner, but really it would be awesome not to have to limit the memory
use under Apache either.
So, Monday, may 9th, hmmmm. I'll see if I can get my schedule clear of IAM
duties.

mark at geekhive

May 9, 2011, 10:07 PM

Post #16 of 16 (2331 views)
Permalink

Re: setting up bric_queued [In reply to]

Hi all,

I spent some time with the publishing system last night, without any
immediate success. However I do have a plan for capturing memory-use data
from a running bric_queued.

Naturally data from the production system would be best, since it's under
the heaviest load.

Now I just have to talk our web team into it.

-mark

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads