Mailing List Archive: RSyslog: users

rsyslog 4.6.4 stops logging to local files

Index | Next | Previous | View Threaded

Juerg.Haefliger at hp

Jun 14, 2012, 8:04 AM

Post #1 of 6 (216 views)
Permalink

rsyslog 4.6.4 stops logging to local files

Hi,

I'm experiencing sporadic kernel hangs on different machines running Ubuntu Natty (rsyslog 4.6.4). The machines need to be rebooted to recover. What I noticed after the machines come back is that syslog and kern.log have not been updated for some time (anywhere from a few minutes to several hours) prior to the hang. Interestingly, the updates of syslog and kern.log stopped at different times. Whether this has anything to do with the later hang or is an early indication that something is wrong, I don't know. If I were to catch a machine in this state, what can I look at to try to figure out why logging to the filesystem stopped? I can't run rsyslog in debug mode. I could attach gdb to the running rsyslog process but don't know what to look for.

Thanks
...Juerg

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

rgerhards at hq

Jun 14, 2012, 8:11 AM

Post #2 of 6 (210 views)
Permalink

Re: rsyslog 4.6.4 stops logging to local files [In reply to]

> I'm experiencing sporadic kernel hangs on different machines running
> Ubuntu Natty (rsyslog 4.6.4). The machines need to be rebooted to
> recover. What I noticed after the machines come back is that syslog and
> kern.log have not been updated for some time (anywhere from a few
> minutes to several hours) prior to the hang. Interestingly, the updates
> of syslog and kern.log stopped at different times. Whether this has
> anything to do with the later hang or is an early indication that
> something is wrong, I don't know. If I were to catch a machine in this
> state, what can I look at to try to figure out why logging to the
> filesystem stopped? I can't run rsyslog in debug mode. I could attach
> gdb to the running rsyslog process but don't know what to look for.

I think that is one of the many bugs that has been fixed in the past years since that version was released. I suggest that you open a bug tracker with ubuntu. Maybe they solve this by re-creating the packages with recent builds.

I am not bashing: that's most probably really the solution, and it would be great if they did. AFAIK there are recent .deb packages, so this should not cause too much trouble to them...

I understand this is not the answer you were looking for. Sorry for that. But even if I dug out which patch you need, it wouldn't go into the ubuntu package, so there is no point in that. If you can build from source, you can do that, but then simply use the most recent released version (4.8.somewhat for the v4 tree). Note that v4 is officially too old to be supported by the project itself. We are right now at 6.2.2.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Juerg.Haefliger at hp

Jun 14, 2012, 8:26 AM

Post #3 of 6 (208 views)
Permalink

Re: rsyslog 4.6.4 stops logging to local files [In reply to]

> > I'm experiencing sporadic kernel hangs on different machines running
> > Ubuntu Natty (rsyslog 4.6.4). The machines need to be rebooted to
> > recover. What I noticed after the machines come back is that syslog and
> > kern.log have not been updated for some time (anywhere from a few
> > minutes to several hours) prior to the hang. Interestingly, the updates
> > of syslog and kern.log stopped at different times. Whether this has
> > anything to do with the later hang or is an early indication that
> > something is wrong, I don't know. If I were to catch a machine in this
> > state, what can I look at to try to figure out why logging to the
> > filesystem stopped? I can't run rsyslog in debug mode. I could attach
> > gdb to the running rsyslog process but don't know what to look for.
>
> I think that is one of the many bugs that has been fixed in the past
> years since that version was released.

I was afraid you were going to say that :-)

> I suggest that you open a bug
> tracker with ubuntu.

Yes, that is on my todo list.

> Maybe they solve this by re-creating the packages
> with recent builds.

They usually don't just pull in later builds for older distros, they backport patches that fix known problems. We're currently stuck with Natty so upgrading to a later distro is not an option at the moment.

> I am not bashing: that's most probably really the solution, and it would
> be great if they did. AFAIK there are recent .deb packages, so this
> should not cause too much trouble to them...
>
> I understand this is not the answer you were looking for. Sorry for that.
> But even if I dug out which patch you need, it wouldn't go into the
> ubuntu package, so there is no point in that.

In fact it would. We have a contract with Canonical and if I had a patch, we can ask them to backport and build a new Natty package. Do you happen to remember what the problem was (or might have been)? Some keyword maybe so that I can go hunt for the patch? Note that I'm only seeing this problem on machines that hung so I'm wondering if something else is going on that causes rsyslog to stop logging. I'm not trying to blame rsyslog, I'm just trying to find the cause of the hang and the syslog funkiness is all I've got at the moment to go after.

> If you can build from
> source, you can do that, but then simply use the most recent released
> version (4.8.somewhat for the v4 tree). Note that v4 is officially too
> old to be supported by the project itself. We are right now at 6.2.2.

Understood. But these are production machines and I can't install unsupported (by Canonical) packages.

...Juerg

> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

rgerhards at hq

Jun 14, 2012, 9:36 AM

Post #4 of 6 (211 views)
Permalink

Re: rsyslog 4.6.4 stops logging to local files [In reply to]

> > I understand this is not the answer you were looking for. Sorry for that.
> > But even if I dug out which patch you need, it wouldn't go into the
> > ubuntu package, so there is no point in that.
>
> In fact it would. We have a contract with Canonical and if I had a patch, we can
> ask them to backport and build a new Natty package.

I find this interesting. For folks that have a contract with us, we go find bugs&patches ourselves when problems are reported (even in old versions). Do you mean you have a contract with Canonical and they will only act under it if you provide the patch yourself? Strange...

>Do you happen to
> remember what the problem was (or might have been)? Some keyword maybe
> so that I can go hunt for the patch? Note that I'm only seeing this problem on
> machines that hung so I'm wondering if something else is going on that causes
> rsyslog to stop logging. I'm not trying to blame rsyslog, I'm just trying to find the
> cause of the hang and the syslog funkiness is all I've got at the moment to go
> after.

My guess is that the problem is related to light delay mode, probably in imuxsock. At least that immediately triggered my mind. But I have no idea when exactly this was addressed. The ChangeLog (retrieve via git if you don't have it otherwise) will possibly help.

Rainer
>
> > If you can build from
> > source, you can do that, but then simply use the most recent released
> > version (4.8.somewhat for the v4 tree). Note that v4 is officially too
> > old to be supported by the project itself. We are right now at 6.2.2.
>
> Understood. But these are production machines and I can't install unsupported
> (by Canonical) packages.
>
> ...Juerg
>
>
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Juerg.Haefliger at hp

Jun 14, 2012, 9:40 AM

Post #5 of 6 (210 views)
Permalink

Re: rsyslog 4.6.4 stops logging to local files [In reply to]

> > > I understand this is not the answer you were looking for. Sorry for that.
> > > But even if I dug out which patch you need, it wouldn't go into the
> > > ubuntu package, so there is no point in that.
> >
> > In fact it would. We have a contract with Canonical and if I had a patch, we can
> > ask them to backport and build a new Natty package.
>
> I find this interesting. For folks that have a contract with us, we go
> find bugs&patches ourselves when problems are reported (even in old
> versions). Do you mean you have a contract with Canonical and they will
> only act under it if you provide the patch yourself? Strange...

No they'll act on their own but the turn-around-time is much shorter if I'll present them the patch on a silver plate :-)

> >Do you happen to
> > remember what the problem was (or might have been)? Some keyword maybe
> > so that I can go hunt for the patch? Note that I'm only seeing this problem on
> > machines that hung so I'm wondering if something else is going on that causes
> > rsyslog to stop logging. I'm not trying to blame rsyslog, I'm just trying to find the
> > cause of the hang and the syslog funkiness is all I've got at the moment to go
> > after.
>
> My guess is that the problem is related to light delay mode, probably in
> imuxsock. At least that immediately triggered my mind. But I have no idea
> when exactly this was addressed. The ChangeLog (retrieve via git if you
> don't have it otherwise) will possibly help.

Very good. Will do. Thanks a lot!

...Juerg

> Rainer
> >
> > > If you can build from
> > > source, you can do that, but then simply use the most recent released
> > > version (4.8.somewhat for the v4 tree). Note that v4 is officially too
> > > old to be supported by the project itself. We are right now at 6.2.2.
> >
> > Understood. But these are production machines and I can't install unsupported
> > (by Canonical) packages.
> >
> > ...Juerg
> >
> >
> > > Rainer
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

rgerhards at hq

Jun 14, 2012, 9:52 AM

Post #6 of 6 (208 views)
Permalink

Re: rsyslog 4.6.4 stops logging to local files [In reply to]

> > I find this interesting. For folks that have a contract with us, we go
> > find bugs&patches ourselves when problems are reported (even in old
> > versions). Do you mean you have a contract with Canonical and they will
> > only act under it if you provide the patch yourself? Strange...
>
> No they'll act on their own but the turn-around-time is much shorter if I'll
> present them the patch on a silver plate :-)

I guess I now know why I receive so many patches from Red Hat and so few (any? Mmhhhh) from canonical ;)

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads