Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

rsyslog-5.8.11 omit syncing is always on?

 

 

First page Previous page 1 2 Next page Last page  View All RSyslog users RSS feed   Index | Next | Previous | View Threaded


marcin at mejor

May 4, 2012, 2:13 AM

Post #1 of 28 (1742 views)
Permalink
rsyslog-5.8.11 omit syncing is always on?

Hello,
i'm noticing messages appears in log with huge delay. For example when
i run `logger test` it appears in message.log after 5-10 minutes (or
longer) or appears when i shutdown rsyslog.
I tried with and without "-" before file name
*.info;mail.none;authpriv.none;cron.none /var/log/messages.log
and it didn't change behavior.
What could i miss in configuration?
I've got omrelp and "$ActionQueueType LinkedList" activated if it has
any signification.
Regards,
Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 4, 2012, 2:23 AM

Post #2 of 28 (1728 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

This sounds like async writing is turned on. I guess you did not activate it?

None of this code (or the sync) has been touched in 5.8.11.
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Friday, May 04, 2012 11:14 AM
> To: rsyslog [at] lists
> Subject: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> Hello,
> i'm noticing messages appears in log with huge delay. For example when
> i run `logger test` it appears in message.log after 5-10 minutes (or
> longer) or appears when i shutdown rsyslog.
> I tried with and without "-" before file name
> *.info;mail.none;authpriv.none;cron.none /var/log/messages.log
> and it didn't change behavior.
> What could i miss in configuration?
> I've got omrelp and "$ActionQueueType LinkedList" activated if it has
> any signification.
> Regards,
> Marcin
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

May 4, 2012, 2:37 AM

Post #3 of 28 (1722 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 2012-05-04 11:23, Rainer Gerhards napisał(a):
> This sounds like async writing is turned on. I guess you did not
> activate it?
>
> None of this code (or the sync) has been touched in 5.8.11.

I added now: $OMFileAsyncWriting off
and there is no change.
I don't know in which version something changed, i've upgraded from
5.6.5 to brand new 5.8.11
and noticed such situation.
BTW, i'm still testing but it looks 5.8.11 has no problem with quiting.
Many thanks for it!
Marcin

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 4, 2012, 4:00 AM

Post #4 of 28 (1719 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Friday, May 04, 2012 11:38 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 2012-05-04 11:23, Rainer Gerhards napisał(a):
> > This sounds like async writing is turned on. I guess you did not
> > activate it?
> >
> > None of this code (or the sync) has been touched in 5.8.11.
>
> I added now: $OMFileAsyncWriting off
> and there is no change.
> I don't know in which version something changed, i've upgraded from
> 5.6.5 to brand new 5.8.11
> and noticed such situation.

Can you provide me a debug log (private mail is OK)? I just re-tried it, and
everything works for me. Probably environment-induced, in which case the
debug log should shed some light...

Rainer

> BTW, i'm still testing but it looks 5.8.11 has no problem with quiting.
> Many thanks for it!
> Marcin
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

May 4, 2012, 7:34 AM

Post #5 of 28 (1721 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 2012-05-04 13:00, Rainer Gerhards pisze:
> Can you provide me a debug log (private mail is OK)? I just re-tried it, and
> everything works for me. Probably environment-induced, in which case the
> debug log should shed some light...

When i run rsyslog in debug mode ( LC_ALL="en_US.utf.8"
RSYSLOG_DEBUG="debug PrintFuncDB PrintMutexAction" rsyslogd -c5 -dn
>rsyslogd.deb.log ), problem didn't occur :(
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 4, 2012, 9:12 AM

Post #6 of 28 (1727 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Friday, May 04, 2012 4:34 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 2012-05-04 13:00, Rainer Gerhards pisze:
> > Can you provide me a debug log (private mail is OK)? I just re-tried
> it, and
> > everything works for me. Probably environment-induced, in which case
> the
> > debug log should shed some light...
>
> When i run rsyslog in debug mode ( LC_ALL="en_US.utf.8"
> RSYSLOG_DEBUG="debug PrintFuncDB PrintMutexAction" rsyslogd -c5 -dn
> >rsyslogd.deb.log ), problem didn't occur :(

Mhhh, that's bad. Can you set RSYSLOG_DEBUG just to "debug" but nothing else?
Maybe this helps...

Probably it also explains why I don't see the issue.

Rainer

> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


ju2wheels at gmail

May 16, 2012, 11:01 PM

Post #7 of 28 (1709 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

Hi Rainer,

Im not sure if its caused by the same thing as OP below, but I have this
same issue with 5.8.10 that ive been working through for the past few
hours where any messages sent with 'logger' are extremely delayed or
disappear into the ether all together. Its happening with a default
unmodified config.

I have a set of bash functions I use and I have a generic log function
that uses 'logger' to send detailed output to syslog, but even after
having ran commands for about 4 hrs theres tons of missing messages in
/var/log/messages. Only when I enabled '$ActionFileEnableSync on' do all
the log messages start appearing normally. Is that supposed to be normal
behavior? I was under the impression from the comment on this setting
that the messages would be delayed since it was related to rsyslog
forcing a sync on write, not lost (from what I can tell).

Thanks,

On 05/04/2012 12:12 PM, Rainer Gerhards wrote:
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Marcin Miroslaw
>> Sent: Friday, May 04, 2012 4:34 PM
>> To: rsyslog [at] lists
>> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>>
>> W dniu 2012-05-04 13:00, Rainer Gerhards pisze:
>>> Can you provide me a debug log (private mail is OK)? I just re-tried
>> it, and
>>> everything works for me. Probably environment-induced, in which case
>> the
>>> debug log should shed some light...
>> When i run rsyslog in debug mode ( LC_ALL="en_US.utf.8"
>> RSYSLOG_DEBUG="debug PrintFuncDB PrintMutexAction" rsyslogd -c5 -dn
>>> rsyslogd.deb.log ), problem didn't occur :(
> Mhhh, that's bad. Can you set RSYSLOG_DEBUG just to "debug" but nothing else?
> Maybe this helps...
>
> Probably it also explains why I don't see the issue.
>
> Rainer
>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

May 17, 2012, 1:58 PM

Post #8 of 28 (1706 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

I use logger a lot (a couple hundred servers writing a few dozen lines of
logs each every minute) and have never run into a problem of it now
showing up in the log.

syncing should only matter if the system crashes, normal operation should
not need syncing enabled (and enabling it will cripple your performance)

David Lang

On Thu, 17 May 2012, Julio Lajara wrote:

> Date: Thu, 17 May 2012 02:01:59 -0400
> From: Julio Lajara <ju2wheels [at] gmail>
> Reply-To: julio.lajara [at] alum, rsyslog-users <rsyslog [at] lists>
> To: rsyslog-users <rsyslog [at] lists>
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> Hi Rainer,
>
> Im not sure if its caused by the same thing as OP below, but I have this same
> issue with 5.8.10 that ive been working through for the past few hours where
> any messages sent with 'logger' are extremely delayed or disappear into the
> ether all together. Its happening with a default unmodified config.
>
> I have a set of bash functions I use and I have a generic log function that
> uses 'logger' to send detailed output to syslog, but even after having ran
> commands for about 4 hrs theres tons of missing messages in
> /var/log/messages. Only when I enabled '$ActionFileEnableSync on' do all the
> log messages start appearing normally. Is that supposed to be normal
> behavior? I was under the impression from the comment on this setting that
> the messages would be delayed since it was related to rsyslog forcing a sync
> on write, not lost (from what I can tell).
>
> Thanks,
>
> On 05/04/2012 12:12 PM, Rainer Gerhards wrote:
>>> -----Original Message-----
>>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>>> bounces [at] lists] On Behalf Of Marcin Miroslaw
>>> Sent: Friday, May 04, 2012 4:34 PM
>>> To: rsyslog [at] lists
>>> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>>>
>>> W dniu 2012-05-04 13:00, Rainer Gerhards pisze:
>>>> Can you provide me a debug log (private mail is OK)? I just re-tried
>>> it, and
>>>> everything works for me. Probably environment-induced, in which case
>>> the
>>>> debug log should shed some light...
>>> When i run rsyslog in debug mode ( LC_ALL="en_US.utf.8"
>>> RSYSLOG_DEBUG="debug PrintFuncDB PrintMutexAction" rsyslogd -c5 -dn
>>>> rsyslogd.deb.log ), problem didn't occur :(
>> Mhhh, that's bad. Can you set RSYSLOG_DEBUG just to "debug" but nothing
>> else?
>> Maybe this helps...
>>
>> Probably it also explains why I don't see the issue.
>>
>> Rainer
>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

May 22, 2012, 4:29 AM

Post #9 of 28 (1686 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 17.05.2012 08:01, Julio Lajara pisze:
Hi Julio!

> Im not sure if its caused by the same thing as OP below, but I have this
> same issue with 5.8.10 that ive been working through for the past few
> hours where any messages sent with 'logger' are extremely delayed or
> disappear into the ether all together. Its happening with a default
> unmodified config.

My issue seems to be very similar to yours, i noticed disappearing
messages too. I've created debug log (and sended offline to Rainer) then
i did downgrade.
Julio, do you use relp or do you forward logs to another host?
Regards,
Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 22, 2012, 4:48 AM

Post #10 of 28 (1680 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

Hi Folks,

just to break the silence: this issue is on my list of things to look at. As
usual, it's just quite busy (and thankfully with paid work as well), so this
waits on a timeslot...

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Tuesday, May 22, 2012 1:29 PM
> To: julio.lajara [at] alum; rsyslog-users
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 17.05.2012 08:01, Julio Lajara pisze:
> Hi Julio!
>
> > Im not sure if its caused by the same thing as OP below, but I have
> this
> > same issue with 5.8.10 that ive been working through for the past few
> > hours where any messages sent with 'logger' are extremely delayed or
> > disappear into the ether all together. Its happening with a default
> > unmodified config.
>
> My issue seems to be very similar to yours, i noticed disappearing
> messages too. I've created debug log (and sended offline to Rainer)
> then
> i did downgrade.
> Julio, do you use relp or do you forward logs to another host?
> Regards,
> Marcin
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

May 22, 2012, 8:25 AM

Post #11 of 28 (1683 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 22.05.2012 13:48, Rainer Gerhards pisze:
> Hi Folks,
>
> just to break the silence: this issue is on my list of things to look at. As
> usual, it's just quite busy (and thankfully with paid work as well), so this
> waits on a timeslot...

Hi!
I know you have many work. I asked Julio about his use case because i
have a little problem to reproduce situation on test box.
Now meseems i can reproduce problem with maximally stripped config file[1].
This steps should allow to reproduce situation:
1. start rsyslog with given config
2. cat "some text file, a several dozen of lines should be enough" >>
/var/log/exim/exim_main.log
3. try to use logger and log any message
4. if it works wait sometime, eventually repeat step 2
5. try to use logger

David, you didn't hit such problem because you probably doesn't monitor
files and send logs via relp. (In my tests it looks like both situation
are needed: filemonitor and sending via relp).

Regards,
Marcin

[1] - config file:
$ModLoad immark.so
$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imrelp.so
$ModLoad omrelp.so
$ModLoad imfile

$WorkDirectory /var/spool/rsyslog
$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1
$MainMsgQueueMaxDiskSpace 312M
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueCheckpointInterval 80
$MainMsgQueueSyncQueueFiles on
$MainMsgQueueTimeoutShutdown 10000

$InputFileName /var/log/exim/exim_main.log
$InputFileTag meteor.exim_main:
$InputFileStateFile meteor.exim_main
$InputFileFacility mail
$InputFilePollInterval 1
$InputRunFileMonitor

*.* :omrelp:10.10.100.101:20514
*.* /var/log/wszystko.log

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 22, 2012, 8:27 AM

Post #12 of 28 (1690 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

Excellent, small config is always good! I'll try to reproduce this tomorrow
morning. Thanks for your persistence!
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Tuesday, May 22, 2012 5:25 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 22.05.2012 13:48, Rainer Gerhards pisze:
> > Hi Folks,
> >
> > just to break the silence: this issue is on my list of things to look
> at. As
> > usual, it's just quite busy (and thankfully with paid work as well),
> so this
> > waits on a timeslot...
>
> Hi!
> I know you have many work. I asked Julio about his use case because i
> have a little problem to reproduce situation on test box.
> Now meseems i can reproduce problem with maximally stripped config
> file[1].
> This steps should allow to reproduce situation:
> 1. start rsyslog with given config
> 2. cat "some text file, a several dozen of lines should be enough" >>
> /var/log/exim/exim_main.log
> 3. try to use logger and log any message
> 4. if it works wait sometime, eventually repeat step 2
> 5. try to use logger
>
> David, you didn't hit such problem because you probably doesn't monitor
> files and send logs via relp. (In my tests it looks like both situation
> are needed: filemonitor and sending via relp).
>
> Regards,
> Marcin
>
> [1] - config file:
> $ModLoad immark.so
> $ModLoad imuxsock.so
> $ModLoad imklog.so
> $ModLoad imrelp.so
> $ModLoad omrelp.so
> $ModLoad imfile
>
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueType LinkedList
> $ActionQueueFileName dbq
> $ActionResumeRetryCount -1
> $MainMsgQueueMaxDiskSpace 312M
> $MainMsgQueueSaveOnShutdown on
> $MainMsgQueueCheckpointInterval 80
> $MainMsgQueueSyncQueueFiles on
> $MainMsgQueueTimeoutShutdown 10000
>
> $InputFileName /var/log/exim/exim_main.log
> $InputFileTag meteor.exim_main:
> $InputFileStateFile meteor.exim_main
> $InputFileFacility mail
> $InputFilePollInterval 1
> $InputRunFileMonitor
>
> *.* :omrelp:10.10.100.101:20514
> *.* /var/log/wszystko.log
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


ju2wheels at gmail

May 22, 2012, 8:43 AM

Post #13 of 28 (1695 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

Hi Marcin, glad to see im not the only one experiencing the problem. I
have the same setup you do, I am using RELP and file monitoring as well.

The main reason I noticed it was because im migrating from sysklogd to
rsyslog on our prod RHEL5 boxes and boxes that normally had a lot of
syslog output suddenly had none or extremely delayed logging. I never
added any config changes to the queue's as I didnt think we had enough
volume to merit changing the defaults but thats what im trying first on
my end.

Ive also had a weird issue this weekend where rsyslog/pdns are not
playing well together and testing has proven (by using restoring rsyslog
configs to defaults) that the delay related to rsyslog caused pdns to be
unable to respond to dns lookups after a reboot.

Ill try your test config on one of my lab boxes and follow up.

Thanks,

On 05/22/2012 11:27 AM, Rainer Gerhards wrote:
> Excellent, small config is always good! I'll try to reproduce this tomorrow
> morning. Thanks for your persistence!
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Marcin Miroslaw
>> Sent: Tuesday, May 22, 2012 5:25 PM
>> To: rsyslog [at] lists
>> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>>
>> W dniu 22.05.2012 13:48, Rainer Gerhards pisze:
>>> Hi Folks,
>>>
>>> just to break the silence: this issue is on my list of things to look
>> at. As
>>> usual, it's just quite busy (and thankfully with paid work as well),
>> so this
>>> waits on a timeslot...
>> Hi!
>> I know you have many work. I asked Julio about his use case because i
>> have a little problem to reproduce situation on test box.
>> Now meseems i can reproduce problem with maximally stripped config
>> file[1].
>> This steps should allow to reproduce situation:
>> 1. start rsyslog with given config
>> 2. cat "some text file, a several dozen of lines should be enough">>
>> /var/log/exim/exim_main.log
>> 3. try to use logger and log any message
>> 4. if it works wait sometime, eventually repeat step 2
>> 5. try to use logger
>>
>> David, you didn't hit such problem because you probably doesn't monitor
>> files and send logs via relp. (In my tests it looks like both situation
>> are needed: filemonitor and sending via relp).
>>
>> Regards,
>> Marcin
>>
>> [1] - config file:
>> $ModLoad immark.so
>> $ModLoad imuxsock.so
>> $ModLoad imklog.so
>> $ModLoad imrelp.so
>> $ModLoad omrelp.so
>> $ModLoad imfile
>>
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueType LinkedList
>> $ActionQueueFileName dbq
>> $ActionResumeRetryCount -1
>> $MainMsgQueueMaxDiskSpace 312M
>> $MainMsgQueueSaveOnShutdown on
>> $MainMsgQueueCheckpointInterval 80
>> $MainMsgQueueSyncQueueFiles on
>> $MainMsgQueueTimeoutShutdown 10000
>>
>> $InputFileName /var/log/exim/exim_main.log
>> $InputFileTag meteor.exim_main:
>> $InputFileStateFile meteor.exim_main
>> $InputFileFacility mail
>> $InputFilePollInterval 1
>> $InputRunFileMonitor
>>
>> *.* :omrelp:10.10.100.101:20514
>> *.* /var/log/wszystko.log
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


ju2wheels at gmail

May 23, 2012, 12:25 PM

Post #14 of 28 (1694 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

Hi all, just wanted to follow up on my results with performance tweaking
here:

I resolved my pdns issue and log delay/loss issue with the following
changes:

1. Disabled imux socket rate limiting (this seemed to have the most
effect on stopping the loss of messages, on the pdns server I found
through debugging to have been losing about 1,000 msgs every few seconds
to this and the other default queue timeout settings. The delay on the
pdns server was enough to cause pdns to stop working within an hr of
reboot.):
#Disable imuxsock rate limit
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0

2. Changed the queue types, timeouts, and limits to ensure it was
caching to disk so I wouldnt lose any messages on MainQueue
(client/server) and also all ActionQueue on client.
$MainMsgQueueFileName client_main_queue
$MainMsgQueueType LinkedList
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueMaxFileSize 1,000M
$MainMsgQueueTimeoutActionCompletion 10,000
$MainMsgQueueTimeoutEnqueue 10,000
$MainMsgQueueDiscardMark 80,000
$MainMsgQueueSize 100,000

$ActionQueueType LinkedList
$ActionQueueFileName clientlocal_messages
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueMaxFileSize 500M
$ActionQueueTimeoutActionCompletion 10,000
$ActionQueueTimeoutEnqueue 10,000
$ActionQueueSize 20,000
$ActionQueueDiscardMark 18,000

Only thing I havent determined now is why my rsyslogd server process is
stuck at 100% cpu.

Hope this helps resolve your issue.

Thanks,

On 05/22/2012 11:27 AM, Rainer Gerhards wrote:
> Excellent, small config is always good! I'll try to reproduce this tomorrow
> morning. Thanks for your persistence!
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Marcin Miroslaw
>> Sent: Tuesday, May 22, 2012 5:25 PM
>> To: rsyslog [at] lists
>> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>>
>> W dniu 22.05.2012 13:48, Rainer Gerhards pisze:
>>> Hi Folks,
>>>
>>> just to break the silence: this issue is on my list of things to look
>> at. As
>>> usual, it's just quite busy (and thankfully with paid work as well),
>> so this
>>> waits on a timeslot...
>> Hi!
>> I know you have many work. I asked Julio about his use case because i
>> have a little problem to reproduce situation on test box.
>> Now meseems i can reproduce problem with maximally stripped config
>> file[1].
>> This steps should allow to reproduce situation:
>> 1. start rsyslog with given config
>> 2. cat "some text file, a several dozen of lines should be enough">>
>> /var/log/exim/exim_main.log
>> 3. try to use logger and log any message
>> 4. if it works wait sometime, eventually repeat step 2
>> 5. try to use logger
>>
>> David, you didn't hit such problem because you probably doesn't monitor
>> files and send logs via relp. (In my tests it looks like both situation
>> are needed: filemonitor and sending via relp).
>>
>> Regards,
>> Marcin
>>
>> [1] - config file:
>> $ModLoad immark.so
>> $ModLoad imuxsock.so
>> $ModLoad imklog.so
>> $ModLoad imrelp.so
>> $ModLoad omrelp.so
>> $ModLoad imfile
>>
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueType LinkedList
>> $ActionQueueFileName dbq
>> $ActionResumeRetryCount -1
>> $MainMsgQueueMaxDiskSpace 312M
>> $MainMsgQueueSaveOnShutdown on
>> $MainMsgQueueCheckpointInterval 80
>> $MainMsgQueueSyncQueueFiles on
>> $MainMsgQueueTimeoutShutdown 10000
>>
>> $InputFileName /var/log/exim/exim_main.log
>> $InputFileTag meteor.exim_main:
>> $InputFileStateFile meteor.exim_main
>> $InputFileFacility mail
>> $InputFilePollInterval 1
>> $InputRunFileMonitor
>>
>> *.* :omrelp:10.10.100.101:20514
>> *.* /var/log/wszystko.log
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

May 31, 2012, 3:33 AM

Post #15 of 28 (1660 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> Hi!
> I know you have many work. I asked Julio about his use case because i
> have a little problem to reproduce situation on test box.
> Now meseems i can reproduce problem with maximally stripped config file[1].
> This steps should allow to reproduce situation:
> 1. start rsyslog with given config
> 2. cat "some text file, a several dozen of lines should be enough" >>
> /var/log/exim/exim_main.log
> 3. try to use logger and log any message
> 4. if it works wait sometime, eventually repeat step 2
> 5. try to use logger

I tried to reproduce the problem today, but had no success with that. I went through several cycles but finally gave up.

Then, I reviewed the debug log that you sent with some mail in this thread. In that log, it looks like the action queue fills up and then the main queue begins to fill up as well. When the main queue is full, messages are discarded with the usual 1 (or 2) second timeout. This seems to be the cause of the problem.

HOWEVER, that is a different scenario than what you describe above, with only "several dozen of lines". Would it be possible that you provide me a debug log for the above scenario? As I said, all works perfectly for me. I even tried to shut down the remote RELP server, but that had no influence at all (as, of course, it should be).

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

Jun 1, 2012, 3:01 AM

Post #16 of 28 (1668 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 31.05.2012 12:33, Rainer Gerhards pisze:

Hello!

> I tried to reproduce the problem today, but had no success with that. I went through several cycles but finally gave up.
>
> Then, I reviewed the debug log that you sent with some mail in this thread. In that log, it looks like the action queue fills up and then the main queue begins to fill up as well. When the main queue is full, messages are discarded with the usual 1 (or 2) second timeout. This seems to be the cause of the problem.
>
> HOWEVER, that is a different scenario than what you describe above, with only "several dozen of lines". Would it be possible that you provide me a debug log for the above scenario? As I said, all works perfectly for me. I even tried to shut down the remote RELP server, but that had no influence at all (as, of course, it should be).

Rainer,
i've used one-liner:
for x in $(seq 1 1000); do logger "seq:${x} time:$(date +%T)" ; echo
"write_to_exim_log:${x} time:$(date +%T)" >> /var/log/exim/exim_main.log
; sleep 0.4; done
(Thunderbird wraps lines:/)
After seq:206 rsyslogd stops to write to wszystko.log. I've forgot to
say earlier i'm testing with relp destination unavailable.
Before i run test i've done:
rm /var/log/exim/exim_main.log && touch /var/log/exim/exim_main.log &&
rm /var/spool/rsyslog/*

After rsyslogd stops write to wszystko.log there in debug log i can see
message "wti worker in full delay timed out, checking termination..."
I'll send you debug.log offlist, there is no reason to send 330kB to
every subscriber.

Julio,
i tested your settings. Sadly, in my case it doesn't change anything.

Furthermore i can see rsyslogd doesn't create queue files with messages
which wasn't send to relp receiver.

Regards,
Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 3:49 AM

Post #17 of 28 (1656 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> -----Original Message-----
> From: Marcin Mirosław [mailto:marcin [at] mejor]
> Sent: Friday, June 01, 2012 12:01 PM
> To: rsyslog-users
> Cc: Rainer Gerhards
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 31.05.2012 12:33, Rainer Gerhards pisze:
>
> Hello!
>
> > I tried to reproduce the problem today, but had no success with that. I went
> through several cycles but finally gave up.
> >
> > Then, I reviewed the debug log that you sent with some mail in this thread. In
> that log, it looks like the action queue fills up and then the main queue begins to
> fill up as well. When the main queue is full, messages are discarded with the
> usual 1 (or 2) second timeout. This seems to be the cause of the problem.
> >
> > HOWEVER, that is a different scenario than what you describe above, with
> only "several dozen of lines". Would it be possible that you provide me a debug
> log for the above scenario? As I said, all works perfectly for me. I even tried to
> shut down the remote RELP server, but that had no influence at all (as, of
> course, it should be).
>
> Rainer,
> i've used one-liner:
> for x in $(seq 1 1000); do logger "seq:${x} time:$(date +%T)" ; echo
> "write_to_exim_log:${x} time:$(date +%T)" >> /var/log/exim/exim_main.log
> ; sleep 0.4; done
> (Thunderbird wraps lines:/)
> After seq:206 rsyslogd stops to write to wszystko.log. I've forgot to
> say earlier i'm testing with relp destination unavailable.

Ah, OK, that is quite a bit more than a handful of messages. I never tried with that volume.

> Before i run test i've done:
> rm /var/log/exim/exim_main.log && touch /var/log/exim/exim_main.log &&
> rm /var/spool/rsyslog/*
>
> After rsyslogd stops write to wszystko.log there in debug log i can see
> message "wti worker in full delay timed out, checking termination..."
> I'll send you debug.log offlist, there is no reason to send 330kB to
> every subscriber.

Thx, the debug log is actually quite interesting. At first, it looks like the queue limit is hit in full delay mode (which makes some sense), but on second look the exact occurrence looks strange. I'll try to reproduce the problem with your new instructions. If I still cannot get the problem to show up, I'll probably create an instrumented version for you (I need some more insight than standard debug messages provide).


> Julio,
> i tested your settings. Sadly, in my case it doesn't change anything.
>
> Furthermore i can see rsyslogd doesn't create queue files with messages
> which wasn't send to relp receiver.

That's intentional. There is little point in having imfile pull all messages into the queue files. So imfile should be delayed. HOWEVER, this is intended to happen in a way that does not affect other traffic. Obviously something fails here. It would also probably good to have a way to change the default behavior, in case you want it differently (e.g. the file is rotated away too quickly, so that you actually want to copy data to the queue files).

Rainer
>
> Regards,
> Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 4:06 AM

Post #18 of 28 (1653 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> > for x in $(seq 1 1000); do logger "seq:${x} time:$(date +%T)" ; echo
> > "write_to_exim_log:${x} time:$(date +%T)" >> /var/log/exim/exim_main.log
> > ; sleep 0.4; done
> > (Thunderbird wraps lines:/)
> > After seq:206 rsyslogd stops to write to wszystko.log. I've forgot to
> > say earlier i'm testing with relp destination unavailable.
>
> Ah, OK, that is quite a bit more than a handful of messages. I never tried with
> that volume.

OK, this did the trick. I see the same issue. The sleep isn't even necessary. The problem is strictly related to a non-responsive RELP server. In that case the action queue fills up, and light delay blocking kicks in. However, something seems to go wrong with that, and the queue may fill up. Or the real cause may simply be that the logger-created messages fill it completely.

Anyhow, now that I can reproduce, I can check the specifics. Thanks for your help to getting to this point. Will keep you posted, but it may take me a while to figure out what is going wrong.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 4:12 AM

Post #19 of 28 (1654 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

FYI: I can even reproduce a case where rsyslog does not terminate...

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Friday, June 01, 2012 1:06 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> > > for x in $(seq 1 1000); do logger "seq:${x} time:$(date +%T)" ; echo
> > > "write_to_exim_log:${x} time:$(date +%T)" >> /var/log/exim/exim_main.log
> > > ; sleep 0.4; done
> > > (Thunderbird wraps lines:/)
> > > After seq:206 rsyslogd stops to write to wszystko.log. I've forgot to
> > > say earlier i'm testing with relp destination unavailable.
> >
> > Ah, OK, that is quite a bit more than a handful of messages. I never tried with
> > that volume.
>
> OK, this did the trick. I see the same issue. The sleep isn't even necessary. The
> problem is strictly related to a non-responsive RELP server. In that case the
> action queue fills up, and light delay blocking kicks in. However, something
> seems to go wrong with that, and the queue may fill up. Or the real cause may
> simply be that the logger-created messages fill it completely.
>
> Anyhow, now that I can reproduce, I can check the specifics. Thanks for your
> help to getting to this point. Will keep you posted, but it may take me a while to
> figure out what is going wrong.
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

Jun 1, 2012, 4:17 AM

Post #20 of 28 (1654 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 01.06.2012 13:06, Rainer Gerhards pisze:
>>> for x in $(seq 1 1000); do logger "seq:${x} time:$(date +%T)" ; echo
>>> "write_to_exim_log:${x} time:$(date +%T)" >> /var/log/exim/exim_main.log
>>> ; sleep 0.4; done
>>> (Thunderbird wraps lines:/)
>>> After seq:206 rsyslogd stops to write to wszystko.log. I've forgot to
>>> say earlier i'm testing with relp destination unavailable.
>>
>> Ah, OK, that is quite a bit more than a handful of messages. I never tried with
>> that volume.


> OK, this did the trick. I see the same issue. The sleep isn't even necessary. The problem is strictly related to a non-responsive RELP server. In that case the action queue fills up, and light delay blocking kicks in. However, something seems to go wrong with that, and the queue may fill up. Or the real cause may simply be that the logger-created messages fill it completely.
>
> Anyhow, now that I can reproduce, I can check the specifics. Thanks for your help to getting to this point. Will keep you posted, but it may take me a while to figure out what is going wrong.

Sleep was used for convenience of my eyes;)
It's good to hear you can reproduce issue, usual this is problem for me
to create 100% successful testcase.
Thanks,
Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

Jun 1, 2012, 4:19 AM

Post #21 of 28 (1652 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 01.06.2012 13:12, Rainer Gerhards pisze:
> FYI: I can even reproduce a case where rsyslog does not terminate...

Bug 299 strikes back?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


marcin at mejor

Jun 1, 2012, 4:22 AM

Post #22 of 28 (1654 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

W dniu 01.06.2012 12:49, Rainer Gerhards pisze:
>> Furthermore i can see rsyslogd doesn't create queue files with messages
>> which wasn't send to relp receiver.
>
> That's intentional. There is little point in having imfile pull all messages into the queue files. So imfile should be delayed. HOWEVER, this is intended to happen in a way that does not affect other traffic. Obviously something fails here. It would also probably good to have a way to change the default behavior, in case you want it differently (e.g. the file is rotated away too quickly, so that you actually want to copy data to the queue files).

Hmm, what about messages received from logger? Should be saved in spool
when rsyslogd receives SIGTERM? Those messages can't be replayed from
imfile.
Regards,
Marcin
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 5:55 AM

Post #23 of 28 (1659 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Marcin Miroslaw
> Sent: Friday, June 01, 2012 1:23 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] rsyslog-5.8.11 omit syncing is always on?
>
> W dniu 01.06.2012 12:49, Rainer Gerhards pisze:
> >> Furthermore i can see rsyslogd doesn't create queue files with messages
> >> which wasn't send to relp receiver.
> >
> > That's intentional. There is little point in having imfile pull all messages into the
> queue files. So imfile should be delayed. HOWEVER, this is intended to happen in
> a way that does not affect other traffic. Obviously something fails here. It would
> also probably good to have a way to change the default behavior, in case you
> want it differently (e.g. the file is rotated away too quickly, so that you actually
> want to copy data to the queue files).
>
> Hmm, what about messages received from logger?

That's a different story, I talked about imfile.

> Should be saved in spool
> when rsyslogd receives SIGTERM? Those messages can't be replayed from
> imfile.

Exactly. My main thought error, as I currently see it, is that while it is great to delay inputs when they submit to main queues, it is counter-productive to try the same with action queues. After all, the messages are already accepted and we would actually delay the main q worker (this is exactly what happens). I guess this is at the root of the problems we currently see. But I need to do some more analysis...
Rainer
> Regards,
> Marcin
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 6:24 AM

Post #24 of 28 (1651 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> Exactly. My main thought error, as I currently see it, is that while it is great to
> delay inputs when they submit to main queues, it is counter-productive to try the
> same with action queues. After all, the messages are already accepted and we
> would actually delay the main q worker (this is exactly what happens). I guess
> this is at the root of the problems we currently see. But I need to do some more
> analysis...

Yes, that was exactly the problem. A patch is available here:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=fd46d10a4f49da35758ab9201f6f863d69d5d4f5

It works for me, at least during quick testing. Would be great if you could also try it out.

Note, however, that it does not (and cannot) solve the issue that a filled-up action message queue will cause delays on the main queue. You can reproduce this problem if you remove the disk queue and let your script run. There is no way to prevent this without discarding messages in the action queue. However, I need to check if the action queue is configurable enough so that one can configure immediate discarding on queue full state. I think that's currently not possible.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Jun 1, 2012, 6:55 AM

Post #25 of 28 (1651 views)
Permalink
Re: rsyslog-5.8.11 omit syncing is always on? [In reply to]

> Note, however, that it does not (and cannot) solve the issue that a filled-up
> action message queue will cause delays on the main queue. You can reproduce
> this problem if you remove the disk queue and let your script run. There is no
> way to prevent this without discarding messages in the action queue. However, I
> need to check if the action queue is configurable enough so that one can
> configure immediate discarding on queue full state. I think that's currently not
> possible.

But I was wrong ;)

Use

$ActionQueueTimeoutEnqueue 0

To set the action queue to discard messages immediately (or use any other timeout you like, but keep in mind that this will block the main queue). The current default is 2000ms, which is rather long. In v6, I'll reduce it to 50ms, which is still quite a lot of time in terms of what the machine can do within it...

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

First page Previous page 1 2 Next page Last page  View All RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.