Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

rsyslog stops logging when disk cache is filled

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


bodik at civ

Mar 9, 2012, 7:02 AM

Post #1 of 5 (801 views)
Permalink
rsyslog stops logging when disk cache is filled

hi,

we'r testing remote logging with omgssapi (git://v5-devel+v5-omgsss..) , and
tests shows that when cacahe is full, rsyslog stops delivering messages to local
storage ;(

while(true); do
logger `dd if=/dev/urandom bs=10 count=1 2>/dev/null | base64;
perl -e 'print("A"x100000)'`;
done

does anybody have any idea ?

thanks
bodik


---------------------------------
# /etc/rsyslog.conf Configuration file for rsyslog.
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err

news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice

*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages

*.emerg :omusrmsg:*

daemon.*;mail.*;\
news.err;\
*.=debug;*.=info;\
*.=notice;*.=warn |/dev/xconsole

$RepeatedMsgReduction on
$ModLoad omgssapi
$GSSForwardServiceName host
$WorkDirectory /var/cache/rsyslog # default location for work (spool) files

#forward1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionQueueMaxDiskSpace 1m
*.* :omgssapi:srvr1:515



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/


rgerhards at hq

Mar 9, 2012, 7:11 AM

Post #2 of 5 (769 views)
Permalink
Re: rsyslog stops logging when disk cache is filled [In reply to]

Is this gssapi specific? Can you try with the regular or TLS tcp output?
(gssapi is contributed, I don't even have an env).

Rainer


> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of bodik
> Sent: Friday, March 09, 2012 4:02 PM
> To: rsyslog-users
> Subject: [rsyslog] rsyslog stops logging when disk cache is filled
>
> hi,
>
> we'r testing remote logging with omgssapi (git://v5-devel+v5-omgsss..)
> , and
> tests shows that when cacahe is full, rsyslog stops delivering messages
> to local
> storage ;(
>
> while(true); do
> logger `dd if=/dev/urandom bs=10 count=1 2>/dev/null | base64;
> perl -e 'print("A"x100000)'`;
> done
>
> does anybody have any idea ?
>
> thanks
> bodik
>
>
> ---------------------------------
> # /etc/rsyslog.conf Configuration file for rsyslog.
> $ModLoad imuxsock # provides support for local system logging
> $ModLoad imklog # provides kernel logging support (previously done by
> rklogd)
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
> auth,authpriv.* /var/log/auth.log
> *.*;auth,authpriv.none -/var/log/syslog
> #cron.* /var/log/cron.log
> daemon.* -/var/log/daemon.log
> kern.* -/var/log/kern.log
> lpr.* -/var/log/lpr.log
> mail.* -/var/log/mail.log
> user.* -/var/log/user.log
>
> mail.info -/var/log/mail.info
> mail.warn -/var/log/mail.warn
> mail.err /var/log/mail.err
>
> news.crit /var/log/news/news.crit
> news.err /var/log/news/news.err
> news.notice -/var/log/news/news.notice
>
> *.=debug;\
> auth,authpriv.none;\
> news.none;mail.none -/var/log/debug
> *.=info;*.=notice;*.=warn;\
> auth,authpriv.none;\
> cron,daemon.none;\
> mail,news.none -/var/log/messages
>
> *.emerg :omusrmsg:*
>
> daemon.*;mail.*;\
> news.err;\
> *.=debug;*.=info;\
> *.=notice;*.=warn |/dev/xconsole
>
> $RepeatedMsgReduction on
> $ModLoad omgssapi
> $GSSForwardServiceName host
> $WorkDirectory /var/cache/rsyslog # default location for work (spool)
> files
>
> #forward1
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName srvrfwd1 # set file name, also enables disk
> mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog
> shuts down
> $ActionQueueMaxDiskSpace 1m
> *.* :omgssapi:srvr1:515
>
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/


bodik at civ

Mar 9, 2012, 7:31 AM

Post #3 of 5 (791 views)
Permalink
Re: rsyslog stops logging when disk cache is filled [In reply to]

it behaves the same with

$ModLoad imtcp
$InputTCPServerRun 1514

and

*.* @@server:1514

we'r simulating network errors with netfilter and with the same results for
-j DROP and -j REJECT


watch 'ps --no-headers -o vsize $(pidof rsyslogd); du -sh /var/cache/rsyslog/;
ls -l /var/cache/rsyslog/'


bodik

Dne 03/09/12 16:11, Rainer Gerhards napsal(a):
> Is this gssapi specific? Can you try with the regular or TLS tcp output?
> (gssapi is contributed, I don't even have an env).
>
> Rainer
>
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of bodik
>> Sent: Friday, March 09, 2012 4:02 PM
>> To: rsyslog-users
>> Subject: [rsyslog] rsyslog stops logging when disk cache is filled
>>
>> hi,
>>
>> we'r testing remote logging with omgssapi (git://v5-devel+v5-omgsss..)
>> , and
>> tests shows that when cacahe is full, rsyslog stops delivering messages
>> to local
>> storage ;(
>>
>> while(true); do
>> logger `dd if=/dev/urandom bs=10 count=1 2>/dev/null | base64;
>> perl -e 'print("A"x100000)'`;
>> done
>>
>> does anybody have any idea ?
>>
>> thanks
>> bodik
>>
>>
>> ---------------------------------
>> # /etc/rsyslog.conf Configuration file for rsyslog.
>> $ModLoad imuxsock # provides support for local system logging
>> $ModLoad imklog # provides kernel logging support (previously done by
>> rklogd)
>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> auth,authpriv.* /var/log/auth.log
>> *.*;auth,authpriv.none -/var/log/syslog
>> #cron.* /var/log/cron.log
>> daemon.* -/var/log/daemon.log
>> kern.* -/var/log/kern.log
>> lpr.* -/var/log/lpr.log
>> mail.* -/var/log/mail.log
>> user.* -/var/log/user.log
>>
>> mail.info -/var/log/mail.info
>> mail.warn -/var/log/mail.warn
>> mail.err /var/log/mail.err
>>
>> news.crit /var/log/news/news.crit
>> news.err /var/log/news/news.err
>> news.notice -/var/log/news/news.notice
>>
>> *.=debug;\
>> auth,authpriv.none;\
>> news.none;mail.none -/var/log/debug
>> *.=info;*.=notice;*.=warn;\
>> auth,authpriv.none;\
>> cron,daemon.none;\
>> mail,news.none -/var/log/messages
>>
>> *.emerg :omusrmsg:*
>>
>> daemon.*;mail.*;\
>> news.err;\
>> *.=debug;*.=info;\
>> *.=notice;*.=warn |/dev/xconsole
>>
>> $RepeatedMsgReduction on
>> $ModLoad omgssapi
>> $GSSForwardServiceName host
>> $WorkDirectory /var/cache/rsyslog # default location for work (spool)
>> files
>>
>> #forward1
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName srvrfwd1 # set file name, also enables disk
>> mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog
>> shuts down
>> $ActionQueueMaxDiskSpace 1m
>> *.* :omgssapi:srvr1:515
>>
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/


--
------------------------------------------------------------------------
Ing. Radoslav Bodo | http://bodik.nawebu.info
|
Laboratory for Computer Science | e-mail: bodik [at] civ
University of West Bohemia | phone: +420 37763 2852
Univerzitni 20, 306 14 Pilsen | fax: +420 37763 2702
Czech Republic | location: IC Bory, UI 402
------------------------------------------------------------------------
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/


theinric at redhat

Mar 14, 2012, 2:32 AM

Post #4 of 5 (758 views)
Permalink
Re: rsyslog stops logging when disk cache is filled [In reply to]

Hi, just a wild guess:

On 03/09/2012 04:31 PM, bodik wrote:
> it behaves the same with
>
> $ModLoad imtcp
> $InputTCPServerRun 1514
>
> and
>
> *.* @@server:1514

Local logging stops completely, or is just very slow?
If the later, can't this be a simple case of rate limiting?
There's an option '$IMUXSockRateLimitInterval' to control this.
The default has been changed to zero, but just recently (v5.9.6?).

You have a separate queue

>>> $ActionQueueType LinkedList # use asynchronous processing

but don't drop any messages

>>> $ActionResumeRetryCount -1 # infinite retries on insert failure

so after it fills up, also the main queue fills up.

"$ActionQueueTimeoutEnqueue 1" might also be helpful.

Tomas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/


bodik at civ

Apr 12, 2012, 12:40 AM

Post #5 of 5 (720 views)
Permalink
Re: rsyslog stops logging when disk cache is filled - solved [In reply to]

On 03/14/12 10:32, Tomas Heinrich wrote:
> Hi, just a wild guess:
>
> On 03/09/2012 04:31 PM, bodik wrote:
>> it behaves the same with
>>
>> $ModLoad imtcp
>> $InputTCPServerRun 1514
>>
>> and
>>
>> *.* @@server:1514
>
> Local logging stops completely, or is just very slow?
> If the later, can't this be a simple case of rate limiting?
> There's an option '$IMUXSockRateLimitInterval' to control this.
> The default has been changed to zero, but just recently (v5.9.6?).
>
> You have a separate queue
>
>>>> $ActionQueueType LinkedList # use asynchronous processing
>
> but don't drop any messages
>
>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>
> so after it fills up, also the main queue fills up.
>
> "$ActionQueueTimeoutEnqueue 1" might also be helpful.

hi,

thank for advice. Finally i manage to get to this issue and I also been through
queues/lanes docs to get real sense what's going on there.

with "$ActionQueueTimeoutEnqueue 1" everything works fine when cache fills up.

But I just set that to "100ms". just in case the machine is very busy and cache
is not filled up. Does it make more sense than to have it just "1ms" ? because
it's actualy an DA, so disk write could take some time under stress


bodik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.