Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

Unable to implement on-disk queuing

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


steve.chupack at dealer

Sep 11, 2011, 1:50 PM

Post #1 of 11 (830 views)
Permalink
Unable to implement on-disk queuing

´╗┐Hi,

I have rsyslogd up and running and logging to mysql with loganalyzer as a front end. Very cool.

However, I can't seem to get on-disk queuing working, which would be nice if mysql goes down or a table is locked because I'm purging records during a nightly maintenance cron.

I never see any queue files created. I dropped the size of the memory queue to something ridiculously small, but still no queue files even after 20 minutes.

I followed the instructions given here: http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a couple of other tutorials on rsyslogd and queuing.

I'm quite sure I'm just completely missing something obvious. Details below... And thanks to anyone who might be able to point out what I'm doing wrong.

-Steve



MY CONFIG, please, no flames re the horrendously complex rules ;-)

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.

$ModLoad immark # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # kernel logging (formerly provided by rklogd)
$ModLoad ommysql

# SC 2010.11.11: configure disk caching in case mysql is unavailable

$MainMsgQueueSize 5

$WorkDirectory /var/log/rsyslogq # default location for work (spool) files

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName dbq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure

# send snmpd INFO messages to the dustbin

if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~

$template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw
& ~
#if ($fromhost-ip contains '10.128.255') then ~

$template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
#if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
& ~
#if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~

$template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
& ~
#if $fromhost-ip contains '10.128.244' then ~

$template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
& ~

if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~


#*.* :ommysql:localhost,Syslog,root,mysqldb44
*.notice :ommysql:localhost,Syslog,root,mysqldb44

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none -/var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* -/var/log/cron

# Everybody gets emergency messages
*.emerg *

# Save news errors of level crit and higher in a special file.
uucp,news.crit -/var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /rsyslog/spool # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514


# ######### Receiving Messages from Remote Hosts ##########
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API (if compiled to support it)
$ModLoad imtcp.so # load module
$InputTCPServerRun 1470 # start up TCP listener at port 514

# UDP Syslog Server:
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514


SOME DEBUG OUTPUT:

(see attached)









Steve Chupack | IT Systems Administrator
V: 877.327.8422 x 1242
Steve.Chupack [at] dealer | www.dealer.com
Attachments: rsyslog-debug.txt (12.6 KB)


steve.chupack at dealer

Sep 11, 2011, 2:10 PM

Post #2 of 11 (799 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Ooops... previous message's attachment was the config, not the debug output. Debug attached.



On Sun, 11 Sep 2011 16:50:57 -0400
Steve Chupack <steve.chupack [at] dealer> wrote:

> Hi,
>
> I have rsyslogd up and running and logging to mysql with loganalyzer as a front end. Very cool.
>
> However, I can't seem to get on-disk queuing working, which would be nice if mysql goes down or a table is locked because I'm purging records during a nightly maintenance cron.
>
> I never see any queue files created. I dropped the size of the memory queue to something ridiculously small, but still no queue files even after 20 minutes.
>
> I followed the instructions given here: http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a couple of other tutorials on rsyslogd and queuing.
>
> I'm quite sure I'm just completely missing something obvious. Details below... And thanks to anyone who might be able to point out what I'm doing wrong.
>
> -Steve
>
>
>
> MY CONFIG, please, no flames re the horrendously complex rules ;-)
>
> # if you experience problems, check
> # http://www.rsyslog.com/troubleshoot for assistance
>
> # rsyslog v3: load input modules
> # If you do not load inputs, nothing happens!
> # You may need to set the module load path if modules are not found.
>
> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
> $ModLoad imklog # kernel logging (formerly provided by rklogd)
> $ModLoad ommysql
>
> # SC 2010.11.11: configure disk caching in case mysql is unavailable
>
> $MainMsgQueueSize 5
>
> $WorkDirectory /var/log/rsyslogq # default location for work (spool) files
>
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName dbq # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
>
> # send snmpd INFO messages to the dustbin
>
> if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' then ~
>
> $template vtfw,"insert into vtfw (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
>
> if ($fromhost-ip contains '10.128.255') then :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> & ~
> #if ($fromhost-ip contains '10.128.255') then ~
>
> $template vt1hs1_switches,"insert into vt1hs1_switches (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
>
> if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> & ~
> #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01' or $hostname contains_i 'core02' then ~
>
> $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
>
> if $fromhost-ip contains '10.128.244' then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> & ~
> #if $fromhost-ip contains '10.128.244' then ~
>
> $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
>
> if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic') then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> & ~
>
> if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44' or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and ($syslogseverity-text == 'notice') then ~
>
>
> #*.* :ommysql:localhost,Syslog,root,mysqldb44
> *.notice :ommysql:localhost,Syslog,root,mysqldb44
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none -/var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
>
>
> # Log cron stuff
> cron.* -/var/log/cron
>
> # Everybody gets emergency messages
> *.emerg *
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit -/var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
> # Remote Logging (we use TCP for reliable delivery)
> # An on-disk queue is created for this action. If the remote host is
> # down, messages are spooled to disk and sent when it is up again.
> #$WorkDirectory /rsyslog/spool # where to place spool files
> #$ActionQueueFileName uniqName # unique name prefix for spool files
> #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> #$ActionQueueType LinkedList # run asynchronously
> #$ActionResumeRetryCount -1 # infinite retries if host is down
> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> #*.* @@remote-host:514
>
>
> # ######### Receiving Messages from Remote Hosts ##########
> # TCP Syslog Server:
> # provides TCP syslog reception and GSS-API (if compiled to support it)
> $ModLoad imtcp.so # load module
> $InputTCPServerRun 1470 # start up TCP listener at port 514
>
> # UDP Syslog Server:
> $ModLoad imudp.so # provides UDP syslog reception
> $UDPServerRun 514 # start a UDP syslog server at standard port 514
>
>
> SOME DEBUG OUTPUT:
>
> (see attached)
>
>
>
>
>
>
>
>
>
> Steve Chupack | IT Systems Administrator
> V: 877.327.8422 x 1242
> Steve.Chupack [at] dealer | www.dealer.com
>
>
Attachments: rsyslog-debug.txt (23.0 KB)


rgerhards at hq

Sep 12, 2011, 2:13 AM

Post #3 of 11 (804 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Queue files are only created if necessary. The debug log does not contain any
such situation.

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Steve Chupack
> Sent: Sunday, September 11, 2011 11:11 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Unable to implement on-disk queuing
>
> Ooops... previous message's attachment was the config, not the debug
> output. Debug attached.
>
>
>
> On Sun, 11 Sep 2011 16:50:57 -0400
> Steve Chupack <steve.chupack [at] dealer> wrote:
>
> > Hi,
> >
> > I have rsyslogd up and running and logging to mysql with loganalyzer
> as a front end. Very cool.
> >
> > However, I can't seem to get on-disk queuing working, which would be
> nice if mysql goes down or a table is locked because I'm purging
> records during a nightly maintenance cron.
> >
> > I never see any queue files created. I dropped the size of the memory
> queue to something ridiculously small, but still no queue files even
> after 20 minutes.
> >
> > I followed the instructions given here:
> http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a
> couple of other tutorials on rsyslogd and queuing.
> >
> > I'm quite sure I'm just completely missing something obvious. Details
> below... And thanks to anyone who might be able to point out what I'm
> doing wrong.
> >
> > -Steve
> >
> >
> >
> > MY CONFIG, please, no flames re the horrendously complex rules ;-)
> >
> > # if you experience problems, check
> > # http://www.rsyslog.com/troubleshoot for assistance
> >
> > # rsyslog v3: load input modules
> > # If you do not load inputs, nothing happens!
> > # You may need to set the module load path if modules are not found.
> >
> > $ModLoad immark # provides --MARK-- message capability
> > $ModLoad imuxsock # provides support for local system logging (e.g.
> via logger command)
> > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > $ModLoad ommysql
> >
> > # SC 2010.11.11: configure disk caching in case mysql is unavailable
> >
> > $MainMsgQueueSize 5
> >
> > $WorkDirectory /var/log/rsyslogq # default location for work (spool)
> files
> >
> > $ActionQueueType LinkedList # use asynchronous processing
> > $ActionQueueFileName dbq # set file name, also enables disk mode
> > $ActionResumeRetryCount -1 # infinite retries on insert failure
> >
> > # send snmpd INFO messages to the dustbin
> >
> > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> then ~
> >
> > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values
> ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%,
> '%syslogtag%')",SQL
> >
> > if ($fromhost-ip contains '10.128.255') then
> :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > & ~
> > #if ($fromhost-ip contains '10.128.255') then ~
> >
> > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> %syslogpriority%, '%timereported:::date-mysql%',
> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> >
> > if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01'
> or $hostname contains_i 'core02' then
> :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > & ~
> > #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01'
> or $hostname contains_i 'core02' then ~
> >
> > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility,
> FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> %syslogpriority%, '%timereported:::date-mysql%',
> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> >
> > if $fromhost-ip contains '10.128.244' then
> :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > & ~
> > #if $fromhost-ip contains '10.128.244' then ~
> >
> > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> %syslogpriority%, '%timereported:::date-mysql%',
> '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> >
> > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44'
> or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or
> $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and
> ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or
> $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or
> $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic')
> then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > & ~
> >
> > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44'
> or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or
> $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and
> ($syslogseverity-text == 'notice') then ~
> >
> >
> > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> >
> > # Log all kernel messages to the console.
> > # Logging much else clutters up the screen.
> > #kern.* /dev/console
> >
> > # Log anything (except mail) of level info or higher.
> > # Don't log private authentication messages!
> > *.info;mail.none;authpriv.none;cron.none -
> /var/log/messages
> >
> > # The authpriv file has restricted access.
> > authpriv.*
> /var/log/secure
> >
> > # Log all the mail messages in one place.
> > mail.* -
> /var/log/maillog
> >
> >
> > # Log cron stuff
> > cron.* -
> /var/log/cron
> >
> > # Everybody gets emergency messages
> > *.emerg *
> >
> > # Save news errors of level crit and higher in a special file.
> > uucp,news.crit -
> /var/log/spooler
> >
> > # Save boot messages also to boot.log
> > local7.*
> /var/log/boot.log
> >
> > # Remote Logging (we use TCP for reliable delivery)
> > # An on-disk queue is created for this action. If the remote host is
> > # down, messages are spooled to disk and sent when it is up again.
> > #$WorkDirectory /rsyslog/spool # where to place spool files
> > #$ActionQueueFileName uniqName # unique name prefix for spool files
> > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> possible)
> > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> > #$ActionQueueType LinkedList # run asynchronously
> > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> > #*.* @@remote-host:514
> >
> >
> > # ######### Receiving Messages from Remote Hosts ##########
> > # TCP Syslog Server:
> > # provides TCP syslog reception and GSS-API (if compiled to support
> it)
> > $ModLoad imtcp.so # load module
> > $InputTCPServerRun 1470 # start up TCP listener at port 514
> >
> > # UDP Syslog Server:
> > $ModLoad imudp.so # provides UDP syslog reception
> > $UDPServerRun 514 # start a UDP syslog server at standard port 514
> >
> >
> > SOME DEBUG OUTPUT:
> >
> > (see attached)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Steve Chupack | IT Systems Administrator
> > V: 877.327.8422 x 1242
> > Steve.Chupack [at] dealer | www.dealer.com
> >
> >
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


steve.chupack at dealer

Sep 12, 2011, 6:02 AM

Post #4 of 11 (791 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql turned off for 20 minutes... And my message flow rate is up around 1 per second. So I would have expected some queuing to happen. Is there anything else I should look at or try?

On Mon, 12 Sep 2011 11:13:28 +0200
Rainer Gerhards <rgerhards [at] hq> wrote:

> Queue files are only created if necessary. The debug log does not contain any
> such situation.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Steve Chupack
> > Sent: Sunday, September 11, 2011 11:11 PM
> > To: rsyslog [at] lists
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Ooops... previous message's attachment was the config, not the debug
> > output. Debug attached.
> >
> >
> >
> > On Sun, 11 Sep 2011 16:50:57 -0400
> > Steve Chupack <steve.chupack [at] dealer> wrote:
> >
> > > Hi,
> > >
> > > I have rsyslogd up and running and logging to mysql with loganalyzer
> > as a front end. Very cool.
> > >
> > > However, I can't seem to get on-disk queuing working, which would be
> > nice if mysql goes down or a table is locked because I'm purging
> > records during a nightly maintenance cron.
> > >
> > > I never see any queue files created. I dropped the size of the memory
> > queue to something ridiculously small, but still no queue files even
> > after 20 minutes.
> > >
> > > I followed the instructions given here:
> > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along with a
> > couple of other tutorials on rsyslogd and queuing.
> > >
> > > I'm quite sure I'm just completely missing something obvious. Details
> > below... And thanks to anyone who might be able to point out what I'm
> > doing wrong.
> > >
> > > -Steve
> > >
> > >
> > >
> > > MY CONFIG, please, no flames re the horrendously complex rules ;-)
> > >
> > > # if you experience problems, check
> > > # http://www.rsyslog.com/troubleshoot for assistance
> > >
> > > # rsyslog v3: load input modules
> > > # If you do not load inputs, nothing happens!
> > > # You may need to set the module load path if modules are not found.
> > >
> > > $ModLoad immark # provides --MARK-- message capability
> > > $ModLoad imuxsock # provides support for local system logging (e.g.
> > via logger command)
> > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > $ModLoad ommysql
> > >
> > > # SC 2010.11.11: configure disk caching in case mysql is unavailable
> > >
> > > $MainMsgQueueSize 5
> > >
> > > $WorkDirectory /var/log/rsyslogq # default location for work (spool)
> > files
> > >
> > > $ActionQueueType LinkedList # use asynchronous processing
> > > $ActionQueueFileName dbq # set file name, also enables disk mode
> > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > >
> > > # send snmpd INFO messages to the dustbin
> > >
> > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > then ~
> > >
> > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values
> > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%,
> > '%syslogtag%')",SQL
> > >
> > > if ($fromhost-ip contains '10.128.255') then
> > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > & ~
> > > #if ($fromhost-ip contains '10.128.255') then ~
> > >
> > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > %syslogpriority%, '%timereported:::date-mysql%',
> > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > >
> > > if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01'
> > or $hostname contains_i 'core02' then
> > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > > & ~
> > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i 'core01'
> > or $hostname contains_i 'core02' then ~
> > >
> > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, Facility,
> > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > %syslogpriority%, '%timereported:::date-mysql%',
> > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > >
> > > if $fromhost-ip contains '10.128.244' then
> > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > & ~
> > > #if $fromhost-ip contains '10.128.244' then ~
> > >
> > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > %syslogpriority%, '%timereported:::date-mysql%',
> > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > >
> > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44'
> > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or
> > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and
> > ($syslogseverity-text == 'warning' or $syslogseverity-text == 'warn' or
> > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' or
> > $syslogseverity-text == 'emergency' or $syslogseverity-text == 'panic')
> > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > & ~
> > >
> > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == '10.128.24.44'
> > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' or
> > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') and
> > ($syslogseverity-text == 'notice') then ~
> > >
> > >
> > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > >
> > > # Log all kernel messages to the console.
> > > # Logging much else clutters up the screen.
> > > #kern.* /dev/console
> > >
> > > # Log anything (except mail) of level info or higher.
> > > # Don't log private authentication messages!
> > > *.info;mail.none;authpriv.none;cron.none -
> > /var/log/messages
> > >
> > > # The authpriv file has restricted access.
> > > authpriv.*
> > /var/log/secure
> > >
> > > # Log all the mail messages in one place.
> > > mail.* -
> > /var/log/maillog
> > >
> > >
> > > # Log cron stuff
> > > cron.* -
> > /var/log/cron
> > >
> > > # Everybody gets emergency messages
> > > *.emerg *
> > >
> > > # Save news errors of level crit and higher in a special file.
> > > uucp,news.crit -
> > /var/log/spooler
> > >
> > > # Save boot messages also to boot.log
> > > local7.*
> > /var/log/boot.log
> > >
> > > # Remote Logging (we use TCP for reliable delivery)
> > > # An on-disk queue is created for this action. If the remote host is
> > > # down, messages are spooled to disk and sent when it is up again.
> > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > #$ActionQueueFileName uniqName # unique name prefix for spool files
> > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > possible)
> > > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> > > #$ActionQueueType LinkedList # run asynchronously
> > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> > > #*.* @@remote-host:514
> > >
> > >
> > > # ######### Receiving Messages from Remote Hosts ##########
> > > # TCP Syslog Server:
> > > # provides TCP syslog reception and GSS-API (if compiled to support
> > it)
> > > $ModLoad imtcp.so # load module
> > > $InputTCPServerRun 1470 # start up TCP listener at port 514
> > >
> > > # UDP Syslog Server:
> > > $ModLoad imudp.so # provides UDP syslog reception
> > > $UDPServerRun 514 # start a UDP syslog server at standard port 514
> > >
> > >
> > > SOME DEBUG OUTPUT:
> > >
> > > (see attached)
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Steve Chupack | IT Systems Administrator
> > > V: 877.327.8422 x 1242
> > > Steve.Chupack [at] dealer | www.dealer.com
> > >
> > >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


rgerhards at hq

Sep 12, 2011, 6:18 AM

Post #5 of 11 (796 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Steve Chupack
> Sent: Monday, September 12, 2011 3:02 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Unable to implement on-disk queuing
>
> Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql
> turned off for 20 minutes... And my message flow rate is up around 1
> per second. So I would have expected some queuing to happen. Is there
> anything else I should look at or try?

I think the unusual extremely low queue sizes results in some watermarks set
to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500
or 1000 and try again. Note that you can use logger or the tcpflood tool to
injects lots of message in a short period.

Rainer

>
> On Mon, 12 Sep 2011 11:13:28 +0200
> Rainer Gerhards <rgerhards [at] hq> wrote:
>
> > Queue files are only created if necessary. The debug log does not
> contain any
> > such situation.
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Steve Chupack
> > > Sent: Sunday, September 11, 2011 11:11 PM
> > > To: rsyslog [at] lists
> > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > >
> > > Ooops... previous message's attachment was the config, not the
> debug
> > > output. Debug attached.
> > >
> > >
> > >
> > > On Sun, 11 Sep 2011 16:50:57 -0400
> > > Steve Chupack <steve.chupack [at] dealer> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have rsyslogd up and running and logging to mysql with
> loganalyzer
> > > as a front end. Very cool.
> > > >
> > > > However, I can't seem to get on-disk queuing working, which would
> be
> > > nice if mysql goes down or a table is locked because I'm purging
> > > records during a nightly maintenance cron.
> > > >
> > > > I never see any queue files created. I dropped the size of the
> memory
> > > queue to something ridiculously small, but still no queue files
> even
> > > after 20 minutes.
> > > >
> > > > I followed the instructions given here:
> > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> with a
> > > couple of other tutorials on rsyslogd and queuing.
> > > >
> > > > I'm quite sure I'm just completely missing something obvious.
> Details
> > > below... And thanks to anyone who might be able to point out what
> I'm
> > > doing wrong.
> > > >
> > > > -Steve
> > > >
> > > >
> > > >
> > > > MY CONFIG, please, no flames re the horrendously complex rules ;-
> )
> > > >
> > > > # if you experience problems, check
> > > > # http://www.rsyslog.com/troubleshoot for assistance
> > > >
> > > > # rsyslog v3: load input modules
> > > > # If you do not load inputs, nothing happens!
> > > > # You may need to set the module load path if modules are not
> found.
> > > >
> > > > $ModLoad immark # provides --MARK-- message capability
> > > > $ModLoad imuxsock # provides support for local system logging
> (e.g.
> > > via logger command)
> > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > $ModLoad ommysql
> > > >
> > > > # SC 2010.11.11: configure disk caching in case mysql is
> unavailable
> > > >
> > > > $MainMsgQueueSize 5
> > > >
> > > > $WorkDirectory /var/log/rsyslogq # default location for work
> (spool)
> > > files
> > > >
> > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > $ActionQueueFileName dbq # set file name, also enables disk
> mode
> > > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > > >
> > > > # send snmpd INFO messages to the dustbin
> > > >
> > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > then ~
> > > >
> > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> values
> > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> %iut%,
> > > '%syslogtag%')",SQL
> > > >
> > > > if ($fromhost-ip contains '10.128.255') then
> > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > & ~
> > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > >
> > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> '%HOSTNAME%',
> > > %syslogpriority%, '%timereported:::date-mysql%',
> > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > >
> > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> 'core01'
> > > or $hostname contains_i 'core02' then
> > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > > > & ~
> > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> 'core01'
> > > or $hostname contains_i 'core02' then ~
> > > >
> > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> Facility,
> > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > %syslogpriority%, '%timereported:::date-mysql%',
> > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > >
> > > > if $fromhost-ip contains '10.128.244' then
> > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > & ~
> > > > #if $fromhost-ip contains '10.128.244' then ~
> > > >
> > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> '%HOSTNAME%',
> > > %syslogpriority%, '%timereported:::date-mysql%',
> > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > >
> > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> '10.128.24.44'
> > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> or
> > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> and
> > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> 'warn' or
> > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert'
> or
> > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> 'panic')
> > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > & ~
> > > >
> > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> '10.128.24.44'
> > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> or
> > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> and
> > > ($syslogseverity-text == 'notice') then ~
> > > >
> > > >
> > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > >
> > > > # Log all kernel messages to the console.
> > > > # Logging much else clutters up the screen.
> > > > #kern.*
> /dev/console
> > > >
> > > > # Log anything (except mail) of level info or higher.
> > > > # Don't log private authentication messages!
> > > > *.info;mail.none;authpriv.none;cron.none -
> > > /var/log/messages
> > > >
> > > > # The authpriv file has restricted access.
> > > > authpriv.*
> > > /var/log/secure
> > > >
> > > > # Log all the mail messages in one place.
> > > > mail.* -
> > > /var/log/maillog
> > > >
> > > >
> > > > # Log cron stuff
> > > > cron.* -
> > > /var/log/cron
> > > >
> > > > # Everybody gets emergency messages
> > > > *.emerg *
> > > >
> > > > # Save news errors of level crit and higher in a special file.
> > > > uucp,news.crit -
> > > /var/log/spooler
> > > >
> > > > # Save boot messages also to boot.log
> > > > local7.*
> > > /var/log/boot.log
> > > >
> > > > # Remote Logging (we use TCP for reliable delivery)
> > > > # An on-disk queue is created for this action. If the remote host
> is
> > > > # down, messages are spooled to disk and sent when it is up
> again.
> > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> files
> > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > possible)
> > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> shutdown
> > > > #$ActionQueueType LinkedList # run asynchronously
> > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> optional
> > > > #*.* @@remote-host:514
> > > >
> > > >
> > > > # ######### Receiving Messages from Remote Hosts ##########
> > > > # TCP Syslog Server:
> > > > # provides TCP syslog reception and GSS-API (if compiled to
> support
> > > it)
> > > > $ModLoad imtcp.so # load module
> > > > $InputTCPServerRun 1470 # start up TCP listener at port 514
> > > >
> > > > # UDP Syslog Server:
> > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> 514
> > > >
> > > >
> > > > SOME DEBUG OUTPUT:
> > > >
> > > > (see attached)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Steve Chupack | IT Systems Administrator
> > > > V: 877.327.8422 x 1242
> > > > Steve.Chupack [at] dealer | www.dealer.com
> > > >
> > > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


steve.chupack at dealer

Sep 12, 2011, 7:39 AM

Post #6 of 11 (793 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Thanks for taking the time to respond, I really appreciate it.

OK, I think I can better describe this now. You are correct, rsyslog is not seeing a need to queue. My debug log is filled with the following entries, which clearly indicate it sees no need to queue or spool. So again, I think I am missing something very basic here...


7878.780426000:455e1940: main Q: entry added, size now 1 entries
7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries

I used the very basic config from your article on buffering.

$MainMsgQueueSize 500
$WorkDirectory /var/log/rsyslogq # default location for work (spool) files
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure


Mysql is definitely not running, as show by the following:

8220.440850000:41537940: Called LogError, msg: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
rsyslogd: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
8220.441040000:41537940: Called action, logging to ommysql
8220.573450000:41537940: Called action, logging to ommysql
8220.574801000:41537940: Called action, logging to ommysql
8220.602406000:41537940: Called action, logging to ommysql
8220.604330000:41537940: Called action, logging to ommysql



On Mon, 12 Sep 2011 15:18:32 +0200
Rainer Gerhards <rgerhards [at] hq> wrote:

>
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Steve Chupack
> > Sent: Monday, September 12, 2011 3:02 PM
> > To: rsyslog [at] lists
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql
> > turned off for 20 minutes... And my message flow rate is up around 1
> > per second. So I would have expected some queuing to happen. Is there
> > anything else I should look at or try?
>
> I think the unusual extremely low queue sizes results in some watermarks set
> to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500
> or 1000 and try again. Note that you can use logger or the tcpflood tool to
> injects lots of message in a short period.
>
> Rainer
>
> >
> > On Mon, 12 Sep 2011 11:13:28 +0200
> > Rainer Gerhards <rgerhards [at] hq> wrote:
> >
> > > Queue files are only created if necessary. The debug log does not
> > contain any
> > > such situation.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > To: rsyslog [at] lists
> > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > >
> > > > Ooops... previous message's attachment was the config, not the
> > debug
> > > > output. Debug attached.
> > > >
> > > >
> > > >
> > > > On Sun, 11 Sep 2011 16:50:57 -0400
> > > > Steve Chupack <steve.chupack [at] dealer> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have rsyslogd up and running and logging to mysql with
> > loganalyzer
> > > > as a front end. Very cool.
> > > > >
> > > > > However, I can't seem to get on-disk queuing working, which would
> > be
> > > > nice if mysql goes down or a table is locked because I'm purging
> > > > records during a nightly maintenance cron.
> > > > >
> > > > > I never see any queue files created. I dropped the size of the
> > memory
> > > > queue to something ridiculously small, but still no queue files
> > even
> > > > after 20 minutes.
> > > > >
> > > > > I followed the instructions given here:
> > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > with a
> > > > couple of other tutorials on rsyslogd and queuing.
> > > > >
> > > > > I'm quite sure I'm just completely missing something obvious.
> > Details
> > > > below... And thanks to anyone who might be able to point out what
> > I'm
> > > > doing wrong.
> > > > >
> > > > > -Steve
> > > > >
> > > > >
> > > > >
> > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-
> > )
> > > > >
> > > > > # if you experience problems, check
> > > > > # http://www.rsyslog.com/troubleshoot for assistance
> > > > >
> > > > > # rsyslog v3: load input modules
> > > > > # If you do not load inputs, nothing happens!
> > > > > # You may need to set the module load path if modules are not
> > found.
> > > > >
> > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > $ModLoad imuxsock # provides support for local system logging
> > (e.g.
> > > > via logger command)
> > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > $ModLoad ommysql
> > > > >
> > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > unavailable
> > > > >
> > > > > $MainMsgQueueSize 5
> > > > >
> > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > (spool)
> > > > files
> > > > >
> > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > mode
> > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > > > >
> > > > > # send snmpd INFO messages to the dustbin
> > > > >
> > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > then ~
> > > > >
> > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > values
> > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > %iut%,
> > > > '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip contains '10.128.255') then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > & ~
> > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > >
> > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then ~
> > > > >
> > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > Facility,
> > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.244' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > >
> > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > 'warn' or
> > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert'
> > or
> > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > 'panic')
> > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > & ~
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'notice') then ~
> > > > >
> > > > >
> > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > >
> > > > > # Log all kernel messages to the console.
> > > > > # Logging much else clutters up the screen.
> > > > > #kern.*
> > /dev/console
> > > > >
> > > > > # Log anything (except mail) of level info or higher.
> > > > > # Don't log private authentication messages!
> > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > /var/log/messages
> > > > >
> > > > > # The authpriv file has restricted access.
> > > > > authpriv.*
> > > > /var/log/secure
> > > > >
> > > > > # Log all the mail messages in one place.
> > > > > mail.* -
> > > > /var/log/maillog
> > > > >
> > > > >
> > > > > # Log cron stuff
> > > > > cron.* -
> > > > /var/log/cron
> > > > >
> > > > > # Everybody gets emergency messages
> > > > > *.emerg *
> > > > >
> > > > > # Save news errors of level crit and higher in a special file.
> > > > > uucp,news.crit -
> > > > /var/log/spooler
> > > > >
> > > > > # Save boot messages also to boot.log
> > > > > local7.*
> > > > /var/log/boot.log
> > > > >
> > > > > # Remote Logging (we use TCP for reliable delivery)
> > > > > # An on-disk queue is created for this action. If the remote host
> > is
> > > > > # down, messages are spooled to disk and sent when it is up
> > again.
> > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > files
> > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > possible)
> > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > shutdown
> > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > optional
> > > > > #*.* @@remote-host:514
> > > > >
> > > > >
> > > > > # ######### Receiving Messages from Remote Hosts ##########
> > > > > # TCP Syslog Server:
> > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > support
> > > > it)
> > > > > $ModLoad imtcp.so # load module
> > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514
> > > > >
> > > > > # UDP Syslog Server:
> > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > 514
> > > > >
> > > > >
> > > > > SOME DEBUG OUTPUT:
> > > > >
> > > > > (see attached)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Steve Chupack | IT Systems Administrator
> > > > > V: 877.327.8422 x 1242
> > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > >
> > > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


rgerhards at hq

Sep 12, 2011, 8:24 AM

Post #7 of 11 (800 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Can you mail me the complete debug log, please.

Rainer

Steve Chupack <steve.chupack [at] dealer> hat geschrieben:Thanks for taking the time to respond, I really appreciate it.

OK, I think I can better describe this now. You are correct, rsyslog is not seeing a need to queue. My debug log is filled with the following entries, which clearly indicate it sees no need to queue or spool. So again, I think I am missing something very basic here...


7878.780426000:455e1940: main Q: entry added, size now 1 entries
7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries

I used the very basic config from your article on buffering.

$MainMsgQueueSize 500
$WorkDirectory /var/log/rsyslogq # default location for work (spool) files
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure


Mysql is definitely not running, as show by the following:

8220.440850000:41537940: Called LogError, msg: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
rsyslogd: db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg db error (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
8220.441040000:41537940: Called action, logging to ommysql
8220.573450000:41537940: Called action, logging to ommysql
8220.574801000:41537940: Called action, logging to ommysql
8220.602406000:41537940: Called action, logging to ommysql
8220.604330000:41537940: Called action, logging to ommysql



On Mon, 12 Sep 2011 15:18:32 +0200
Rainer Gerhards <rgerhards [at] hq> wrote:

>
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Steve Chupack
> > Sent: Monday, September 12, 2011 3:02 PM
> > To: rsyslog [at] lists
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql
> > turned off for 20 minutes... And my message flow rate is up around 1
> > per second. So I would have expected some queuing to happen. Is there
> > anything else I should look at or try?
>
> I think the unusual extremely low queue sizes results in some watermarks set
> to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500
> or 1000 and try again. Note that you can use logger or the tcpflood tool to
> injects lots of message in a short period.
>
> Rainer
>
> >
> > On Mon, 12 Sep 2011 11:13:28 +0200
> > Rainer Gerhards <rgerhards [at] hq> wrote:
> >
> > > Queue files are only created if necessary. The debug log does not
> > contain any
> > > such situation.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > To: rsyslog [at] lists
> > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > >
> > > > Ooops... previous message's attachment was the config, not the
> > debug
> > > > output. Debug attached.
> > > >
> > > >
> > > >
> > > > On Sun, 11 Sep 2011 16:50:57 -0400
> > > > Steve Chupack <steve.chupack [at] dealer> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have rsyslogd up and running and logging to mysql with
> > loganalyzer
> > > > as a front end. Very cool.
> > > > >
> > > > > However, I can't seem to get on-disk queuing working, which would
> > be
> > > > nice if mysql goes down or a table is locked because I'm purging
> > > > records during a nightly maintenance cron.
> > > > >
> > > > > I never see any queue files created. I dropped the size of the
> > memory
> > > > queue to something ridiculously small, but still no queue files
> > even
> > > > after 20 minutes.
> > > > >
> > > > > I followed the instructions given here:
> > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > with a
> > > > couple of other tutorials on rsyslogd and queuing.
> > > > >
> > > > > I'm quite sure I'm just completely missing something obvious.
> > Details
> > > > below... And thanks to anyone who might be able to point out what
> > I'm
> > > > doing wrong.
> > > > >
> > > > > -Steve
> > > > >
> > > > >
> > > > >
> > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-
> > )
> > > > >
> > > > > # if you experience problems, check
> > > > > # http://www.rsyslog.com/troubleshoot for assistance
> > > > >
> > > > > # rsyslog v3: load input modules
> > > > > # If you do not load inputs, nothing happens!
> > > > > # You may need to set the module load path if modules are not
> > found.
> > > > >
> > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > $ModLoad imuxsock # provides support for local system logging
> > (e.g.
> > > > via logger command)
> > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > $ModLoad ommysql
> > > > >
> > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > unavailable
> > > > >
> > > > > $MainMsgQueueSize 5
> > > > >
> > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > (spool)
> > > > files
> > > > >
> > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > mode
> > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > > > >
> > > > > # send snmpd INFO messages to the dustbin
> > > > >
> > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > then ~
> > > > >
> > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > values
> > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > %iut%,
> > > > '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip contains '10.128.255') then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > & ~
> > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > >
> > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then ~
> > > > >
> > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > Facility,
> > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.244' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > >
> > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > 'warn' or
> > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert'
> > or
> > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > 'panic')
> > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > & ~
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'notice') then ~
> > > > >
> > > > >
> > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > >
> > > > > # Log all kernel messages to the console.
> > > > > # Logging much else clutters up the screen.
> > > > > #kern.*
> > /dev/console
> > > > >
> > > > > # Log anything (except mail) of level info or higher.
> > > > > # Don't log private authentication messages!
> > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > /var/log/messages
> > > > >
> > > > > # The authpriv file has restricted access.
> > > > > authpriv.*
> > > > /var/log/secure
> > > > >
> > > > > # Log all the mail messages in one place.
> > > > > mail.* -
> > > > /var/log/maillog
> > > > >
> > > > >
> > > > > # Log cron stuff
> > > > > cron.* -
> > > > /var/log/cron
> > > > >
> > > > > # Everybody gets emergency messages
> > > > > *.emerg *
> > > > >
> > > > > # Save news errors of level crit and higher in a special file.
> > > > > uucp,news.crit -
> > > > /var/log/spooler
> > > > >
> > > > > # Save boot messages also to boot.log
> > > > > local7.*
> > > > /var/log/boot.log
> > > > >
> > > > > # Remote Logging (we use TCP for reliable delivery)
> > > > > # An on-disk queue is created for this action. If the remote host
> > is
> > > > > # down, messages are spooled to disk and sent when it is up
> > again.
> > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > files
> > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > possible)
> > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > shutdown
> > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > optional
> > > > > #*.* @@remote-host:514
> > > > >
> > > > >
> > > > > # ######### Receiving Messages from Remote Hosts ##########
> > > > > # TCP Syslog Server:
> > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > support
> > > > it)
> > > > > $ModLoad imtcp.so # load module
> > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514
> > > > >
> > > > > # UDP Syslog Server:
> > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > 514
> > > > >
> > > > >
> > > > > SOME DEBUG OUTPUT:
> > > > >
> > > > > (see attached)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Steve Chupack | IT Systems Administrator
> > > > > V: 877.327.8422 x 1242
> > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > >
> > > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


rgerhards at hq

Sep 12, 2011, 10:00 AM

Post #8 of 11 (793 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

I think I saw the problem in the config as exposed by the debug log. If I saw
correctly, you use

$ActionResumeRetryCount -1
Some filter ~
Some filter :ommysql:

The retry count is of type auto-reset. So the unlimited retries apply to the
first action (the discard) and the sql writer uses the default value, which
is to give up fairly quickly. It is important that you group these statements
right in front of the actual action you want them to apply to -- actually,
they *are* part of that action. I know its cumbersome, and things have and
will continue to improve greatly in v6.

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Steve Chupack
> Sent: Monday, September 12, 2011 4:39 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Unable to implement on-disk queuing
>
> Thanks for taking the time to respond, I really appreciate it.
>
> OK, I think I can better describe this now. You are correct, rsyslog is not
> seeing a need to queue. My debug log is filled with the following entries,
> which clearly indicate it sees no need to queue or spool. So again, I think
I am
> missing something very basic here...
>
>
> 7878.780426000:455e1940: main Q: entry added, size now 1 entries
> 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
> 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
> 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries
>
> I used the very basic config from your article on buffering.
>
> $MainMsgQueueSize 500
> $WorkDirectory /var/log/rsyslogq # default location for work (spool) files
> $MainMsgQueueFileName mainq # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
>
>
> Mysql is definitely not running, as show by the following:
>
> 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't
> connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
(2)
> rsyslogd: db error (2002): Can't connect to local MySQL server through
socket
> '/var/lib/mysql/mysql.sock' (2)
> 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg
> db error (2002): Can't connect to local MySQL server through socket
> '/var/lib/mysql/mysql.sock' (2)
> 8220.441040000:41537940: Called action, logging to ommysql
> 8220.573450000:41537940: Called action, logging to ommysql
> 8220.574801000:41537940: Called action, logging to ommysql
> 8220.602406000:41537940: Called action, logging to ommysql
> 8220.604330000:41537940: Called action, logging to ommysql
>
>
>
> On Mon, 12 Sep 2011 15:18:32 +0200
> Rainer Gerhards <rgerhards [at] hq> wrote:
>
> >
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Steve Chupack
> > > Sent: Monday, September 12, 2011 3:02 PM
> > > To: rsyslog [at] lists
> > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > >
> > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had
> > > mysql turned off for 20 minutes... And my message flow rate is up
> > > around 1 per second. So I would have expected some queuing to
> > > happen. Is there anything else I should look at or try?
> >
> > I think the unusual extremely low queue sizes results in some
> > watermarks set to 0 and inhibiting DA queue startup. Set the main
> > queue size to, say, 500 or 1000 and try again. Note that you can use
> > logger or the tcpflood tool to injects lots of message in a short period.
> >
> > Rainer
> >
> > >
> > > On Mon, 12 Sep 2011 11:13:28 +0200
> > > Rainer Gerhards <rgerhards [at] hq> wrote:
> > >
> > > > Queue files are only created if necessary. The debug log does not
> > > contain any
> > > > such situation.
> > > >
> > > > Rainer
> > > >
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > > To: rsyslog [at] lists
> > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > > >
> > > > > Ooops... previous message's attachment was the config, not the
> > > debug
> > > > > output. Debug attached.
> > > > >
> > > > >
> > > > >
> > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack
> > > > > <steve.chupack [at] dealer> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have rsyslogd up and running and logging to mysql with
> > > loganalyzer
> > > > > as a front end. Very cool.
> > > > > >
> > > > > > However, I can't seem to get on-disk queuing working, which
> > > > > > would
> > > be
> > > > > nice if mysql goes down or a table is locked because I'm purging
> > > > > records during a nightly maintenance cron.
> > > > > >
> > > > > > I never see any queue files created. I dropped the size of the
> > > memory
> > > > > queue to something ridiculously small, but still no queue files
> > > even
> > > > > after 20 minutes.
> > > > > >
> > > > > > I followed the instructions given here:
> > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > > with a
> > > > > couple of other tutorials on rsyslogd and queuing.
> > > > > >
> > > > > > I'm quite sure I'm just completely missing something obvious.
> > > Details
> > > > > below... And thanks to anyone who might be able to point out
> > > > > what
> > > I'm
> > > > > doing wrong.
> > > > > >
> > > > > > -Steve
> > > > > >
> > > > > >
> > > > > >
> > > > > > MY CONFIG, please, no flames re the horrendously complex rules
> > > > > > ;-
> > > )
> > > > > >
> > > > > > # if you experience problems, check #
> > > > > > http://www.rsyslog.com/troubleshoot for assistance
> > > > > >
> > > > > > # rsyslog v3: load input modules # If you do not load inputs,
> > > > > > nothing happens!
> > > > > > # You may need to set the module load path if modules are not
> > > found.
> > > > > >
> > > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > > $ModLoad imuxsock # provides support for local system logging
> > > (e.g.
> > > > > via logger command)
> > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > > $ModLoad ommysql
> > > > > >
> > > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > > unavailable
> > > > > >
> > > > > > $MainMsgQueueSize 5
> > > > > >
> > > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > > (spool)
> > > > > files
> > > > > >
> > > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > > mode
> > > > > > $ActionResumeRetryCount -1 # infinite retries on insert
> > > > > > failure
> > > > > >
> > > > > > # send snmpd INFO messages to the dustbin
> > > > > >
> > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > > then ~
> > > > > >
> > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > > values
> > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > > %iut%,
> > > > > '%syslogtag%')",SQL
> > > > > >
> > > > > > if ($fromhost-ip contains '10.128.255') then
> > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > > & ~
> > > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > > >
> > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches
> > > > > > (Message,
> > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > '%HOSTNAME%',
> > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > >
> > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > 'core01'
> > > > > or $hostname contains_i 'core02' then
> > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > > #if ($fromhost-ip contains '10.128.0' or
> > > > > > $source=='vt1hs1-dc216-
> > > > > core01' or $source=='vt1hs1-dc216-core02') then
> > > > > /var/log/switches
> > > > > > & ~
> > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > 'core01'
> > > > > or $hostname contains_i 'core02' then ~
> > > > > >
> > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > > Facility,
> > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > >
> > > > > > if $fromhost-ip contains '10.128.244' then
> > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > > & ~
> > > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > > >
> > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > '%HOSTNAME%',
> > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > >
> > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > '10.128.24.44'
> > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > or
> > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > '10.128.24.84')
> > > and
> > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > > 'warn' or
> > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error'
> > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text ==
'alert'
> > > or
> > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > > 'panic')
> > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > > & ~
> > > > > >
> > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > '10.128.24.44'
> > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > or
> > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > '10.128.24.84')
> > > and
> > > > > ($syslogseverity-text == 'notice') then ~
> > > > > >
> > > > > >
> > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > > >
> > > > > > # Log all kernel messages to the console.
> > > > > > # Logging much else clutters up the screen.
> > > > > > #kern.*
> > > /dev/console
> > > > > >
> > > > > > # Log anything (except mail) of level info or higher.
> > > > > > # Don't log private authentication messages!
> > > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > > /var/log/messages
> > > > > >
> > > > > > # The authpriv file has restricted access.
> > > > > > authpriv.*
> > > > > /var/log/secure
> > > > > >
> > > > > > # Log all the mail messages in one place.
> > > > > > mail.* -
> > > > > /var/log/maillog
> > > > > >
> > > > > >
> > > > > > # Log cron stuff
> > > > > > cron.* -
> > > > > /var/log/cron
> > > > > >
> > > > > > # Everybody gets emergency messages
> > > > > > *.emerg *
> > > > > >
> > > > > > # Save news errors of level crit and higher in a special file.
> > > > > > uucp,news.crit -
> > > > > /var/log/spooler
> > > > > >
> > > > > > # Save boot messages also to boot.log
> > > > > > local7.*
> > > > > /var/log/boot.log
> > > > > >
> > > > > > # Remote Logging (we use TCP for reliable delivery) # An
> > > > > > on-disk queue is created for this action. If the remote host
> > > is
> > > > > > # down, messages are spooled to disk and sent when it is up
> > > again.
> > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > > files
> > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > > possible)
> > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > > shutdown
> > > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > > optional
> > > > > > #*.* @@remote-host:514
> > > > > >
> > > > > >
> > > > > > # ######### Receiving Messages from Remote Hosts ##########
> #
> > > > > > TCP Syslog Server:
> > > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > > support
> > > > > it)
> > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 #
> > > > > > start up TCP listener at port 514
> > > > > >
> > > > > > # UDP Syslog Server:
> > > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > > 514
> > > > > >
> > > > > >
> > > > > > SOME DEBUG OUTPUT:
> > > > > >
> > > > > > (see attached)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Steve Chupack | IT Systems Administrator
> > > > > > V: 877.327.8422 x 1242
> > > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > > >
> > > > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


steve.chupack at dealer

Sep 12, 2011, 1:06 PM

Post #9 of 11 (794 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Rainier,

Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior.

Again, your help is hugely appreciated.

Steve



On Mon, 12 Sep 2011 19:00:28 +0200
Rainer Gerhards <rgerhards [at] hq> wrote:

> I think I saw the problem in the config as exposed by the debug log. If I saw
> correctly, you use
>
> $ActionResumeRetryCount -1
> Some filter ~
> Some filter :ommysql:
>
> The retry count is of type auto-reset. So the unlimited retries apply to the
> first action (the discard) and the sql writer uses the default value, which
> is to give up fairly quickly. It is important that you group these statements
> right in front of the actual action you want them to apply to -- actually,
> they *are* part of that action. I know its cumbersome, and things have and
> will continue to improve greatly in v6.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Steve Chupack
> > Sent: Monday, September 12, 2011 4:39 PM
> > To: rsyslog [at] lists
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Thanks for taking the time to respond, I really appreciate it.
> >
> > OK, I think I can better describe this now. You are correct, rsyslog is not
> > seeing a need to queue. My debug log is filled with the following entries,
> > which clearly indicate it sees no need to queue or spool. So again, I think
> I am
> > missing something very basic here...
> >
> >
> > 7878.780426000:455e1940: main Q: entry added, size now 1 entries
> > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
> > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
> > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries
> >
> > I used the very basic config from your article on buffering.
> >
> > $MainMsgQueueSize 500
> > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files
> > $MainMsgQueueFileName mainq # set file name, also enables disk mode
> > $ActionResumeRetryCount -1 # infinite retries on insert failure
> >
> >
> > Mysql is definitely not running, as show by the following:
> >
> > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't
> > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
> (2)
> > rsyslogd: db error (2002): Can't connect to local MySQL server through
> socket
> > '/var/lib/mysql/mysql.sock' (2)
> > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg
> > db error (2002): Can't connect to local MySQL server through socket
> > '/var/lib/mysql/mysql.sock' (2)
> > 8220.441040000:41537940: Called action, logging to ommysql
> > 8220.573450000:41537940: Called action, logging to ommysql
> > 8220.574801000:41537940: Called action, logging to ommysql
> > 8220.602406000:41537940: Called action, logging to ommysql
> > 8220.604330000:41537940: Called action, logging to ommysql
> >
> >
> >
> > On Mon, 12 Sep 2011 15:18:32 +0200
> > Rainer Gerhards <rgerhards [at] hq> wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > Sent: Monday, September 12, 2011 3:02 PM
> > > > To: rsyslog [at] lists
> > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > >
> > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had
> > > > mysql turned off for 20 minutes... And my message flow rate is up
> > > > around 1 per second. So I would have expected some queuing to
> > > > happen. Is there anything else I should look at or try?
> > >
> > > I think the unusual extremely low queue sizes results in some
> > > watermarks set to 0 and inhibiting DA queue startup. Set the main
> > > queue size to, say, 500 or 1000 and try again. Note that you can use
> > > logger or the tcpflood tool to injects lots of message in a short period.
> > >
> > > Rainer
> > >
> > > >
> > > > On Mon, 12 Sep 2011 11:13:28 +0200
> > > > Rainer Gerhards <rgerhards [at] hq> wrote:
> > > >
> > > > > Queue files are only created if necessary. The debug log does not
> > > > contain any
> > > > > such situation.
> > > > >
> > > > > Rainer
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > > > To: rsyslog [at] lists
> > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > > > >
> > > > > > Ooops... previous message's attachment was the config, not the
> > > > debug
> > > > > > output. Debug attached.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack
> > > > > > <steve.chupack [at] dealer> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have rsyslogd up and running and logging to mysql with
> > > > loganalyzer
> > > > > > as a front end. Very cool.
> > > > > > >
> > > > > > > However, I can't seem to get on-disk queuing working, which
> > > > > > > would
> > > > be
> > > > > > nice if mysql goes down or a table is locked because I'm purging
> > > > > > records during a nightly maintenance cron.
> > > > > > >
> > > > > > > I never see any queue files created. I dropped the size of the
> > > > memory
> > > > > > queue to something ridiculously small, but still no queue files
> > > > even
> > > > > > after 20 minutes.
> > > > > > >
> > > > > > > I followed the instructions given here:
> > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > > > with a
> > > > > > couple of other tutorials on rsyslogd and queuing.
> > > > > > >
> > > > > > > I'm quite sure I'm just completely missing something obvious.
> > > > Details
> > > > > > below... And thanks to anyone who might be able to point out
> > > > > > what
> > > > I'm
> > > > > > doing wrong.
> > > > > > >
> > > > > > > -Steve
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > MY CONFIG, please, no flames re the horrendously complex rules
> > > > > > > ;-
> > > > )
> > > > > > >
> > > > > > > # if you experience problems, check #
> > > > > > > http://www.rsyslog.com/troubleshoot for assistance
> > > > > > >
> > > > > > > # rsyslog v3: load input modules # If you do not load inputs,
> > > > > > > nothing happens!
> > > > > > > # You may need to set the module load path if modules are not
> > > > found.
> > > > > > >
> > > > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > > > $ModLoad imuxsock # provides support for local system logging
> > > > (e.g.
> > > > > > via logger command)
> > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > > > $ModLoad ommysql
> > > > > > >
> > > > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > > > unavailable
> > > > > > >
> > > > > > > $MainMsgQueueSize 5
> > > > > > >
> > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > > > (spool)
> > > > > > files
> > > > > > >
> > > > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > > > mode
> > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert
> > > > > > > failure
> > > > > > >
> > > > > > > # send snmpd INFO messages to the dustbin
> > > > > > >
> > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > > > then ~
> > > > > > >
> > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > > > values
> > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > > > %iut%,
> > > > > > '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if ($fromhost-ip contains '10.128.255') then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > > > & ~
> > > > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > > > >
> > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches
> > > > > > > (Message,
> > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > 'core01'
> > > > > > or $hostname contains_i 'core02' then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > > > #if ($fromhost-ip contains '10.128.0' or
> > > > > > > $source=='vt1hs1-dc216-
> > > > > > core01' or $source=='vt1hs1-dc216-core02') then
> > > > > > /var/log/switches
> > > > > > > & ~
> > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > 'core01'
> > > > > > or $hostname contains_i 'core02' then ~
> > > > > > >
> > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > > > Facility,
> > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if $fromhost-ip contains '10.128.244' then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > > > & ~
> > > > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > > > >
> > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > '10.128.24.44'
> > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > or
> > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > '10.128.24.84')
> > > > and
> > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > > > 'warn' or
> > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error'
> > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text ==
> 'alert'
> > > > or
> > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > > > 'panic')
> > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > > > & ~
> > > > > > >
> > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > '10.128.24.44'
> > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > or
> > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > '10.128.24.84')
> > > > and
> > > > > > ($syslogseverity-text == 'notice') then ~
> > > > > > >
> > > > > > >
> > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > >
> > > > > > > # Log all kernel messages to the console.
> > > > > > > # Logging much else clutters up the screen.
> > > > > > > #kern.*
> > > > /dev/console
> > > > > > >
> > > > > > > # Log anything (except mail) of level info or higher.
> > > > > > > # Don't log private authentication messages!
> > > > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > > > /var/log/messages
> > > > > > >
> > > > > > > # The authpriv file has restricted access.
> > > > > > > authpriv.*
> > > > > > /var/log/secure
> > > > > > >
> > > > > > > # Log all the mail messages in one place.
> > > > > > > mail.* -
> > > > > > /var/log/maillog
> > > > > > >
> > > > > > >
> > > > > > > # Log cron stuff
> > > > > > > cron.* -
> > > > > > /var/log/cron
> > > > > > >
> > > > > > > # Everybody gets emergency messages
> > > > > > > *.emerg *
> > > > > > >
> > > > > > > # Save news errors of level crit and higher in a special file.
> > > > > > > uucp,news.crit -
> > > > > > /var/log/spooler
> > > > > > >
> > > > > > > # Save boot messages also to boot.log
> > > > > > > local7.*
> > > > > > /var/log/boot.log
> > > > > > >
> > > > > > > # Remote Logging (we use TCP for reliable delivery) # An
> > > > > > > on-disk queue is created for this action. If the remote host
> > > > is
> > > > > > > # down, messages are spooled to disk and sent when it is up
> > > > again.
> > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > > > files
> > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > > > possible)
> > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > > > shutdown
> > > > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > > > optional
> > > > > > > #*.* @@remote-host:514
> > > > > > >
> > > > > > >
> > > > > > > # ######### Receiving Messages from Remote Hosts ##########
> > #
> > > > > > > TCP Syslog Server:
> > > > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > > > support
> > > > > > it)
> > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 #
> > > > > > > start up TCP listener at port 514
> > > > > > >
> > > > > > > # UDP Syslog Server:
> > > > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > > > 514
> > > > > > >
> > > > > > >
> > > > > > > SOME DEBUG OUTPUT:
> > > > > > >
> > > > > > > (see attached)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Steve Chupack | IT Systems Administrator
> > > > > > > V: 877.327.8422 x 1242
> > > > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > > > >
> > > > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


steve.chupack at dealer

Sep 12, 2011, 1:06 PM

Post #10 of 11 (795 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

Rainer,

Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior.

Again, your help is hugely appreciated.

Steve



On Mon, 12 Sep 2011 19:00:28 +0200
Rainer Gerhards <rgerhards [at] hq> wrote:

> I think I saw the problem in the config as exposed by the debug log. If I saw
> correctly, you use
>
> $ActionResumeRetryCount -1
> Some filter ~
> Some filter :ommysql:
>
> The retry count is of type auto-reset. So the unlimited retries apply to the
> first action (the discard) and the sql writer uses the default value, which
> is to give up fairly quickly. It is important that you group these statements
> right in front of the actual action you want them to apply to -- actually,
> they *are* part of that action. I know its cumbersome, and things have and
> will continue to improve greatly in v6.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Steve Chupack
> > Sent: Monday, September 12, 2011 4:39 PM
> > To: rsyslog [at] lists
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Thanks for taking the time to respond, I really appreciate it.
> >
> > OK, I think I can better describe this now. You are correct, rsyslog is not
> > seeing a need to queue. My debug log is filled with the following entries,
> > which clearly indicate it sees no need to queue or spool. So again, I think
> I am
> > missing something very basic here...
> >
> >
> > 7878.780426000:455e1940: main Q: entry added, size now 1 entries
> > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
> > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
> > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries
> >
> > I used the very basic config from your article on buffering.
> >
> > $MainMsgQueueSize 500
> > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files
> > $MainMsgQueueFileName mainq # set file name, also enables disk mode
> > $ActionResumeRetryCount -1 # infinite retries on insert failure
> >
> >
> > Mysql is definitely not running, as show by the following:
> >
> > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't
> > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
> (2)
> > rsyslogd: db error (2002): Can't connect to local MySQL server through
> socket
> > '/var/lib/mysql/mysql.sock' (2)
> > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg
> > db error (2002): Can't connect to local MySQL server through socket
> > '/var/lib/mysql/mysql.sock' (2)
> > 8220.441040000:41537940: Called action, logging to ommysql
> > 8220.573450000:41537940: Called action, logging to ommysql
> > 8220.574801000:41537940: Called action, logging to ommysql
> > 8220.602406000:41537940: Called action, logging to ommysql
> > 8220.604330000:41537940: Called action, logging to ommysql
> >
> >
> >
> > On Mon, 12 Sep 2011 15:18:32 +0200
> > Rainer Gerhards <rgerhards [at] hq> wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > Sent: Monday, September 12, 2011 3:02 PM
> > > > To: rsyslog [at] lists
> > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > >
> > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had
> > > > mysql turned off for 20 minutes... And my message flow rate is up
> > > > around 1 per second. So I would have expected some queuing to
> > > > happen. Is there anything else I should look at or try?
> > >
> > > I think the unusual extremely low queue sizes results in some
> > > watermarks set to 0 and inhibiting DA queue startup. Set the main
> > > queue size to, say, 500 or 1000 and try again. Note that you can use
> > > logger or the tcpflood tool to injects lots of message in a short period.
> > >
> > > Rainer
> > >
> > > >
> > > > On Mon, 12 Sep 2011 11:13:28 +0200
> > > > Rainer Gerhards <rgerhards [at] hq> wrote:
> > > >
> > > > > Queue files are only created if necessary. The debug log does not
> > > > contain any
> > > > > such situation.
> > > > >
> > > > > Rainer
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > > > To: rsyslog [at] lists
> > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > > > >
> > > > > > Ooops... previous message's attachment was the config, not the
> > > > debug
> > > > > > output. Debug attached.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack
> > > > > > <steve.chupack [at] dealer> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have rsyslogd up and running and logging to mysql with
> > > > loganalyzer
> > > > > > as a front end. Very cool.
> > > > > > >
> > > > > > > However, I can't seem to get on-disk queuing working, which
> > > > > > > would
> > > > be
> > > > > > nice if mysql goes down or a table is locked because I'm purging
> > > > > > records during a nightly maintenance cron.
> > > > > > >
> > > > > > > I never see any queue files created. I dropped the size of the
> > > > memory
> > > > > > queue to something ridiculously small, but still no queue files
> > > > even
> > > > > > after 20 minutes.
> > > > > > >
> > > > > > > I followed the instructions given here:
> > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > > > with a
> > > > > > couple of other tutorials on rsyslogd and queuing.
> > > > > > >
> > > > > > > I'm quite sure I'm just completely missing something obvious.
> > > > Details
> > > > > > below... And thanks to anyone who might be able to point out
> > > > > > what
> > > > I'm
> > > > > > doing wrong.
> > > > > > >
> > > > > > > -Steve
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > MY CONFIG, please, no flames re the horrendously complex rules
> > > > > > > ;-
> > > > )
> > > > > > >
> > > > > > > # if you experience problems, check #
> > > > > > > http://www.rsyslog.com/troubleshoot for assistance
> > > > > > >
> > > > > > > # rsyslog v3: load input modules # If you do not load inputs,
> > > > > > > nothing happens!
> > > > > > > # You may need to set the module load path if modules are not
> > > > found.
> > > > > > >
> > > > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > > > $ModLoad imuxsock # provides support for local system logging
> > > > (e.g.
> > > > > > via logger command)
> > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > > > $ModLoad ommysql
> > > > > > >
> > > > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > > > unavailable
> > > > > > >
> > > > > > > $MainMsgQueueSize 5
> > > > > > >
> > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > > > (spool)
> > > > > > files
> > > > > > >
> > > > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > > > mode
> > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert
> > > > > > > failure
> > > > > > >
> > > > > > > # send snmpd INFO messages to the dustbin
> > > > > > >
> > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > > > then ~
> > > > > > >
> > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > > > values
> > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > > > %iut%,
> > > > > > '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if ($fromhost-ip contains '10.128.255') then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > > > & ~
> > > > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > > > >
> > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches
> > > > > > > (Message,
> > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > 'core01'
> > > > > > or $hostname contains_i 'core02' then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > > > #if ($fromhost-ip contains '10.128.0' or
> > > > > > > $source=='vt1hs1-dc216-
> > > > > > core01' or $source=='vt1hs1-dc216-core02') then
> > > > > > /var/log/switches
> > > > > > > & ~
> > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > 'core01'
> > > > > > or $hostname contains_i 'core02' then ~
> > > > > > >
> > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > > > Facility,
> > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if $fromhost-ip contains '10.128.244' then
> > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > > > & ~
> > > > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > > > >
> > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > '%HOSTNAME%',
> > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > >
> > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > '10.128.24.44'
> > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > or
> > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > '10.128.24.84')
> > > > and
> > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > > > 'warn' or
> > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error'
> > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text ==
> 'alert'
> > > > or
> > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > > > 'panic')
> > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > > > & ~
> > > > > > >
> > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > '10.128.24.44'
> > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > or
> > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > '10.128.24.84')
> > > > and
> > > > > > ($syslogseverity-text == 'notice') then ~
> > > > > > >
> > > > > > >
> > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > >
> > > > > > > # Log all kernel messages to the console.
> > > > > > > # Logging much else clutters up the screen.
> > > > > > > #kern.*
> > > > /dev/console
> > > > > > >
> > > > > > > # Log anything (except mail) of level info or higher.
> > > > > > > # Don't log private authentication messages!
> > > > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > > > /var/log/messages
> > > > > > >
> > > > > > > # The authpriv file has restricted access.
> > > > > > > authpriv.*
> > > > > > /var/log/secure
> > > > > > >
> > > > > > > # Log all the mail messages in one place.
> > > > > > > mail.* -
> > > > > > /var/log/maillog
> > > > > > >
> > > > > > >
> > > > > > > # Log cron stuff
> > > > > > > cron.* -
> > > > > > /var/log/cron
> > > > > > >
> > > > > > > # Everybody gets emergency messages
> > > > > > > *.emerg *
> > > > > > >
> > > > > > > # Save news errors of level crit and higher in a special file.
> > > > > > > uucp,news.crit -
> > > > > > /var/log/spooler
> > > > > > >
> > > > > > > # Save boot messages also to boot.log
> > > > > > > local7.*
> > > > > > /var/log/boot.log
> > > > > > >
> > > > > > > # Remote Logging (we use TCP for reliable delivery) # An
> > > > > > > on-disk queue is created for this action. If the remote host
> > > > is
> > > > > > > # down, messages are spooled to disk and sent when it is up
> > > > again.
> > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > > > files
> > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > > > possible)
> > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > > > shutdown
> > > > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > > > optional
> > > > > > > #*.* @@remote-host:514
> > > > > > >
> > > > > > >
> > > > > > > # ######### Receiving Messages from Remote Hosts ##########
> > #
> > > > > > > TCP Syslog Server:
> > > > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > > > support
> > > > > > it)
> > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 #
> > > > > > > start up TCP listener at port 514
> > > > > > >
> > > > > > > # UDP Syslog Server:
> > > > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > > > 514
> > > > > > >
> > > > > > >
> > > > > > > SOME DEBUG OUTPUT:
> > > > > > >
> > > > > > > (see attached)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Steve Chupack | IT Systems Administrator
> > > > > > > V: 877.327.8422 x 1242
> > > > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > > > >
> > > > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com


steve.chupack at dealer

Sep 12, 2011, 1:09 PM

Post #11 of 11 (798 views)
Permalink
Re: Unable to implement on-disk queuing [In reply to]

sorry, i incorrectly spelled your name the first time around on this. ;-)

On Mon, 12 Sep 2011 16:06:27 -0400
Steve Chupack <steve.chupack [at] dealer> wrote:

> Rainer,
>
> Thank you! I just tested and spool files are created and grow when mysql is off, then disappear when mysql comes back. The debug log also confirmed this behavior.
>
> Again, your help is hugely appreciated.
>
> Steve
>
>
>
> On Mon, 12 Sep 2011 19:00:28 +0200
> Rainer Gerhards <rgerhards [at] hq> wrote:
>
> > I think I saw the problem in the config as exposed by the debug log. If I saw
> > correctly, you use
> >
> > $ActionResumeRetryCount -1
> > Some filter ~
> > Some filter :ommysql:
> >
> > The retry count is of type auto-reset. So the unlimited retries apply to the
> > first action (the discard) and the sql writer uses the default value, which
> > is to give up fairly quickly. It is important that you group these statements
> > right in front of the actual action you want them to apply to -- actually,
> > they *are* part of that action. I know its cumbersome, and things have and
> > will continue to improve greatly in v6.
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Steve Chupack
> > > Sent: Monday, September 12, 2011 4:39 PM
> > > To: rsyslog [at] lists
> > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > >
> > > Thanks for taking the time to respond, I really appreciate it.
> > >
> > > OK, I think I can better describe this now. You are correct, rsyslog is not
> > > seeing a need to queue. My debug log is filled with the following entries,
> > > which clearly indicate it sees no need to queue or spool. So again, I think
> > I am
> > > missing something very basic here...
> > >
> > >
> > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries
> > > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
> > > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
> > > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries
> > >
> > > I used the very basic config from your article on buffering.
> > >
> > > $MainMsgQueueSize 500
> > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files
> > > $MainMsgQueueFileName mainq # set file name, also enables disk mode
> > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > >
> > >
> > > Mysql is definitely not running, as show by the following:
> > >
> > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't
> > > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
> > (2)
> > > rsyslogd: db error (2002): Can't connect to local MySQL server through
> > socket
> > > '/var/lib/mysql/mysql.sock' (2)
> > > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg
> > > db error (2002): Can't connect to local MySQL server through socket
> > > '/var/lib/mysql/mysql.sock' (2)
> > > 8220.441040000:41537940: Called action, logging to ommysql
> > > 8220.573450000:41537940: Called action, logging to ommysql
> > > 8220.574801000:41537940: Called action, logging to ommysql
> > > 8220.602406000:41537940: Called action, logging to ommysql
> > > 8220.604330000:41537940: Called action, logging to ommysql
> > >
> > >
> > >
> > > On Mon, 12 Sep 2011 15:18:32 +0200
> > > Rainer Gerhards <rgerhards [at] hq> wrote:
> > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > > Sent: Monday, September 12, 2011 3:02 PM
> > > > > To: rsyslog [at] lists
> > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > > >
> > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had
> > > > > mysql turned off for 20 minutes... And my message flow rate is up
> > > > > around 1 per second. So I would have expected some queuing to
> > > > > happen. Is there anything else I should look at or try?
> > > >
> > > > I think the unusual extremely low queue sizes results in some
> > > > watermarks set to 0 and inhibiting DA queue startup. Set the main
> > > > queue size to, say, 500 or 1000 and try again. Note that you can use
> > > > logger or the tcpflood tool to injects lots of message in a short period.
> > > >
> > > > Rainer
> > > >
> > > > >
> > > > > On Mon, 12 Sep 2011 11:13:28 +0200
> > > > > Rainer Gerhards <rgerhards [at] hq> wrote:
> > > > >
> > > > > > Queue files are only created if necessary. The debug log does not
> > > > > contain any
> > > > > > such situation.
> > > > > >
> > > > > > Rainer
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > > > bounces [at] lists] On Behalf Of Steve Chupack
> > > > > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > > > > To: rsyslog [at] lists
> > > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > > > > >
> > > > > > > Ooops... previous message's attachment was the config, not the
> > > > > debug
> > > > > > > output. Debug attached.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack
> > > > > > > <steve.chupack [at] dealer> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have rsyslogd up and running and logging to mysql with
> > > > > loganalyzer
> > > > > > > as a front end. Very cool.
> > > > > > > >
> > > > > > > > However, I can't seem to get on-disk queuing working, which
> > > > > > > > would
> > > > > be
> > > > > > > nice if mysql goes down or a table is locked because I'm purging
> > > > > > > records during a nightly maintenance cron.
> > > > > > > >
> > > > > > > > I never see any queue files created. I dropped the size of the
> > > > > memory
> > > > > > > queue to something ridiculously small, but still no queue files
> > > > > even
> > > > > > > after 20 minutes.
> > > > > > > >
> > > > > > > > I followed the instructions given here:
> > > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > > > > with a
> > > > > > > couple of other tutorials on rsyslogd and queuing.
> > > > > > > >
> > > > > > > > I'm quite sure I'm just completely missing something obvious.
> > > > > Details
> > > > > > > below... And thanks to anyone who might be able to point out
> > > > > > > what
> > > > > I'm
> > > > > > > doing wrong.
> > > > > > > >
> > > > > > > > -Steve
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules
> > > > > > > > ;-
> > > > > )
> > > > > > > >
> > > > > > > > # if you experience problems, check #
> > > > > > > > http://www.rsyslog.com/troubleshoot for assistance
> > > > > > > >
> > > > > > > > # rsyslog v3: load input modules # If you do not load inputs,
> > > > > > > > nothing happens!
> > > > > > > > # You may need to set the module load path if modules are not
> > > > > found.
> > > > > > > >
> > > > > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > > > > $ModLoad imuxsock # provides support for local system logging
> > > > > (e.g.
> > > > > > > via logger command)
> > > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > > > > $ModLoad ommysql
> > > > > > > >
> > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > > > > unavailable
> > > > > > > >
> > > > > > > > $MainMsgQueueSize 5
> > > > > > > >
> > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > > > > (spool)
> > > > > > > files
> > > > > > > >
> > > > > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > > > > mode
> > > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert
> > > > > > > > failure
> > > > > > > >
> > > > > > > > # send snmpd INFO messages to the dustbin
> > > > > > > >
> > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > > > > then ~
> > > > > > > >
> > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > > > > values
> > > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > > > > %iut%,
> > > > > > > '%syslogtag%')",SQL
> > > > > > > >
> > > > > > > > if ($fromhost-ip contains '10.128.255') then
> > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > > > > & ~
> > > > > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > > > > >
> > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches
> > > > > > > > (Message,
> > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > > '%HOSTNAME%',
> > > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > > >
> > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > > 'core01'
> > > > > > > or $hostname contains_i 'core02' then
> > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > > > > #if ($fromhost-ip contains '10.128.0' or
> > > > > > > > $source=='vt1hs1-dc216-
> > > > > > > core01' or $source=='vt1hs1-dc216-core02') then
> > > > > > > /var/log/switches
> > > > > > > > & ~
> > > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > > > > 'core01'
> > > > > > > or $hostname contains_i 'core02' then ~
> > > > > > > >
> > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > > > > Facility,
> > > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > > >
> > > > > > > > if $fromhost-ip contains '10.128.244' then
> > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > > > > & ~
> > > > > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > > > > >
> > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > > > > '%HOSTNAME%',
> > > > > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > > > > >
> > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > > '10.128.24.44'
> > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > > or
> > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > > '10.128.24.84')
> > > > > and
> > > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > > > > 'warn' or
> > > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error'
> > > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text ==
> > 'alert'
> > > > > or
> > > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > > > > 'panic')
> > > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > > > > & ~
> > > > > > > >
> > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > > > > '10.128.24.44'
> > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > > > > or
> > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76'
> > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip ==
> > > > > > > '10.128.24.84')
> > > > > and
> > > > > > > ($syslogseverity-text == 'notice') then ~
> > > > > > > >
> > > > > > > >
> > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > > > > >
> > > > > > > > # Log all kernel messages to the console.
> > > > > > > > # Logging much else clutters up the screen.
> > > > > > > > #kern.*
> > > > > /dev/console
> > > > > > > >
> > > > > > > > # Log anything (except mail) of level info or higher.
> > > > > > > > # Don't log private authentication messages!
> > > > > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > > > > /var/log/messages
> > > > > > > >
> > > > > > > > # The authpriv file has restricted access.
> > > > > > > > authpriv.*
> > > > > > > /var/log/secure
> > > > > > > >
> > > > > > > > # Log all the mail messages in one place.
> > > > > > > > mail.* -
> > > > > > > /var/log/maillog
> > > > > > > >
> > > > > > > >
> > > > > > > > # Log cron stuff
> > > > > > > > cron.* -
> > > > > > > /var/log/cron
> > > > > > > >
> > > > > > > > # Everybody gets emergency messages
> > > > > > > > *.emerg *
> > > > > > > >
> > > > > > > > # Save news errors of level crit and higher in a special file.
> > > > > > > > uucp,news.crit -
> > > > > > > /var/log/spooler
> > > > > > > >
> > > > > > > > # Save boot messages also to boot.log
> > > > > > > > local7.*
> > > > > > > /var/log/boot.log
> > > > > > > >
> > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An
> > > > > > > > on-disk queue is created for this action. If the remote host
> > > > > is
> > > > > > > > # down, messages are spooled to disk and sent when it is up
> > > > > again.
> > > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > > > > files
> > > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > > > > possible)
> > > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > > > > shutdown
> > > > > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > > > > optional
> > > > > > > > #*.* @@remote-host:514
> > > > > > > >
> > > > > > > >
> > > > > > > > # ######### Receiving Messages from Remote Hosts ##########
> > > #
> > > > > > > > TCP Syslog Server:
> > > > > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > > > > support
> > > > > > > it)
> > > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 #
> > > > > > > > start up TCP listener at port 514
> > > > > > > >
> > > > > > > > # UDP Syslog Server:
> > > > > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > > > > 514
> > > > > > > >
> > > > > > > >
> > > > > > > > SOME DEBUG OUTPUT:
> > > > > > > >
> > > > > > > > (see attached)
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Steve Chupack | IT Systems Administrator
> > > > > > > > V: 877.327.8422 x 1242
> > > > > > > > Steve.Chupack [at] dealer | www.dealer.com
> > > > > > > >
> > > > > > > >
> > > > > > _______________________________________________
> > > > > > rsyslog mailing list
> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > http://www.rsyslog.com
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.