Mailing List Archive: RSyslog: users

Issues with configuring rsyslog with TCP configuration. Please help

Index | Next | Previous | View Threaded

velu1980 at gmail

Dec 9, 2011, 12:15 PM

Post #1 of 17 (699 views)
Permalink

Issues with configuring rsyslog with TCP configuration. Please help

Hi,

I am a newbie to Rsyslog and am having a difficult time with configuring
Rsyslog with TCP.

My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
centralise log server and logged into mysql DB.
Every thing works fine when running in TCP. But all the messages from the
clients are lost when shutdown the server.
The messages are not buffered in TCP. I have been struck with this for long
time. Any help is highly appreciated.
I have pasted the conf files and the log file. I noticed the following:

error forwarding via tcp, suspending.

I don't know how to resolve this. Please help.

*My client rsyslog.conf:*

$ModLoad imuxsock # local message reception
$ModLoad imudp
$ModLoad immark # provides --MARK-- message capability
$ModLoad imklog # provides kernel logging support (previously done by
rklogd)

$EscapeControlCharactersOnReceive off
$RepeatedMsgReduction on
$MarkMessagePeriod 0

#ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # the traditional
forwarding format with low-precision timestamps

*.info;mail.none;authpriv.none;cron.none /var/log/messages
# Log all kernel messages clutters everything
kern.* ~
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log

$WorkDirectory /rsyslog/work

$ActionQueueType LinkedList
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileEnableSync on

*.* @@10.XXX.XX.XX:10514

*Server rsyslog.conf file:*

$ModLoad imtcp
$ModLoad immark # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog # kernel logging (formerly provided by rklogd)
$ModLoad imudp # provides UDP syslog reception
$ModLoad MySQL

#InputTCPMaxSessions 500
$InputTCPServerRun 10514

$SystemLogSocketFlowControl on
$SystemLogSocketName syslog

$EscapeControlCharactersOnReceive off

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

local0.* /var/log/ltm
local4.* /var/log/ltm

$WorkDirectory /rsyslog/work

$RepeatedMsgReduction on

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName tcpfile # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down

:*.* :ommysql:127.0.0.1,DB,DBUser,dbpass

Log snippet:
9937.418783000:main queue:Reg/w0: (/var/log/messages)
9937.418798000:main queue:Reg/w0: Filter: check for property 'msg' (value '
#eNpVUV1P4zAQ/CuWn0Aiwet8tISnuxa4StcKCU48Rq7jthZJ7PNHrwL1v7Nu0Qkkv3hndmd29p22IobdH6/coqMNAFzRNuhB0YbeO31F2A2ZK0k4AyBQNmXVFBNy9/RMkefUYIJaWORCDpznUOdVkRAct8eJJ4SdoOkkn8K5yVszejUzHYpwxj7pWibNIXotE21QYWfQERX9Og5zFYTufQKkGYM6BETs/pAKerQxeNq8n6mtTl0lL/DVSfGQRZv5uB4N1lft6ueifZo/tMDLEirOqgljDFhdFbwG/p+/9WOmk/+6wLVwPShzXqZ5Tv2NygfrzEb3yXPAX2aT64iLZGKrxmRvad5034vrKmfk4kWPnfnnyeqZVDncErdvsH5JHpR8NdeYbvIA5F47tTGH1PPFuZdOr3GyNLgmfRQOBUQ/wyCc6X/p7Q65nQii3ToTrT+F1mNNSKlsyOROOK+SpRg22ZQeMbT47eDKOePOB6HL5exx/uM3hnK6ZBedCNqMtCnh+AGmDKd7$')
contains 'pvx': FALSE
9937.418810000:main queue:Reg/w0: Filter: check for property 'syslogtag'
(value '[ReportingSysLog]') startswith 'pvx': FALSE
9937.418821000:main queue:Reg/w0: Filter: check for property 'syslogtag'
(value '[ReportingSysLog]') isequal '[ReportingSysLog]': TRUE
9937.418835000:main queue:Reg/w0: Called action, logging to builtin-fwd
9937.418852000:main queue:Reg/w0: 10.122.87.75
9937.418862000:main queue:Reg/w0: 10.122.87.75:10514/tcp
9937.418879000:main queue:Reg/w0: CheckConnection detected broken
connection - closing it
9937.418924000:main queue:Reg/w0: file netstrms.c released module
'lmnsd_ptcp', reference count now 0
9937.418934000:main queue:Reg/w0: module 'lmnsd_ptcp' has zero reference
count, unloading...
9937.418943000:main queue:Reg/w0: Unloading module lmnsd_ptcp
9937.418954000:main queue:Reg/w0: file nsd_ptcp.c released module
'lmnetstrms', reference count now 2
9937.419019000:main queue:Reg/w0: caller requested object 'nsd_ptcp', not
found (iRet -3003)
9937.419033000:main queue:Reg/w0: Requested to load module 'lmnsd_ptcp'
9937.419044000:main queue:Reg/w0: loading module
'/usr/local/lib/rsyslog/lmnsd_ptcp.so'
9937.419154000:main queue:Reg/w0: source file nsd_ptcp.c requested
reference for module 'lmnetstrms', reference count now 3
9937.419169000:main queue:Reg/w0: module of type 2 being loaded.
9937.419179000:main queue:Reg/w0: source file netstrms.c requested
reference for module 'lmnsd_ptcp', reference count now 1
9937.419316000:main queue:Reg/w0: file netstrms.c released module
'lmnsd_ptcp', reference count now 0
9937.419326000:main queue:Reg/w0: module 'lmnsd_ptcp' has zero reference
count, unloading...
9937.419347000:main queue:Reg/w0: Unloading module lmnsd_ptcp
9937.419358000:main queue:Reg/w0: file nsd_ptcp.c released module
'lmnetstrms', reference count now 2
9937.419384000:main queue:Reg/w0: error forwarding via tcp, suspending
9937.419393000:main queue:Reg/w0: Action requested to be suspended, done
that.
9937.419403000:main queue:Reg/w0: Filter: check for property 'syslogtag'
(value '[ReportingSysLog]') isequal '[QPassReportingSysLog]': FALSE
9937.419416000:main queue:Reg/w0: main queue:Reg/w0: worker IDLE, waiting
for work.
9942.951659000:imudp.c: CmpHost returns 0

Thanks in advance again.

Thanks,
Velu
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 12:30 PM

Post #2 of 17 (663 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Fri, 9 Dec 2011, Velu S wrote:

> I am a newbie to Rsyslog and am having a difficult time with configuring
> Rsyslog with TCP.
>
> My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
> centralise log server and logged into mysql DB.
> Every thing works fine when running in TCP. But all the messages from the
> clients are lost when shutdown the server.
> The messages are not buffered in TCP. I have been struck with this for long
> time. Any help is highly appreciated.

With TCP the sender thinks the message has been delivered as soon as it
passes it to the TCP stack on the sending machine. As a result, when the
server goes down, any messages that the sender has passed to the TCP
stack.

To avoid this loss, you will need to use RELP instead of TCP. RELP doesn't
consider the message delivered until the receiving server acknowledges it.

This is part of the problem you are having.

In addition, you may be having additional problems with your sending
systems not buffering the messages, but it's a little hard to tell without
either resolving this issue, or at least quantifying how much of the issue
is this problem.

> I have pasted the conf files and the log file. I noticed the following:
>
> error forwarding via tcp, suspending.

This is exactly what you would expect to see when the connection goes
down. The sender will fill up the TCP buffers on the sending machine and
then it will find that it can't send any more and it will suspend the
channel.

This is normal and what should be happening. What then should happen when
the server comes back up is that the sender should notice that it can
re-establish the connection it should resume the channel

what version of rsyslog are you using on the sender and receiver.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 12:43 PM

Post #3 of 17 (664 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Hi David,

Thank you very much for your quick reply. I am using rsyslog version 4.4.2
in both sender and receiver.
I guess the TCP message doesn't happen on the sending machine. The messages
are not send even
when the receiver is restarted. Do i need to set any parameter other than
the parameters in my client
config file to buffer the messages.

*Client config file*

$ModLoad imuxsock # local message reception
$ModLoad imudp
$ModLoad immark # provides --MARK-- message capability
$ModLoad imklog # provides kernel logging support (previously done by
rklogd)

$EscapeControlCharactersOnReceive off
$RepeatedMsgReduction on
$MarkMessagePeriod 0

#ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # the traditional
forwarding format with low-precision timestamps

*.info;mail.none;authpriv.none;cron.none /var/log/messages
# Log all kernel messages clutters everything
kern.* ~
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log

$WorkDirectory /rsyslog/work

$ActionQueueType LinkedList
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionFileEnableSync on

*.* @@10.XXX.XX.XX:10514

Thanks,
Anand

On Fri, Dec 9, 2011 at 3:30 PM, <david [at] lang> wrote:

> On Fri, 9 Dec 2011, Velu S wrote:
>
> I am a newbie to Rsyslog and am having a difficult time with configuring
>> Rsyslog with TCP.
>>
>> My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
>> centralise log server and logged into mysql DB.
>> Every thing works fine when running in TCP. But all the messages from the
>> clients are lost when shutdown the server.
>> The messages are not buffered in TCP. I have been struck with this for
>> long
>> time. Any help is highly appreciated.
>>
>
> With TCP the sender thinks the message has been delivered as soon as it
> passes it to the TCP stack on the sending machine. As a result, when the
> server goes down, any messages that the sender has passed to the TCP stack.
>
> To avoid this loss, you will need to use RELP instead of TCP. RELP doesn't
> consider the message delivered until the receiving server acknowledges it.
>
>
> This is part of the problem you are having.
>
> In addition, you may be having additional problems with your sending
> systems not buffering the messages, but it's a little hard to tell without
> either resolving this issue, or at least quantifying how much of the issue
> is this problem.
>
>
> I have pasted the conf files and the log file. I noticed the following:
>>
>> error forwarding via tcp, suspending.
>>
>
> This is exactly what you would expect to see when the connection goes
> down. The sender will fill up the TCP buffers on the sending machine and
> then it will find that it can't send any more and it will suspend the
> channel.
>
> This is normal and what should be happening. What then should happen when
> the server comes back up is that the sender should notice that it can
> re-establish the connection it should resume the channel
>
> what version of rsyslog are you using on the sender and receiver.
>
> David Lang
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 1:04 PM

Post #4 of 17 (652 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Fri, 9 Dec 2011, Velu S wrote:

> Thank you very much for your quick reply. I am using rsyslog version 4.4.2
> in both sender and receiver.

I seem to remember hearing about problems with TCP not restarting the
dataflow properly and those problems getting fixed sometime in the last
year or two (4.4.2 is rather old)

if you can go to the latest 5.x it would be best.

David Lang

> I guess the TCP message doesn't happen on the sending machine. The messages
> are not send even
> when the receiver is restarted. Do i need to set any parameter other than
> the parameters in my client
> config file to buffer the messages.
>
> *Client config file*
>
> $ModLoad imuxsock # local message reception
> $ModLoad imudp
> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imklog # provides kernel logging support (previously done by
> rklogd)
>
> $EscapeControlCharactersOnReceive off
> $RepeatedMsgReduction on
> $MarkMessagePeriod 0
>
> #ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # the traditional
> forwarding format with low-precision timestamps
>
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
> # Log all kernel messages clutters everything
> kern.* ~
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
> # Log all the mail messages in one place.
> mail.* -/var/log/maillog
> # Log cron stuff
> cron.* /var/log/cron
> # Everybody gets emergency messages
> *.emerg *
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
> $WorkDirectory /rsyslog/work
>
> $ActionQueueType LinkedList
> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
> $ActionFileEnableSync on
>
> *.* @@10.XXX.XX.XX:10514
>
>
> Thanks,
> Anand
>
> On Fri, Dec 9, 2011 at 3:30 PM, <david [at] lang> wrote:
>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>
>> I am a newbie to Rsyslog and am having a difficult time with configuring
>>> Rsyslog with TCP.
>>>
>>> My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
>>> centralise log server and logged into mysql DB.
>>> Every thing works fine when running in TCP. But all the messages from the
>>> clients are lost when shutdown the server.
>>> The messages are not buffered in TCP. I have been struck with this for
>>> long
>>> time. Any help is highly appreciated.
>>>
>>
>> With TCP the sender thinks the message has been delivered as soon as it
>> passes it to the TCP stack on the sending machine. As a result, when the
>> server goes down, any messages that the sender has passed to the TCP stack.
>>
>> To avoid this loss, you will need to use RELP instead of TCP. RELP doesn't
>> consider the message delivered until the receiving server acknowledges it.
>>
>>
>> This is part of the problem you are having.
>>
>> In addition, you may be having additional problems with your sending
>> systems not buffering the messages, but it's a little hard to tell without
>> either resolving this issue, or at least quantifying how much of the issue
>> is this problem.
>>
>>
>> I have pasted the conf files and the log file. I noticed the following:
>>>
>>> error forwarding via tcp, suspending.
>>>
>>
>> This is exactly what you would expect to see when the connection goes
>> down. The sender will fill up the TCP buffers on the sending machine and
>> then it will find that it can't send any more and it will suspend the
>> channel.
>>
>> This is normal and what should be happening. What then should happen when
>> the server comes back up is that the sender should notice that it can
>> re-establish the connection it should resume the channel
>>
>> what version of rsyslog are you using on the sender and receiver.
>>
>> David Lang
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 1:17 PM

Post #5 of 17 (659 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Thank you David again for the quick reply. Really appreciate it!! I am
trying to install rsyslog-5.8.5 in my system.
I was able to run sudo ./configure succefully however when i make i get the
following error:
sudo make
make all-recursive
make[1]: Entering directory `/home/mfadmin/rsyslog-5.8.5'
Making all in doc
make[2]: Entering directory `/home/mfadmin/rsyslog-5.8.5/doc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/home/mfadmin/rsyslog-5.8.5/doc'
Making all in runtime
make[2]: Entering directory `/home/mfadmin/rsyslog-5.8.5/runtime'
CC librsyslog_la-rsyslog.lo
In file included from stream.h:72,
from obj.h:50,
from rsyslog.h:452,
from rsyslog.c:63:
zlibw.h:28:18: error: zlib.h: No such file or directory
In file included from stream.h:72,
from obj.h:50,
from rsyslog.h:452,
from rsyslog.c:63:
zlibw.h:32: error: expected ')' before 'strm'
zlibw.h:33: error: expected ';' before 'int'
In file included from obj.h:50,
from rsyslog.h:452,
from rsyslog.c:63:
stream.h:123: error: expected specifier-qualifier-list before 'Bytef'
make[2]: *** [librsyslog_la-rsyslog.lo] Error 1
make[2]: Leaving directory `/home/mfadmin/rsyslog-5.8.5/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/mfadmin/rsyslog-5.8.5'
make: *** [all] Error 2

I have made the rsyslog version 4.4.2 same way. It worked fine. Am i
missing any dependencies.

Thanks,
Velu

On Fri, Dec 9, 2011 at 4:04 PM, <david [at] lang> wrote:

> On Fri, 9 Dec 2011, Velu S wrote:
>
> Thank you very much for your quick reply. I am using rsyslog version 4.4.2
>> in both sender and receiver.
>>
>
> I seem to remember hearing about problems with TCP not restarting the
> dataflow properly and those problems getting fixed sometime in the last
> year or two (4.4.2 is rather old)
>
> if you can go to the latest 5.x it would be best.
>
> David Lang
>
> I guess the TCP message doesn't happen on the sending machine. The
>> messages
>> are not send even
>> when the receiver is restarted. Do i need to set any parameter other than
>> the parameters in my client
>> config file to buffer the messages.
>>
>> *Client config file*
>>
>>
>> $ModLoad imuxsock # local message reception
>> $ModLoad imudp
>> $ModLoad immark # provides --MARK-- message capability
>> $ModLoad imklog # provides kernel logging support (previously done by
>> rklogd)
>>
>> $**EscapeControlCharactersOnRecei**ve off
>> $RepeatedMsgReduction on
>> $MarkMessagePeriod 0
>>
>> #ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # the traditional
>> forwarding format with low-precision timestamps
>>
>> *.info;mail.none;authpriv.**none;cron.none
>> /var/log/messages
>> # Log all kernel messages clutters everything
>> kern.* ~
>> # The authpriv file has restricted access.
>> authpriv.* /var/log/secure
>> # Log all the mail messages in one place.
>> mail.* -/var/log/maillog
>> # Log cron stuff
>> cron.* /var/log/cron
>> # Everybody gets emergency messages
>> *.emerg *
>> # Save news errors of level crit and higher in a special file.
>> uucp,news.crit /var/log/spooler
>> # Save boot messages also to boot.log
>> local7.* /var/log/boot.log
>>
>> $WorkDirectory /rsyslog/work
>>
>> $ActionQueueType LinkedList
>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>> $ActionFileEnableSync on
>>
>> *.* @@10.XXX.XX.XX:10514
>>
>>
>> Thanks,
>> Anand
>>
>> On Fri, Dec 9, 2011 at 3:30 PM, <david [at] lang> wrote:
>>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>>
>>> I am a newbie to Rsyslog and am having a difficult time with configuring
>>>
>>>> Rsyslog with TCP.
>>>>
>>>> My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
>>>> centralise log server and logged into mysql DB.
>>>> Every thing works fine when running in TCP. But all the messages from
>>>> the
>>>> clients are lost when shutdown the server.
>>>> The messages are not buffered in TCP. I have been struck with this for
>>>> long
>>>> time. Any help is highly appreciated.
>>>>
>>>>
>>> With TCP the sender thinks the message has been delivered as soon as it
>>> passes it to the TCP stack on the sending machine. As a result, when the
>>> server goes down, any messages that the sender has passed to the TCP
>>> stack.
>>>
>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>> doesn't
>>> consider the message delivered until the receiving server acknowledges
>>> it.
>>>
>>>
>>> This is part of the problem you are having.
>>>
>>> In addition, you may be having additional problems with your sending
>>> systems not buffering the messages, but it's a little hard to tell
>>> without
>>> either resolving this issue, or at least quantifying how much of the
>>> issue
>>> is this problem.
>>>
>>>
>>> I have pasted the conf files and the log file. I noticed the following:
>>>
>>>>
>>>> error forwarding via tcp, suspending.
>>>>
>>>>
>>> This is exactly what you would expect to see when the connection goes
>>> down. The sender will fill up the TCP buffers on the sending machine and
>>> then it will find that it can't send any more and it will suspend the
>>> channel.
>>>
>>> This is normal and what should be happening. What then should happen when
>>> the server comes back up is that the sender should notice that it can
>>> re-establish the connection it should resume the channel
>>>
>>> what version of rsyslog are you using on the sender and receiver.
>>>
>>> David Lang
>>> ______________________________****_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 1:37 PM

Post #6 of 17 (698 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

this sounds like you are missing the zlib development library. you may not
have had compression enabled with your 4.x build in the config step.

David Lang

On Fri, 9 Dec 2011, Velu S wrote:

> Date: Fri, 9 Dec 2011 16:17:14 -0500
> From: Velu S <velu1980 [at] gmail>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: rsyslog-users <rsyslog [at] lists>
> Subject: Re: [rsyslog] Issues with configuring rsyslog with TCP configuration.
> Please help
>
> Thank you David again for the quick reply. Really appreciate it!! I am
> trying to install rsyslog-5.8.5 in my system.
> I was able to run sudo ./configure succefully however when i make i get the
> following error:
> sudo make
> make all-recursive
> make[1]: Entering directory `/home/mfadmin/rsyslog-5.8.5'
> Making all in doc
> make[2]: Entering directory `/home/mfadmin/rsyslog-5.8.5/doc'
> make[2]: Nothing to be done for `all'.
> make[2]: Leaving directory `/home/mfadmin/rsyslog-5.8.5/doc'
> Making all in runtime
> make[2]: Entering directory `/home/mfadmin/rsyslog-5.8.5/runtime'
> CC librsyslog_la-rsyslog.lo
> In file included from stream.h:72,
> from obj.h:50,
> from rsyslog.h:452,
> from rsyslog.c:63:
> zlibw.h:28:18: error: zlib.h: No such file or directory
> In file included from stream.h:72,
> from obj.h:50,
> from rsyslog.h:452,
> from rsyslog.c:63:
> zlibw.h:32: error: expected ')' before 'strm'
> zlibw.h:33: error: expected ';' before 'int'
> In file included from obj.h:50,
> from rsyslog.h:452,
> from rsyslog.c:63:
> stream.h:123: error: expected specifier-qualifier-list before 'Bytef'
> make[2]: *** [librsyslog_la-rsyslog.lo] Error 1
> make[2]: Leaving directory `/home/mfadmin/rsyslog-5.8.5/runtime'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/home/mfadmin/rsyslog-5.8.5'
> make: *** [all] Error 2
>
> I have made the rsyslog version 4.4.2 same way. It worked fine. Am i
> missing any dependencies.
>
> Thanks,
> Velu
>
>
> On Fri, Dec 9, 2011 at 4:04 PM, <david [at] lang> wrote:
>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>
>> Thank you very much for your quick reply. I am using rsyslog version 4.4.2
>>> in both sender and receiver.
>>>
>>
>> I seem to remember hearing about problems with TCP not restarting the
>> dataflow properly and those problems getting fixed sometime in the last
>> year or two (4.4.2 is rather old)
>>
>> if you can go to the latest 5.x it would be best.
>>
>> David Lang
>>
>> I guess the TCP message doesn't happen on the sending machine. The
>>> messages
>>> are not send even
>>> when the receiver is restarted. Do i need to set any parameter other than
>>> the parameters in my client
>>> config file to buffer the messages.
>>>
>>> *Client config file*
>>>
>>>
>>> $ModLoad imuxsock # local message reception
>>> $ModLoad imudp
>>> $ModLoad immark # provides --MARK-- message capability
>>> $ModLoad imklog # provides kernel logging support (previously done by
>>> rklogd)
>>>
>>> $**EscapeControlCharactersOnRecei**ve off
>>> $RepeatedMsgReduction on
>>> $MarkMessagePeriod 0
>>>
>>> #ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # the traditional
>>> forwarding format with low-precision timestamps
>>>
>>> *.info;mail.none;authpriv.**none;cron.none
>>> /var/log/messages
>>> # Log all kernel messages clutters everything
>>> kern.* ~
>>> # The authpriv file has restricted access.
>>> authpriv.* /var/log/secure
>>> # Log all the mail messages in one place.
>>> mail.* -/var/log/maillog
>>> # Log cron stuff
>>> cron.* /var/log/cron
>>> # Everybody gets emergency messages
>>> *.emerg *
>>> # Save news errors of level crit and higher in a special file.
>>> uucp,news.crit /var/log/spooler
>>> # Save boot messages also to boot.log
>>> local7.* /var/log/boot.log
>>>
>>> $WorkDirectory /rsyslog/work
>>>
>>> $ActionQueueType LinkedList
>>> $ActionQueueFileName srvrfwd # set file name, also enables disk mode
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
>>> $ActionFileEnableSync on
>>>
>>> *.* @@10.XXX.XX.XX:10514
>>>
>>>
>>> Thanks,
>>> Anand
>>>
>>> On Fri, Dec 9, 2011 at 3:30 PM, <david [at] lang> wrote:
>>>
>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>
>>>> I am a newbie to Rsyslog and am having a difficult time with configuring
>>>>
>>>>> Rsyslog with TCP.
>>>>>
>>>>> My environment : Rsyslog 4.4.2 in RHEL 5. Log messages are logged to a
>>>>> centralise log server and logged into mysql DB.
>>>>> Every thing works fine when running in TCP. But all the messages from
>>>>> the
>>>>> clients are lost when shutdown the server.
>>>>> The messages are not buffered in TCP. I have been struck with this for
>>>>> long
>>>>> time. Any help is highly appreciated.
>>>>>
>>>>>
>>>> With TCP the sender thinks the message has been delivered as soon as it
>>>> passes it to the TCP stack on the sending machine. As a result, when the
>>>> server goes down, any messages that the sender has passed to the TCP
>>>> stack.
>>>>
>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>> doesn't
>>>> consider the message delivered until the receiving server acknowledges
>>>> it.
>>>>
>>>>
>>>> This is part of the problem you are having.
>>>>
>>>> In addition, you may be having additional problems with your sending
>>>> systems not buffering the messages, but it's a little hard to tell
>>>> without
>>>> either resolving this issue, or at least quantifying how much of the
>>>> issue
>>>> is this problem.
>>>>
>>>>
>>>> I have pasted the conf files and the log file. I noticed the following:
>>>>
>>>>>
>>>>> error forwarding via tcp, suspending.
>>>>>
>>>>>
>>>> This is exactly what you would expect to see when the connection goes
>>>> down. The sender will fill up the TCP buffers on the sending machine and
>>>> then it will find that it can't send any more and it will suspend the
>>>> channel.
>>>>
>>>> This is normal and what should be happening. What then should happen when
>>>> the server comes back up is that the sender should notice that it can
>>>> re-establish the connection it should resume the channel
>>>>
>>>> what version of rsyslog are you using on the sender and receiver.
>>>>
>>>> David Lang
>>>> ______________________________****_________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

gkra at unnerving

Dec 9, 2011, 1:54 PM

Post #7 of 17 (659 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:

> To avoid this loss, you will need to use RELP instead of TCP. RELP doesn't consider the message delivered until the receiving server acknowledges it.

I'm currently using TCP delivery because I couldn't make sense of RELP when I put it in place. Currently, I have clients doing:

$WorkDirectory /var/spool/rsyslog
$ActionQueueFileName fwd_queue
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1
*.* @@logserver:1234;SiteIDForwardFormat

Would switching to RELP be as simple as replacing that last line with:

*.* :omrelp:logserver:2345;SiteIDForwardFormat

And adding a RELP listener on the new port on the server?

Thanks,

Gregory

--
Gregory K. Ruiz-Ade <gkra [at] unnerving>
OpenPGP Key ID: EAF4844B keyserver: pgpkeys.mit.edu

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 2:03 PM

Post #8 of 17 (653 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:

> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>
>> To avoid this loss, you will need to use RELP instead of TCP. RELP doesn't consider the message delivered until the receiving server acknowledges it.
>
> I'm currently using TCP delivery because I couldn't make sense of RELP when I put it in place. Currently, I have clients doing:
>
> $WorkDirectory /var/spool/rsyslog
> $ActionQueueFileName fwd_queue
> $ActionQueueSaveOnShutdown on
> $ActionQueueType LinkedList
> $ActionResumeRetryCount -1
> *.* @@logserver:1234;SiteIDForwardFormat
>
> Would switching to RELP be as simple as replacing that last line with:
>
> *.* :omrelp:logserver:2345;SiteIDForwardFormat
>
> And adding a RELP listener on the new port on the server?

other than the need to load the relp module, it should be about that
simple.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 2:35 PM

Post #9 of 17 (681 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Thank you David again for your quick reply. I installed Rsyslog version
5.8.6 in both the sender and receiver. But My TCP messages are still not
logged when the central server is down. Here is my log snippet:

9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
reference count now 2
9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
9717.969857000:4266b940: earliest retry=1323469718
9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
9717.969877000:4266b940: tryDoAction: unexpected error code -2007[nElem 1,
Commited UpTo 0], finalizing
9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1, nElem 1
9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
next retry (if applicable): 1323469718 [now 1323469717]
9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
9717.969911000:4266b940: tryDoAction: unexpected error code -2123[nElem 1,
Commited UpTo 0], finalizing
9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
next retry (if applicable): 1323469718 [now 1323469717]
9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
9717.969936000:4266b940: Processing next rule

Thanks,
Velu

On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:

> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>
> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>
>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>> doesn't consider the message delivered until the receiving server
>>> acknowledges it.
>>>
>>
>> I'm currently using TCP delivery because I couldn't make sense of RELP
>> when I put it in place. Currently, I have clients doing:
>>
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueFileName fwd_queue
>> $ActionQueueSaveOnShutdown on
>> $ActionQueueType LinkedList
>> $ActionResumeRetryCount -1
>> *.* @@logserver:1234;**SiteIDForwardFormat
>>
>> Would switching to RELP be as simple as replacing that last line with:
>>
>> *.* :omrelp:logserver:2345;**SiteIDForwardFormat
>>
>> And adding a RELP listener on the new port on the server?
>>
>
> other than the need to load the relp module, it should be about that
> simple.
>
> David Lang
>
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 3:55 PM

Post #10 of 17 (656 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Fri, 9 Dec 2011, Velu S wrote:

> Thank you David again for your quick reply. I installed Rsyslog version
> 5.8.6 in both the sender and receiver. But My TCP messages are still not
> logged when the central server is down. Here is my log snippet:

OK, I don't understand the problem

while the central server is down I would not expect the messages to be
logged there. but with a disk assist mode they should show up after the
server comes up.

I'm getting ready to head out for the evening, but will check in again in
4-5 hours, so let's try to clarify what's happening.

are you using RELP or just TCP?

In the ideal world, what should happen is:

everything up, logs are written locally and sent to the remote server.

remote server goes down.

with TCP, messages in flight are lost, with RELP nothing is lost

you should see the transport be suspended, and it will retry, failing each
time and eventually retry less frequently (to keep from wasting resources
on the sending box)

log messages should still be written locally, but queue up to be sent to
the remote server later.

note that if you don't enable the disk assisted queue this will still
happen, but only up to the point where you fill memory.

verify that logs are still being written locally, and look for files to be
created in your working directory for the ones queued up to be sent later.
you may need to generate a bunch of log messages to get to the point where
they queue to disk.

then bring up the remote server. you should have logs start arriving there
within a couple minutes, and then the older logs will start arriving.

the debug logs around the time that the remote server comes up should be
interesting. I don't remember what the retry time is, but I seem to
remember that by default it includes some sort of logrithmic backoff
logic, so if you are down for a long time, it may take a long time before
it decides to try again.

David Lang

> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
> reference count now 2
> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
> 9717.969857000:4266b940: earliest retry=1323469718
> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
> 9717.969877000:4266b940: tryDoAction: unexpected error code -2007[nElem 1,
> Commited UpTo 0], finalizing
> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1, nElem 1
> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
> next retry (if applicable): 1323469718 [now 1323469717]
> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
> 9717.969911000:4266b940: tryDoAction: unexpected error code -2123[nElem 1,
> Commited UpTo 0], finalizing
> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
> next retry (if applicable): 1323469718 [now 1323469717]
> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
> 9717.969936000:4266b940: Processing next rule
>
>
> Thanks,
> Velu
>
>
>
> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>
>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>
>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>
>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>> doesn't consider the message delivered until the receiving server
>>>> acknowledges it.
>>>>
>>>
>>> I'm currently using TCP delivery because I couldn't make sense of RELP
>>> when I put it in place. Currently, I have clients doing:
>>>
>>> $WorkDirectory /var/spool/rsyslog
>>> $ActionQueueFileName fwd_queue
>>> $ActionQueueSaveOnShutdown on
>>> $ActionQueueType LinkedList
>>> $ActionResumeRetryCount -1
>>> *.* @@logserver:1234;**SiteIDForwardFormat
>>>
>>> Would switching to RELP be as simple as replacing that last line with:
>>>
>>> *.* :omrelp:logserver:2345;**SiteIDForwardFormat
>>>
>>> And adding a RELP listener on the new port on the server?
>>>
>>
>> other than the need to load the relp module, it should be about that
>> simple.
>>
>> David Lang
>>
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 8:49 PM

Post #11 of 17 (660 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Thank you David for the reply. Sorry for not explaining clearly in my
previous emails.
I use disk assisted mode and TCP both on the client and the server. When
all the servers are up everything works fine.
When the server goes down the message send from the client are not logged
in. Thats expected. But these messages are lost even when the server comes
back online. I will get the entire log of the client (When the center goes
down and up).
I haven't tried installing RELP yet. Will try installing it and test the
same scenario again. Thanks again for your timely help and time. Really
appreciate it.

Thanks,
Velu

On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:

> On Fri, 9 Dec 2011, Velu S wrote:
>
> Thank you David again for your quick reply. I installed Rsyslog version
>> 5.8.6 in both the sender and receiver. But My TCP messages are still not
>> logged when the central server is down. Here is my log snippet:
>>
>
> OK, I don't understand the problem
>
> while the central server is down I would not expect the messages to be
> logged there. but with a disk assist mode they should show up after the
> server comes up.
>
> I'm getting ready to head out for the evening, but will check in again in
> 4-5 hours, so let's try to clarify what's happening.
>
> are you using RELP or just TCP?
>
> In the ideal world, what should happen is:
>
> everything up, logs are written locally and sent to the remote server.
>
> remote server goes down.
>
> with TCP, messages in flight are lost, with RELP nothing is lost
>
> you should see the transport be suspended, and it will retry, failing each
> time and eventually retry less frequently (to keep from wasting resources
> on the sending box)
>
> log messages should still be written locally, but queue up to be sent to
> the remote server later.
>
> note that if you don't enable the disk assisted queue this will still
> happen, but only up to the point where you fill memory.
>
> verify that logs are still being written locally, and look for files to be
> created in your working directory for the ones queued up to be sent later.
> you may need to generate a bunch of log messages to get to the point where
> they queue to disk.
>
>
> then bring up the remote server. you should have logs start arriving there
> within a couple minutes, and then the older logs will start arriving.
>
> the debug logs around the time that the remote server comes up should be
> interesting. I don't remember what the retry time is, but I seem to
> remember that by default it includes some sort of logrithmic backoff logic,
> so if you are down for a long time, it may take a long time before it
> decides to try again.
>
> David Lang
>
> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
>> reference count now 2
>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
>> 9717.969857000:4266b940: earliest retry=1323469718
>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>> 9717.969877000:4266b940: tryDoAction: unexpected error code -2007[nElem 1,
>> Commited UpTo 0], finalizing
>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1, nElem 1
>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>> next retry (if applicable): 1323469718 [now 1323469717]
>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>> 9717.969911000:4266b940: tryDoAction: unexpected error code -2123[nElem 1,
>> Commited UpTo 0], finalizing
>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>> next retry (if applicable): 1323469718 [now 1323469717]
>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>> 9717.969936000:4266b940: Processing next rule
>>
>>
>> Thanks,
>> Velu
>>
>>
>>
>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>
>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>
>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>
>>>>
>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>
>>>>> doesn't consider the message delivered until the receiving server
>>>>> acknowledges it.
>>>>>
>>>>>
>>>> I'm currently using TCP delivery because I couldn't make sense of RELP
>>>> when I put it in place. Currently, I have clients doing:
>>>>
>>>> $WorkDirectory /var/spool/rsyslog
>>>> $ActionQueueFileName fwd_queue
>>>> $ActionQueueSaveOnShutdown on
>>>> $ActionQueueType LinkedList
>>>> $ActionResumeRetryCount -1
>>>> *.* @@logserver:1234;****SiteIDForwardFormat
>>>>
>>>>
>>>> Would switching to RELP be as simple as replacing that last line with:
>>>>
>>>> *.* :omrelp:logserver:2345;****SiteIDForwardFormat
>>>>
>>>>
>>>> And adding a RELP listener on the new port on the server?
>>>>
>>>>
>>> other than the need to load the relp module, it should be about that
>>> simple.
>>>
>>> David Lang
>>>
>>> ______________________________****_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 9, 2011, 9:23 PM

Post #12 of 17 (654 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Fri, 9 Dec 2011, Velu S wrote:

> Thank you David for the reply. Sorry for not explaining clearly in my
> previous emails.
> I use disk assisted mode and TCP both on the client and the server. When
> all the servers are up everything works fine.
> When the server goes down the message send from the client are not logged
> in. Thats expected. But these messages are lost even when the server comes
> back online. I will get the entire log of the client (When the center goes
> down and up).

Ok, then the first thing that I would do to troubleshoot this is to
simplify things.

try removing all of the actionqueue configuraiton entries and see what
happens. you will only be able to store a relativly small number of
messages before things block, but if you stop the server, wait until you
get the suspend message, and then startup the server again, you should see
the messages resume. If not, there is some sort of retry option that will
need to be set

after we have the suspend/resume function working, then the next thing to
do is configure the main queue to be disk assisted (even if you want a
separate action queue eventually, changing the main queue is changing
fewer things). make sure that when there are enough messages queued that
without the disk assist things would block, you instead have files created
in your work directory and that after the server is restarted, it gets the
messages.

only at this point (where we know that suspend/resume works, and we know
that a disk assisted queue works), would I then change the main queue back
to the default and create an action queue.

David Lang

> I haven't tried installing RELP yet. Will try installing it and test the
> same scenario again. Thanks again for your timely help and time. Really
> appreciate it.
>
> Thanks,
> Velu
>
> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>
>> Thank you David again for your quick reply. I installed Rsyslog version
>>> 5.8.6 in both the sender and receiver. But My TCP messages are still not
>>> logged when the central server is down. Here is my log snippet:
>>>
>>
>> OK, I don't understand the problem
>>
>> while the central server is down I would not expect the messages to be
>> logged there. but with a disk assist mode they should show up after the
>> server comes up.
>>
>> I'm getting ready to head out for the evening, but will check in again in
>> 4-5 hours, so let's try to clarify what's happening.
>>
>> are you using RELP or just TCP?
>>
>> In the ideal world, what should happen is:
>>
>> everything up, logs are written locally and sent to the remote server.
>>
>> remote server goes down.
>>
>> with TCP, messages in flight are lost, with RELP nothing is lost
>>
>> you should see the transport be suspended, and it will retry, failing each
>> time and eventually retry less frequently (to keep from wasting resources
>> on the sending box)
>>
>> log messages should still be written locally, but queue up to be sent to
>> the remote server later.
>>
>> note that if you don't enable the disk assisted queue this will still
>> happen, but only up to the point where you fill memory.
>>
>> verify that logs are still being written locally, and look for files to be
>> created in your working directory for the ones queued up to be sent later.
>> you may need to generate a bunch of log messages to get to the point where
>> they queue to disk.
>>
>>
>> then bring up the remote server. you should have logs start arriving there
>> within a couple minutes, and then the older logs will start arriving.
>>
>> the debug logs around the time that the remote server comes up should be
>> interesting. I don't remember what the retry time is, but I seem to
>> remember that by default it includes some sort of logrithmic backoff logic,
>> so if you are down for a long time, it may take a long time before it
>> decides to try again.
>>
>> David Lang
>>
>> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
>>> reference count now 2
>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
>>> 9717.969857000:4266b940: earliest retry=1323469718
>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>> 9717.969877000:4266b940: tryDoAction: unexpected error code -2007[nElem 1,
>>> Commited UpTo 0], finalizing
>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1, nElem 1
>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>>> next retry (if applicable): 1323469718 [now 1323469717]
>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>> 9717.969911000:4266b940: tryDoAction: unexpected error code -2123[nElem 1,
>>> Commited UpTo 0], finalizing
>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>>> next retry (if applicable): 1323469718 [now 1323469717]
>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>> 9717.969936000:4266b940: Processing next rule
>>>
>>>
>>> Thanks,
>>> Velu
>>>
>>>
>>>
>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>
>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>
>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>
>>>>>
>>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>>
>>>>>> doesn't consider the message delivered until the receiving server
>>>>>> acknowledges it.
>>>>>>
>>>>>>
>>>>> I'm currently using TCP delivery because I couldn't make sense of RELP
>>>>> when I put it in place. Currently, I have clients doing:
>>>>>
>>>>> $WorkDirectory /var/spool/rsyslog
>>>>> $ActionQueueFileName fwd_queue
>>>>> $ActionQueueSaveOnShutdown on
>>>>> $ActionQueueType LinkedList
>>>>> $ActionResumeRetryCount -1
>>>>> *.* @@logserver:1234;****SiteIDForwardFormat
>>>>>
>>>>>
>>>>> Would switching to RELP be as simple as replacing that last line with:
>>>>>
>>>>> *.* :omrelp:logserver:2345;****SiteIDForwardFormat
>>>>>
>>>>>
>>>>> And adding a RELP listener on the new port on the server?
>>>>>
>>>>>
>>>> other than the need to load the relp module, it should be about that
>>>> simple.
>>>>
>>>> David Lang
>>>>
>>>> ______________________________****_________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 9:56 PM

Post #13 of 17 (658 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Thank you David again for the help. Here is what i tried. The messages are
still not buffered when the central server is down.
I disabled all the action queue messages. I restarted both the sender and
receiver.
When both servers are up the messages gets logged in fine, but when the
central server is down,
the messages don't get logged in. I tried by setting the
following parameters on, but still couldn't succed:

$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionFileEnableSync on
$ActionSendResendLastMsgOnReconnect on

Thanks,
Anand

On Sat, Dec 10, 2011 at 12:23 AM, <david [at] lang> wrote:

> On Fri, 9 Dec 2011, Velu S wrote:
>
> Thank you David for the reply. Sorry for not explaining clearly in my
>> previous emails.
>> I use disk assisted mode and TCP both on the client and the server. When
>> all the servers are up everything works fine.
>> When the server goes down the message send from the client are not logged
>> in. Thats expected. But these messages are lost even when the server comes
>> back online. I will get the entire log of the client (When the center goes
>> down and up).
>>
>
> Ok, then the first thing that I would do to troubleshoot this is to
> simplify things.
>
> try removing all of the actionqueue configuraiton entries and see what
> happens. you will only be able to store a relativly small number of
> messages before things block, but if you stop the server, wait until you
> get the suspend message, and then startup the server again, you should see
> the messages resume. If not, there is some sort of retry option that will
> need to be set
>
> after we have the suspend/resume function working, then the next thing to
> do is configure the main queue to be disk assisted (even if you want a
> separate action queue eventually, changing the main queue is changing fewer
> things). make sure that when there are enough messages queued that without
> the disk assist things would block, you instead have files created in your
> work directory and that after the server is restarted, it gets the messages.
>
> only at this point (where we know that suspend/resume works, and we know
> that a disk assisted queue works), would I then change the main queue back
> to the default and create an action queue.
>
> David Lang
>
> I haven't tried installing RELP yet. Will try installing it and test the
>> same scenario again. Thanks again for your timely help and time. Really
>> appreciate it.
>>
>> Thanks,
>> Velu
>>
>> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>>
>>> Thank you David again for your quick reply. I installed Rsyslog version
>>>
>>>> 5.8.6 in both the sender and receiver. But My TCP messages are still not
>>>> logged when the central server is down. Here is my log snippet:
>>>>
>>>>
>>> OK, I don't understand the problem
>>>
>>> while the central server is down I would not expect the messages to be
>>> logged there. but with a disk assist mode they should show up after the
>>> server comes up.
>>>
>>> I'm getting ready to head out for the evening, but will check in again in
>>> 4-5 hours, so let's try to clarify what's happening.
>>>
>>> are you using RELP or just TCP?
>>>
>>> In the ideal world, what should happen is:
>>>
>>> everything up, logs are written locally and sent to the remote server.
>>>
>>> remote server goes down.
>>>
>>> with TCP, messages in flight are lost, with RELP nothing is lost
>>>
>>> you should see the transport be suspended, and it will retry, failing
>>> each
>>> time and eventually retry less frequently (to keep from wasting resources
>>> on the sending box)
>>>
>>> log messages should still be written locally, but queue up to be sent to
>>> the remote server later.
>>>
>>> note that if you don't enable the disk assisted queue this will still
>>> happen, but only up to the point where you fill memory.
>>>
>>> verify that logs are still being written locally, and look for files to
>>> be
>>> created in your working directory for the ones queued up to be sent
>>> later.
>>> you may need to generate a bunch of log messages to get to the point
>>> where
>>> they queue to disk.
>>>
>>>
>>> then bring up the remote server. you should have logs start arriving
>>> there
>>> within a couple minutes, and then the older logs will start arriving.
>>>
>>> the debug logs around the time that the remote server comes up should be
>>> interesting. I don't remember what the retry time is, but I seem to
>>> remember that by default it includes some sort of logrithmic backoff
>>> logic,
>>> so if you are down for a long time, it may take a long time before it
>>> decides to try again.
>>>
>>> David Lang
>>>
>>> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
>>>
>>>> reference count now 2
>>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
>>>> 9717.969857000:4266b940: earliest retry=1323469718
>>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>>> 9717.969877000:4266b940: tryDoAction: unexpected error code -2007[nElem
>>>> 1,
>>>> Commited UpTo 0], finalizing
>>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1,
>>>> nElem 1
>>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>>> 9717.969911000:4266b940: tryDoAction: unexpected error code -2123[nElem
>>>> 1,
>>>> Commited UpTo 0], finalizing
>>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state: susp,
>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>>> 9717.969936000:4266b940: Processing next rule
>>>>
>>>>
>>>> Thanks,
>>>> Velu
>>>>
>>>>
>>>>
>>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>>
>>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>
>>>>>
>>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>>
>>>>>
>>>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>>>
>>>>>> doesn't consider the message delivered until the receiving server
>>>>>>> acknowledges it.
>>>>>>>
>>>>>>>
>>>>>>> I'm currently using TCP delivery because I couldn't make sense of
>>>>>> RELP
>>>>>> when I put it in place. Currently, I have clients doing:
>>>>>>
>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>> $ActionQueueFileName fwd_queue
>>>>>> $ActionQueueSaveOnShutdown on
>>>>>> $ActionQueueType LinkedList
>>>>>> $ActionResumeRetryCount -1
>>>>>> *.* @@logserver:1234;******SiteIDForwardFormat
>>>>>>
>>>>>>
>>>>>>
>>>>>> Would switching to RELP be as simple as replacing that last line with:
>>>>>>
>>>>>> *.* :omrelp:logserver:2345;******SiteIDForwardFormat
>>>>>>
>>>>>>
>>>>>>
>>>>>> And adding a RELP listener on the new port on the server?
>>>>>>
>>>>>>
>>>>>> other than the need to load the relp module, it should be about that
>>>>> simple.
>>>>>
>>>>> David Lang
>>>>>
>>>>> ______________________________******_________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> >
>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> >
>>>>>
>>>>>>
>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> >
>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>> >
>>>>>
>>>>>>
>>>>>>
>>>>> ______________________________****_________________
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> >
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>> >
>>>>
>>>> ______________________________****_________________
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 9, 2011, 10:22 PM

Post #14 of 17 (661 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Hi,

Btw i tried the RELP installation too in Rsyslog 5.8.6. I was configure,
make and make install from the tar file. But
When i start the server in debug mode i get the following error:

dlopen: /usr/local/lib/rsyslog/imrelp.so: cannot open shared object file:
No such file or directory

Is there any additional steps i need to take for installing RELP and make
it work. Here is my Server config for RELP:

$ModLoad imtcp
$ModLoad immark # provides --MARK-- message capability
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog # kernel logging (formerly provided by rklogd)
$ModLoad imudp # provides UDP syslog reception
$ModLoad MySQL
$ModLoad imrelp

#InputTCPMaxSessions 500
$InputTCPServerRun 10514

$InputRELPServerRun 10514

$SystemLogSocketFlowControl on
$SystemLogSocketName syslog

$EscapeControlCharactersOnReceive off

Thanks,
Velu

On Sat, Dec 10, 2011 at 12:56 AM, Velu S <velu1980 [at] gmail> wrote:

> Thank you David again for the help. Here is what i tried. The messages are
> still not buffered when the central server is down.
> I disabled all the action queue messages. I restarted both the sender and
> receiver.
> When both servers are up the messages gets logged in fine, but when the
> central server is down,
> the messages don't get logged in. I tried by setting the
> following parameters on, but still couldn't succed:
>
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> $ActionFileEnableSync on
> $ActionSendResendLastMsgOnReconnect on
>
>
>
> Thanks,
> Anand
>
>
> On Sat, Dec 10, 2011 at 12:23 AM, <david [at] lang> wrote:
>
>> On Fri, 9 Dec 2011, Velu S wrote:
>>
>> Thank you David for the reply. Sorry for not explaining clearly in my
>>> previous emails.
>>> I use disk assisted mode and TCP both on the client and the server. When
>>> all the servers are up everything works fine.
>>> When the server goes down the message send from the client are not logged
>>> in. Thats expected. But these messages are lost even when the server
>>> comes
>>> back online. I will get the entire log of the client (When the center
>>> goes
>>> down and up).
>>>
>>
>> Ok, then the first thing that I would do to troubleshoot this is to
>> simplify things.
>>
>> try removing all of the actionqueue configuraiton entries and see what
>> happens. you will only be able to store a relativly small number of
>> messages before things block, but if you stop the server, wait until you
>> get the suspend message, and then startup the server again, you should see
>> the messages resume. If not, there is some sort of retry option that will
>> need to be set
>>
>> after we have the suspend/resume function working, then the next thing to
>> do is configure the main queue to be disk assisted (even if you want a
>> separate action queue eventually, changing the main queue is changing fewer
>> things). make sure that when there are enough messages queued that without
>> the disk assist things would block, you instead have files created in your
>> work directory and that after the server is restarted, it gets the messages.
>>
>> only at this point (where we know that suspend/resume works, and we know
>> that a disk assisted queue works), would I then change the main queue back
>> to the default and create an action queue.
>>
>> David Lang
>>
>> I haven't tried installing RELP yet. Will try installing it and test the
>>> same scenario again. Thanks again for your timely help and time. Really
>>> appreciate it.
>>>
>>> Thanks,
>>> Velu
>>>
>>> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>>>
>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>
>>>> Thank you David again for your quick reply. I installed Rsyslog version
>>>>
>>>>> 5.8.6 in both the sender and receiver. But My TCP messages are still
>>>>> not
>>>>> logged when the central server is down. Here is my log snippet:
>>>>>
>>>>>
>>>> OK, I don't understand the problem
>>>>
>>>> while the central server is down I would not expect the messages to be
>>>> logged there. but with a disk assist mode they should show up after the
>>>> server comes up.
>>>>
>>>> I'm getting ready to head out for the evening, but will check in again
>>>> in
>>>> 4-5 hours, so let's try to clarify what's happening.
>>>>
>>>> are you using RELP or just TCP?
>>>>
>>>> In the ideal world, what should happen is:
>>>>
>>>> everything up, logs are written locally and sent to the remote server.
>>>>
>>>> remote server goes down.
>>>>
>>>> with TCP, messages in flight are lost, with RELP nothing is lost
>>>>
>>>> you should see the transport be suspended, and it will retry, failing
>>>> each
>>>> time and eventually retry less frequently (to keep from wasting
>>>> resources
>>>> on the sending box)
>>>>
>>>> log messages should still be written locally, but queue up to be sent to
>>>> the remote server later.
>>>>
>>>> note that if you don't enable the disk assisted queue this will still
>>>> happen, but only up to the point where you fill memory.
>>>>
>>>> verify that logs are still being written locally, and look for files to
>>>> be
>>>> created in your working directory for the ones queued up to be sent
>>>> later.
>>>> you may need to generate a bunch of log messages to get to the point
>>>> where
>>>> they queue to disk.
>>>>
>>>>
>>>> then bring up the remote server. you should have logs start arriving
>>>> there
>>>> within a couple minutes, and then the older logs will start arriving.
>>>>
>>>> the debug logs around the time that the remote server comes up should be
>>>> interesting. I don't remember what the retry time is, but I seem to
>>>> remember that by default it includes some sort of logrithmic backoff
>>>> logic,
>>>> so if you are down for a long time, it may take a long time before it
>>>> decides to try again.
>>>>
>>>> David Lang
>>>>
>>>> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
>>>>
>>>>> reference count now 2
>>>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
>>>>> 9717.969857000:4266b940: earliest retry=1323469718
>>>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>>>> 9717.969877000:4266b940: tryDoAction: unexpected error code
>>>>> -2007[nElem 1,
>>>>> Commited UpTo 0], finalizing
>>>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1,
>>>>> nElem 1
>>>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>> susp,
>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>>>> 9717.969911000:4266b940: tryDoAction: unexpected error code
>>>>> -2123[nElem 1,
>>>>> Commited UpTo 0], finalizing
>>>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>> susp,
>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>>>> 9717.969936000:4266b940: Processing next rule
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Velu
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>>>
>>>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>>
>>>>>>
>>>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>>>
>>>>>>
>>>>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>>>>
>>>>>>> doesn't consider the message delivered until the receiving server
>>>>>>>> acknowledges it.
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm currently using TCP delivery because I couldn't make sense of
>>>>>>> RELP
>>>>>>> when I put it in place. Currently, I have clients doing:
>>>>>>>
>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>> $ActionQueueFileName fwd_queue
>>>>>>> $ActionQueueSaveOnShutdown on
>>>>>>> $ActionQueueType LinkedList
>>>>>>> $ActionResumeRetryCount -1
>>>>>>> *.* @@logserver:1234;******SiteIDForwardFormat
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Would switching to RELP be as simple as replacing that last line
>>>>>>> with:
>>>>>>>
>>>>>>> *.* :omrelp:logserver:2345;******SiteIDForwardFormat
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> And adding a RELP listener on the new port on the server?
>>>>>>>
>>>>>>>
>>>>>>> other than the need to load the relp module, it should be about that
>>>>>> simple.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> ______________________________******_________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>> >
>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>> >
>>>>>>
>>>>>>>
>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>> >
>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>> >
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ______________________________****_________________
>>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> >
>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>> >
>>>>>
>>>>> ______________________________****_________________
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> >
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>> >
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Dec 10, 2011, 5:08 AM

Post #15 of 17 (655 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

On Sat, 10 Dec 2011, Velu S wrote:

>
> Btw i tried the RELP installation too in Rsyslog 5.8.6. I was configure,
> make and make install from the tar file. But
> When i start the server in debug mode i get the following error:
>
> dlopen: /usr/local/lib/rsyslog/imrelp.so: cannot open shared object file:
> No such file or directory
>
> Is there any additional steps i need to take for installing RELP and make
> it work. Here is my Server config for RELP:

it sounds as if it's not compiled in. go back to the configure step and
look at it's final output to see if you have relp enabled. you may need to
do --enable-relp to make it compile.

David Lang

> $ModLoad imtcp
> $ModLoad immark # provides --MARK-- message capability
> $ModLoad imuxsock # provides support for local system logging (e.g. via
> logger command)
> $ModLoad imklog # kernel logging (formerly provided by rklogd)
> $ModLoad imudp # provides UDP syslog reception
> $ModLoad MySQL
> $ModLoad imrelp
>
>
> #InputTCPMaxSessions 500
> $InputTCPServerRun 10514
>
> $InputRELPServerRun 10514
>
> $SystemLogSocketFlowControl on
> $SystemLogSocketName syslog
>
> $EscapeControlCharactersOnReceive off
>
> Thanks,
> Velu
>
> On Sat, Dec 10, 2011 at 12:56 AM, Velu S <velu1980 [at] gmail> wrote:
>
>> Thank you David again for the help. Here is what i tried. The messages are
>> still not buffered when the central server is down.
>> I disabled all the action queue messages. I restarted both the sender and
>> receiver.
>> When both servers are up the messages gets logged in fine, but when the
>> central server is down,
>> the messages don't get logged in. I tried by setting the
>> following parameters on, but still couldn't succed:
>>
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> $ActionFileEnableSync on
>> $ActionSendResendLastMsgOnReconnect on
>>
>>
>>
>> Thanks,
>> Anand
>>
>>
>> On Sat, Dec 10, 2011 at 12:23 AM, <david [at] lang> wrote:
>>
>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>
>>> Thank you David for the reply. Sorry for not explaining clearly in my
>>>> previous emails.
>>>> I use disk assisted mode and TCP both on the client and the server. When
>>>> all the servers are up everything works fine.
>>>> When the server goes down the message send from the client are not logged
>>>> in. Thats expected. But these messages are lost even when the server
>>>> comes
>>>> back online. I will get the entire log of the client (When the center
>>>> goes
>>>> down and up).
>>>>
>>>
>>> Ok, then the first thing that I would do to troubleshoot this is to
>>> simplify things.
>>>
>>> try removing all of the actionqueue configuraiton entries and see what
>>> happens. you will only be able to store a relativly small number of
>>> messages before things block, but if you stop the server, wait until you
>>> get the suspend message, and then startup the server again, you should see
>>> the messages resume. If not, there is some sort of retry option that will
>>> need to be set
>>>
>>> after we have the suspend/resume function working, then the next thing to
>>> do is configure the main queue to be disk assisted (even if you want a
>>> separate action queue eventually, changing the main queue is changing fewer
>>> things). make sure that when there are enough messages queued that without
>>> the disk assist things would block, you instead have files created in your
>>> work directory and that after the server is restarted, it gets the messages.
>>>
>>> only at this point (where we know that suspend/resume works, and we know
>>> that a disk assisted queue works), would I then change the main queue back
>>> to the default and create an action queue.
>>>
>>> David Lang
>>>
>>> I haven't tried installing RELP yet. Will try installing it and test the
>>>> same scenario again. Thanks again for your timely help and time. Really
>>>> appreciate it.
>>>>
>>>> Thanks,
>>>> Velu
>>>>
>>>> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>>>>
>>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>>
>>>>> Thank you David again for your quick reply. I installed Rsyslog version
>>>>>
>>>>>> 5.8.6 in both the sender and receiver. But My TCP messages are still
>>>>>> not
>>>>>> logged when the central server is down. Here is my log snippet:
>>>>>>
>>>>>>
>>>>> OK, I don't understand the problem
>>>>>
>>>>> while the central server is down I would not expect the messages to be
>>>>> logged there. but with a disk assist mode they should show up after the
>>>>> server comes up.
>>>>>
>>>>> I'm getting ready to head out for the evening, but will check in again
>>>>> in
>>>>> 4-5 hours, so let's try to clarify what's happening.
>>>>>
>>>>> are you using RELP or just TCP?
>>>>>
>>>>> In the ideal world, what should happen is:
>>>>>
>>>>> everything up, logs are written locally and sent to the remote server.
>>>>>
>>>>> remote server goes down.
>>>>>
>>>>> with TCP, messages in flight are lost, with RELP nothing is lost
>>>>>
>>>>> you should see the transport be suspended, and it will retry, failing
>>>>> each
>>>>> time and eventually retry less frequently (to keep from wasting
>>>>> resources
>>>>> on the sending box)
>>>>>
>>>>> log messages should still be written locally, but queue up to be sent to
>>>>> the remote server later.
>>>>>
>>>>> note that if you don't enable the disk assisted queue this will still
>>>>> happen, but only up to the point where you fill memory.
>>>>>
>>>>> verify that logs are still being written locally, and look for files to
>>>>> be
>>>>> created in your working directory for the ones queued up to be sent
>>>>> later.
>>>>> you may need to generate a bunch of log messages to get to the point
>>>>> where
>>>>> they queue to disk.
>>>>>
>>>>>
>>>>> then bring up the remote server. you should have logs start arriving
>>>>> there
>>>>> within a couple minutes, and then the older logs will start arriving.
>>>>>
>>>>> the debug logs around the time that the remote server comes up should be
>>>>> interesting. I don't remember what the retry time is, but I seem to
>>>>> remember that by default it includes some sort of logrithmic backoff
>>>>> logic,
>>>>> so if you are down for a long time, it may take a long time before it
>>>>> decides to try again.
>>>>>
>>>>> David Lang
>>>>>
>>>>> 9717.969783000:4266b940: file nsd_ptcp.c released module 'lmnetstrms',
>>>>>
>>>>>> reference count now 2
>>>>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state: susp
>>>>>> 9717.969857000:4266b940: earliest retry=1323469718
>>>>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>>>>> 9717.969877000:4266b940: tryDoAction: unexpected error code
>>>>>> -2007[nElem 1,
>>>>>> Commited UpTo 0], finalizing
>>>>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1,
>>>>>> nElem 1
>>>>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>> susp,
>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>>>>> 9717.969911000:4266b940: tryDoAction: unexpected error code
>>>>>> -2123[nElem 1,
>>>>>> Commited UpTo 0], finalizing
>>>>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>> susp,
>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>>>>> 9717.969936000:4266b940: Processing next rule
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Velu
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>>>>
>>>>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>>>
>>>>>>>
>>>>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>>>>
>>>>>>>
>>>>>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>>>>>
>>>>>>>> doesn't consider the message delivered until the receiving server
>>>>>>>>> acknowledges it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm currently using TCP delivery because I couldn't make sense of
>>>>>>>> RELP
>>>>>>>> when I put it in place. Currently, I have clients doing:
>>>>>>>>
>>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>>> $ActionQueueFileName fwd_queue
>>>>>>>> $ActionQueueSaveOnShutdown on
>>>>>>>> $ActionQueueType LinkedList
>>>>>>>> $ActionResumeRetryCount -1
>>>>>>>> *.* @@logserver:1234;******SiteIDForwardFormat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Would switching to RELP be as simple as replacing that last line
>>>>>>>> with:
>>>>>>>>
>>>>>>>> *.* :omrelp:logserver:2345;******SiteIDForwardFormat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> And adding a RELP listener on the new port on the server?
>>>>>>>>
>>>>>>>>
>>>>>>>> other than the need to load the relp module, it should be about that
>>>>>>> simple.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> ______________________________******_________________
>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>>>
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>>
>>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________****_________________
>>>>>>>
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>
>>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>>>
>>>>>>
>>>>>> ______________________________****_________________
>>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>
>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>>
>>>>>
>>>>> ______________________________**_________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 13, 2011, 7:47 AM

Post #16 of 17 (678 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Hi David,

Sorry for the late reply. I was travelling and struck up with other stuffs.
I was running my rsyslog in debug mode both in the client and the server.
Here is the log snippet when the client communicates tries to communicate
to the server, when the central server is down:

7073.730513000:429eb940: Called action 0xbbe2870 (complex case), logging to
builtin-file
7073.730534000:429eb940: action 0xbbe2870: filterOK:0 state:0
execWhenPrev:0 prevWasSusp:0
7073.730542000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
7073.730550000:429eb940: Processing next rule
7073.730561000:429eb940: Filter: check for property 'msg' (value '
#eNpVUV2P0zAQ/CuWn0C65LxO0pbwBC0clWh10hXxGLnOtrVIYuOPUnG6/37rFiGQn7wzuzM7+8w7leLpW0C/7nkLAHe8i2ZE3vJdwjsGFVuhZlIAMPGureZtU7FPTztOPI+jjbh2xIUSpCxhUcoM0LQzDbwC4oos5uUCbj3B2Sng0vakIYX4Qzc6S44pGJ1pI8aTJUNcDfs0rjAqM4QMaDtFvERC3PmSC2ZyKQbePt+oncldIKv8suKlSK4IaT9Zqm+77cd197R66EDWNTRSNHMhBIhZU8kZyL/8Y5gKk/3PqhJmtB3UpazzPI8/E4bovD2YIXuO9Ctcdp1okUIdccr2Nva3GQZ135SCvflupt7+Cmy7Y00J75k/t1R/yx5Q/7D3FG72AOyz8Xiwl9zzj/OgvdnTZG1pTf6oPAmoYUlBeDt8MccTcXsVVXf0NrlwDW2gmtIaXSz0SfmA2VKKh2LBXyi09N+90Xvrbwfhm83ycfXhK4VS5XT75FU0duJtXb+8AjuBp00=$')
contains 'pvx': FALSE
7073.730573000:429eb940: Processing next action
7073.730582000:429eb940: Called action 0xbbe40e0 (complex case), logging to
builtin-fwd
7073.730590000:429eb940: action 0xbbe40e0: filterOK:0 state:0
execWhenPrev:0 prevWasSusp:0
7073.730598000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
7073.730606000:429eb940: Processing next rule
7073.730615000:429eb940: Filter: check for property 'syslogtag' (value
'[ReportingSysLog]') startswith 'pvx': FALSE
7073.730626000:429eb940: Processing next action
7073.730634000:429eb940: Called action 0xbbe8600 (complex case), logging to
builtin-fwd
7073.730642000:429eb940: action 0xbbe8600: filterOK:0 state:0
execWhenPrev:0 prevWasSusp:0
7073.730650000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
7073.730657000:429eb940: Processing next rule
7073.730666000:429eb940: Filter: check for property 'syslogtag' (value
'[ReportingSysLog]') isequal '[ReportingSysLog]': TRUE
7073.730676000:429eb940: Processing next action
7073.730684000:429eb940: Called action 0xbbe8d50 (complex case), logging to
builtin-fwd
7073.730692000:429eb940: action 0xbbe8d50: filterOK:1 state:0
execWhenPrev:0 prevWasSusp:0
7073.730702000:429eb940: Called action(complex case), logging to builtin-fwd
7073.730714000:429eb940: XXXXX: tryDoAction 0xbbe8d50, pnElem 1, nElem 1
7073.730722000:429eb940: Action 0xbbe8d50 transitioned to state: rtry
7073.730731000:429eb940: 10.122.87.75
7073.730747000:429eb940: caller requested object 'nsd_ptcp', not found
(iRet -3003)
7073.730757000:429eb940: Requested to load module 'lmnsd_ptcp'
7073.730769000:429eb940: loading module
'/usr/local/lib/rsyslog/lmnsd_ptcp.so'
7073.730927000:429eb940: source file nsd_ptcp.c requested reference for
module 'lmnetstrms', reference count now 3
7073.730946000:429eb940: module of type 2 being loaded.
7073.730955000:429eb940: entry point 'isCompatibleWithFeature' not present
in module
7073.730964000:429eb940: source file netstrms.c requested reference for
module 'lmnsd_ptcp', reference count now 1
7073.731126000:429eb940: file netstrms.c released module 'lmnsd_ptcp',
reference count now 0
7073.731136000:429eb940: module 'lmnsd_ptcp' has zero reference count,
unloading...
7073.731144000:429eb940: Unloading module lmnsd_ptcp
7073.731155000:429eb940: file nsd_ptcp.c released module 'lmnetstrms',
reference count now 2
7073.731205000:429eb940: Action 0xbbe8d50 transitioned to state: susp
7073.731214000:429eb940: *earliest retry=1323787074*
7073.731222000:429eb940: action 0xbbe8d50 call returned -2007
7073.731231000:429eb940: tryDoAction: unexpected error code -2007[nElem 1,
Commited UpTo 0], finalizing
7073.731239000:429eb940: XXXXX: tryDoAction 0xbbe8d50, pnElem 1, nElem 1
7073.731249000:429eb940: actionTryResume: action 0xbbe8d50 state: susp,
next retry (if applicable): 1323787074 [now 1323787073]
7073.731257000:429eb940: action 0xbbe8d50 call returned -2123
7073.731265000:429eb940: tryDoAction: unexpected error code -2123[nElem 1,
Commited UpTo 0], finalizing
7073.731274000:429eb940: actionTryResume: action 0xbbe8d50 state: susp,
next retry (if applicable): 1323787074 [now 1323787073]

I noticed the earliest retry=1323787074, but i have set it
to $ActionResumeInterval 1. Am not sure if earliest retry is the number of
seconds rsyslog takes before sending the message to the server. Please let
me know if i need to set any other parameter. Thanks in advance. Any help
is highly appreciated.

Thanks,
Anand

On Sat, Dec 10, 2011 at 8:08 AM, <david [at] lang> wrote:

> On Sat, 10 Dec 2011, Velu S wrote:
>
>
>> Btw i tried the RELP installation too in Rsyslog 5.8.6. I was configure,
>> make and make install from the tar file. But
>> When i start the server in debug mode i get the following error:
>>
>> dlopen: /usr/local/lib/rsyslog/imrelp.**so: cannot open shared object
>> file:
>> No such file or directory
>>
>> Is there any additional steps i need to take for installing RELP and make
>> it work. Here is my Server config for RELP:
>>
>
> it sounds as if it's not compiled in. go back to the configure step and
> look at it's final output to see if you have relp enabled. you may need to
> do --enable-relp to make it compile.
>
> David Lang
>
> $ModLoad imtcp
>> $ModLoad immark # provides --MARK-- message capability
>> $ModLoad imuxsock # provides support for local system logging (e.g. via
>> logger command)
>> $ModLoad imklog # kernel logging (formerly provided by rklogd)
>> $ModLoad imudp # provides UDP syslog reception
>> $ModLoad MySQL
>> $ModLoad imrelp
>>
>>
>> #InputTCPMaxSessions 500
>> $InputTCPServerRun 10514
>>
>> $InputRELPServerRun 10514
>>
>> $SystemLogSocketFlowControl on
>> $SystemLogSocketName syslog
>>
>> $**EscapeControlCharactersOnRecei**ve off
>>
>> Thanks,
>> Velu
>>
>> On Sat, Dec 10, 2011 at 12:56 AM, Velu S <velu1980 [at] gmail> wrote:
>>
>> Thank you David again for the help. Here is what i tried. The messages
>>> are
>>> still not buffered when the central server is down.
>>> I disabled all the action queue messages. I restarted both the sender and
>>> receiver.
>>> When both servers are up the messages gets logged in fine, but when the
>>> central server is down,
>>> the messages don't get logged in. I tried by setting the
>>> following parameters on, but still couldn't succed:
>>>
>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>> $ActionFileEnableSync on
>>> $**ActionSendResendLastMsgOnRecon**nect on
>>>
>>>
>>>
>>> Thanks,
>>> Anand
>>>
>>>
>>> On Sat, Dec 10, 2011 at 12:23 AM, <david [at] lang> wrote:
>>>
>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>
>>>> Thank you David for the reply. Sorry for not explaining clearly in my
>>>>
>>>>> previous emails.
>>>>> I use disk assisted mode and TCP both on the client and the server.
>>>>> When
>>>>> all the servers are up everything works fine.
>>>>> When the server goes down the message send from the client are not
>>>>> logged
>>>>> in. Thats expected. But these messages are lost even when the server
>>>>> comes
>>>>> back online. I will get the entire log of the client (When the center
>>>>> goes
>>>>> down and up).
>>>>>
>>>>>
>>>> Ok, then the first thing that I would do to troubleshoot this is to
>>>> simplify things.
>>>>
>>>> try removing all of the actionqueue configuraiton entries and see what
>>>> happens. you will only be able to store a relativly small number of
>>>> messages before things block, but if you stop the server, wait until you
>>>> get the suspend message, and then startup the server again, you should
>>>> see
>>>> the messages resume. If not, there is some sort of retry option that
>>>> will
>>>> need to be set
>>>>
>>>> after we have the suspend/resume function working, then the next thing
>>>> to
>>>> do is configure the main queue to be disk assisted (even if you want a
>>>> separate action queue eventually, changing the main queue is changing
>>>> fewer
>>>> things). make sure that when there are enough messages queued that
>>>> without
>>>> the disk assist things would block, you instead have files created in
>>>> your
>>>> work directory and that after the server is restarted, it gets the
>>>> messages.
>>>>
>>>> only at this point (where we know that suspend/resume works, and we know
>>>> that a disk assisted queue works), would I then change the main queue
>>>> back
>>>> to the default and create an action queue.
>>>>
>>>> David Lang
>>>>
>>>> I haven't tried installing RELP yet. Will try installing it and test
>>>> the
>>>>
>>>>> same scenario again. Thanks again for your timely help and time. Really
>>>>> appreciate it.
>>>>>
>>>>> Thanks,
>>>>> Velu
>>>>>
>>>>> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>>>>>
>>>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>>
>>>>>>
>>>>>> Thank you David again for your quick reply. I installed Rsyslog
>>>>>> version
>>>>>>
>>>>>> 5.8.6 in both the sender and receiver. But My TCP messages are still
>>>>>>> not
>>>>>>> logged when the central server is down. Here is my log snippet:
>>>>>>>
>>>>>>>
>>>>>>> OK, I don't understand the problem
>>>>>>
>>>>>> while the central server is down I would not expect the messages to be
>>>>>> logged there. but with a disk assist mode they should show up after
>>>>>> the
>>>>>> server comes up.
>>>>>>
>>>>>> I'm getting ready to head out for the evening, but will check in again
>>>>>> in
>>>>>> 4-5 hours, so let's try to clarify what's happening.
>>>>>>
>>>>>> are you using RELP or just TCP?
>>>>>>
>>>>>> In the ideal world, what should happen is:
>>>>>>
>>>>>> everything up, logs are written locally and sent to the remote server.
>>>>>>
>>>>>> remote server goes down.
>>>>>>
>>>>>> with TCP, messages in flight are lost, with RELP nothing is lost
>>>>>>
>>>>>> you should see the transport be suspended, and it will retry, failing
>>>>>> each
>>>>>> time and eventually retry less frequently (to keep from wasting
>>>>>> resources
>>>>>> on the sending box)
>>>>>>
>>>>>> log messages should still be written locally, but queue up to be sent
>>>>>> to
>>>>>> the remote server later.
>>>>>>
>>>>>> note that if you don't enable the disk assisted queue this will still
>>>>>> happen, but only up to the point where you fill memory.
>>>>>>
>>>>>> verify that logs are still being written locally, and look for files
>>>>>> to
>>>>>> be
>>>>>> created in your working directory for the ones queued up to be sent
>>>>>> later.
>>>>>> you may need to generate a bunch of log messages to get to the point
>>>>>> where
>>>>>> they queue to disk.
>>>>>>
>>>>>>
>>>>>> then bring up the remote server. you should have logs start arriving
>>>>>> there
>>>>>> within a couple minutes, and then the older logs will start arriving.
>>>>>>
>>>>>> the debug logs around the time that the remote server comes up should
>>>>>> be
>>>>>> interesting. I don't remember what the retry time is, but I seem to
>>>>>> remember that by default it includes some sort of logrithmic backoff
>>>>>> logic,
>>>>>> so if you are down for a long time, it may take a long time before it
>>>>>> decides to try again.
>>>>>>
>>>>>> David Lang
>>>>>>
>>>>>> 9717.969783000:4266b940: file nsd_ptcp.c released module
>>>>>> 'lmnetstrms',
>>>>>>
>>>>>> reference count now 2
>>>>>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state:
>>>>>>> susp
>>>>>>> 9717.969857000:4266b940: earliest retry=1323469718
>>>>>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>>>>>> 9717.969877000:4266b940: tryDoAction: unexpected error code
>>>>>>> -2007[nElem 1,
>>>>>>> Commited UpTo 0], finalizing
>>>>>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1,
>>>>>>> nElem 1
>>>>>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>>> susp,
>>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>>>>>> 9717.969911000:4266b940: tryDoAction: unexpected error code
>>>>>>> -2123[nElem 1,
>>>>>>> Commited UpTo 0], finalizing
>>>>>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>>> susp,
>>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>>>>>> 9717.969936000:4266b940: Processing next rule
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Velu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>>>>>
>>>>>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>>>>
>>>>>>>
>>>>>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> To avoid this loss, you will need to use RELP instead of TCP. RELP
>>>>>>>>>
>>>>>>>>> doesn't consider the message delivered until the receiving server
>>>>>>>>>
>>>>>>>>>> acknowledges it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm currently using TCP delivery because I couldn't make sense of
>>>>>>>>>>
>>>>>>>>> RELP
>>>>>>>>> when I put it in place. Currently, I have clients doing:
>>>>>>>>>
>>>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>>>> $ActionQueueFileName fwd_queue
>>>>>>>>> $ActionQueueSaveOnShutdown on
>>>>>>>>> $ActionQueueType LinkedList
>>>>>>>>> $ActionResumeRetryCount -1
>>>>>>>>> *.* @@logserver:1234;********SiteIDForwardFormat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Would switching to RELP be as simple as replacing that last line
>>>>>>>>> with:
>>>>>>>>>
>>>>>>>>> *.* :omrelp:logserver:2345;********SiteIDForwardFormat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And adding a RELP listener on the new port on the server?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> other than the need to load the relp module, it should be about
>>>>>>>>> that
>>>>>>>>>
>>>>>>>> simple.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>> ______________________________********_________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/********mailman/listinfo/rsyslog<http://lists.adiscon.net/******mailman/listinfo/rsyslog>
>>>>>>>> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>>> >
>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog>
>>>>>>>> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog>
>>>>>>>> <htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>>> >
>>>>>>>> <htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>> http://www.rsyslog.com/********professional-services/<http://www.rsyslog.com/******professional-services/>
>>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>> <http://**www.rsyslog.com/****professional-**services/<http://www.rsyslog.com/**professional-**services/>
>>>>>>>> <http:**//www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>>> >
>>>>>>>> <http:**//www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ______________________________******_________________
>>>>>>>>
>>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>> >
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>> >
>>>>>>>
>>>>>>>>
>>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>> >
>>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>>> >
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________******_________________
>>>>>>>
>>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>> >
>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>> >
>>>>>>
>>>>>>>
>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>> >
>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>> >
>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ______________________________****_________________
>>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> >
>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>> >
>>>>>
>>>>> ______________________________****_________________
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> >
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>> >
>>>>
>>>>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

velu1980 at gmail

Dec 13, 2011, 2:55 PM

Post #17 of 17 (683 views)
Permalink

Re: Issues with configuring rsyslog with TCP configuration. Please help [In reply to]

Hi David,

I tried installing RELP:

I was able to install and build it without any errors.I am trying to log
all my messages
to a central log server. Am trying to use RELP for better reliability.

When i start my Centralised server in debug mode i get the following error:

rsyslogd: could not load module '/usr/local/lib/rsyslog/imrelp.so',
dlopen: /usr/local/lib/rsyslog/imrelp.so: undefined symbol:
relpEngineAddListner
[try http://www.rsyslog.com/e/2066 ]
rsyslogd: the last error occured in /etc/rsyslog.conf, line 7:"$ModLoad
imrelp"
rsyslogd: invalid or yet-unknown config file command - have you forgotten
to load a module? [tryhttp://www.rsyslog.com/e/3003 ]
rsyslogd: the last error occured in /etc/rsyslog.conf, line
12:"$InputRELPServerRun 10514"
rsyslogd: CONFIG ERROR: could not interpret master config file
'/etc/rsyslog.conf'. [tryhttp://www.rsyslog.com/e/2124 ]

The imrelp.so does exist in the specified directory with correct
permissions.
Did anyone had similar issue. Any help will be highly appreciated

Thanks,
Velu

On Tue, Dec 13, 2011 at 10:47 AM, Velu S <velu1980 [at] gmail> wrote:

> Hi David,
>
> Sorry for the late reply. I was travelling and struck up with other
> stuffs. I was running my rsyslog in debug mode both in the client and the
> server. Here is the log snippet when the client communicates tries to
> communicate to the server, when the central server is down:
>
> 7073.730513000:429eb940: Called action 0xbbe2870 (complex case), logging
> to builtin-file
> 7073.730534000:429eb940: action 0xbbe2870: filterOK:0 state:0
> execWhenPrev:0 prevWasSusp:0
> 7073.730542000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
> 7073.730550000:429eb940: Processing next rule
> 7073.730561000:429eb940: Filter: check for property 'msg' (value '
> #eNpVUV2P0zAQ/CuWn0C65LxO0pbwBC0clWh10hXxGLnOtrVIYuOPUnG6/37rFiGQn7wzuzM7+8w7leLpW0C/7nkLAHe8i2ZE3vJdwjsGFVuhZlIAMPGureZtU7FPTztOPI+jjbh2xIUSpCxhUcoM0LQzDbwC4oos5uUCbj3B2Sng0vakIYX4Qzc6S44pGJ1pI8aTJUNcDfs0rjAqM4QMaDtFvERC3PmSC2ZyKQbePt+oncldIKv8suKlSK4IaT9Zqm+77cd197R66EDWNTRSNHMhBIhZU8kZyL/8Y5gKk/3PqhJmtB3UpazzPI8/E4bovD2YIXuO9Ctcdp1okUIdccr2Nva3GQZ135SCvflupt7+Cmy7Y00J75k/t1R/yx5Q/7D3FG72AOyz8Xiwl9zzj/OgvdnTZG1pTf6oPAmoYUlBeDt8MccTcXsVVXf0NrlwDW2gmtIaXSz0SfmA2VKKh2LBXyi09N+90Xvrbwfhm83ycfXhK4VS5XT75FU0duJtXb+8AjuBp00=$')
> contains 'pvx': FALSE
> 7073.730573000:429eb940: Processing next action
> 7073.730582000:429eb940: Called action 0xbbe40e0 (complex case), logging
> to builtin-fwd
> 7073.730590000:429eb940: action 0xbbe40e0: filterOK:0 state:0
> execWhenPrev:0 prevWasSusp:0
> 7073.730598000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
> 7073.730606000:429eb940: Processing next rule
> 7073.730615000:429eb940: Filter: check for property 'syslogtag' (value
> '[ReportingSysLog]') startswith 'pvx': FALSE
> 7073.730626000:429eb940: Processing next action
> 7073.730634000:429eb940: Called action 0xbbe8600 (complex case), logging
> to builtin-fwd
> 7073.730642000:429eb940: action 0xbbe8600: filterOK:0 state:0
> execWhenPrev:0 prevWasSusp:0
> 7073.730650000:429eb940: ruleset: get iRet 0 from rule.ProcessMsg()
> 7073.730657000:429eb940: Processing next rule
> 7073.730666000:429eb940: Filter: check for property 'syslogtag' (value
> '[ReportingSysLog]') isequal '[ReportingSysLog]': TRUE
> 7073.730676000:429eb940: Processing next action
> 7073.730684000:429eb940: Called action 0xbbe8d50 (complex case), logging
> to builtin-fwd
> 7073.730692000:429eb940: action 0xbbe8d50: filterOK:1 state:0
> execWhenPrev:0 prevWasSusp:0
> 7073.730702000:429eb940: Called action(complex case), logging to
> builtin-fwd
> 7073.730714000:429eb940: XXXXX: tryDoAction 0xbbe8d50, pnElem 1, nElem 1
> 7073.730722000:429eb940: Action 0xbbe8d50 transitioned to state: rtry
> 7073.730731000:429eb940: 10.122.87.75
> 7073.730747000:429eb940: caller requested object 'nsd_ptcp', not found
> (iRet -3003)
> 7073.730757000:429eb940: Requested to load module 'lmnsd_ptcp'
> 7073.730769000:429eb940: loading module
> '/usr/local/lib/rsyslog/lmnsd_ptcp.so'
> 7073.730927000:429eb940: source file nsd_ptcp.c requested reference for
> module 'lmnetstrms', reference count now 3
> 7073.730946000:429eb940: module of type 2 being loaded.
> 7073.730955000:429eb940: entry point 'isCompatibleWithFeature' not present
> in module
> 7073.730964000:429eb940: source file netstrms.c requested reference for
> module 'lmnsd_ptcp', reference count now 1
> 7073.731126000:429eb940: file netstrms.c released module 'lmnsd_ptcp',
> reference count now 0
> 7073.731136000:429eb940: module 'lmnsd_ptcp' has zero reference count,
> unloading...
> 7073.731144000:429eb940: Unloading module lmnsd_ptcp
> 7073.731155000:429eb940: file nsd_ptcp.c released module 'lmnetstrms',
> reference count now 2
> 7073.731205000:429eb940: Action 0xbbe8d50 transitioned to state: susp
> 7073.731214000:429eb940: *earliest retry=1323787074*
> 7073.731222000:429eb940: action 0xbbe8d50 call returned -2007
> 7073.731231000:429eb940: tryDoAction: unexpected error code -2007[nElem 1,
> Commited UpTo 0], finalizing
> 7073.731239000:429eb940: XXXXX: tryDoAction 0xbbe8d50, pnElem 1, nElem 1
> 7073.731249000:429eb940: actionTryResume: action 0xbbe8d50 state: susp,
> next retry (if applicable): 1323787074 [now 1323787073]
> 7073.731257000:429eb940: action 0xbbe8d50 call returned -2123
> 7073.731265000:429eb940: tryDoAction: unexpected error code -2123[nElem 1,
> Commited UpTo 0], finalizing
> 7073.731274000:429eb940: actionTryResume: action 0xbbe8d50 state: susp,
> next retry (if applicable): 1323787074 [now 1323787073]
>
> I noticed the earliest retry=1323787074, but i have set it
> to $ActionResumeInterval 1. Am not sure if earliest retry is the number of
> seconds rsyslog takes before sending the message to the server. Please let
> me know if i need to set any other parameter. Thanks in advance. Any help
> is highly appreciated.
>
> Thanks,
> Anand
>
>
>
>
>
> On Sat, Dec 10, 2011 at 8:08 AM, <david [at] lang> wrote:
>
>> On Sat, 10 Dec 2011, Velu S wrote:
>>
>>
>>> Btw i tried the RELP installation too in Rsyslog 5.8.6. I was configure,
>>> make and make install from the tar file. But
>>> When i start the server in debug mode i get the following error:
>>>
>>> dlopen: /usr/local/lib/rsyslog/imrelp.**so: cannot open shared object
>>> file:
>>> No such file or directory
>>>
>>> Is there any additional steps i need to take for installing RELP and make
>>> it work. Here is my Server config for RELP:
>>>
>>
>> it sounds as if it's not compiled in. go back to the configure step and
>> look at it's final output to see if you have relp enabled. you may need to
>> do --enable-relp to make it compile.
>>
>> David Lang
>>
>> $ModLoad imtcp
>>> $ModLoad immark # provides --MARK-- message capability
>>> $ModLoad imuxsock # provides support for local system logging (e.g. via
>>> logger command)
>>> $ModLoad imklog # kernel logging (formerly provided by rklogd)
>>> $ModLoad imudp # provides UDP syslog reception
>>> $ModLoad MySQL
>>> $ModLoad imrelp
>>>
>>>
>>> #InputTCPMaxSessions 500
>>> $InputTCPServerRun 10514
>>>
>>> $InputRELPServerRun 10514
>>>
>>> $SystemLogSocketFlowControl on
>>> $SystemLogSocketName syslog
>>>
>>> $**EscapeControlCharactersOnRecei**ve off
>>>
>>> Thanks,
>>> Velu
>>>
>>> On Sat, Dec 10, 2011 at 12:56 AM, Velu S <velu1980 [at] gmail> wrote:
>>>
>>> Thank you David again for the help. Here is what i tried. The messages
>>>> are
>>>> still not buffered when the central server is down.
>>>> I disabled all the action queue messages. I restarted both the sender
>>>> and
>>>> receiver.
>>>> When both servers are up the messages gets logged in fine, but when the
>>>> central server is down,
>>>> the messages don't get logged in. I tried by setting the
>>>> following parameters on, but still couldn't succed:
>>>>
>>>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>>>> $ActionFileEnableSync on
>>>> $**ActionSendResendLastMsgOnRecon**nect on
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Anand
>>>>
>>>>
>>>> On Sat, Dec 10, 2011 at 12:23 AM, <david [at] lang> wrote:
>>>>
>>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>>
>>>>> Thank you David for the reply. Sorry for not explaining clearly in my
>>>>>
>>>>>> previous emails.
>>>>>> I use disk assisted mode and TCP both on the client and the server.
>>>>>> When
>>>>>> all the servers are up everything works fine.
>>>>>> When the server goes down the message send from the client are not
>>>>>> logged
>>>>>> in. Thats expected. But these messages are lost even when the server
>>>>>> comes
>>>>>> back online. I will get the entire log of the client (When the center
>>>>>> goes
>>>>>> down and up).
>>>>>>
>>>>>>
>>>>> Ok, then the first thing that I would do to troubleshoot this is to
>>>>> simplify things.
>>>>>
>>>>> try removing all of the actionqueue configuraiton entries and see what
>>>>> happens. you will only be able to store a relativly small number of
>>>>> messages before things block, but if you stop the server, wait until
>>>>> you
>>>>> get the suspend message, and then startup the server again, you should
>>>>> see
>>>>> the messages resume. If not, there is some sort of retry option that
>>>>> will
>>>>> need to be set
>>>>>
>>>>> after we have the suspend/resume function working, then the next thing
>>>>> to
>>>>> do is configure the main queue to be disk assisted (even if you want a
>>>>> separate action queue eventually, changing the main queue is changing
>>>>> fewer
>>>>> things). make sure that when there are enough messages queued that
>>>>> without
>>>>> the disk assist things would block, you instead have files created in
>>>>> your
>>>>> work directory and that after the server is restarted, it gets the
>>>>> messages.
>>>>>
>>>>> only at this point (where we know that suspend/resume works, and we
>>>>> know
>>>>> that a disk assisted queue works), would I then change the main queue
>>>>> back
>>>>> to the default and create an action queue.
>>>>>
>>>>> David Lang
>>>>>
>>>>> I haven't tried installing RELP yet. Will try installing it and test
>>>>> the
>>>>>
>>>>>> same scenario again. Thanks again for your timely help and time.
>>>>>> Really
>>>>>> appreciate it.
>>>>>>
>>>>>> Thanks,
>>>>>> Velu
>>>>>>
>>>>>> On Fri, Dec 9, 2011 at 6:55 PM, <david [at] lang> wrote:
>>>>>>
>>>>>> On Fri, 9 Dec 2011, Velu S wrote:
>>>>>>
>>>>>>>
>>>>>>> Thank you David again for your quick reply. I installed Rsyslog
>>>>>>> version
>>>>>>>
>>>>>>> 5.8.6 in both the sender and receiver. But My TCP messages are still
>>>>>>>> not
>>>>>>>> logged when the central server is down. Here is my log snippet:
>>>>>>>>
>>>>>>>>
>>>>>>>> OK, I don't understand the problem
>>>>>>>
>>>>>>> while the central server is down I would not expect the messages to
>>>>>>> be
>>>>>>> logged there. but with a disk assist mode they should show up after
>>>>>>> the
>>>>>>> server comes up.
>>>>>>>
>>>>>>> I'm getting ready to head out for the evening, but will check in
>>>>>>> again
>>>>>>> in
>>>>>>> 4-5 hours, so let's try to clarify what's happening.
>>>>>>>
>>>>>>> are you using RELP or just TCP?
>>>>>>>
>>>>>>> In the ideal world, what should happen is:
>>>>>>>
>>>>>>> everything up, logs are written locally and sent to the remote
>>>>>>> server.
>>>>>>>
>>>>>>> remote server goes down.
>>>>>>>
>>>>>>> with TCP, messages in flight are lost, with RELP nothing is lost
>>>>>>>
>>>>>>> you should see the transport be suspended, and it will retry, failing
>>>>>>> each
>>>>>>> time and eventually retry less frequently (to keep from wasting
>>>>>>> resources
>>>>>>> on the sending box)
>>>>>>>
>>>>>>> log messages should still be written locally, but queue up to be
>>>>>>> sent to
>>>>>>> the remote server later.
>>>>>>>
>>>>>>> note that if you don't enable the disk assisted queue this will
>>>>>>> still
>>>>>>> happen, but only up to the point where you fill memory.
>>>>>>>
>>>>>>> verify that logs are still being written locally, and look for files
>>>>>>> to
>>>>>>> be
>>>>>>> created in your working directory for the ones queued up to be sent
>>>>>>> later.
>>>>>>> you may need to generate a bunch of log messages to get to the point
>>>>>>> where
>>>>>>> they queue to disk.
>>>>>>>
>>>>>>>
>>>>>>> then bring up the remote server. you should have logs start arriving
>>>>>>> there
>>>>>>> within a couple minutes, and then the older logs will start arriving.
>>>>>>>
>>>>>>> the debug logs around the time that the remote server comes up
>>>>>>> should be
>>>>>>> interesting. I don't remember what the retry time is, but I seem to
>>>>>>> remember that by default it includes some sort of logrithmic backoff
>>>>>>> logic,
>>>>>>> so if you are down for a long time, it may take a long time before it
>>>>>>> decides to try again.
>>>>>>>
>>>>>>> David Lang
>>>>>>>
>>>>>>> 9717.969783000:4266b940: file nsd_ptcp.c released module
>>>>>>> 'lmnetstrms',
>>>>>>>
>>>>>>> reference count now 2
>>>>>>>> 9717.969836000:4266b940: Action 0x1e25ae30 transitioned to state:
>>>>>>>> susp
>>>>>>>> 9717.969857000:4266b940: earliest retry=1323469718
>>>>>>>> 9717.969865000:4266b940: action 0x1e25ae30 call returned -2007
>>>>>>>> 9717.969877000:4266b940: tryDoAction: unexpected error code
>>>>>>>> -2007[nElem 1,
>>>>>>>> Commited UpTo 0], finalizing
>>>>>>>> 9717.969885000:4266b940: XXXXX: tryDoAction 0x1e25ae30, pnElem 1,
>>>>>>>> nElem 1
>>>>>>>> 9717.969894000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>>>> susp,
>>>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>>>> 9717.969903000:4266b940: action 0x1e25ae30 call returned -2123
>>>>>>>> 9717.969911000:4266b940: tryDoAction: unexpected error code
>>>>>>>> -2123[nElem 1,
>>>>>>>> Commited UpTo 0], finalizing
>>>>>>>> 9717.969919000:4266b940: actionTryResume: action 0x1e25ae30 state:
>>>>>>>> susp,
>>>>>>>> next retry (if applicable): 1323469718 [now 1323469717]
>>>>>>>> 9717.969928000:4266b940: ruleset: get iRet 0 from rule.ProcessMsg()
>>>>>>>> 9717.969936000:4266b940: Processing next rule
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Velu
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 9, 2011 at 5:03 PM, <david [at] lang> wrote:
>>>>>>>>
>>>>>>>> On Fri, 9 Dec 2011, Gregory K. Ruiz-Ade wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Dec 9, 2011, at 12:30 PM, david [at] lang wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> To avoid this loss, you will need to use RELP instead of TCP.
>>>>>>>>>> RELP
>>>>>>>>>>
>>>>>>>>>> doesn't consider the message delivered until the receiving server
>>>>>>>>>>
>>>>>>>>>>> acknowledges it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm currently using TCP delivery because I couldn't make sense
>>>>>>>>>>> of
>>>>>>>>>>>
>>>>>>>>>> RELP
>>>>>>>>>> when I put it in place. Currently, I have clients doing:
>>>>>>>>>>
>>>>>>>>>> $WorkDirectory /var/spool/rsyslog
>>>>>>>>>> $ActionQueueFileName fwd_queue
>>>>>>>>>> $ActionQueueSaveOnShutdown on
>>>>>>>>>> $ActionQueueType LinkedList
>>>>>>>>>> $ActionResumeRetryCount -1
>>>>>>>>>> *.* @@logserver:1234;********SiteIDForwardFormat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Would switching to RELP be as simple as replacing that last line
>>>>>>>>>> with:
>>>>>>>>>>
>>>>>>>>>> *.* :omrelp:logserver:2345;********SiteIDForwardFormat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And adding a RELP listener on the new port on the server?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> other than the need to load the relp module, it should be about
>>>>>>>>>> that
>>>>>>>>>>
>>>>>>>>> simple.
>>>>>>>>>
>>>>>>>>> David Lang
>>>>>>>>>
>>>>>>>>> ______________________________********_________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/********mailman/listinfo/rsyslog<http://lists.adiscon.net/******mailman/listinfo/rsyslog>
>>>>>>>>> <http**://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>>>> >
>>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog>
>>>>>>>>> <htt**p://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <http:**//lists.adiscon.net/****mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/**listinfo/rsyslog>
>>>>>>>>> <htt**p://lists.adiscon.net/mailman/****listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>>>> >
>>>>>>>>> <htt**p://lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> http://www.rsyslog.com/********professional-services/<http://www.rsyslog.com/******professional-services/>
>>>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>> <http://**www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <http://**www.rsyslog.com/****professional-**services/<http://www.rsyslog.com/**professional-**services/>
>>>>>>>>> <http:**//www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>>>> >
>>>>>>>>> <http:**//www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ______________________________******_________________
>>>>>>>>>
>>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>>> >
>>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>>> >
>>>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ______________________________******_________________
>>>>>>>>
>>>>>>>> rsyslog mailing list
>>>>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<http://lists.adiscon.net/****mailman/listinfo/rsyslog>
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>>> >
>>>>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/**listinfo/rsyslog>
>>>>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>> >
>>>>>>>
>>>>>>>>
>>>>>>>> http://www.rsyslog.com/******professional-services/<http://www.rsyslog.com/****professional-services/>
>>>>>>> <http://**www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>>> >
>>>>>>> <http://**www.rsyslog.com/**professional-**services/<http://www.rsyslog.com/professional-**services/>
>>>>>>> <http:**//www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>>>>> >
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ______________________________****_________________
>>>>>>>
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>> >
>>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>> >
>>>>>>
>>>>>> ______________________________****_________________
>>>>>>
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> >
>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>> >
>>>>>
>>>>>
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads