Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

templates

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


swillis at compete

Apr 24, 2012, 1:57 PM

Post #1 of 9 (962 views)
Permalink
templates

I'm having some trouble with templates and remote logging. When I add this to my conf:

$template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
:msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat
& ~

Then I created a test log message with:

$ logger -p daemon.info -t DataNode "hadoop hi"

The message gets to the remote host, but the output in /var/log/syslog on that host is:

Apr 24 15:59:22 172.29.208.56 hostname: {SWILLIS-E6320} rawmsg:{<30>Apr 24 15:59:22 DataNode: hadoop hi}

Somehow, "hostname:" in my template is replaced with "172.29.208.56 hostname: ". I then tried the following in my conf:

$template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
$template myFormat2,"%timereported% hostname{%hostname%} rawmsg:{%rawmsg%}\n"
:msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat
:msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat2
& ~

And the output in syslog is then:

Apr 24 16:38:19 172.29.208.56 hostname: {SWILLIS-E6320} rawmsg:{<30>Apr 24 16:38:19 DataNode: hadoop hi}
Apr 24 16:38:19 hostname{SWILLIS-E6320} rawmsg: {<30>Apr 24 16:38:19 DataNode: hadoop hi}

So there seems to be a bug with having "hostname:" in the format, that isn't brought out with just "hostname".

-Steven Willis
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 24, 2012, 2:01 PM

Post #2 of 9 (923 views)
Permalink
Re: templates [In reply to]

On Tue, 24 Apr 2012, Steven Willis wrote:

> I'm having some trouble with templates and remote logging. When I add this to my conf:
>
> $template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
> :msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat
> & ~
>
> Then I created a test log message with:
>
> $ logger -p daemon.info -t DataNode "hadoop hi"
>
> The message gets to the remote host, but the output in /var/log/syslog on that host is:
>
> Apr 24 15:59:22 172.29.208.56 hostname: {SWILLIS-E6320} rawmsg:{<30>Apr 24 15:59:22 DataNode: hadoop hi}
>
> Somehow, "hostname:" in my template is replaced with "172.29.208.56 hostname: ". I then tried the following in my conf:
>
> $template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
> $template myFormat2,"%timereported% hostname{%hostname%} rawmsg:{%rawmsg%}\n"
> :msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat
> :msg,contains,"hadoop" :omrelp:kaptain:20514;myFormat2
> & ~
>
> And the output in syslog is then:
>
> Apr 24 16:38:19 172.29.208.56 hostname: {SWILLIS-E6320} rawmsg:{<30>Apr 24 16:38:19 DataNode: hadoop hi}
> Apr 24 16:38:19 hostname{SWILLIS-E6320} rawmsg: {<30>Apr 24 16:38:19 DataNode: hadoop hi}
>
> So there seems to be a bug with having "hostname:" in the format, that isn't brought out with just "hostname".

I think you are forgetting about the format that the remote host is
configured to log to disk with.

If it is logging with the traditional file format, what you show as being
in the log would be correct.

remember that each instance of rsyslog has it's own templates for logs,
you can't just change the template on the first system and expect systems
downstream to use it.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


swillis at compete

Apr 24, 2012, 4:05 PM

Post #3 of 9 (924 views)
Permalink
Re: templates [In reply to]

> I think you are forgetting about the format that the remote host is
> configured to log to disk with.
>
> If it is logging with the traditional file format, what you show as
> being in the log would be correct.
>
> remember that each instance of rsyslog has it's own templates for logs,
> you can't just change the template on the first system and expect
> systems downstream to use it.
>
> David Lang

I don't think I fully understand. I get that there can be issues between each syslog reading in an writing out in a different format, but how does the addition of a single colon drastically change the output? These were my two formats:

$template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
$template myFormat2,"%timereported% hostname{%hostname%} rawmsg:{%rawmsg%}\n"

And the final output when using the second is exactly what I naively expected, but the first got garbled. When you use remote logging, is the remote rsyslog daemon always going to parse the input based on a strict format? And if so, what is that format, I've seen a couple mentions of different forward formats.

-Steve
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 24, 2012, 4:23 PM

Post #4 of 9 (924 views)
Permalink
Re: templates [In reply to]

On Tue, 24 Apr 2012, Steven Willis wrote:

>> I think you are forgetting about the format that the remote host is
>> configured to log to disk with.
>>
>> If it is logging with the traditional file format, what you show as
>> being in the log would be correct.
>>
>> remember that each instance of rsyslog has it's own templates for logs,
>> you can't just change the template on the first system and expect
>> systems downstream to use it.
>>
>> David Lang
>
> I don't think I fully understand. I get that there can be issues between each syslog reading in an writing out in a different format, but how does the addition of a single colon drastically change the output? These were my two formats:
>
> $template myFormat,"%timereported% hostname:{%hostname%} rawmsg:{%rawmsg%}\n"
> $template myFormat2,"%timereported% hostname{%hostname%} rawmsg:{%rawmsg%}\n"
>
> And the final output when using the second is exactly what I naively
> expected, but the first got garbled. When you use remote logging, is the
> remote rsyslog daemon always going to parse the input based on a strict
> format? And if so, what is that format, I've seen a couple mentions of
> different forward formats.

when the remote system receives the message, if you have not loaded a
specific parser module it assumes that what it's receiving is in the
traditional syslog format.

in your first example, hostname:{%hostname%} triggers something in the
parsing logic that says that this can't be a legitimate hostname, so it
puts the IP address of the sender in the hostname field instead.

In the second case, this heuristic doesn't get triggered, so it puts the
result of 'hostname{%hostname%}' in the hostname field, so it does what
you are expecting.

The short version is not to muck with the formatting until you arrive at
your final destination (unless you need to fix something that's broken)

I'll bet that if you use the default format on your sending machine, and
your custom format on the recieving machine, it will do what you want.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 24, 2012, 10:38 PM

Post #5 of 9 (920 views)
Permalink
Re: templates [In reply to]

> when the remote system receives the message, if you have not loaded a
> specific parser module it assumes that what it's receiving is in the
> traditional syslog format.
>
> in your first example, hostname:{%hostname%} triggers something in the
> parsing logic that says that this can't be a legitimate hostname, so it
> puts the IP address of the sender in the hostname field instead.

Jup, colon cannot appear in a hostname, so that field cannot be a hostname
(see relevant RFCs).

Also, the PRI part of the message is missing. For forwarding, you need to
start with

"<%PRI%>..."

(The PRI property may actually have a different name, I did not check this
against the property list).

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


swillis at compete

Apr 25, 2012, 8:11 AM

Post #6 of 9 (917 views)
Permalink
Re: templates [In reply to]

> when the remote system receives the message, if you have not loaded a
> specific parser module it assumes that what it's receiving is in the
> traditional syslog format.
>
> in your first example, hostname:{%hostname%} triggers something in the
> parsing logic that says that this can't be a legitimate hostname, so it
> puts the IP address of the sender in the hostname field instead.
>
> In the second case, this heuristic doesn't get triggered, so it puts
> the result of 'hostname{%hostname%}' in the hostname field, so it does
> what you are expecting.
>
> The short version is not to muck with the formatting until you arrive
> at your final destination (unless you need to fix something that's
> broken)
>
> I'll bet that if you use the default format on your sending machine,
> and your custom format on the recieving machine, it will do what you
> want.

Ahh, thanks David. I got it now. But, for the forwarding, should I use RSYSLOG_TraditionalForwardFormat, RSYSLOG_ForwardFormat, or RSYSLOG_SyslogProtocol23Format (is that even a forwarding format)? We're using rsyslog 4.6.2, and there's no chance that we'll be sending to any other syslogs or earlier version of rsyslog.

-Steve
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 25, 2012, 8:13 AM

Post #7 of 9 (920 views)
Permalink
Re: templates [In reply to]

On Wed, 25 Apr 2012, Steven Willis wrote:

>> when the remote system receives the message, if you have not loaded a
>> specific parser module it assumes that what it's receiving is in the
>> traditional syslog format.
>>
>> in your first example, hostname:{%hostname%} triggers something in the
>> parsing logic that says that this can't be a legitimate hostname, so it
>> puts the IP address of the sender in the hostname field instead.
>>
>> In the second case, this heuristic doesn't get triggered, so it puts
>> the result of 'hostname{%hostname%}' in the hostname field, so it does
>> what you are expecting.
>>
>> The short version is not to muck with the formatting until you arrive
>> at your final destination (unless you need to fix something that's
>> broken)
>>
>> I'll bet that if you use the default format on your sending machine,
>> and your custom format on the recieving machine, it will do what you
>> want.
>
> Ahh, thanks David. I got it now. But, for the forwarding, should I use RSYSLOG_TraditionalForwardFormat, RSYSLOG_ForwardFormat, or RSYSLOG_SyslogProtocol23Format (is that even a forwarding format)? We're using rsyslog 4.6.2, and there's no chance that we'll be sending to any other syslogs or earlier version of rsyslog.

the difference between Tradtional and other is the accracy of the
timestamp. If you are happy with the standard month day hh:mm:ss
timestamp, traditional works. If you want the timestamp to include the
timezone and sub-second accuracy, then use the RSYSLOG_ForwardFormat

David Lang

> -Steve
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


tmortensen at gmail

Apr 5, 2013, 9:27 AM

Post #8 of 9 (704 views)
Permalink
Re: Templates [In reply to]

You can make the template name anything you want. It doesn't even need to
begin with DYN.



On Fri, Apr 5, 2013 at 9:21 AM, Josh Bitto <jbitto [at] onlineschool> wrote:

> When creating a template is there a certain way to name them or could I
> make anything and use that in my if then statements.
>
> Example...
>
> $template DYNmessages,"/var/log/hosts/%HOSTNAME%/messages"
>
> if \
> $source != 'syslog.onlineschool.ca' \
> and \
> $syslogseverity <= '6' \
> and ( \
> $syslogfacility-text != 'mail' \
> and \
> $syslogfacility-text != 'authpriv' \
> and \
> $syslogfacility-text != 'cron' \
> ) \
> then ?DYNmessages
>
>
> where the $template is DYNmessages----could this be anything? Like
> DYNbumbleetuna
>
>
>
> Joshua Bitto
> Information Technologist
> KCC
>
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


jbitto at onlineschool

Apr 5, 2013, 9:27 AM

Post #9 of 9 (701 views)
Permalink
Re: Templates [In reply to]

Ok cool thanks!


-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Todd Mortensen
Sent: Friday, April 05, 2013 9:27 AM
To: rsyslog-users
Subject: Re: [rsyslog] Templates

You can make the template name anything you want. It doesn't even need to begin with DYN.



On Fri, Apr 5, 2013 at 9:21 AM, Josh Bitto <jbitto [at] onlineschool> wrote:

> When creating a template is there a certain way to name them or could
> I make anything and use that in my if then statements.
>
> Example...
>
> $template DYNmessages,"/var/log/hosts/%HOSTNAME%/messages"
>
> if \
> $source != 'syslog.onlineschool.ca' \
> and \
> $syslogseverity <= '6' \
> and ( \
> $syslogfacility-text != 'mail' \
> and \
> $syslogfacility-text != 'authpriv' \
> and \
> $syslogfacility-text != 'cron' \
> ) \
> then ?DYNmessages
>
>
> where the $template is DYNmessages----could this be anything? Like
> DYNbumbleetuna
>
>
>
> Joshua Bitto
> Information Technologist
> KCC
>
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.