Mailing List Archive: RSyslog: users

Modifying the hostname format that is logged (remove subdomains)

Index | Next | Previous | View Threaded

shaded4 at gmail

Jul 29, 2012, 7:11 AM

Post #1 of 3 (343 views)
Permalink

Modifying the hostname format that is logged (remove subdomains)

Hi,
We want to upgrade our syslog servers from an old sysklog version
to rsyslog 6.3.11.

I need to change the hostname format which is logged by rsyslog:

All of our servers have the format x.y.z.com .
If z is either domain1,domain2,domain3 then I need rsyslog to write just
'x.y' and nothing else. (i.e. strip off z.com)
I came up with this:
$template TemplateStrip,"%TIMESTAMP%
%HOSTNAME:R,ERE,1,FIELD:^(.*)\.(domain1\.com|domain2\.com|domain3\.com)$--end%
%syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate TemplateStrip
i.e. it uses a regular expression to strip off the domain.com part.

This appears to work correctly but is this the most efficient way to do it,
or are there better ways?
The Property Replacer page (
http://www.rsyslog.com/doc/property_replacer.html) suggests regular
expressions can be slower than other methods,
but I think the other methods on that page aren't really applicable in my
case?

Our old system receives about 10000 msgs/s so I'm hoping the new rsyslog
server can do the same.

P.S. I tried using rsyslog's -s commandline option, but couldn't get it to
strip subdomains like this -
I could only get it to strip either y.z.com or nothing.

Is the -s switch _supposed_ to be able to strip subdomains?
The rsyslogd man page (below) is a bit confusing: is it saying that it
CAN strip off just north.de (by doing something like "-s north.de:i
nfodrom.north.de" as it says),
or that it CAN'T do it?

-s domainlist
Specify a domainname that should be stripped off before logging.
Multiple domains may be specified using the colon ('':'') separa-
tor. Please be advised that no sub-domains may be specified but
only entire domains. For example if -s north.de is specified and
the host logging resolves to satu.infodrom.north.de no domain would
be cut, you will have to specify two domains like:
-s north.de:infodrom.north.de.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

Jul 29, 2012, 7:48 PM

Post #2 of 3 (318 views)
Permalink

Re: Modifying the hostname format that is logged (remove subdomains) [In reply to]

On Mon, 30 Jul 2012, shaded 4 wrote:

> Hi,
> We want to upgrade our syslog servers from an old sysklog version
> to rsyslog 6.3.11.
>
> I need to change the hostname format which is logged by rsyslog:
>
> All of our servers have the format x.y.z.com .
> If z is either domain1,domain2,domain3 then I need rsyslog to write just
> 'x.y' and nothing else. (i.e. strip off z.com)
> I came up with this:
> $template TemplateStrip,"%TIMESTAMP%
> %HOSTNAME:R,ERE,1,FIELD:^(.*)\.(domain1\.com|domain2\.com|domain3\.com)$--end%
> %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
> $ActionFileDefaultTemplate TemplateStrip
> i.e. it uses a regular expression to strip off the domain.com part.
>
> This appears to work correctly but is this the most efficient way to do it,
> or are there better ways?
> The Property Replacer page (
> http://www.rsyslog.com/doc/property_replacer.html) suggests regular
> expressions can be slower than other methods,
> but I think the other methods on that page aren't really applicable in my
> case?

regex is a slow way of doing things, but sometimes it's the only generic
way to do it.

You may want to look at the liblognorm option, it's FAR more efficient
than regex parsing (especially if you end up with lots of rules)

> Our old system receives about 10000 msgs/s so I'm hoping the new rsyslog
> server can do the same.

With equivalent configs, the new one should be faster. If you're having
the new one do a lot more, you may run into issues, but on a reasonably
modern system, 10K logs/sec should not be a problem.

I'm not familiar with the -s option, but it's a holdover from some really
old stuff, and I would not be surprised if nobody used it and as a result
it broke without anyone noticing. Rainer would need to comment on this.

David Lang

> P.S. I tried using rsyslog's -s commandline option, but couldn't get it to
> strip subdomains like this -
> I could only get it to strip either y.z.com or nothing.
>
> Is the -s switch _supposed_ to be able to strip subdomains?
> The rsyslogd man page (below) is a bit confusing: is it saying that it
> CAN strip off just north.de (by doing something like "-s north.de:i
> nfodrom.north.de" as it says),
> or that it CAN'T do it?
>
> -s domainlist
> Specify a domainname that should be stripped off before logging.
> Multiple domains may be specified using the colon ('':'') separa-
> tor. Please be advised that no sub-domains may be specified but
> only entire domains. For example if -s north.de is specified and
> the host logging resolves to satu.infodrom.north.de no domain would
> be cut, you will have to specify two domains like:
> -s north.de:infodrom.north.de.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

rgerhards at hq

Jul 30, 2012, 12:20 AM

Post #3 of 3 (328 views)
Permalink

Re: Modifying the hostname format that is logged (remove subdomains) [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of david [at] lang
> Sent: Monday, July 30, 2012 4:48 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Modifying the hostname format that is logged
> (remove subdomains)
>
> On Mon, 30 Jul 2012, shaded 4 wrote:
>
> > Hi,
> > We want to upgrade our syslog servers from an old sysklog version
> > to rsyslog 6.3.11.
> >
> > I need to change the hostname format which is logged by rsyslog:
> >
> > All of our servers have the format x.y.z.com .
> > If z is either domain1,domain2,domain3 then I need rsyslog to write
> just
> > 'x.y' and nothing else. (i.e. strip off z.com)
> > I came up with this:
> > $template TemplateStrip,"%TIMESTAMP%
> >
> %HOSTNAME:R,ERE,1,FIELD:^(.*)\.(domain1\.com|domain2\.com|domain3\.com)
> $--end%
> > %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
> > $ActionFileDefaultTemplate TemplateStrip
> > i.e. it uses a regular expression to strip off the domain.com part.
> >
> > This appears to work correctly but is this the most efficient way to
> do it,
> > or are there better ways?
> > The Property Replacer page (
> > http://www.rsyslog.com/doc/property_replacer.html) suggests regular
> > expressions can be slower than other methods,
> > but I think the other methods on that page aren't really applicable
> in my
> > case?
>
> regex is a slow way of doing things, but sometimes it's the only
> generic
> way to do it.
>
> You may want to look at the liblognorm option, it's FAR more efficient
> than regex parsing (especially if you end up with lots of rules)
>
> > Our old system receives about 10000 msgs/s so I'm hoping the new
> rsyslog
> > server can do the same.
>
> With equivalent configs, the new one should be faster. If you're having
> the new one do a lot more, you may run into issues, but on a reasonably
> modern system, 10K logs/sec should not be a problem.
>
> I'm not familiar with the -s option, but it's a holdover from some
> really
> old stuff, and I would not be surprised if nobody used it and as a
> result
> it broke without anyone noticing. Rainer would need to comment on this.

It's like you say: it may be broken over time. Unfortunately, I cannot look into it right now, as I have some priority stuff to finish (and will shortly head to vacation ;)).

Rainer
>
> David Lang
>
> > P.S. I tried using rsyslog's -s commandline option, but couldn't get
> it to
> > strip subdomains like this -
> > I could only get it to strip either y.z.com or nothing.
> >
> > Is the -s switch _supposed_ to be able to strip subdomains?
> > The rsyslogd man page (below) is a bit confusing: is it saying that
> it
> > CAN strip off just north.de (by doing something like "-s north.de:i
> > nfodrom.north.de" as it says),
> > or that it CAN'T do it?
> >
> > -s domainlist
> > Specify a domainname that should be stripped off before logging.
> > Multiple domains may be specified using the colon ('':'') separa-
> > tor. Please be advised that no sub-domains may be specified but
> > only entire domains. For example if -s north.de is specified and
> > the host logging resolves to satu.infodrom.north.de no domain would
> > be cut, you will have to specify two domains like:
> > -s north.de:infodrom.north.de.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads