Mailing List Archive: RSyslog: users

Replacing regular expression for particular tag

Index | Next | Previous | View Threaded

xfustero at gmail

May 17, 2012, 12:52 AM

Post #1 of 9 (356 views)
Permalink

Replacing regular expression for particular tag

Hi,

I want to ask what would be the best way to implement the following. I have
a several nodes identified with a particular ID (e.g: 01-9291212,
01-823HHK1). Those servers send their logs to a central rsyslog server
(RELP + stunnel). I want to create a directory entry on the server with
this ID name. Like rsyslog-server:/var/logs/01-9291212,
/var/logs/01-823HHK1 and so on.

My first attempt was to create a template on the client side and add this
ID manually

$template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
%syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
*.* :omrelp:127.0.0.1:port_number;ID

On the server side, I have created a regular expression to match a string
like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates for
each particular log: messages, maillog, cron, secure, etc.

E.g.: $Template Dyn_messages,
"/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
$template
Dyn_cron,"/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/cron"
...

I have a sequence of if/else where depending on facilities it sends to one
or another dynamic template. However, I would like to replace regular
expression for something like a %my_particular_tag%. I can't see the way I
can create this particular tag. They seem to be hardcoded. I also try to
modify property names (hostname,syslogtag,etc) and replace it for a
completely new name (my ID) but I can't find how to do this.
%propname:fromChar:toChar:options:fieldname% doesn't seem to allow this.

I would like to get ridd off regular expressions. They have an impact in
performance and complicate my templates on the server side. They also
created the directory **NO MATCH** which I would like to avoid. Using
tags, templates on server side would be something like:

$Template Dyn_messages, "/var/log/%mytag%/messages"
...

Does anybody know how to do this?

Thanks in advance,
Xavi
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

May 17, 2012, 1:50 PM

Post #2 of 9 (357 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

There are some new features in version 6 that will allow you to create
your own tags (either the liblognorm stuff or the project lumberjack
stuff)

you can't modify any existing tags once the log is received, but you could
change the sender to put a tag in the right place so that it will get
parsed by the central server as one of those tags.

how many different types of tags are you talking about here? is it a
handful (where you could create specific rules for each tag)? or are there
a lot (where you really need to use the dynafile to create all the
destination directories)

David Lang

On Thu, 17 May 2012, Xavier Fustero wrote:

> Date: Thu, 17 May 2012 09:52:38 +0200
> From: Xavier Fustero <xfustero [at] gmail>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: rsyslog [at] lists
> Subject: [rsyslog] Replacing regular expression for particular tag
>
> Hi,
>
> I want to ask what would be the best way to implement the following. I have
> a several nodes identified with a particular ID (e.g: 01-9291212,
> 01-823HHK1). Those servers send their logs to a central rsyslog server
> (RELP + stunnel). I want to create a directory entry on the server with
> this ID name. Like rsyslog-server:/var/logs/01-9291212,
> /var/logs/01-823HHK1 and so on.
>
> My first attempt was to create a template on the client side and add this
> ID manually
>
> $template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
> %syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
> *.* :omrelp:127.0.0.1:port_number;ID
>
> On the server side, I have created a regular expression to match a string
> like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates for
> each particular log: messages, maillog, cron, secure, etc.
>
> E.g.: $Template Dyn_messages,
> "/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
> $template
> Dyn_cron,"/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/cron"
> ...
>
> I have a sequence of if/else where depending on facilities it sends to one
> or another dynamic template. However, I would like to replace regular
> expression for something like a %my_particular_tag%. I can't see the way I
> can create this particular tag. They seem to be hardcoded. I also try to
> modify property names (hostname,syslogtag,etc) and replace it for a
> completely new name (my ID) but I can't find how to do this.
> %propname:fromChar:toChar:options:fieldname% doesn't seem to allow this.
>
> I would like to get ridd off regular expressions. They have an impact in
> performance and complicate my templates on the server side. They also
> created the directory **NO MATCH** which I would like to avoid. Using
> tags, templates on server side would be something like:
>
> $Template Dyn_messages, "/var/log/%mytag%/messages"
> ...
>
> Does anybody know how to do this?
>
> Thanks in advance,
> Xavi
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

xfustero at gmail

May 23, 2012, 1:12 AM

Post #3 of 9 (326 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

Hi,

On 17 May 2012 22:50, <david [at] lang> wrote:

> There are some new features in version 6 that will allow you to create
> your own tags (either the liblognorm stuff or the project lumberjack stuff)
>

thanks. I will check if it makes sense for us to move to version 6.

>
> you can't modify any existing tags once the log is received, but you could
> change the sender to put a tag in the right place so that it will get
> parsed by the central server as one of those tags.
>

That's exactly what I am trying to do. Creating a tag from sender. I can
create a template and put the text I want but I can't find through the docs
how to extract this as a tag.

I have something like this in my client: $template lala,"%syslogtag%
HOST_ID %msg%"

My problem is I would like to parse this HOST_ID as a tag but I couldn't
find how so I am using a regular expression on the server to do this. This
HOST_ID is always 01-(+7 alphanumeric characters).

$Template Dyn_messages,
"/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"

and I would like to replace for something like

$Template Dyn_messages, "/var/log/%HOST_ID%/messages"

> how many different types of tags are you talking about here? is it a
> handful (where you could create specific rules for each tag)? or are there
> a lot (where you really need to use the dynafile to create all the
> destination directories)
>

There will be a lot. This is a project for launching nodes in the cloud.

Thanks a lot,
Xavi

>
> David Lang
>
> On Thu, 17 May 2012, Xavier Fustero wrote:
>
> Date: Thu, 17 May 2012 09:52:38 +0200
>> From: Xavier Fustero <xfustero [at] gmail>
>> Reply-To: rsyslog-users <rsyslog [at] lists>
>> To: rsyslog [at] lists
>> Subject: [rsyslog] Replacing regular expression for particular tag
>>
>>
>> Hi,
>>
>> I want to ask what would be the best way to implement the following. I
>> have
>> a several nodes identified with a particular ID (e.g: 01-9291212,
>> 01-823HHK1). Those servers send their logs to a central rsyslog server
>> (RELP + stunnel). I want to create a directory entry on the server with
>> this ID name. Like rsyslog-server:/var/logs/01-**9291212,
>> /var/logs/01-823HHK1 and so on.
>>
>> My first attempt was to create a template on the client side and add this
>> ID manually
>>
>> $template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
>> %syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
>>
>> *.* :omrelp:127.0.0.1:port_number;**ID
>>
>> On the server side, I have created a regular expression to match a string
>> like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates for
>> each particular log: messages, maillog, cron, secure, etc.
>>
>> E.g.: $Template Dyn_messages,
>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>> $template
>> Dyn_cron,"/var/log/%msg:R,ERE,**0,DFLT:01\-[0-9A-Z]{7}--end%/**cron"
>> ...
>>
>> I have a sequence of if/else where depending on facilities it sends to one
>> or another dynamic template. However, I would like to replace regular
>> expression for something like a %my_particular_tag%. I can't see the way I
>> can create this particular tag. They seem to be hardcoded. I also try to
>> modify property names (hostname,syslogtag,etc) and replace it for a
>> completely new name (my ID) but I can't find how to do this.
>> %propname:fromChar:toChar:**options:fieldname% doesn't seem to allow
>> this.
>>
>> I would like to get ridd off regular expressions. They have an impact in
>> performance and complicate my templates on the server side. They also
>> created the directory **NO MATCH** which I would like to avoid. Using
>> tags, templates on server side would be something like:
>>
>> $Template Dyn_messages, "/var/log/%mytag%/messages"
>> ...
>>
>> Does anybody know how to do this?
>>
>> Thanks in advance,
>> Xavi
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

May 23, 2012, 11:41 AM

Post #4 of 9 (322 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

On Wed, 23 May 2012, Xavier Fustero wrote:

> Hi,
>
> On 17 May 2012 22:50, <david [at] lang> wrote:
>
>> There are some new features in version 6 that will allow you to create
>> your own tags (either the liblognorm stuff or the project lumberjack stuff)
>>
>
> thanks. I will check if it makes sense for us to move to version 6.
>
>
>
>>
>> you can't modify any existing tags once the log is received, but you could
>> change the sender to put a tag in the right place so that it will get
>> parsed by the central server as one of those tags.
>>
>
> That's exactly what I am trying to do. Creating a tag from sender. I can
> create a template and put the text I want but I can't find through the docs
> how to extract this as a tag.
>
> I have something like this in my client: $template lala,"%syslogtag%
> HOST_ID %msg%"
>
> My problem is I would like to parse this HOST_ID as a tag but I couldn't
> find how so I am using a regular expression on the server to do this. This
> HOST_ID is always 01-(+7 alphanumeric characters).
>
> $Template Dyn_messages,
> "/var/log/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
>
> and I would like to replace for something like
>
> $Template Dyn_messages, "/var/log/%HOST_ID%/messages"

right now you have two choices.

1. put the HOST_ID in place of the servername in your template so that it
gets parsed as %hostname%

2. use version 6 with either the project lumberjack parsing of JSON
messages or the mmnormalize module to create custom tags.

>> how many different types of tags are you talking about here? is it a
>> handful (where you could create specific rules for each tag)? or are there
>> a lot (where you really need to use the dynafile to create all the
>> destination directories)
>>
>
> There will be a lot. This is a project for launching nodes in the cloud.

In that case, you probably want to go with the version 6.3+ stuff that
lets you create custom tags by either parsing JSON formatted messages or
with the mmnormalize module

David Lang

> Thanks a lot,
> Xavi
>
>
>>
>> David Lang
>>
>> On Thu, 17 May 2012, Xavier Fustero wrote:
>>
>> Date: Thu, 17 May 2012 09:52:38 +0200
>>> From: Xavier Fustero <xfustero [at] gmail>
>>> Reply-To: rsyslog-users <rsyslog [at] lists>
>>> To: rsyslog [at] lists
>>> Subject: [rsyslog] Replacing regular expression for particular tag
>>>
>>>
>>> Hi,
>>>
>>> I want to ask what would be the best way to implement the following. I
>>> have
>>> a several nodes identified with a particular ID (e.g: 01-9291212,
>>> 01-823HHK1). Those servers send their logs to a central rsyslog server
>>> (RELP + stunnel). I want to create a directory entry on the server with
>>> this ID name. Like rsyslog-server:/var/logs/01-**9291212,
>>> /var/logs/01-823HHK1 and so on.
>>>
>>> My first attempt was to create a template on the client side and add this
>>> ID manually
>>>
>>> $template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
>>> %syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
>>>
>>> *.* :omrelp:127.0.0.1:port_number;**ID
>>>
>>> On the server side, I have created a regular expression to match a string
>>> like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates for
>>> each particular log: messages, maillog, cron, secure, etc.
>>>
>>> E.g.: $Template Dyn_messages,
>>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>>> $template
>>> Dyn_cron,"/var/log/%msg:R,ERE,**0,DFLT:01\-[0-9A-Z]{7}--end%/**cron"
>>> ...
>>>
>>> I have a sequence of if/else where depending on facilities it sends to one
>>> or another dynamic template. However, I would like to replace regular
>>> expression for something like a %my_particular_tag%. I can't see the way I
>>> can create this particular tag. They seem to be hardcoded. I also try to
>>> modify property names (hostname,syslogtag,etc) and replace it for a
>>> completely new name (my ID) but I can't find how to do this.
>>> %propname:fromChar:toChar:**options:fieldname% doesn't seem to allow
>>> this.
>>>
>>> I would like to get ridd off regular expressions. They have an impact in
>>> performance and complicate my templates on the server side. They also
>>> created the directory **NO MATCH** which I would like to avoid. Using
>>> tags, templates on server side would be something like:
>>>
>>> $Template Dyn_messages, "/var/log/%mytag%/messages"
>>> ...
>>>
>>> Does anybody know how to do this?
>>>
>>> Thanks in advance,
>>> Xavi
>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

xfustero at gmail

May 24, 2012, 3:59 AM

Post #5 of 9 (322 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

HI,

Inline ...

On 23 May 2012 20:41, <david [at] lang> wrote:

> On Wed, 23 May 2012, Xavier Fustero wrote:
>
> Hi,
>>
>> On 17 May 2012 22:50, <david [at] lang> wrote:
>>
>> There are some new features in version 6 that will allow you to create
>>> your own tags (either the liblognorm stuff or the project lumberjack
>>> stuff)
>>>
>>>
>> thanks. I will check if it makes sense for us to move to version 6.
>>
>>
>>
>>
>>> you can't modify any existing tags once the log is received, but you
>>> could
>>> change the sender to put a tag in the right place so that it will get
>>> parsed by the central server as one of those tags.
>>>
>>>
>> That's exactly what I am trying to do. Creating a tag from sender. I can
>> create a template and put the text I want but I can't find through the
>> docs
>> how to extract this as a tag.
>>
>> I have something like this in my client: $template lala,"%syslogtag%
>> HOST_ID %msg%"
>>
>> My problem is I would like to parse this HOST_ID as a tag but I couldn't
>> find how so I am using a regular expression on the server to do this.
>> This
>> HOST_ID is always 01-(+7 alphanumeric characters).
>>
>> $Template Dyn_messages,
>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>>
>> and I would like to replace for something like
>>
>> $Template Dyn_messages, "/var/log/%HOST_ID%/messages"
>>
>
> right now you have two choices.
>
> 1. put the HOST_ID in place of the servername in your template so that it
> gets parsed as %hostname%
>

Correct me if I am wrong. Do you mean I should change something like (in
the client):

$template hostID,"%TIMESTAMP% *%HOSTNAME%* %syslogtag%
%syslogfacility-text% %syslogseverity% %msg%\n"

to

$template hostID,"%TIMESTAMP% *01-1V8IMU1* %syslogtag%
%syslogfacility-text% %syslogseverity% %msg%\n" ?

...

*.* :omrelp:127.0.0.1:20500;hostID

and then, in the server, I will be able to replace the regular expression

$Template Dyn_messages,
"/var/log//xavi/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"

for

$Template Dyn_messages, "/var/log/xavi/%HOSTNAME%/messages" ?

I don't understand how rsyslog from server knows %HOSTNAME% is the tag I
hardcoded in the client template.

I might be missing something...

Thanks,
Xavi

> 2. use version 6 with either the project lumberjack parsing of JSON
> messages or the mmnormalize module to create custom tags.
>
>
> how many different types of tags are you talking about here? is it a
>>> handful (where you could create specific rules for each tag)? or are
>>> there
>>> a lot (where you really need to use the dynafile to create all the
>>> destination directories)
>>>
>>>
>> There will be a lot. This is a project for launching nodes in the cloud.
>>
>
> In that case, you probably want to go with the version 6.3+ stuff that
> lets you create custom tags by either parsing JSON formatted messages or
> with the mmnormalize module
>
> David Lang
>
> Thanks a lot,
>> Xavi
>>
>>
>>
>>> David Lang
>>>
>>> On Thu, 17 May 2012, Xavier Fustero wrote:
>>>
>>> Date: Thu, 17 May 2012 09:52:38 +0200
>>>
>>>> From: Xavier Fustero <xfustero [at] gmail>
>>>> Reply-To: rsyslog-users <rsyslog [at] lists>
>>>> To: rsyslog [at] lists
>>>> Subject: [rsyslog] Replacing regular expression for particular tag
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I want to ask what would be the best way to implement the following. I
>>>> have
>>>> a several nodes identified with a particular ID (e.g: 01-9291212,
>>>> 01-823HHK1). Those servers send their logs to a central rsyslog server
>>>> (RELP + stunnel). I want to create a directory entry on the server with
>>>> this ID name. Like rsyslog-server:/var/logs/01-****9291212,
>>>>
>>>> /var/logs/01-823HHK1 and so on.
>>>>
>>>> My first attempt was to create a template on the client side and add
>>>> this
>>>> ID manually
>>>>
>>>> $template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
>>>> %syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
>>>>
>>>> *.* :omrelp:127.0.0.1:port_number;****ID
>>>>
>>>>
>>>> On the server side, I have created a regular expression to match a
>>>> string
>>>> like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates
>>>> for
>>>> each particular log: messages, maillog, cron, secure, etc.
>>>>
>>>> E.g.: $Template Dyn_messages,
>>>> "/var/log/%msg:R,ERE,0,DFLT:****01\-[0-9A-Z]{7}--end%/****messages"
>>>> $template
>>>> Dyn_cron,"/var/log/%msg:R,ERE,****0,DFLT:01\-[0-9A-Z]{7}--end%**
>>>> /**cron"
>>>>
>>>> ...
>>>>
>>>> I have a sequence of if/else where depending on facilities it sends to
>>>> one
>>>> or another dynamic template. However, I would like to replace regular
>>>> expression for something like a %my_particular_tag%. I can't see the
>>>> way I
>>>> can create this particular tag. They seem to be hardcoded. I also try to
>>>> modify property names (hostname,syslogtag,etc) and replace it for a
>>>> completely new name (my ID) but I can't find how to do this.
>>>> %propname:fromChar:toChar:****options:fieldname% doesn't seem to allow
>>>>
>>>> this.
>>>>
>>>> I would like to get ridd off regular expressions. They have an impact in
>>>> performance and complicate my templates on the server side. They also
>>>> created the directory **NO MATCH** which I would like to avoid. Using
>>>> tags, templates on server side would be something like:
>>>>
>>>> $Template Dyn_messages, "/var/log/%mytag%/messages"
>>>> ...
>>>>
>>>> Does anybody know how to do this?
>>>>
>>>> Thanks in advance,
>>>> Xavi
>>>> ______________________________****_________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> >
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>> >
>>>>
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>
>>>> ______________________________****_________________
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

xfustero at gmail

May 24, 2012, 7:22 AM

Post #6 of 9 (322 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

Hi,

>>>
>>>> you can't modify any existing tags once the log is received, but you
>>>> could
>>>> change the sender to put a tag in the right place so that it will get
>>>> parsed by the central server as one of those tags.
>>>>
>>>>
>>> That's exactly what I am trying to do. Creating a tag from sender. I can
>>> create a template and put the text I want but I can't find through the
>>> docs
>>> how to extract this as a tag.
>>>
>>> I have something like this in my client: $template lala,"%syslogtag%
>>> HOST_ID %msg%"
>>>
>>> My problem is I would like to parse this HOST_ID as a tag but I couldn't
>>> find how so I am using a regular expression on the server to do this.
>>> This
>>> HOST_ID is always 01-(+7 alphanumeric characters).
>>>
>>> $Template Dyn_messages,
>>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>>>
>>> and I would like to replace for something like
>>>
>>> $Template Dyn_messages, "/var/log/%HOST_ID%/messages"
>>>
>>
>> right now you have two choices.
>>
>> 1. put the HOST_ID in place of the servername in your template so that it
>> gets parsed as %hostname%
>>
>
> Correct me if I am wrong. Do you mean I should change something like (in
> the client):
>
> $template hostID,"%TIMESTAMP% *%HOSTNAME%* %syslogtag%
> %syslogfacility-text% %syslogseverity% %msg%\n"
>
> to
>
> $template hostID,"%TIMESTAMP% *01-1V8IMU1* %syslogtag%
> %syslogfacility-text% %syslogseverity% %msg%\n" ?
>
> ...
>
> *.* :omrelp:127.0.0.1:20500;hostID
>
>
> and then, in the server, I will be able to replace the regular expression
>
> $Template Dyn_messages,
> "/var/log//xavi/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
>
> for
>
> $Template Dyn_messages, "/var/log/xavi/%HOSTNAME%/messages" ?
>
> I don't understand how rsyslog from server knows %HOSTNAME% is the tag I
> hardcoded in the client template.
>
> I might be missing something...
>
> Thanks,
> Xavi
>
>

Should be the solution something like this?

CLIENT:
$template hostID,"%TIMESTAMP% *HOST_ID=01-1V8IMU1* %syslogtag%
%syslogfacility-text% %syslogseverity% %msg%\n" ?

SERVER: write a rule like :msg contains HOST_ID .... ?

Thanks,
Xavi

>
>
>
>
>
>> 2. use version 6 with either the project lumberjack parsing of JSON
>> messages or the mmnormalize module to create custom tags.
>>
>>
>> how many different types of tags are you talking about here? is it a
>>>> handful (where you could create specific rules for each tag)? or are
>>>> there
>>>> a lot (where you really need to use the dynafile to create all the
>>>> destination directories)
>>>>
>>>>
>>> There will be a lot. This is a project for launching nodes in the cloud.
>>>
>>
>> In that case, you probably want to go with the version 6.3+ stuff that
>> lets you create custom tags by either parsing JSON formatted messages or
>> with the mmnormalize module
>>
>> David Lang
>>
>> Thanks a lot,
>>> Xavi
>>>
>>>
>>>
>>>> David Lang
>>>>
>>>> On Thu, 17 May 2012, Xavier Fustero wrote:
>>>>
>>>> Date: Thu, 17 May 2012 09:52:38 +0200
>>>>
>>>>> From: Xavier Fustero <xfustero [at] gmail>
>>>>> Reply-To: rsyslog-users <rsyslog [at] lists>
>>>>> To: rsyslog [at] lists
>>>>> Subject: [rsyslog] Replacing regular expression for particular tag
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I want to ask what would be the best way to implement the following. I
>>>>> have
>>>>> a several nodes identified with a particular ID (e.g: 01-9291212,
>>>>> 01-823HHK1). Those servers send their logs to a central rsyslog server
>>>>> (RELP + stunnel). I want to create a directory entry on the server with
>>>>> this ID name. Like rsyslog-server:/var/logs/01-****9291212,
>>>>>
>>>>> /var/logs/01-823HHK1 and so on.
>>>>>
>>>>> My first attempt was to create a template on the client side and add
>>>>> this
>>>>> ID manually
>>>>>
>>>>> $template ID,"%TIMESTAMP% %HOSTNAME% %syslogtag% %syslogfacility-text%
>>>>> %syslogseverity-text% *ID: 01-XXXXXXX* %syslogtag% %msg%\n
>>>>>
>>>>> *.* :omrelp:127.0.0.1:port_number;****ID
>>>>>
>>>>>
>>>>> On the server side, I have created a regular expression to match a
>>>>> string
>>>>> like 01-([0-9A-Za-a]{7} (my ID's format) and created dynamic templates
>>>>> for
>>>>> each particular log: messages, maillog, cron, secure, etc.
>>>>>
>>>>> E.g.: $Template Dyn_messages,
>>>>> "/var/log/%msg:R,ERE,0,DFLT:****01\-[0-9A-Z]{7}--end%/****messages"
>>>>> $template
>>>>> Dyn_cron,"/var/log/%msg:R,ERE,****0,DFLT:01\-[0-9A-Z]{7}--end%**
>>>>> /**cron"
>>>>>
>>>>> ...
>>>>>
>>>>> I have a sequence of if/else where depending on facilities it sends to
>>>>> one
>>>>> or another dynamic template. However, I would like to replace regular
>>>>> expression for something like a %my_particular_tag%. I can't see the
>>>>> way I
>>>>> can create this particular tag. They seem to be hardcoded. I also try
>>>>> to
>>>>> modify property names (hostname,syslogtag,etc) and replace it for a
>>>>> completely new name (my ID) but I can't find how to do this.
>>>>> %propname:fromChar:toChar:****options:fieldname% doesn't seem to allow
>>>>>
>>>>> this.
>>>>>
>>>>> I would like to get ridd off regular expressions. They have an impact
>>>>> in
>>>>> performance and complicate my templates on the server side. They also
>>>>> created the directory **NO MATCH** which I would like to avoid. Using
>>>>> tags, templates on server side would be something like:
>>>>>
>>>>> $Template Dyn_messages, "/var/log/%mytag%/messages"
>>>>> ...
>>>>>
>>>>> Does anybody know how to do this?
>>>>>
>>>>> Thanks in advance,
>>>>> Xavi
>>>>> ______________________________****_________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>> >
>>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>> >
>>>>>
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>
>>>>> ______________________________****_________________
>>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>> >
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>> >
>>>>
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>
>>>> ______________________________**_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

May 24, 2012, 12:40 PM

Post #7 of 9 (323 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

On Thu, 24 May 2012, Xavier Fustero wrote:

> HI,
>
> Inline ...
>
>
>
> On 23 May 2012 20:41, <david [at] lang> wrote:
>
>> On Wed, 23 May 2012, Xavier Fustero wrote:
>>
>> Hi,
>>>
>>> On 17 May 2012 22:50, <david [at] lang> wrote:
>>>
>>> There are some new features in version 6 that will allow you to create
>>>> your own tags (either the liblognorm stuff or the project lumberjack
>>>> stuff)
>>>>
>>>>
>>> thanks. I will check if it makes sense for us to move to version 6.
>>>
>>>
>>>
>>>
>>>> you can't modify any existing tags once the log is received, but you
>>>> could
>>>> change the sender to put a tag in the right place so that it will get
>>>> parsed by the central server as one of those tags.
>>>>
>>>>
>>> That's exactly what I am trying to do. Creating a tag from sender. I can
>>> create a template and put the text I want but I can't find through the
>>> docs
>>> how to extract this as a tag.
>>>
>>> I have something like this in my client: $template lala,"%syslogtag%
>>> HOST_ID %msg%"
>>>
>>> My problem is I would like to parse this HOST_ID as a tag but I couldn't
>>> find how so I am using a regular expression on the server to do this.
>>> This
>>> HOST_ID is always 01-(+7 alphanumeric characters).
>>>
>>> $Template Dyn_messages,
>>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>>>
>>> and I would like to replace for something like
>>>
>>> $Template Dyn_messages, "/var/log/%HOST_ID%/messages"
>>>
>>
>> right now you have two choices.
>>
>> 1. put the HOST_ID in place of the servername in your template so that it
>> gets parsed as %hostname%
>>
>
> Correct me if I am wrong. Do you mean I should change something like (in
> the client):
>
> $template hostID,"%TIMESTAMP% *%HOSTNAME%* %syslogtag%
> %syslogfacility-text% %syslogseverity% %msg%\n"
>
> to
>
> $template hostID,"%TIMESTAMP% *01-1V8IMU1* %syslogtag%
> %syslogfacility-text% %syslogseverity% %msg%\n" ?
>
> ...
>
> *.* :omrelp:127.0.0.1:20500;hostID
>
>
> and then, in the server, I will be able to replace the regular expression
>
> $Template Dyn_messages,
> "/var/log//xavi/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
>
> for
>
> $Template Dyn_messages, "/var/log/xavi/%HOSTNAME%/messages" ?

yes. or more precisely, almost, the template should have <%PRI%> before
the timestamp when forwarding.

> I don't understand how rsyslog from server knows %HOSTNAME% is the tag I
> hardcoded in the client template.

the receiving syslog daemon parses the inbound message according to the
RFC standards. This say that the string after the timestamp is the
hostname. It doesn't know if it's really the hostname or not (the sender
can lie, as I am suggesting that you do in this case), but it assumes that
it is the hostname and puts it in the hostname property.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

May 24, 2012, 12:43 PM

Post #8 of 9 (319 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

On Thu, 24 May 2012, Xavier Fustero wrote:

> Hi,
>
>
>
>>>>
>>>>> you can't modify any existing tags once the log is received, but you
>>>>> could
>>>>> change the sender to put a tag in the right place so that it will get
>>>>> parsed by the central server as one of those tags.
>>>>>
>>>>>
>>>> That's exactly what I am trying to do. Creating a tag from sender. I can
>>>> create a template and put the text I want but I can't find through the
>>>> docs
>>>> how to extract this as a tag.
>>>>
>>>> I have something like this in my client: $template lala,"%syslogtag%
>>>> HOST_ID %msg%"
>>>>
>>>> My problem is I would like to parse this HOST_ID as a tag but I couldn't
>>>> find how so I am using a regular expression on the server to do this.
>>>> This
>>>> HOST_ID is always 01-(+7 alphanumeric characters).
>>>>
>>>> $Template Dyn_messages,
>>>> "/var/log/%msg:R,ERE,0,DFLT:**01\-[0-9A-Z]{7}--end%/**messages"
>>>>
>>>> and I would like to replace for something like
>>>>
>>>> $Template Dyn_messages, "/var/log/%HOST_ID%/messages"
>>>>
>>>
>>> right now you have two choices.
>>>
>>> 1. put the HOST_ID in place of the servername in your template so that it
>>> gets parsed as %hostname%
>>>
>>
>> Correct me if I am wrong. Do you mean I should change something like (in
>> the client):
>>
>> $template hostID,"%TIMESTAMP% *%HOSTNAME%* %syslogtag%
>> %syslogfacility-text% %syslogseverity% %msg%\n"
>>
>> to
>>
>> $template hostID,"%TIMESTAMP% *01-1V8IMU1* %syslogtag%
>> %syslogfacility-text% %syslogseverity% %msg%\n" ?
>>
>> ...
>>
>> *.* :omrelp:127.0.0.1:20500;hostID
>>
>>
>> and then, in the server, I will be able to replace the regular expression
>>
>> $Template Dyn_messages,
>> "/var/log//xavi/%msg:R,ERE,0,DFLT:01\-[0-9A-Z]{7}--end%/messages"
>>
>> for
>>
>> $Template Dyn_messages, "/var/log/xavi/%HOSTNAME%/messages" ?
>>
>> I don't understand how rsyslog from server knows %HOSTNAME% is the tag I
>> hardcoded in the client template.
>>
>> I might be missing something...
>>
>> Thanks,
>> Xavi
>>
>>
>
> Should be the solution something like this?
>
> CLIENT:
> $template hostID,"%TIMESTAMP% *HOST_ID=01-1V8IMU1* %syslogtag%
> %syslogfacility-text% %syslogseverity% %msg%\n" ?
>
> SERVER: write a rule like :msg contains HOST_ID .... ?

no, this isn't the way to do it.

you are using characters in the hostname field that are not valid there (=
and possibly _) so this would be generating malformed messages.

I think that the current rsyslog message parser would put that in the
message on the server, but given that it's only trying to do it's best to
deal with something that's not formatted properly, this isn't a good thing
to rely on (some future version may deal with it in a slightly different
way, and since what you are sending isn't valid by any definition, there's
no way to have it tested)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

xfustero at gmail

May 25, 2012, 3:46 AM

Post #9 of 9 (321 views)
Permalink

Re: Replacing regular expression for particular tag [In reply to]

David,

thanks a lot. Everything is working fine now! Thanks for clarifying me this.

Xavi

>>> right now you have two choices.
>>>
>>> 1. put the HOST_ID in place of the servername in your template so that it
>>> gets parsed as %hostname%
>>>
>>>
>> Correct me if I am wrong. Do you mean I should change something like (in
>> the client):
>>
>> $template hostID,"%TIMESTAMP% *%HOSTNAME%* %syslogtag%
>>
>> %syslogfacility-text% %syslogseverity% %msg%\n"
>>
>> to
>>
>> $template hostID,"%TIMESTAMP% *01-1V8IMU1* %syslogtag%
>>
>> %syslogfacility-text% %syslogseverity% %msg%\n" ?
>>
>> ...
>>
>> *.* :omrelp:127.0.0.1:20500;hostID
>>
>>
>> and then, in the server, I will be able to replace the regular expression
>>
>> $Template Dyn_messages,
>> "/var/log//xavi/%msg:R,ERE,0,**DFLT:01\-[0-9A-Z]{7}--end%/**messages"
>>
>> for
>>
>> $Template Dyn_messages, "/var/log/xavi/%HOSTNAME%/**messages" ?
>>
>
> yes. or more precisely, almost, the template should have <%PRI%> before
> the timestamp when forwarding.
>
>
> I don't understand how rsyslog from server knows %HOSTNAME% is the tag I
>> hardcoded in the client template.
>>
>
> the receiving syslog daemon parses the inbound message according to the
> RFC standards. This say that the string after the timestamp is the
> hostname. It doesn't know if it's really the hostname or not (the sender
> can lie, as I am suggesting that you do in this case), but it assumes that
> it is the hostname and puts it in the hostname property.
>
> David Lang
>
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads