Mailing List Archive: RSyslog: users

statsobj.c

Index | Next | Previous | View Threaded

briank at talksum

Jun 13, 2012, 12:31 PM

Post #1 of 5 (160 views)
Permalink

statsobj.c

Rainer -

I'm working on getting imstats data emitted by the module into a
"flattened" cee-like json list. (This is part of what I was doing earlier
today when I emailed about the mmjsonparse issue).

So (of course speaking of after mmjsonparse in a rsyslog 6 style block is
working properly) if I set $PStatCEE on, and pass the resulting message to
mmjson parse, and then output using a template that uses "%$!all-json%\n",
the resulting message is like:

{"name": "imuxsock", "submitted": "0", "ratelimit.discarded": "0",
"ratelimit.numratelimiters": "0"}

Which is of course pretty much the same as the original messages:

2012-06-13T11:33:10.894045-04:00 localhost rsyslogd-pstats: @cee:
{"name":"emerg_action","processed":0,"failed":0}

What I would like to do at this point is get the time the stat was
generated into the message. I want to do this in order to get a totally
flat JSON message versus the way I've been doing it in the past, which was
to use a template to create a nested document like:

{"time":1339035848,
"stat":{"name":"authpriv_action","processed":10,"failed":0}}

My end goal is:

{"time": 1339035848, "name":"authpriv_action", "processed":10, "failed":0}

I could of course forward the message to a service that rewrites the json,
but this is an extra parsing step I'd rather not do.

It seems to me my options would be to somehow get the time of the emitted
message from the impstats module itself, and pass it to the the call that
in return calls getAllStatsLines so it can be inserted into the message, or
do it directly in statsobj if the time is available there.

So I suppose my questions are:

1) is there an easier way to do this, and
2) if not, is there a more preferred way to do this, and
3) does this sound reasonable :)

Brian
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

briank at talksum

Jun 13, 2012, 12:37 PM

Post #2 of 5 (148 views)
Permalink

Re: statsobj.c [In reply to]

Thinking about it for a couple another option would be something like a
PStatOpenJSON directive, that would emit an json message without the
brackets:

"name":"imuxsock", "submitted":0, "ratelimit.discarded":0

that could then be combined with an output template like...

"{ \"foo\":\"bar\", \"baz\":\"bat\", %msg% }\n"

On Wed, Jun 13, 2012 at 3:31 PM, Brian Knox <briank [at] talksum> wrote:

> Rainer -
>
> I'm working on getting imstats data emitted by the module into a
> "flattened" cee-like json list. (This is part of what I was doing earlier
> today when I emailed about the mmjsonparse issue).
>
> So (of course speaking of after mmjsonparse in a rsyslog 6 style block is
> working properly) if I set $PStatCEE on, and pass the resulting message to
> mmjson parse, and then output using a template that uses "%$!all-json%\n",
> the resulting message is like:
>
> {"name": "imuxsock", "submitted": "0", "ratelimit.discarded": "0",
> "ratelimit.numratelimiters": "0"}
>
> Which is of course pretty much the same as the original messages:
>
> 2012-06-13T11:33:10.894045-04:00 localhost rsyslogd-pstats: @cee:
> {"name":"emerg_action","processed":0,"failed":0}
>
> What I would like to do at this point is get the time the stat was
> generated into the message. I want to do this in order to get a totally
> flat JSON message versus the way I've been doing it in the past, which was
> to use a template to create a nested document like:
>
> {"time":1339035848,
> "stat":{"name":"authpriv_action","processed":10,"failed":0}}
>
> My end goal is:
>
> {"time": 1339035848, "name":"authpriv_action", "processed":10, "failed":0}
>
> I could of course forward the message to a service that rewrites the json,
> but this is an extra parsing step I'd rather not do.
>
> It seems to me my options would be to somehow get the time of the emitted
> message from the impstats module itself, and pass it to the the call that
> in return calls getAllStatsLines so it can be inserted into the message, or
> do it directly in statsobj if the time is available there.
>
> So I suppose my questions are:
>
> 1) is there an easier way to do this, and
> 2) if not, is there a more preferred way to do this, and
> 3) does this sound reasonable :)
>
> Brian
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

rgerhards at hq

Jun 14, 2012, 3:43 AM

Post #3 of 5 (143 views)
Permalink

Re: statsobj.c [In reply to]

I see where you are coming from. I also think this makes sense. However, it looks for me like an incarnation of improved JSON support inside the template processor, which takes some more time...

So question now how to create a work around... Need to think a bit about that.

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Wednesday, June 13, 2012 9:32 PM
> To: rsyslog-users
> Subject: [rsyslog] statsobj.c
>
> Rainer -
>
> I'm working on getting imstats data emitted by the module into a
> "flattened" cee-like json list. (This is part of what I was doing earlier
> today when I emailed about the mmjsonparse issue).
>
> So (of course speaking of after mmjsonparse in a rsyslog 6 style block is
> working properly) if I set $PStatCEE on, and pass the resulting message to
> mmjson parse, and then output using a template that uses "%$!all-json%\n",
> the resulting message is like:
>
> {"name": "imuxsock", "submitted": "0", "ratelimit.discarded": "0",
> "ratelimit.numratelimiters": "0"}
>
> Which is of course pretty much the same as the original messages:
>
> 2012-06-13T11:33:10.894045-04:00 localhost rsyslogd-pstats: @cee:
> {"name":"emerg_action","processed":0,"failed":0}
>
> What I would like to do at this point is get the time the stat was
> generated into the message. I want to do this in order to get a totally
> flat JSON message versus the way I've been doing it in the past, which was
> to use a template to create a nested document like:
>
> {"time":1339035848,
> "stat":{"name":"authpriv_action","processed":10,"failed":0}}
>
> My end goal is:
>
> {"time": 1339035848, "name":"authpriv_action", "processed":10, "failed":0}
>
> I could of course forward the message to a service that rewrites the json,
> but this is an extra parsing step I'd rather not do.
>
> It seems to me my options would be to somehow get the time of the emitted
> message from the impstats module itself, and pass it to the the call that
> in return calls getAllStatsLines so it can be inserted into the message, or
> do it directly in statsobj if the time is available there.
>
> So I suppose my questions are:
>
> 1) is there an easier way to do this, and
> 2) if not, is there a more preferred way to do this, and
> 3) does this sound reasonable :)
>
> Brian
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

briank at talksum

Jun 14, 2012, 5:23 AM

Post #4 of 5 (142 views)
Permalink

Re: statsobj.c [In reply to]

If you have large plans for improved JSON support, for now I could just
patch our rsyslog's locally to emit the "inner json" that I need. That
would certainly be an acceptable option if you don't want to pollute the
source with temporary fixes.

Brian

On Thu, Jun 14, 2012 at 6:43 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> I see where you are coming from. I also think this makes sense. However,
> it looks for me like an incarnation of improved JSON support inside the
> template processor, which takes some more time...
>
> So question now how to create a work around... Need to think a bit about
> that.
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Brian Knox
> > Sent: Wednesday, June 13, 2012 9:32 PM
> > To: rsyslog-users
> > Subject: [rsyslog] statsobj.c
> >
> > Rainer -
> >
> > I'm working on getting imstats data emitted by the module into a
> > "flattened" cee-like json list. (This is part of what I was doing
> earlier
> > today when I emailed about the mmjsonparse issue).
> >
> > So (of course speaking of after mmjsonparse in a rsyslog 6 style block is
> > working properly) if I set $PStatCEE on, and pass the resulting message
> to
> > mmjson parse, and then output using a template that uses
> "%$!all-json%\n",
> > the resulting message is like:
> >
> > {"name": "imuxsock", "submitted": "0", "ratelimit.discarded": "0",
> > "ratelimit.numratelimiters": "0"}
> >
> > Which is of course pretty much the same as the original messages:
> >
> > 2012-06-13T11:33:10.894045-04:00 localhost rsyslogd-pstats: @cee:
> > {"name":"emerg_action","processed":0,"failed":0}
> >
> > What I would like to do at this point is get the time the stat was
> > generated into the message. I want to do this in order to get a totally
> > flat JSON message versus the way I've been doing it in the past, which
> was
> > to use a template to create a nested document like:
> >
> > {"time":1339035848,
> > "stat":{"name":"authpriv_action","processed":10,"failed":0}}
> >
> > My end goal is:
> >
> > {"time": 1339035848, "name":"authpriv_action", "processed":10,
> "failed":0}
> >
> > I could of course forward the message to a service that rewrites the
> json,
> > but this is an extra parsing step I'd rather not do.
> >
> > It seems to me my options would be to somehow get the time of the emitted
> > message from the impstats module itself, and pass it to the the call that
> > in return calls getAllStatsLines so it can be inserted into the message,
> or
> > do it directly in statsobj if the time is available there.
> >
> > So I suppose my questions are:
> >
> > 1) is there an easier way to do this, and
> > 2) if not, is there a more preferred way to do this, and
> > 3) does this sound reasonable :)
> >
> > Brian
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

briank at talksum

Jun 14, 2012, 9:04 AM

Post #5 of 5 (142 views)
Permalink

Re: statsobj.c [In reply to]

So, I've got a patch locally that lets me do this:

$PStatsInterval 10
$PStatSeverity 7
$PStatLIST on

$template stat_test,"{\"time\":%timestamp:::date-unixtimestamp%, %msg% }\n"

with the following result:

{"time":1339689727,
"name":"TestTcpToFile","size":0,"enqueued":0,"full":0,"discarded.full":0,"
discarded.nf":0,"maxqsize":0 }

which solves my problem for the moment.

Brian

On Thu, Jun 14, 2012 at 8:23 AM, Brian Knox <briank [at] talksum> wrote:

> If you have large plans for improved JSON support, for now I could just
> patch our rsyslog's locally to emit the "inner json" that I need. That
> would certainly be an acceptable option if you don't want to pollute the
> source with temporary fixes.
>
> Brian
>
>
> On Thu, Jun 14, 2012 at 6:43 AM, Rainer Gerhards <rgerhards [at] hq
> > wrote:
>
>> I see where you are coming from. I also think this makes sense. However,
>> it looks for me like an incarnation of improved JSON support inside the
>> template processor, which takes some more time...
>>
>> So question now how to create a work around... Need to think a bit about
>> that.
>>
>> Rainer
>>
>> > -----Original Message-----
>> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > bounces [at] lists] On Behalf Of Brian Knox
>> > Sent: Wednesday, June 13, 2012 9:32 PM
>> > To: rsyslog-users
>> > Subject: [rsyslog] statsobj.c
>> >
>> > Rainer -
>> >
>> > I'm working on getting imstats data emitted by the module into a
>> > "flattened" cee-like json list. (This is part of what I was doing
>> earlier
>> > today when I emailed about the mmjsonparse issue).
>> >
>> > So (of course speaking of after mmjsonparse in a rsyslog 6 style block
>> is
>> > working properly) if I set $PStatCEE on, and pass the resulting message
>> to
>> > mmjson parse, and then output using a template that uses
>> "%$!all-json%\n",
>> > the resulting message is like:
>> >
>> > {"name": "imuxsock", "submitted": "0", "ratelimit.discarded": "0",
>> > "ratelimit.numratelimiters": "0"}
>> >
>> > Which is of course pretty much the same as the original messages:
>> >
>> > 2012-06-13T11:33:10.894045-04:00 localhost rsyslogd-pstats: @cee:
>> > {"name":"emerg_action","processed":0,"failed":0}
>> >
>> > What I would like to do at this point is get the time the stat was
>> > generated into the message. I want to do this in order to get a totally
>> > flat JSON message versus the way I've been doing it in the past, which
>> was
>> > to use a template to create a nested document like:
>> >
>> > {"time":1339035848,
>> > "stat":{"name":"authpriv_action","processed":10,"failed":0}}
>> >
>> > My end goal is:
>> >
>> > {"time": 1339035848, "name":"authpriv_action", "processed":10,
>> "failed":0}
>> >
>> > I could of course forward the message to a service that rewrites the
>> json,
>> > but this is an extra parsing step I'd rather not do.
>> >
>> > It seems to me my options would be to somehow get the time of the
>> emitted
>> > message from the impstats module itself, and pass it to the the call
>> that
>> > in return calls getAllStatsLines so it can be inserted into the
>> message, or
>> > do it directly in statsobj if the time is available there.
>> >
>> > So I suppose my questions are:
>> >
>> > 1) is there an easier way to do this, and
>> > 2) if not, is there a more preferred way to do this, and
>> > 3) does this sound reasonable :)
>> >
>> > Brian
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads