Mailing List Archive: RSyslog: users

next question: templates, JSON and MongoDB

1 2

View All

Index | Next | Previous | View Threaded

rgerhards at hq

Mar 15, 2012, 2:46 AM

Post #1 of 50 (1174 views)
Permalink

next question: templates, JSON and MongoDB

Hi all,

another question I am facing while implementing the new stuff:

http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html

Feedback is appreciated :-)

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 3:03 AM

Post #2 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

I posted this on the blog as well -

For my immediate needs, being able to JSON escape properties as a property
option is satisfactory. I'm experimenting with templates such as:

$template cee_enhanced,"@cee:
{\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"time\":\"%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
26 *.* /var/log/cee_events.log;cee_enhanced

Which works for a lot of my messages, the escaping is the only issue.
Adding a JSON escape property option would allow me to move pretty quickly
as I prototype things.

Longer term, having some sort of JSON call where I could pass a property
list would be nice... although the templates do give me the flexibility of
specifying nested document structures, which might get trickier. I'll think
more about it.

On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> Hi all,
>
> another question I am facing while implementing the new stuff:
>
> http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>
> Feedback is appreciated :-)
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 3:08 AM

Post #3 of 50 (1136 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

I'll add an option, hopefully later today. Should be fairly simple ;)
Please also think a little bit about the MongoDB interface, which is a bit
trickier (I don'T like the idea of formatting JSON to immediately parse it
again, even though this may be a working first instance solution).

Thx,
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Thursday, March 15, 2012 11:04 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> I posted this on the blog as well -
>
> For my immediate needs, being able to JSON escape properties as a
> property
> option is satisfactory. I'm experimenting with templates such as:
>
> $template cee_enhanced,"@cee:
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> e\":\"%timestamp:::date-
> rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> 26 *.* /var/log/cee_events.log;cee_enhanced
>
> Which works for a lot of my messages, the escaping is the only issue.
> Adding a JSON escape property option would allow me to move pretty
> quickly
> as I prototype things.
>
> Longer term, having some sort of JSON call where I could pass a
> property
> list would be nice... although the templates do give me the flexibility
> of
> specifying nested document structures, which might get trickier. I'll
> think
> more about it.
>
>
> On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> <rgerhards [at] hq>wrote:
>
> > Hi all,
> >
> > another question I am facing while implementing the new stuff:
> >
> > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> >
> > Feedback is appreciated :-)
> >
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 3:19 AM

Post #4 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Lol... some month ago, I was smart enough to add a ",json" option to the
template system, which seems to do what you urgently need. It's not a real
JSON-encoder, but handles the quote char. So:

$template cee_enhanced,"@cee:
{\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"time\":\"
%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n",json
*.* /var/log/cee_events.log;cee_enhanced

Note the end of the $template line!

... should do what you need to get started. This is very experimental code,
and I think it will not properly work with already-JSON escaped data (like in
$!all-json!) and it will also not do proper full JSON escaping. But it really
looks like it is what you right now need for your PoC ;)

But please expect syntax changes, so you probably need to adjust your
rsyslog.conf later. I do not like to be forced to keep supporting this even
if we go into a totally different direction (and I think this experimental
solution is not clean in the light of what has changed since I implemented
it).

Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Thursday, March 15, 2012 11:08 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> I'll add an option, hopefully later today. Should be fairly simple ;)
> Please also think a little bit about the MongoDB interface, which is a
> bit
> trickier (I don'T like the idea of formatting JSON to immediately parse
> it
> again, even though this may be a working first instance solution).
>
> Thx,
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Brian Knox
> > Sent: Thursday, March 15, 2012 11:04 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > I posted this on the blog as well -
> >
> > For my immediate needs, being able to JSON escape properties as a
> > property
> > option is satisfactory. I'm experimenting with templates such as:
> >
> > $template cee_enhanced,"@cee:
> >
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > e\":\"%timestamp:::date-
> > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > 26 *.* /var/log/cee_events.log;cee_enhanced
> >
> > Which works for a lot of my messages, the escaping is the only issue.
> > Adding a JSON escape property option would allow me to move pretty
> > quickly
> > as I prototype things.
> >
> > Longer term, having some sort of JSON call where I could pass a
> > property
> > list would be nice... although the templates do give me the
> flexibility
> > of
> > specifying nested document structures, which might get trickier. I'll
> > think
> > more about it.
> >
> >
> > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > <rgerhards [at] hq>wrote:
> >
> > > Hi all,
> > >
> > > another question I am facing while implementing the new stuff:
> > >
> > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > >
> > > Feedback is appreciated :-)
> > >
> > > Rainer
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 3:21 AM

Post #5 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

I think the MongoDB question ties into a larger question, of external
sources that use text based protocols in general. For instance, Redis,
which does not use JSON but does use a text based protocol over TCP.

In my case, our primary concern when interacting with external data stores
is always performance over convenience. Double parsing a JSON document
(first into CEE, and then from CEE into a MongoDB command) would definitely
be a performance hit.

Thinking about it though, I would like a solution that allows formatting
the message independently of the specific output. For instance - in the
past I've connected rsyslog to mongodb by passing the message over ZMQ to a
daemon that handled the mongodb loading.

I'll have to think about all of this more after I have some coffee!

Brian

On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> I'll add an option, hopefully later today. Should be fairly simple ;)
> Please also think a little bit about the MongoDB interface, which is a bit
> trickier (I don'T like the idea of formatting JSON to immediately parse it
> again, even though this may be a working first instance solution).
>
> Thx,
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Brian Knox
> > Sent: Thursday, March 15, 2012 11:04 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > I posted this on the blog as well -
> >
> > For my immediate needs, being able to JSON escape properties as a
> > property
> > option is satisfactory. I'm experimenting with templates such as:
> >
> > $template cee_enhanced,"@cee:
> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > e\":\"%timestamp:::date-
> > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > 26 *.* /var/log/cee_events.log;cee_enhanced
> >
> > Which works for a lot of my messages, the escaping is the only issue.
> > Adding a JSON escape property option would allow me to move pretty
> > quickly
> > as I prototype things.
> >
> > Longer term, having some sort of JSON call where I could pass a
> > property
> > list would be nice... although the templates do give me the flexibility
> > of
> > specifying nested document structures, which might get trickier. I'll
> > think
> > more about it.
> >
> >
> > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > <rgerhards [at] hq>wrote:
> >
> > > Hi all,
> > >
> > > another question I am facing while implementing the new stuff:
> > >
> > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > >
> > > Feedback is appreciated :-)
> > >
> > > Rainer
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 3:21 AM

Post #6 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Oh, hah! I'll give this a try this morning then. :)

Brian

On Thu, Mar 15, 2012 at 6:19 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> Lol... some month ago, I was smart enough to add a ",json" option to the
> template system, which seems to do what you urgently need. It's not a real
> JSON-encoder, but handles the quote char. So:
>
> $template cee_enhanced,"@cee:
>
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"time\":\"
> %timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n",json
> *.* /var/log/cee_events.log;cee_enhanced
>
> Note the end of the $template line!
>
> ... should do what you need to get started. This is very experimental code,
> and I think it will not properly work with already-JSON escaped data (like
> in
> $!all-json!) and it will also not do proper full JSON escaping. But it
> really
> looks like it is what you right now need for your PoC ;)
>
> But please expect syntax changes, so you probably need to adjust your
> rsyslog.conf later. I do not like to be forced to keep supporting this even
> if we go into a totally different direction (and I think this experimental
> solution is not clean in the light of what has changed since I implemented
> it).
>
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Rainer Gerhards
> > Sent: Thursday, March 15, 2012 11:08 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > I'll add an option, hopefully later today. Should be fairly simple ;)
> > Please also think a little bit about the MongoDB interface, which is a
> > bit
> > trickier (I don'T like the idea of formatting JSON to immediately parse
> > it
> > again, even though this may be a working first instance solution).
> >
> > Thx,
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Brian Knox
> > > Sent: Thursday, March 15, 2012 11:04 AM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > > I posted this on the blog as well -
> > >
> > > For my immediate needs, being able to JSON escape properties as a
> > > property
> > > option is satisfactory. I'm experimenting with templates such as:
> > >
> > > $template cee_enhanced,"@cee:
> > >
> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > e\":\"%timestamp:::date-
> > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > >
> > > Which works for a lot of my messages, the escaping is the only issue.
> > > Adding a JSON escape property option would allow me to move pretty
> > > quickly
> > > as I prototype things.
> > >
> > > Longer term, having some sort of JSON call where I could pass a
> > > property
> > > list would be nice... although the templates do give me the
> > flexibility
> > > of
> > > specifying nested document structures, which might get trickier. I'll
> > > think
> > > more about it.
> > >
> > >
> > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > <rgerhards [at] hq>wrote:
> > >
> > > > Hi all,
> > > >
> > > > another question I am facing while implementing the new stuff:
> > > >
> > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > > >
> > > > Feedback is appreciated :-)
> > > >
> > > > Rainer
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 3:29 AM

Post #7 of 50 (1138 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

[bknox [at] arthu ~]$ logger "this is a \"test\" of escaping"

@cee:
{"Event":{"p_proc":"rsyslogd-2212","p_sys":"arthur","time":"2012-03-15T06:27:38.747950-04:00"},"Msg":{"raw_msg":"imtcp:
module loaded, but no listeners defined - no input will be gathered [try
http://www.rsyslog.com/e/2212 ]"}}
@cee:
{"Event":{"p_proc":"bknox","p_sys":"arthur","time":"2012-03-15T06:27:47.374394-04:00"},"Msg":{"raw_msg":"<13>Mar
15 06:27:47 bknox: this is a \"test\" of escaping"}}

Well that definitely at least gets me up and running this morning :)

Brian

On Thu, Mar 15, 2012 at 6:21 AM, Brian Knox <briank [at] talksum> wrote:

> Oh, hah! I'll give this a try this morning then. :)
>
> Brian
>
>
> On Thu, Mar 15, 2012 at 6:19 AM, Rainer Gerhards <rgerhards [at] hq
> > wrote:
>
>> Lol... some month ago, I was smart enough to add a ",json" option to the
>> template system, which seems to do what you urgently need. It's not a real
>> JSON-encoder, but handles the quote char. So:
>>
>> $template cee_enhanced,"@cee:
>>
>> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"time\":\"
>> %timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n",json
>> *.* /var/log/cee_events.log;cee_enhanced
>>
>> Note the end of the $template line!
>>
>> ... should do what you need to get started. This is very experimental
>> code,
>> and I think it will not properly work with already-JSON escaped data
>> (like in
>> $!all-json!) and it will also not do proper full JSON escaping. But it
>> really
>> looks like it is what you right now need for your PoC ;)
>>
>> But please expect syntax changes, so you probably need to adjust your
>> rsyslog.conf later. I do not like to be forced to keep supporting this
>> even
>> if we go into a totally different direction (and I think this experimental
>> solution is not clean in the light of what has changed since I implemented
>> it).
>>
>> Rainer
>>
>> > -----Original Message-----
>> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > bounces [at] lists] On Behalf Of Rainer Gerhards
>> > Sent: Thursday, March 15, 2012 11:08 AM
>> > To: rsyslog-users
>> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>> >
>> > I'll add an option, hopefully later today. Should be fairly simple ;)
>> > Please also think a little bit about the MongoDB interface, which is a
>> > bit
>> > trickier (I don'T like the idea of formatting JSON to immediately parse
>> > it
>> > again, even though this may be a working first instance solution).
>> >
>> > Thx,
>> > Rainer
>> >
>> > > -----Original Message-----
>> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > > bounces [at] lists] On Behalf Of Brian Knox
>> > > Sent: Thursday, March 15, 2012 11:04 AM
>> > > To: rsyslog-users
>> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>> > >
>> > > I posted this on the blog as well -
>> > >
>> > > For my immediate needs, being able to JSON escape properties as a
>> > > property
>> > > option is satisfactory. I'm experimenting with templates such as:
>> > >
>> > > $template cee_enhanced,"@cee:
>> > >
>> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
>> > > e\":\"%timestamp:::date-
>> > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
>> > > 26 *.* /var/log/cee_events.log;cee_enhanced
>> > >
>> > > Which works for a lot of my messages, the escaping is the only issue.
>> > > Adding a JSON escape property option would allow me to move pretty
>> > > quickly
>> > > as I prototype things.
>> > >
>> > > Longer term, having some sort of JSON call where I could pass a
>> > > property
>> > > list would be nice... although the templates do give me the
>> > flexibility
>> > > of
>> > > specifying nested document structures, which might get trickier. I'll
>> > > think
>> > > more about it.
>> > >
>> > >
>> > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
>> > > <rgerhards [at] hq>wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > another question I am facing while implementing the new stuff:
>> > > >
>> > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>> > > >
>> > > > Feedback is appreciated :-)
>> > > >
>> > > > Rainer
>> > > > _______________________________________________
>> > > > rsyslog mailing list
>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > http://www.rsyslog.com/professional-services/
>> > > >
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 3:39 AM

Post #8 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Thursday, March 15, 2012 11:21 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> I think the MongoDB question ties into a larger question, of external
> sources that use text based protocols in general. For instance, Redis,
> which does not use JSON but does use a text based protocol over TCP.

Well... not exactly. MongoDB is a bit special as it does not work with JSON
natively but JSON. So I (the output plugin) must create a BSON document, even
if JSON is already present. If I could just simply provide the JSON doc,
there would be limited double processing. Even with pure JSON there would be
duplication effort if the document as such needs to be changed prior to
submitting. However, there I do not see a way around.

> In my case, our primary concern when interacting with external data
> stores
> is always performance over convenience. Double parsing a JSON document
> (first into CEE, and then from CEE into a MongoDB command) would
> definitely
> be a performance hit.

Yep

>
> Thinking about it though, I would like a solution that allows
> formatting
> the message independently of the specific output. For instance - in
> the
> past I've connected rsyslog to mongodb by passing the message over ZMQ
> to a
> daemon that handled the mongodb loading.

That's what the template system does, and it works well. However, the current
system is not doing well at field-like mapping. In essence, it generates
strings (either as one large or an array of them), but it does not provide
mapping information. In SQL datasources that is no problem because we
generate the actual insert command. But I think we need to have some other
way for the JSON structure, maybe something along the lines of explicit
name/value pairs. I have to admit I have no clear idea, and to make it even
harder I'd like to have a couple of ideas where at least one would be doable
without too much modification in order to go to a PoC quickly. Maybe the
double-formatting is an interim solution.

In any case, what would a clean solution look like?

Rainer
>
> I'll have to think about all of this more after I have some coffee!
>
> Brian
>
> On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
> <rgerhards [at] hq>wrote:
>
> > I'll add an option, hopefully later today. Should be fairly simple ;)
> > Please also think a little bit about the MongoDB interface, which is
> a bit
> > trickier (I don'T like the idea of formatting JSON to immediately
> parse it
> > again, even though this may be a working first instance solution).
> >
> > Thx,
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Brian Knox
> > > Sent: Thursday, March 15, 2012 11:04 AM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > > I posted this on the blog as well -
> > >
> > > For my immediate needs, being able to JSON escape properties as a
> > > property
> > > option is satisfactory. I'm experimenting with templates such as:
> > >
> > > $template cee_enhanced,"@cee:
> > >
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > e\":\"%timestamp:::date-
> > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > >
> > > Which works for a lot of my messages, the escaping is the only
> issue.
> > > Adding a JSON escape property option would allow me to move pretty
> > > quickly
> > > as I prototype things.
> > >
> > > Longer term, having some sort of JSON call where I could pass a
> > > property
> > > list would be nice... although the templates do give me the
> flexibility
> > > of
> > > specifying nested document structures, which might get trickier.
> I'll
> > > think
> > > more about it.
> > >
> > >
> > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > <rgerhards [at] hq>wrote:
> > >
> > > > Hi all,
> > > >
> > > > another question I am facing while implementing the new stuff:
> > > >
> > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > > >
> > > > Feedback is appreciated :-)
> > > >
> > > > Rainer
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 3:44 AM

Post #9 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

What immediately comes to mind for me is working with python dictionaries,
and various serialization formats. Basically, a hierarchical data
structure with various serializers. How that translates into something
workable within an rsyslog configuration, I'm not sure yet.

Would we agree the basic requirements would be:

1. The ability to define a hierarchical key / value structure out of
rsyslog properties
2. The ability to define serialization modules that would output those
structures

On Thu, Mar 15, 2012 at 6:39 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Brian Knox
> > Sent: Thursday, March 15, 2012 11:21 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > I think the MongoDB question ties into a larger question, of external
> > sources that use text based protocols in general. For instance, Redis,
> > which does not use JSON but does use a text based protocol over TCP.
>
> Well... not exactly. MongoDB is a bit special as it does not work with JSON
> natively but JSON. So I (the output plugin) must create a BSON document,
> even
> if JSON is already present. If I could just simply provide the JSON doc,
> there would be limited double processing. Even with pure JSON there would
> be
> duplication effort if the document as such needs to be changed prior to
> submitting. However, there I do not see a way around.
>
> > In my case, our primary concern when interacting with external data
> > stores
> > is always performance over convenience. Double parsing a JSON document
> > (first into CEE, and then from CEE into a MongoDB command) would
> > definitely
> > be a performance hit.
>
> Yep
>
> >
> > Thinking about it though, I would like a solution that allows
> > formatting
> > the message independently of the specific output. For instance - in
> > the
> > past I've connected rsyslog to mongodb by passing the message over ZMQ
> > to a
> > daemon that handled the mongodb loading.
>
> That's what the template system does, and it works well. However, the
> current
> system is not doing well at field-like mapping. In essence, it generates
> strings (either as one large or an array of them), but it does not provide
> mapping information. In SQL datasources that is no problem because we
> generate the actual insert command. But I think we need to have some other
> way for the JSON structure, maybe something along the lines of explicit
> name/value pairs. I have to admit I have no clear idea, and to make it even
> harder I'd like to have a couple of ideas where at least one would be
> doable
> without too much modification in order to go to a PoC quickly. Maybe the
> double-formatting is an interim solution.
>
> In any case, what would a clean solution look like?
>
> Rainer
> >
> > I'll have to think about all of this more after I have some coffee!
> >
> > Brian
> >
> > On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
> > <rgerhards [at] hq>wrote:
> >
> > > I'll add an option, hopefully later today. Should be fairly simple ;)
> > > Please also think a little bit about the MongoDB interface, which is
> > a bit
> > > trickier (I don'T like the idea of formatting JSON to immediately
> > parse it
> > > again, even though this may be a working first instance solution).
> > >
> > > Thx,
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > Sent: Thursday, March 15, 2012 11:04 AM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > > >
> > > > I posted this on the blog as well -
> > > >
> > > > For my immediate needs, being able to JSON escape properties as a
> > > > property
> > > > option is satisfactory. I'm experimenting with templates such as:
> > > >
> > > > $template cee_enhanced,"@cee:
> > > >
> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > > e\":\"%timestamp:::date-
> > > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > > >
> > > > Which works for a lot of my messages, the escaping is the only
> > issue.
> > > > Adding a JSON escape property option would allow me to move pretty
> > > > quickly
> > > > as I prototype things.
> > > >
> > > > Longer term, having some sort of JSON call where I could pass a
> > > > property
> > > > list would be nice... although the templates do give me the
> > flexibility
> > > > of
> > > > specifying nested document structures, which might get trickier.
> > I'll
> > > > think
> > > > more about it.
> > > >
> > > >
> > > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > > <rgerhards [at] hq>wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > another question I am facing while implementing the new stuff:
> > > > >
> > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > > > >
> > > > > Feedback is appreciated :-)
> > > > >
> > > > > Rainer
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 3:46 AM

Post #10 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Thursday, March 15, 2012 11:45 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> What immediately comes to mind for me is working with python
> dictionaries,
> and various serialization formats. Basically, a hierarchical data
> structure with various serializers. How that translates into something
> workable within an rsyslog configuration, I'm not sure yet.
>
> Would we agree the basic requirements would be:
>
> 1. The ability to define a hierarchical key / value structure out of
> rsyslog properties
> 2. The ability to define serialization modules that would output those
> structures

I think I can agree (though I would probably use somewhat different terms in
rsyslog speak - but let's start from here on).

Rainer

>
>
> On Thu, Mar 15, 2012 at 6:39 AM, Rainer Gerhards
> <rgerhards [at] hq>wrote:
>
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Brian Knox
> > > Sent: Thursday, March 15, 2012 11:21 AM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > > I think the MongoDB question ties into a larger question, of
> external
> > > sources that use text based protocols in general. For instance,
> Redis,
> > > which does not use JSON but does use a text based protocol over
> TCP.
> >
> > Well... not exactly. MongoDB is a bit special as it does not work
> with JSON
> > natively but JSON. So I (the output plugin) must create a BSON
> document,
> > even
> > if JSON is already present. If I could just simply provide the JSON
> doc,
> > there would be limited double processing. Even with pure JSON there
> would
> > be
> > duplication effort if the document as such needs to be changed prior
> to
> > submitting. However, there I do not see a way around.
> >
> > > In my case, our primary concern when interacting with external data
> > > stores
> > > is always performance over convenience. Double parsing a JSON
> document
> > > (first into CEE, and then from CEE into a MongoDB command) would
> > > definitely
> > > be a performance hit.
> >
> > Yep
> >
> > >
> > > Thinking about it though, I would like a solution that allows
> > > formatting
> > > the message independently of the specific output. For instance -
> in
> > > the
> > > past I've connected rsyslog to mongodb by passing the message over
> ZMQ
> > > to a
> > > daemon that handled the mongodb loading.
> >
> > That's what the template system does, and it works well. However, the
> > current
> > system is not doing well at field-like mapping. In essence, it
> generates
> > strings (either as one large or an array of them), but it does not
> provide
> > mapping information. In SQL datasources that is no problem because we
> > generate the actual insert command. But I think we need to have some
> other
> > way for the JSON structure, maybe something along the lines of
> explicit
> > name/value pairs. I have to admit I have no clear idea, and to make
> it even
> > harder I'd like to have a couple of ideas where at least one would be
> > doable
> > without too much modification in order to go to a PoC quickly. Maybe
> the
> > double-formatting is an interim solution.
> >
> > In any case, what would a clean solution look like?
> >
> > Rainer
> > >
> > > I'll have to think about all of this more after I have some coffee!
> > >
> > > Brian
> > >
> > > On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
> > > <rgerhards [at] hq>wrote:
> > >
> > > > I'll add an option, hopefully later today. Should be fairly
> simple ;)
> > > > Please also think a little bit about the MongoDB interface, which
> is
> > > a bit
> > > > trickier (I don'T like the idea of formatting JSON to immediately
> > > parse it
> > > > again, even though this may be a working first instance
> solution).
> > > >
> > > > Thx,
> > > > Rainer
> > > >
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > > Sent: Thursday, March 15, 2012 11:04 AM
> > > > > To: rsyslog-users
> > > > > Subject: Re: [rsyslog] next question: templates, JSON and
> MongoDB
> > > > >
> > > > > I posted this on the blog as well -
> > > > >
> > > > > For my immediate needs, being able to JSON escape properties as
> a
> > > > > property
> > > > > option is satisfactory. I'm experimenting with templates such
> as:
> > > > >
> > > > > $template cee_enhanced,"@cee:
> > > > >
> > >
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > > > e\":\"%timestamp:::date-
> > > > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > > > >
> > > > > Which works for a lot of my messages, the escaping is the only
> > > issue.
> > > > > Adding a JSON escape property option would allow me to move
> pretty
> > > > > quickly
> > > > > as I prototype things.
> > > > >
> > > > > Longer term, having some sort of JSON call where I could pass a
> > > > > property
> > > > > list would be nice... although the templates do give me the
> > > flexibility
> > > > > of
> > > > > specifying nested document structures, which might get
> trickier.
> > > I'll
> > > > > think
> > > > > more about it.
> > > > >
> > > > >
> > > > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > > > <rgerhards [at] hq>wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > another question I am facing while implementing the new
> stuff:
> > > > > >
> > > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-
> templates.html
> > > > > >
> > > > > > Feedback is appreciated :-)
> > > > > >
> > > > > > Rainer
> > > > > > _______________________________________________
> > > > > > rsyslog mailing list
> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > http://www.rsyslog.com/professional-services/
> > > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 4:13 AM

Post #11 of 50 (1138 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Just an amendment to be explicit: I believe we need the ability to define a
hierarchical data structure that can be a mix of rsyslog properties
(%programname%), the "dynamic" properties returned from mmnormalize and
other parsing modules (%$!foo%), and strings (p:Event).

We then need the ability to pass that structure to a "serializer", that
would then spit out JSON, or some other format.

I'm going to veer off into potentially crazy-talk territory now. If we
want the ability for an output action to spit out a serialized dump of this
hypothetical data structure, what's the potential for having output actions
translate to binary formats as well, say protobuf?

Brian

On Thu, Mar 15, 2012 at 6:46 AM, Rainer Gerhards
<rgerhards [at] hq>wrote:

>
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Brian Knox
> > Sent: Thursday, March 15, 2012 11:45 AM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > What immediately comes to mind for me is working with python
> > dictionaries,
> > and various serialization formats. Basically, a hierarchical data
> > structure with various serializers. How that translates into something
> > workable within an rsyslog configuration, I'm not sure yet.
> >
> > Would we agree the basic requirements would be:
> >
> > 1. The ability to define a hierarchical key / value structure out of
> > rsyslog properties
> > 2. The ability to define serialization modules that would output those
> > structures
>
> I think I can agree (though I would probably use somewhat different terms
> in
> rsyslog speak - but let's start from here on).
>
> Rainer
>
> >
> >
> > On Thu, Mar 15, 2012 at 6:39 AM, Rainer Gerhards
> > <rgerhards [at] hq>wrote:
> >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > Sent: Thursday, March 15, 2012 11:21 AM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > > >
> > > > I think the MongoDB question ties into a larger question, of
> > external
> > > > sources that use text based protocols in general. For instance,
> > Redis,
> > > > which does not use JSON but does use a text based protocol over
> > TCP.
> > >
> > > Well... not exactly. MongoDB is a bit special as it does not work
> > with JSON
> > > natively but JSON. So I (the output plugin) must create a BSON
> > document,
> > > even
> > > if JSON is already present. If I could just simply provide the JSON
> > doc,
> > > there would be limited double processing. Even with pure JSON there
> > would
> > > be
> > > duplication effort if the document as such needs to be changed prior
> > to
> > > submitting. However, there I do not see a way around.
> > >
> > > > In my case, our primary concern when interacting with external data
> > > > stores
> > > > is always performance over convenience. Double parsing a JSON
> > document
> > > > (first into CEE, and then from CEE into a MongoDB command) would
> > > > definitely
> > > > be a performance hit.
> > >
> > > Yep
> > >
> > > >
> > > > Thinking about it though, I would like a solution that allows
> > > > formatting
> > > > the message independently of the specific output. For instance -
> > in
> > > > the
> > > > past I've connected rsyslog to mongodb by passing the message over
> > ZMQ
> > > > to a
> > > > daemon that handled the mongodb loading.
> > >
> > > That's what the template system does, and it works well. However, the
> > > current
> > > system is not doing well at field-like mapping. In essence, it
> > generates
> > > strings (either as one large or an array of them), but it does not
> > provide
> > > mapping information. In SQL datasources that is no problem because we
> > > generate the actual insert command. But I think we need to have some
> > other
> > > way for the JSON structure, maybe something along the lines of
> > explicit
> > > name/value pairs. I have to admit I have no clear idea, and to make
> > it even
> > > harder I'd like to have a couple of ideas where at least one would be
> > > doable
> > > without too much modification in order to go to a PoC quickly. Maybe
> > the
> > > double-formatting is an interim solution.
> > >
> > > In any case, what would a clean solution look like?
> > >
> > > Rainer
> > > >
> > > > I'll have to think about all of this more after I have some coffee!
> > > >
> > > > Brian
> > > >
> > > > On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
> > > > <rgerhards [at] hq>wrote:
> > > >
> > > > > I'll add an option, hopefully later today. Should be fairly
> > simple ;)
> > > > > Please also think a little bit about the MongoDB interface, which
> > is
> > > > a bit
> > > > > trickier (I don'T like the idea of formatting JSON to immediately
> > > > parse it
> > > > > again, even though this may be a working first instance
> > solution).
> > > > >
> > > > > Thx,
> > > > > Rainer
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > > > Sent: Thursday, March 15, 2012 11:04 AM
> > > > > > To: rsyslog-users
> > > > > > Subject: Re: [rsyslog] next question: templates, JSON and
> > MongoDB
> > > > > >
> > > > > > I posted this on the blog as well -
> > > > > >
> > > > > > For my immediate needs, being able to JSON escape properties as
> > a
> > > > > > property
> > > > > > option is satisfactory. I'm experimenting with templates such
> > as:
> > > > > >
> > > > > > $template cee_enhanced,"@cee:
> > > > > >
> > > >
> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > > > > e\":\"%timestamp:::date-
> > > > > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > > > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > > > > >
> > > > > > Which works for a lot of my messages, the escaping is the only
> > > > issue.
> > > > > > Adding a JSON escape property option would allow me to move
> > pretty
> > > > > > quickly
> > > > > > as I prototype things.
> > > > > >
> > > > > > Longer term, having some sort of JSON call where I could pass a
> > > > > > property
> > > > > > list would be nice... although the templates do give me the
> > > > flexibility
> > > > > > of
> > > > > > specifying nested document structures, which might get
> > trickier.
> > > > I'll
> > > > > > think
> > > > > > more about it.
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > > > > <rgerhards [at] hq>wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > another question I am facing while implementing the new
> > stuff:
> > > > > > >
> > > > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-
> > templates.html
> > > > > > >
> > > > > > > Feedback is appreciated :-)
> > > > > > >
> > > > > > > Rainer
> > > > > > > _______________________________________________
> > > > > > > rsyslog mailing list
> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > > http://www.rsyslog.com/professional-services/
> > > > > > >
> > > > > > _______________________________________________
> > > > > > rsyslog mailing list
> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > http://www.rsyslog.com/professional-services/
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 6:00 AM

Post #12 of 50 (1138 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Thinking more about it:

As a user, I'd like the convenience I have of defining nested dicts in a
language like python, and calling json.dumps(foo), in an rsyslog config
friendly syntax. I'm not suggesting the below as the actual syntax, it's
just a pseudocode example:

foo = new event
foo["app"] = %programname%
foo["time"] = %timestamp%
bar = new event
bar["event"] = foo
*.* /var/log/message;bar,json

Warning - I'll go a big extreme if you ask me for feature ideas - what I'd
really ultimately like is a lua interpreter embedded in rsyslog that has
access to reading and setting properties ;)

Brian

On Thu, Mar 15, 2012 at 7:13 AM, Brian Knox <briank [at] talksum> wrote:

> Just an amendment to be explicit: I believe we need the ability to define
> a hierarchical data structure that can be a mix of rsyslog properties
> (%programname%), the "dynamic" properties returned from mmnormalize and
> other parsing modules (%$!foo%), and strings (p:Event).
>
> We then need the ability to pass that structure to a "serializer", that
> would then spit out JSON, or some other format.
>
> I'm going to veer off into potentially crazy-talk territory now. If we
> want the ability for an output action to spit out a serialized dump of this
> hypothetical data structure, what's the potential for having output actions
> translate to binary formats as well, say protobuf?
>
> Brian
>
> On Thu, Mar 15, 2012 at 6:46 AM, Rainer Gerhards <rgerhards [at] hq
> > wrote:
>
>>
>>
>> > -----Original Message-----
>> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > bounces [at] lists] On Behalf Of Brian Knox
>> > Sent: Thursday, March 15, 2012 11:45 AM
>> > To: rsyslog-users
>> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>> >
>> > What immediately comes to mind for me is working with python
>> > dictionaries,
>> > and various serialization formats. Basically, a hierarchical data
>> > structure with various serializers. How that translates into something
>> > workable within an rsyslog configuration, I'm not sure yet.
>> >
>> > Would we agree the basic requirements would be:
>> >
>> > 1. The ability to define a hierarchical key / value structure out of
>> > rsyslog properties
>> > 2. The ability to define serialization modules that would output those
>> > structures
>>
>> I think I can agree (though I would probably use somewhat different terms
>> in
>> rsyslog speak - but let's start from here on).
>>
>> Rainer
>>
>> >
>> >
>> > On Thu, Mar 15, 2012 at 6:39 AM, Rainer Gerhards
>> > <rgerhards [at] hq>wrote:
>> >
>> > > > -----Original Message-----
>> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > > > bounces [at] lists] On Behalf Of Brian Knox
>> > > > Sent: Thursday, March 15, 2012 11:21 AM
>> > > > To: rsyslog-users
>> > > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>> > > >
>> > > > I think the MongoDB question ties into a larger question, of
>> > external
>> > > > sources that use text based protocols in general. For instance,
>> > Redis,
>> > > > which does not use JSON but does use a text based protocol over
>> > TCP.
>> > >
>> > > Well... not exactly. MongoDB is a bit special as it does not work
>> > with JSON
>> > > natively but JSON. So I (the output plugin) must create a BSON
>> > document,
>> > > even
>> > > if JSON is already present. If I could just simply provide the JSON
>> > doc,
>> > > there would be limited double processing. Even with pure JSON there
>> > would
>> > > be
>> > > duplication effort if the document as such needs to be changed prior
>> > to
>> > > submitting. However, there I do not see a way around.
>> > >
>> > > > In my case, our primary concern when interacting with external data
>> > > > stores
>> > > > is always performance over convenience. Double parsing a JSON
>> > document
>> > > > (first into CEE, and then from CEE into a MongoDB command) would
>> > > > definitely
>> > > > be a performance hit.
>> > >
>> > > Yep
>> > >
>> > > >
>> > > > Thinking about it though, I would like a solution that allows
>> > > > formatting
>> > > > the message independently of the specific output. For instance -
>> > in
>> > > > the
>> > > > past I've connected rsyslog to mongodb by passing the message over
>> > ZMQ
>> > > > to a
>> > > > daemon that handled the mongodb loading.
>> > >
>> > > That's what the template system does, and it works well. However, the
>> > > current
>> > > system is not doing well at field-like mapping. In essence, it
>> > generates
>> > > strings (either as one large or an array of them), but it does not
>> > provide
>> > > mapping information. In SQL datasources that is no problem because we
>> > > generate the actual insert command. But I think we need to have some
>> > other
>> > > way for the JSON structure, maybe something along the lines of
>> > explicit
>> > > name/value pairs. I have to admit I have no clear idea, and to make
>> > it even
>> > > harder I'd like to have a couple of ideas where at least one would be
>> > > doable
>> > > without too much modification in order to go to a PoC quickly. Maybe
>> > the
>> > > double-formatting is an interim solution.
>> > >
>> > > In any case, what would a clean solution look like?
>> > >
>> > > Rainer
>> > > >
>> > > > I'll have to think about all of this more after I have some coffee!
>> > > >
>> > > > Brian
>> > > >
>> > > > On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
>> > > > <rgerhards [at] hq>wrote:
>> > > >
>> > > > > I'll add an option, hopefully later today. Should be fairly
>> > simple ;)
>> > > > > Please also think a little bit about the MongoDB interface, which
>> > is
>> > > > a bit
>> > > > > trickier (I don'T like the idea of formatting JSON to immediately
>> > > > parse it
>> > > > > again, even though this may be a working first instance
>> > solution).
>> > > > >
>> > > > > Thx,
>> > > > > Rainer
>> > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> > > > > > bounces [at] lists] On Behalf Of Brian Knox
>> > > > > > Sent: Thursday, March 15, 2012 11:04 AM
>> > > > > > To: rsyslog-users
>> > > > > > Subject: Re: [rsyslog] next question: templates, JSON and
>> > MongoDB
>> > > > > >
>> > > > > > I posted this on the blog as well -
>> > > > > >
>> > > > > > For my immediate needs, being able to JSON escape properties as
>> > a
>> > > > > > property
>> > > > > > option is satisfactory. I'm experimenting with templates such
>> > as:
>> > > > > >
>> > > > > > $template cee_enhanced,"@cee:
>> > > > > >
>> > > >
>> > {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
>> > > > > > e\":\"%timestamp:::date-
>> > > > > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
>> > > > > > 26 *.* /var/log/cee_events.log;cee_enhanced
>> > > > > >
>> > > > > > Which works for a lot of my messages, the escaping is the only
>> > > > issue.
>> > > > > > Adding a JSON escape property option would allow me to move
>> > pretty
>> > > > > > quickly
>> > > > > > as I prototype things.
>> > > > > >
>> > > > > > Longer term, having some sort of JSON call where I could pass a
>> > > > > > property
>> > > > > > list would be nice... although the templates do give me the
>> > > > flexibility
>> > > > > > of
>> > > > > > specifying nested document structures, which might get
>> > trickier.
>> > > > I'll
>> > > > > > think
>> > > > > > more about it.
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
>> > > > > > <rgerhards [at] hq>wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > another question I am facing while implementing the new
>> > stuff:
>> > > > > > >
>> > > > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-
>> > templates.html
>> > > > > > >
>> > > > > > > Feedback is appreciated :-)
>> > > > > > >
>> > > > > > > Rainer
>> > > > > > > _______________________________________________
>> > > > > > > rsyslog mailing list
>> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > > > > http://www.rsyslog.com/professional-services/
>> > > > > > >
>> > > > > > _______________________________________________
>> > > > > > rsyslog mailing list
>> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > > > http://www.rsyslog.com/professional-services/
>> > > > > _______________________________________________
>> > > > > rsyslog mailing list
>> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > > http://www.rsyslog.com/professional-services/
>> > > > >
>> > > > _______________________________________________
>> > > > rsyslog mailing list
>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > > http://www.rsyslog.com/professional-services/
>> > > _______________________________________________
>> > > rsyslog mailing list
>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > > http://www.rsyslog.com/professional-services/
>> > >
>> > _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

vladg at illinois

Mar 15, 2012, 6:28 AM

Post #13 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Well, it seems to me that basically what we're looking for is the ability to pass field names to the current properties. So, would something like this suffice?

$template DNSasJSON,"%timegenerated [at] timestam:::date-rfc3339% %fromhost [at] clien% %$!query$@query%",json

Note - I don't think that @ is currently being used, but the point stands with a different symbol. This template would produce:

{\"timestamp\": \"2012-03-16T06:39:03-06:00\", \"client\": \"192.168.100.1\", \"query\": \"google.com\"}

On 3/15/12 5:39 AM, Rainer Gerhards wrote:
> That's what the template system does, and it works well. However, the current
> system is not doing well at field-like mapping. In essence, it generates
> strings (either as one large or an array of them), but it does not provide
> mapping information. In SQL datasources that is no problem because we
> generate the actual insert command. But I think we need to have some other
> way for the JSON structure, maybe something along the lines of explicit
> name/value pairs. I have to admit I have no clear idea, and to make it even
> harder I'd like to have a couple of ideas where at least one would be doable
> without too much modification in order to go to a PoC quickly. Maybe the
> double-formatting is an interim solution.
>
> In any case, what would a clean solution look like?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 6:53 AM

Post #14 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Vlad Grigorescu
> Sent: Thursday, March 15, 2012 2:29 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> Well, it seems to me that basically what we're looking for is the
> ability to pass field names to the current properties. So, would
> something like this suffice?
>
> $template DNSasJSON,"%timegenerated [at] timestam:::date-rfc3339%
> %fromhost [at] clien% %$!query$@query%",json
>
> Note - I don't think that @ is currently being used, but the point
> stands with a different symbol. This template would produce:
>
> {\"timestamp\": \"2012-03-16T06:39:03-06:00\", \"client\":
> \"192.168.100.1\", \"query\": \"google.com\"}
>
I'Ve not gone through the other mails right now, but this sounds like a
really cool idea. Structure could be expressed in dotted notation (e.g. "a.b"
is a field b below an a object).

Rainer

> On 3/15/12 5:39 AM, Rainer Gerhards wrote:
> > That's what the template system does, and it works well. However, the
> current
> > system is not doing well at field-like mapping. In essence, it
> generates
> > strings (either as one large or an array of them), but it does not
> provide
> > mapping information. In SQL datasources that is no problem because we
> > generate the actual insert command. But I think we need to have some
> other
> > way for the JSON structure, maybe something along the lines of
> explicit
> > name/value pairs. I have to admit I have no clear idea, and to make
> it even
> > harder I'd like to have a couple of ideas where at least one would be
> doable
> > without too much modification in order to go to a PoC quickly. Maybe
> the
> > double-formatting is an interim solution.
> >
> > In any case, what would a clean solution look like?
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

radu0gheorghe at gmail

Mar 15, 2012, 7:02 AM

Post #15 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Hi Rainer,

I've added my comment on your blog. I guess there's no need to
double-post it here.

Now that I've read the other suggestions, I think users should be able
to choose from two scenarios:
- one easy-to-use and less error-prone way of defining templates, like
Vlad suggested
- one that's none of the above, but it's more flexible, which can be
the way it's currently implemented: you write your own JSONs, and
rsyslog only does the escaping

Best regards,
Radu

2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> Hi all,
>
> another question I am facing while implementing the new stuff:
>
> http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>
> Feedback is appreciated :-)
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 8:55 AM

Post #16 of 50 (1135 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Radu Gheorghe
> Sent: Thursday, March 15, 2012 3:03 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> Hi Rainer,
>
> I've added my comment on your blog. I guess there's no need to
> double-post it here.
>
> Now that I've read the other suggestions, I think users should be able
> to choose from two scenarios:
> - one easy-to-use and less error-prone way of defining templates, like
> Vlad suggested
> - one that's none of the above, but it's more flexible, which can be
> the way it's currently implemented: you write your own JSONs, and
> rsyslog only does the escaping

Thanks! Will reply to some details on your comment tomorrow, but I really
like the hybrid model you propose. I guess that's a good solution.

Very productive day today :-) Thanks everyone!

Rainer
>
> Best regards,
> Radu
>
> 2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> > Hi all,
> >
> > another question I am facing while implementing the new stuff:
> >
> > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> >
> > Feedback is appreciated :-)
> >
> > Rainer
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 9:46 AM

Post #17 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

I thought I give upgrading the property replacer a try tomorrow if I am not
distracted. Preparing for this, question comes up. As I said, ",json" is a
bit ugly when fields are already in JSON format (like $!all-json). It may
also have more overhead than actually required (need to verify, though).

As an alternative, would it be OK to have the json formatting option on a
per-field basis. This would also be inline with the other formatting options,
but more elaborate to specify. Using the sample template this would come down
to:

$template cee_enhanced,"@cee:
{\"Event\":{\"p_proc\":\"%programname:::json%\",\"p_sys\":\"%hostname:::json%
\",\"time\":\"%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg:::
json%\"}}\n"

Note that %timestamp:::date-rfc3339,json% would work, but I did not do that
because there is no need for special encoding with this timestamp format
(this is also an example where CPU cycles could be spared).

How does this sound?
Rainer

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Thursday, March 15, 2012 4:56 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
>
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Radu Gheorghe
> > Sent: Thursday, March 15, 2012 3:03 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> > Hi Rainer,
> >
> > I've added my comment on your blog. I guess there's no need to
> > double-post it here.
> >
> > Now that I've read the other suggestions, I think users should be
> able
> > to choose from two scenarios:
> > - one easy-to-use and less error-prone way of defining templates,
> like
> > Vlad suggested
> > - one that's none of the above, but it's more flexible, which can be
> > the way it's currently implemented: you write your own JSONs, and
> > rsyslog only does the escaping
>
> Thanks! Will reply to some details on your comment tomorrow, but I
> really
> like the hybrid model you propose. I guess that's a good solution.
>
> Very productive day today :-) Thanks everyone!
>
> Rainer
> >
> > Best regards,
> > Radu
> >
> > 2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> > > Hi all,
> > >
> > > another question I am facing while implementing the new stuff:
> > >
> > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > >
> > > Feedback is appreciated :-)
> > >
> > > Rainer
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

briank at talksum

Mar 15, 2012, 9:50 AM

Post #18 of 50 (1136 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Having them work as a property option is how I originally envisioned it
working... so while it is a little bit more elaborate, it's fine by me.

Brian

On Thu, Mar 15, 2012 at 12:46 PM, Rainer Gerhards
<rgerhards [at] hq>wrote:

> I thought I give upgrading the property replacer a try tomorrow if I am not
> distracted. Preparing for this, question comes up. As I said, ",json" is a
> bit ugly when fields are already in JSON format (like $!all-json). It may
> also have more overhead than actually required (need to verify, though).
>
> As an alternative, would it be OK to have the json formatting option on a
> per-field basis. This would also be inline with the other formatting
> options,
> but more elaborate to specify. Using the sample template this would come
> down
> to:
>
> $template cee_enhanced,"@cee:
>
> {\"Event\":{\"p_proc\":\"%programname:::json%\",\"p_sys\":\"%hostname:::json%
>
> \",\"time\":\"%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg:::
> json%\"}}\n"
>
> Note that %timestamp:::date-rfc3339,json% would work, but I did not do that
> because there is no need for special encoding with this timestamp format
> (this is also an example where CPU cycles could be spared).
>
> How does this sound?
> Rainer
>
> > -----Original Message-----
> > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > bounces [at] lists] On Behalf Of Rainer Gerhards
> > Sent: Thursday, March 15, 2012 4:56 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> >
> >
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Radu Gheorghe
> > > Sent: Thursday, March 15, 2012 3:03 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > > Hi Rainer,
> > >
> > > I've added my comment on your blog. I guess there's no need to
> > > double-post it here.
> > >
> > > Now that I've read the other suggestions, I think users should be
> > able
> > > to choose from two scenarios:
> > > - one easy-to-use and less error-prone way of defining templates,
> > like
> > > Vlad suggested
> > > - one that's none of the above, but it's more flexible, which can be
> > > the way it's currently implemented: you write your own JSONs, and
> > > rsyslog only does the escaping
> >
> > Thanks! Will reply to some details on your comment tomorrow, but I
> > really
> > like the hybrid model you propose. I guess that's a good solution.
> >
> > Very productive day today :-) Thanks everyone!
> >
> > Rainer
> > >
> > > Best regards,
> > > Radu
> > >
> > > 2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> > > > Hi all,
> > > >
> > > > another question I am facing while implementing the new stuff:
> > > >
> > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
> > > >
> > > > Feedback is appreciated :-)
> > > >
> > > > Rainer
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 9:52 AM

Post #19 of 50 (1139 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Thursday, March 15, 2012 5:50 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> Having them work as a property option is how I originally envisioned it
> working... so while it is a little bit more elaborate, it's fine by me.

Yeah that ",json" option was really a quick hack at the time I did it. So
I'll probably see to fix this up tomorrow (if nobody jumps in with a good
counter-argument ;)).

Thanks!
Rainer

>
> Brian
>
> On Thu, Mar 15, 2012 at 12:46 PM, Rainer Gerhards
> <rgerhards [at] hq>wrote:
>
> > I thought I give upgrading the property replacer a try tomorrow if I
> am not
> > distracted. Preparing for this, question comes up. As I said, ",json"
> is a
> > bit ugly when fields are already in JSON format (like $!all-json). It
> may
> > also have more overhead than actually required (need to verify,
> though).
> >
> > As an alternative, would it be OK to have the json formatting option
> on a
> > per-field basis. This would also be inline with the other formatting
> > options,
> > but more elaborate to specify. Using the sample template this would
> come
> > down
> > to:
> >
> > $template cee_enhanced,"@cee:
> >
> >
> {\"Event\":{\"p_proc\":\"%programname:::json%\",\"p_sys\":\"%hostname::
> :json%
> >
> > \",\"time\":\"%timestamp:::date-
> rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg:::
> > json%\"}}\n"
> >
> > Note that %timestamp:::date-rfc3339,json% would work, but I did not
> do that
> > because there is no need for special encoding with this timestamp
> format
> > (this is also an example where CPU cycles could be spared).
> >
> > How does this sound?
> > Rainer
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Rainer Gerhards
> > > Sent: Thursday, March 15, 2012 4:56 PM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > bounces [at] lists] On Behalf Of Radu Gheorghe
> > > > Sent: Thursday, March 15, 2012 3:03 PM
> > > > To: rsyslog-users
> > > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > > >
> > > > Hi Rainer,
> > > >
> > > > I've added my comment on your blog. I guess there's no need to
> > > > double-post it here.
> > > >
> > > > Now that I've read the other suggestions, I think users should be
> > > able
> > > > to choose from two scenarios:
> > > > - one easy-to-use and less error-prone way of defining templates,
> > > like
> > > > Vlad suggested
> > > > - one that's none of the above, but it's more flexible, which can
> be
> > > > the way it's currently implemented: you write your own JSONs, and
> > > > rsyslog only does the escaping
> > >
> > > Thanks! Will reply to some details on your comment tomorrow, but I
> > > really
> > > like the hybrid model you propose. I guess that's a good solution.
> > >
> > > Very productive day today :-) Thanks everyone!
> > >
> > > Rainer
> > > >
> > > > Best regards,
> > > > Radu
> > > >
> > > > 2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> > > > > Hi all,
> > > > >
> > > > > another question I am facing while implementing the new stuff:
> > > > >
> > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-
> templates.html
> > > > >
> > > > > Feedback is appreciated :-)
> > > > >
> > > > > Rainer
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

rgerhards at hq

Mar 15, 2012, 10:13 AM

Post #20 of 50 (1138 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Thursday, March 15, 2012 12:13 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>
> Just an amendment to be explicit: I believe we need the to
> define a
> hierarchical data structure that can be a mix of rsyslog properties
> (%programname%), the "dynamic" properties returned from mmnormalize and
> other parsing modules (%$!foo%), and strings (p:Event).

That's the idea, but I would still like to physically keep the properties
separate. The reason simply is performance. If they are shuffled into a
single data structure, "traditional" property access would be notably slower.
If I duplicate them... well, then I duplicate them (copy cost, mem cost).

> We then need the ability to pass that structure to a "serializer", that
> would then spit out JSON, or some other format.

Yep, that's actually what I am after. Replace "other format" with "BSON" and
you get the picture. That beast must be capable to take the different sets of
properties and mimic - with as low runtime overhead as possible - them as
being in a single structure. At that point, I admit, it may be necessary to
combine and copy things. On the other hand, there are probably situations
(Mongo/BSON) where it could be avoided (even if it is by the sheer fact that
copying into BSON is required in any case...).
>
> I'm going to veer off into potentially crazy-talk territory now. If we
> want the ability for an output action to spit out a serialized dump of
> this
> hypothetical data structure, what's the potential for having output
> actions
> translate to binary formats as well, say protobuf?

BSON is binary. So ommongodb will probably provide a good template. That's
exactly why I am so interested in doing it right there.

Given the rest of today's discussion, I think we are on a good path. Of
course, this needs to be proven in practice when the experimental code is
ready for (ab)use :-)

Rainer
>
> Brian
>
> On Thu, Mar 15, 2012 at 6:46 AM, Rainer Gerhards
> <rgerhards [at] hq>wrote:
>
> >
> >
> > > -----Original Message-----
> > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > bounces [at] lists] On Behalf Of Brian Knox
> > > Sent: Thursday, March 15, 2012 11:45 AM
> > > To: rsyslog-users
> > > Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
> > >
> > > What immediately comes to mind for me is working with python
> > > dictionaries,
> > > and various serialization formats. Basically, a hierarchical data
> > > structure with various serializers. How that translates into
> something
> > > workable within an rsyslog configuration, I'm not sure yet.
> > >
> > > Would we agree the basic requirements would be:
> > >
> > > 1. The ability to define a hierarchical key / value structure out
> of
> > > rsyslog properties
> > > 2. The ability to define serialization modules that would output
> those
> > > structures
> >
> > I think I can agree (though I would probably use somewhat different
> terms
> > in
> > rsyslog speak - but let's start from here on).
> >
> > Rainer
> >
> > >
> > >
> > > On Thu, Mar 15, 2012 at 6:39 AM, Rainer Gerhards
> > > <rgerhards [at] hq>wrote:
> > >
> > > > > -----Original Message-----
> > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > > Sent: Thursday, March 15, 2012 11:21 AM
> > > > > To: rsyslog-users
> > > > > Subject: Re: [rsyslog] next question: templates, JSON and
> MongoDB
> > > > >
> > > > > I think the MongoDB question ties into a larger question, of
> > > external
> > > > > sources that use text based protocols in general. For
> instance,
> > > Redis,
> > > > > which does not use JSON but does use a text based protocol over
> > > TCP.
> > > >
> > > > Well... not exactly. MongoDB is a bit special as it does not work
> > > with JSON
> > > > natively but JSON. So I (the output plugin) must create a BSON
> > > document,
> > > > even
> > > > if JSON is already present. If I could just simply provide the
> JSON
> > > doc,
> > > > there would be limited double processing. Even with pure JSON
> there
> > > would
> > > > be
> > > > duplication effort if the document as such needs to be changed
> prior
> > > to
> > > > submitting. However, there I do not see a way around.
> > > >
> > > > > In my case, our primary concern when interacting with external
> data
> > > > > stores
> > > > > is always performance over convenience. Double parsing a JSON
> > > document
> > > > > (first into CEE, and then from CEE into a MongoDB command)
> would
> > > > > definitely
> > > > > be a performance hit.
> > > >
> > > > Yep
> > > >
> > > > >
> > > > > Thinking about it though, I would like a solution that allows
> > > > > formatting
> > > > > the message independently of the specific output. For instance
> -
> > > in
> > > > > the
> > > > > past I've connected rsyslog to mongodb by passing the message
> over
> > > ZMQ
> > > > > to a
> > > > > daemon that handled the mongodb loading.
> > > >
> > > > That's what the template system does, and it works well. However,
> the
> > > > current
> > > > system is not doing well at field-like mapping. In essence, it
> > > generates
> > > > strings (either as one large or an array of them), but it does
> not
> > > provide
> > > > mapping information. In SQL datasources that is no problem
> because we
> > > > generate the actual insert command. But I think we need to have
> some
> > > other
> > > > way for the JSON structure, maybe something along the lines of
> > > explicit
> > > > name/value pairs. I have to admit I have no clear idea, and to
> make
> > > it even
> > > > harder I'd like to have a couple of ideas where at least one
> would be
> > > > doable
> > > > without too much modification in order to go to a PoC quickly.
> Maybe
> > > the
> > > > double-formatting is an interim solution.
> > > >
> > > > In any case, what would a clean solution look like?
> > > >
> > > > Rainer
> > > > >
> > > > > I'll have to think about all of this more after I have some
> coffee!
> > > > >
> > > > > Brian
> > > > >
> > > > > On Thu, Mar 15, 2012 at 6:08 AM, Rainer Gerhards
> > > > > <rgerhards [at] hq>wrote:
> > > > >
> > > > > > I'll add an option, hopefully later today. Should be fairly
> > > simple ;)
> > > > > > Please also think a little bit about the MongoDB interface,
> which
> > > is
> > > > > a bit
> > > > > > trickier (I don'T like the idea of formatting JSON to
> immediately
> > > > > parse it
> > > > > > again, even though this may be a working first instance
> > > solution).
> > > > > >
> > > > > > Thx,
> > > > > > Rainer
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: rsyslog-bounces [at] lists [mailto:rsyslog-
> > > > > > > bounces [at] lists] On Behalf Of Brian Knox
> > > > > > > Sent: Thursday, March 15, 2012 11:04 AM
> > > > > > > To: rsyslog-users
> > > > > > > Subject: Re: [rsyslog] next question: templates, JSON and
> > > MongoDB
> > > > > > >
> > > > > > > I posted this on the blog as well -
> > > > > > >
> > > > > > > For my immediate needs, being able to JSON escape
> properties as
> > > a
> > > > > > > property
> > > > > > > option is satisfactory. I'm experimenting with templates
> such
> > > as:
> > > > > > >
> > > > > > > $template cee_enhanced,"@cee:
> > > > > > >
> > > > >
> > >
> {\"Event\":{\"p_proc\":\"%programname%\",\"p_sys\":\"%hostname%\",\"tim
> > > > > > > e\":\"%timestamp:::date-
> > > > > > > rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg%\"}}\n"
> > > > > > > 26 *.* /var/log/cee_events.log;cee_enhanced
> > > > > > >
> > > > > > > Which works for a lot of my messages, the escaping is the
> only
> > > > > issue.
> > > > > > > Adding a JSON escape property option would allow me to move
> > > pretty
> > > > > > > quickly
> > > > > > > as I prototype things.
> > > > > > >
> > > > > > > Longer term, having some sort of JSON call where I could
> pass a
> > > > > > > property
> > > > > > > list would be nice... although the templates do give me the
> > > > > flexibility
> > > > > > > of
> > > > > > > specifying nested document structures, which might get
> > > trickier.
> > > > > I'll
> > > > > > > think
> > > > > > > more about it.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 15, 2012 at 5:46 AM, Rainer Gerhards
> > > > > > > <rgerhards [at] hq>wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > another question I am facing while implementing the new
> > > stuff:
> > > > > > > >
> > > > > > > > http://blog.gerhards.net/2012/03/json-and-rsyslog-
> > > templates.html
> > > > > > > >
> > > > > > > > Feedback is appreciated :-)
> > > > > > > >
> > > > > > > > Rainer
> > > > > > > > _______________________________________________
> > > > > > > > rsyslog mailing list
> > > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > > > http://www.rsyslog.com/professional-services/
> > > > > > > >
> > > > > > > _______________________________________________
> > > > > > > rsyslog mailing list
> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > > http://www.rsyslog.com/professional-services/
> > > > > > _______________________________________________
> > > > > > rsyslog mailing list
> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > > http://www.rsyslog.com/professional-services/
> > > > > >
> > > > > _______________________________________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > > http://www.rsyslog.com/professional-services/
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Mar 15, 2012, 11:49 AM

Post #21 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

On Thu, 15 Mar 2012, Rainer Gerhards wrote:

> Hi all,
>
> another question I am facing while implementing the new stuff:
>
> http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>
> Feedback is appreciated :-)

I think the minumum is to provide a json escaping mechanism. I expect that
there are always going to be cases where something 'odd' needs to be done
(or json needs to be encapsulated inside something else)

I also think that it would be useful to have an way to specify a json
element (name and value), but specify it in such a way that if the value
is blank the entire element disappears from the output.

I also expect that full json formatting is going to be rather complicated
to define in a traditional format string, and therefor expensive to parse
and format.

As a result, I think that there is a need to have a C-based formatter
module (like the built-in formats) to output to json (probably a couple
variations, an 'everything' mode and a 'cee' mode at minimum)

I can see how you could provide a list of properties to a json formatting
module to output a flat list of name=value type things, but I'm having
problems figuring out how you would sanely define a nested structure (so
you could for example have a 'server' item that contained hostname, FQDN,
reverse DNS lookup, and IP address, or a subset of these if you don't know
them all)

As additional structured logging options become available, I expect that
there will be a need to do a very similar thing in formats other than
json. A couple of examples are RESTful interfaces, or SQL interfaces. In
all three cases you have a need to have some sort of structure with a
bunch of different properties in them. As a result I think that you need
to be able to pass a structure to the output module for it to format.

While I can see a good reason to have JSON support nested structures, I
don't see a similar need for RESTful or SQL interfaces, I think that both
of them should be able to work just fine if they can be passed a list of
parameters to include in their output.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Mar 15, 2012, 12:08 PM

Post #22 of 50 (1140 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

On Thu, 15 Mar 2012, Brian Knox wrote:

> What immediately comes to mind for me is working with python dictionaries,
> and various serialization formats. Basically, a hierarchical data
> structure with various serializers. How that translates into something
> workable within an rsyslog configuration, I'm not sure yet.
>
> Would we agree the basic requirements would be:
>
> 1. The ability to define a hierarchical key / value structure out of
> rsyslog properties
> 2. The ability to define serialization modules that would output those
> structures

I think that splitting the task into these two phases is incredibly
useful.

rsyslog already has the logical equivalent of the serialization module in
the form of the C generated templates. This will probably need to be
extended, and it may make sense to make this configurable along the lines
of the parser modules.

This gets to item #1, and there things seem to get harder.

If we just need a flat key/value structure, this seems fairly easy, just
define a 'list' type that lets you specify a list of names (i.e. keys) and
properties (i.e. values). this could be implemented as a list of pointers
to the contents that get passed to the serialzation module.

if you want to make this hierartchical, this gets more complicated as you
also need to define the parent for each name (or that there is no parent).
I saw the other post that suggested defining a.b to make b a child of a,
that seems like it should work reasonably well (at the cost of making '.'
a special character that cannot be used in a field name)

one question, do you need to specify the entire hierarchy, or just the
parent. If you have the relationship a.b and b.c, is that good enough, or
would you need to specify a.b and a.b.c)

I would consider something along the lines of @"name","value","parent" to
specify each item, with parent being optional

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Mar 15, 2012, 12:39 PM

Post #23 of 50 (1136 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

On Thu, 15 Mar 2012, Rainer Gerhards wrote:

> I thought I give upgrading the property replacer a try tomorrow if I am not
> distracted. Preparing for this, question comes up. As I said, ",json" is a
> bit ugly when fields are already in JSON format (like $!all-json). It may
> also have more overhead than actually required (need to verify, though).
>
> As an alternative, would it be OK to have the json formatting option on a
> per-field basis. This would also be inline with the other formatting options,
> but more elaborate to specify. Using the sample template this would come down
> to:
>
> $template cee_enhanced,"@cee:
> {\"Event\":{\"p_proc\":\"%programname:::json%\",\"p_sys\":\"%hostname:::json%
> \",\"time\":\"%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg:::
> json%\"}}\n"
>
> Note that %timestamp:::date-rfc3339,json% would work, but I did not do that
> because there is no need for special encoding with this timestamp format
> (this is also an example where CPU cycles could be spared).
>
> How does this sound?

I think having JSON formatting on a per-field basis is just about a
requirement. There are going to be cases where you want to insert JSON
formatted data into a SQL database for example.

David Lang

> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Rainer Gerhards
>> Sent: Thursday, March 15, 2012 4:56 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>>
>>
>>
>>> -----Original Message-----
>>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>>> bounces [at] lists] On Behalf Of Radu Gheorghe
>>> Sent: Thursday, March 15, 2012 3:03 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] next question: templates, JSON and MongoDB
>>>
>>> Hi Rainer,
>>>
>>> I've added my comment on your blog. I guess there's no need to
>>> double-post it here.
>>>
>>> Now that I've read the other suggestions, I think users should be
>> able
>>> to choose from two scenarios:
>>> - one easy-to-use and less error-prone way of defining templates,
>> like
>>> Vlad suggested
>>> - one that's none of the above, but it's more flexible, which can be
>>> the way it's currently implemented: you write your own JSONs, and
>>> rsyslog only does the escaping
>>
>> Thanks! Will reply to some details on your comment tomorrow, but I
>> really
>> like the hybrid model you propose. I guess that's a good solution.
>>
>> Very productive day today :-) Thanks everyone!
>>
>> Rainer
>>>
>>> Best regards,
>>> Radu
>>>
>>> 2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
>>>> Hi all,
>>>>
>>>> another question I am facing while implementing the new stuff:
>>>>
>>>> http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>>>>
>>>> Feedback is appreciated :-)
>>>>
>>>> Rainer
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

david at lang

Mar 15, 2012, 12:41 PM

Post #24 of 50 (1136 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

On Thu, 15 Mar 2012, david [at] lang wrote:

> On Thu, 15 Mar 2012, Rainer Gerhards wrote:
>
>> I thought I give upgrading the property replacer a try tomorrow if I am not
>> distracted. Preparing for this, question comes up. As I said, ",json" is a
>> bit ugly when fields are already in JSON format (like $!all-json). It may
>> also have more overhead than actually required (need to verify, though).
>>
>> As an alternative, would it be OK to have the json formatting option on a
>> per-field basis. This would also be inline with the other formatting
>> options,
>> but more elaborate to specify. Using the sample template this would come
>> down
>> to:
>>
>> $template cee_enhanced,"@cee:
>> {\"Event\":{\"p_proc\":\"%programname:::json%\",\"p_sys\":\"%hostname:::json%
>> \",\"time\":\"%timestamp:::date-rfc3339%\"},\"Msg\":{\"raw_msg\":\"%rawmsg:::
>> json%\"}}\n"
>>
>> Note that %timestamp:::date-rfc3339,json% would work, but I did not do that
>> because there is no need for special encoding with this timestamp format
>> (this is also an example where CPU cycles could be spared).
>>
>> How does this sound?
>
> I think having JSON formatting on a per-field basis is just about a
> requirement. There are going to be cases where you want to insert JSON
> formatted data into a SQL database for example.

A silly thought I just had, how hard would it be to allow for a format
string to reference another format string? (basically treat it as you
would a property)

If this can be done, it would seem to be a very powerful (if possibly
inefficient) tool for keeping the formats understandable, as well as
maintaining consistancy for cases where you are putting the same data to
different destinations.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

christian at brunner-muc

Mar 15, 2012, 1:16 PM

Post #25 of 50 (1137 views)
Permalink

Re: next question: templates, JSON and MongoDB [In reply to]

Hi Rainer,

did you take a look at my JSON strgen module?

http://lists.adiscon.net/pipermail/rsyslog/2011-June/013383.html

I think adding key:value pairs is much easier than manually crafting
the template string.

The only disadvantage I'm seeing is, that it isn't possible to specify
a hierarchy (parent/child).

Regards,
Christian

2012/3/15 Rainer Gerhards <rgerhards [at] hq>:
> Hi all,
>
> another question I am facing while implementing the new stuff:
>
> http://blog.gerhards.net/2012/03/json-and-rsyslog-templates.html
>
> Feedback is appreciated :-)
>
> Rainer
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads