Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

Rsyslog splitting long-lines into multiple smaller one

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


gentoo at floriancrouzat

Apr 19, 2012, 6:30 AM

Post #1 of 18 (2497 views)
Permalink
Rsyslog splitting long-lines into multiple smaller one

Hi,

This is my first message to the list so please be kind ;)
For the short version of the question, go the bottom.

I'm using CentOS, my tests have been made using 5.6 with
rsyslog-3.22.1-3.el5_6.1 but I aim to install 6.2 with
rsyslog-4.6.2-12.el6.x86_64 ... Old version in both case yeah...

Java log through syslog using a syslog appender in log4j.
I emulated a standard log line format using a log4j pattern, and it
works just fine. Rsyslog add "timestamp %msg" and log4j add "hostname
jboss: blablabla...".

Eg: Apr 19 15:11:10 host.example.com jboss: INFO [ ]
[StandardPctxCacheExitHandler ] - handling standard transaction
expiration for cache id 06570496120419CJ4YAB1

I'm using a log analyser (ossec) who knows how to decode a jboss log
when matching its pattern, but with long-lines, it seems that something
truncate them and create multiple-lines. The first ends with "...".
It means that each splitted line isn't logged through log4j and doesn't
have the pattern I defined ==> I can't parse it, ossec goes crazy, and a
bunch of stuff don't work.

My question in short: is there a way to tell rsyslogd not to split my
long-lines into different smaller ones ?
I tried 3 different thing:
$template JbossFormattest1,"%timegenerated% %msg:0:$%\n"
$template JbossFormattest3,"%timegenerated% %msg:0:3000%\n"
$template JbossFormattest2,"%timegenerated% %msg:drop-cc:%\n"

Sadly, none of them worked.

I'm hoping for some ENV variables, and not to recompile changing a
#define and/or tweaking my kernel.

In the worst case, I'll extract my dynamic log4j pattern in a different
rsyslog templates per server as %HOSTNAME% would be localhost.localdomain.


--
Cheers,
Florian Crouzat
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


jrhett at netconsonance

Apr 19, 2012, 10:37 AM

Post #2 of 18 (2456 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

I can't answer your question, but I can give you a spec file for the latest 5.8 version for CentOS 5 if you want.

On Apr 19, 2012, at 6:30 AM, Florian Crouzat wrote:
> This is my first message to the list so please be kind ;)
> For the short version of the question, go the bottom.
>
> I'm using CentOS, my tests have been made using 5.6 with rsyslog-3.22.1-3.el5_6.1 but I aim to install 6.2 with rsyslog-4.6.2-12.el6.x86_64 ... Old version in both case yeah...
>
> Java log through syslog using a syslog appender in log4j.
> I emulated a standard log line format using a log4j pattern, and it works just fine. Rsyslog add "timestamp %msg" and log4j add "hostname jboss: blablabla...".
>
> Eg: Apr 19 15:11:10 host.example.com jboss: INFO [ ] [StandardPctxCacheExitHandler ] - handling standard transaction expiration for cache id 06570496120419CJ4YAB1
>
> I'm using a log analyser (ossec) who knows how to decode a jboss log when matching its pattern, but with long-lines, it seems that something truncate them and create multiple-lines. The first ends with "...".
> It means that each splitted line isn't logged through log4j and doesn't have the pattern I defined ==> I can't parse it, ossec goes crazy, and a bunch of stuff don't work.
>
> My question in short: is there a way to tell rsyslogd not to split my long-lines into different smaller ones ?
> I tried 3 different thing:
> $template JbossFormattest1,"%timegenerated% %msg:0:$%\n"
> $template JbossFormattest3,"%timegenerated% %msg:0:3000%\n"
> $template JbossFormattest2,"%timegenerated% %msg:drop-cc:%\n"
>
> Sadly, none of them worked.
>
> I'm hoping for some ENV variables, and not to recompile changing a #define and/or tweaking my kernel.
>
> In the worst case, I'll extract my dynamic log4j pattern in a different rsyslog templates per server as %HOSTNAME% would be localhost.localdomain.
>
>
> --
> Cheers,
> Florian Crouzat
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards

--
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other randomness

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


trefalgar at realitybytes

Apr 19, 2012, 10:51 AM

Post #3 of 18 (2455 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Maybe I'm missing something, but your description of "..." followed by
multiple lines of errors, thus split lines and not a single line, is
standard for java logging. Are you sure what you're seeing isn't just
the expected output from java?

Jacob

Quoting Jo Rhett <jrhett [at] netconsonance>:

> I can't answer your question, but I can give you a spec file for the
> latest 5.8 version for CentOS 5 if you want.
>
> On Apr 19, 2012, at 6:30 AM, Florian Crouzat wrote:
>> This is my first message to the list so please be kind ;)
>> For the short version of the question, go the bottom.
>>
>> I'm using CentOS, my tests have been made using 5.6 with
>> rsyslog-3.22.1-3.el5_6.1 but I aim to install 6.2 with
>> rsyslog-4.6.2-12.el6.x86_64 ... Old version in both case yeah...
>>
>> Java log through syslog using a syslog appender in log4j.
>> I emulated a standard log line format using a log4j pattern, and it
>> works just fine. Rsyslog add "timestamp %msg" and log4j add
>> "hostname jboss: blablabla...".
>>
>> Eg: Apr 19 15:11:10 host.example.com jboss: INFO [ ]
>> [StandardPctxCacheExitHandler ] - handling standard transaction
>> expiration for cache id 06570496120419CJ4YAB1
>>
>> I'm using a log analyser (ossec) who knows how to decode a jboss
>> log when matching its pattern, but with long-lines, it seems that
>> something truncate them and create multiple-lines. The first ends
>> with "...".
>> It means that each splitted line isn't logged through log4j and
>> doesn't have the pattern I defined ==> I can't parse it, ossec goes
>> crazy, and a bunch of stuff don't work.
>>
>> My question in short: is there a way to tell rsyslogd not to split
>> my long-lines into different smaller ones ?
>> I tried 3 different thing:
>> $template JbossFormattest1,"%timegenerated% %msg:0:$%\n"
>> $template JbossFormattest3,"%timegenerated% %msg:0:3000%\n"
>> $template JbossFormattest2,"%timegenerated% %msg:drop-cc:%\n"
>>
>> Sadly, none of them worked.
>>
>> I'm hoping for some ENV variables, and not to recompile changing a
>> #define and/or tweaking my kernel.
>>
>> In the worst case, I'll extract my dynamic log4j pattern in a
>> different rsyslog templates per server as %HOSTNAME% would be
>> localhost.localdomain.
>>
>>
>> --
>> Cheers,
>> Florian Crouzat
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
> --
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source
> and other randomness
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 19, 2012, 12:05 PM

Post #4 of 18 (2461 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

there is a max message size parameter that you can set in rsyslog, make
sure it's set long enough for your messages.

David Lang

On Thu, 19 Apr 2012, Florian Crouzat wrote:

> Date: Thu, 19 Apr 2012 15:30:04 +0200
> From: Florian Crouzat <gentoo [at] floriancrouzat>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: rsyslog [at] lists
> Subject: [rsyslog] Rsyslog splitting long-lines into multiple smaller one
>
> Hi,
>
> This is my first message to the list so please be kind ;)
> For the short version of the question, go the bottom.
>
> I'm using CentOS, my tests have been made using 5.6 with
> rsyslog-3.22.1-3.el5_6.1 but I aim to install 6.2 with
> rsyslog-4.6.2-12.el6.x86_64 ... Old version in both case yeah...
>
> Java log through syslog using a syslog appender in log4j.
> I emulated a standard log line format using a log4j pattern, and it works
> just fine. Rsyslog add "timestamp %msg" and log4j add "hostname jboss:
> blablabla...".
>
> Eg: Apr 19 15:11:10 host.example.com jboss: INFO [ ]
> [StandardPctxCacheExitHandler ] - handling standard transaction expiration
> for cache id 06570496120419CJ4YAB1
>
> I'm using a log analyser (ossec) who knows how to decode a jboss log when
> matching its pattern, but with long-lines, it seems that something truncate
> them and create multiple-lines. The first ends with "...".
> It means that each splitted line isn't logged through log4j and doesn't have
> the pattern I defined ==> I can't parse it, ossec goes crazy, and a bunch of
> stuff don't work.
>
> My question in short: is there a way to tell rsyslogd not to split my
> long-lines into different smaller ones ?
> I tried 3 different thing:
> $template JbossFormattest1,"%timegenerated% %msg:0:$%\n"
> $template JbossFormattest3,"%timegenerated% %msg:0:3000%\n"
> $template JbossFormattest2,"%timegenerated% %msg:drop-cc:%\n"
>
> Sadly, none of them worked.
>
> I'm hoping for some ENV variables, and not to recompile changing a #define
> and/or tweaking my kernel.
>
> In the worst case, I'll extract my dynamic log4j pattern in a different
> rsyslog templates per server as %HOSTNAME% would be localhost.localdomain.
>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 19, 2012, 10:54 PM

Post #5 of 18 (2472 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Jacob Steinberger
> Sent: Thursday, April 19, 2012 7:52 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> smaller one
>
> Maybe I'm missing something, but your description of "..." followed by
> multiple lines of errors, thus split lines and not a single line, is
> standard for java logging. Are you sure what you're seeing isn't just
> the expected output from java?

The problem is that the syslog appenders are seriously broken. The emit
malformed messages with wrong framing. The end result is what is being
reported here.

As part of a training project, we have hacked together some appender which
worked at least in lab (YMMV). See here:

http://www.rsyslog.com/tcp-syslog-rfc5424-log4j-appender/

Rainer
>
> Jacob
>
> Quoting Jo Rhett <jrhett [at] netconsonance>:
>
> > I can't answer your question, but I can give you a spec file for the
> > latest 5.8 version for CentOS 5 if you want.
> >
> > On Apr 19, 2012, at 6:30 AM, Florian Crouzat wrote:
> >> This is my first message to the list so please be kind ;)
> >> For the short version of the question, go the bottom.
> >>
> >> I'm using CentOS, my tests have been made using 5.6 with
> >> rsyslog-3.22.1-3.el5_6.1 but I aim to install 6.2 with
> >> rsyslog-4.6.2-12.el6.x86_64 ... Old version in both case yeah...
> >>
> >> Java log through syslog using a syslog appender in log4j.
> >> I emulated a standard log line format using a log4j pattern, and it
> >> works just fine. Rsyslog add "timestamp %msg" and log4j add
> >> "hostname jboss: blablabla...".
> >>
> >> Eg: Apr 19 15:11:10 host.example.com jboss: INFO [ ]
> >> [StandardPctxCacheExitHandler ] - handling standard transaction
> >> expiration for cache id 06570496120419CJ4YAB1
> >>
> >> I'm using a log analyser (ossec) who knows how to decode a jboss
> >> log when matching its pattern, but with long-lines, it seems that
> >> something truncate them and create multiple-lines. The first ends
> >> with "...".
> >> It means that each splitted line isn't logged through log4j and
> >> doesn't have the pattern I defined ==> I can't parse it, ossec goes
> >> crazy, and a bunch of stuff don't work.
> >>
> >> My question in short: is there a way to tell rsyslogd not to split
> >> my long-lines into different smaller ones ?
> >> I tried 3 different thing:
> >> $template JbossFormattest1,"%timegenerated% %msg:0:$%\n"
> >> $template JbossFormattest3,"%timegenerated% %msg:0:3000%\n"
> >> $template JbossFormattest2,"%timegenerated% %msg:drop-cc:%\n"
> >>
> >> Sadly, none of them worked.
> >>
> >> I'm hoping for some ENV variables, and not to recompile changing a
> >> #define and/or tweaking my kernel.
> >>
> >> In the worst case, I'll extract my dynamic log4j pattern in a
> >> different rsyslog templates per server as %HOSTNAME% would be
> >> localhost.localdomain.
> >>
> >>
> >> --
> >> Cheers,
> >> Florian Crouzat
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >
> > --
> > Jo Rhett
> > Net Consonance : consonant endings by net philanthropy, open source
> > and other randomness
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >
>
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


gentoo at floriancrouzat

Apr 20, 2012, 12:32 AM

Post #6 of 18 (2456 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Le 20/04/2012 07:54, Rainer Gerhards a écrit :
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Jacob Steinberger
>> Sent: Thursday, April 19, 2012 7:52 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
>> smaller one
>>
>> Maybe I'm missing something, but your description of "..." followed by
>> multiple lines of errors, thus split lines and not a single line, is
>> standard for java logging. Are you sure what you're seeing isn't just
>> the expected output from java?

Well, I have a log4j FILE appender that cohabits with the syslog
appender for the same logs. When java issue a long log line, the FILE
appender logs it just fine whereas the syslog appender logs
multiple-lines for it (first one ending with '...').

What makes me think it's not java/log4j but syslog is that all of the
multiple lines except the first one aren't prefixed with my log4j layout
as I explained in my initial post.

> The problem is that the syslog appenders are seriously broken. The emit
> malformed messages with wrong framing. The end result is what is being
> reported here.
>
> As part of a training project, we have hacked together some appender which
> worked at least in lab (YMMV). See here:
>
> http://www.rsyslog.com/tcp-syslog-rfc5424-log4j-appender/

I'll see with my dev about that appender, and wether or not we can have
such a thing in production, as I assume it's not maintained and never
updated ? Not sure other appenders are though.

Thanks.

--
Cheers,
Florian Crouzat
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


gentoo at floriancrouzat

Apr 20, 2012, 12:47 AM

Post #7 of 18 (2449 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Le 19/04/2012 21:05, david [at] lang a écrit :
> there is a max message size parameter that you can set in rsyslog, make
> sure it's set long enough for your messages.
>
> David Lang
>

I made some tests with $MaxMessageSize 3K as the log4j syslog appender
forces me to use UDP , it didn't work. I also tried 32K for the record.
Maybe this is related to my old rsyslog version but I tend to blame java.

I'll look into other proposal that have been made.

Thanks.

--
Cheers,
Florian Crouzat
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


briank at talksum

Apr 20, 2012, 2:56 AM

Post #8 of 18 (2452 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Have you tried this: http://syslog4j.org/ ?

It at the least support TCP.


On Fri, Apr 20, 2012 at 3:47 AM, Florian Crouzat
<gentoo [at] floriancrouzat>wrote:

> Le 19/04/2012 21:05, david [at] lang a écrit :
>
> there is a max message size parameter that you can set in rsyslog, make
>> sure it's set long enough for your messages.
>>
>> David Lang
>>
>>
> I made some tests with $MaxMessageSize 3K as the log4j syslog appender
> forces me to use UDP , it didn't work. I also tried 32K for the record.
> Maybe this is related to my old rsyslog version but I tend to blame java.
>
> I'll look into other proposal that have been made.
>
>
> Thanks.
>
> --
> Cheers,
> Florian Crouzat
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 20, 2012, 2:57 AM

Post #9 of 18 (2457 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Brian Knox
> Sent: Friday, April 20, 2012 11:56 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> smaller one
>
> Have you tried this: http://syslog4j.org/ ?
>
> It at the least support TCP.

This is where our implementation is based on. As far as I remember, there
were issues with pure syslog4j as well.

Rainer
>
>
> On Fri, Apr 20, 2012 at 3:47 AM, Florian Crouzat
> <gentoo [at] floriancrouzat>wrote:
>
> > Le 19/04/2012 21:05, david [at] lang a écrit :
> >
> > there is a max message size parameter that you can set in rsyslog,
> make
> >> sure it's set long enough for your messages.
> >>
> >> David Lang
> >>
> >>
> > I made some tests with $MaxMessageSize 3K as the log4j syslog
> appender
> > forces me to use UDP , it didn't work. I also tried 32K for the
> record.
> > Maybe this is related to my old rsyslog version but I tend to blame
> java.
> >
> > I'll look into other proposal that have been made.
> >
> >
> > Thanks.
> >
> > --
> > Cheers,
> > Florian Crouzat
> > ______________________________**_________________
> > rsyslog mailing list
> >
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adisco
> n.net/mailman/listinfo/rsyslog>
> > http://www.rsyslog.com/**professional-
> services/<http://www.rsyslog.com/professional-services/>
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 20, 2012, 3:01 AM

Post #10 of 18 (2460 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Florian Crouzat
> Sent: Friday, April 20, 2012 9:48 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> smaller one
>
> Le 19/04/2012 21:05, david [at] lang a écrit :
> > there is a max message size parameter that you can set in rsyslog,
> make
> > sure it's set long enough for your messages.
> >
> > David Lang
> >
>
> I made some tests with $MaxMessageSize 3K as the log4j syslog appender
> forces me to use UDP , it didn't work. I also tried 32K for the record.
> Maybe this is related to my old rsyslog version but I tend to blame
> java.

Read the source fort he UPD appender: it deliberately truncates at 1k.

Rainer
>
> I'll look into other proposal that have been made.
>
> Thanks.
>
> --
> Cheers,
> Florian Crouzat
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 20, 2012, 3:04 AM

Post #11 of 18 (2484 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Florian Crouzat
> Sent: Friday, April 20, 2012 9:32 AM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> smaller one
>
> Le 20/04/2012 07:54, Rainer Gerhards a écrit :
> >> -----Original Message-----
> >> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> >> bounces [at] lists] On Behalf Of Jacob Steinberger
> >> Sent: Thursday, April 19, 2012 7:52 PM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> >> smaller one
> >>
> >> Maybe I'm missing something, but your description of "..." followed
> by
> >> multiple lines of errors, thus split lines and not a single line, is
> >> standard for java logging. Are you sure what you're seeing isn't
> just
> >> the expected output from java?
>
> Well, I have a log4j FILE appender that cohabits with the syslog
> appender for the same logs. When java issue a long log line, the FILE
> appender logs it just fine whereas the syslog appender logs
> multiple-lines for it (first one ending with '...').
>
> What makes me think it's not java/log4j but syslog is that all of the
> multiple lines except the first one aren't prefixed with my log4j
> layout
> as I) explained in my initial post.

That's what I said: the framing is incorrect. It neither works as far as the
industry standard is concerned (by introducing \n inside the message - as
soon as it is published, you can read about that in RFC6587) nor does it use
standard framing (RFC5425). So it's simply broken.

>
> > The problem is that the syslog appenders are seriously broken. The
> emit
> > malformed messages with wrong framing. The end result is what is
> being
> > reported here.
> >
> > As part of a training project, we have hacked together some appender
> which
> > worked at least in lab (YMMV). See here:
> >
> > http://www.rsyslog.com/tcp-syslog-rfc5424-log4j-appender/
>
> I'll see with my dev about that appender, and wether or not we can have
> such a thing in production, as I assume it's not maintained and never
> updated ? Not sure other appenders are though.

Too early to say. If there is sufficient interest and a user base, it's
status may be promoted. I am not a java wiz, so I have not yet really dug
deeper into that. Obviously I could if it turns out to be useful and nobody
else takes up that task.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


gentoo at floriancrouzat

Apr 20, 2012, 3:10 AM

Post #12 of 18 (2455 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Le 19/04/2012 15:30, Florian Crouzat a écrit :
>
>
> In the worst case, I'll extract my dynamic log4j pattern in a different
> rsyslog templates per server as %HOSTNAME% would be localhost.localdomain.

Finally, I came to do this.

Moving the part of the log4j layout that is used by my log analyser to
know it's jboss logs to my rsyslog template so that even a stacktrace
has each of it's line prefixed my "host.example.net jboss: at
java.foo.bar:13" and gets analyzed correctly.

In my working env, it's a better solution than using your custom log4j
appender Rainer.

Thanks for your propositions, and confirmation about the limitation of
UDP rsyslog message size, etc.

--
Florian Crouzat
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


rgerhards at hq

Apr 20, 2012, 3:12 AM

Post #13 of 18 (2456 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Florian Crouzat
> Sent: Friday, April 20, 2012 12:10 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Rsyslog splitting long-lines into multiple
> smaller one
>
> Le 19/04/2012 15:30, Florian Crouzat a écrit :
> >
> >
> > In the worst case, I'll extract my dynamic log4j pattern in a
> different
> > rsyslog templates per server as %HOSTNAME% would be
> localhost.localdomain.
>
> Finally, I came to do this.
>
> Moving the part of the log4j layout that is used by my log analyser to
> know it's jboss logs to my rsyslog template so that even a stacktrace
> has each of it's line prefixed my "host.example.net jboss: at
> java.foo.bar:13" and gets analyzed correctly.
>
> In my working env, it's a better solution than using your custom log4j
> appender Rainer.

Whatever works best for you :)

> Thanks for your propositions, and confirmation about the limitation of
> UDP rsyslog message size, etc.


I need to correct this. UDP syslog can use at least 4 to 8k without tweaking
and almost 64k with tweaking, as far as syslog the protocol and rsyslog is
involved. It is just the broken log4j syslog append that enforces this
ancient limit.

Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

May 17, 2013, 12:40 AM

Post #14 of 18 (2050 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

On Fri, 17 May 2013, Hanish Bansal wrote:

> I'm using CentOs 6.3.
> rsyslog version : rsyslogd 5.8.10
>
> I am maintaining Java logs through syslog using a syslog appender in log4j.
> But its creating multiple-lines for long lines.
>
> To avoid this i defined MaxMessageSize to 64k in "/etc/rsyslog.conf":
>
> $MaxMessageSize 32768
>
> $ModLoad imudp
>
> $UDPServerRun 514
>
> $ModLoad imtcp
>
> $InputTCPServerRun 514
>
> After that i restarted rsyslog.
> But its not working. Any suggestion?

I strongly suspect that the problem in on the log4j side. I believe that the
default log4j splits syslog messages at the 1K boundry and sends them as
multiple messages. You can see this if you do a 'tcpdump -s 0 -A port 514'

rsyslog has a log4j replacement up that fixes many of the problems in the stock
log4j http://www.rsyslog.com/tag/log4j/

there's also logback http://logback.qos.ch/ which is written by the original
author of log4j

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


hanish.bansal.agarwal at gmail

May 17, 2013, 4:44 AM

Post #15 of 18 (2137 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

In log4j.xml i used:

<appender name="syslog" class="org.apache.log4j.net.SyslogAppender">
<param name="facility" value="local1" />
<param name="facilityPrinting" value="true" />
<param name="syslogHost" value="localhost" />
<param name="threshold" value="debug" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%p] %c{1}:%L - %m%n" />
</layout>
</appender>

Is there any other appender that i can replace here to resolve the problem?



On Fri, May 17, 2013 at 7:40 AM, David Lang <david [at] lang> wrote:

> On Fri, 17 May 2013, Hanish Bansal wrote:
>
> I'm using CentOs 6.3.
>> rsyslog version : rsyslogd 5.8.10
>>
>> I am maintaining Java logs through syslog using a syslog appender in
>> log4j.
>> But its creating multiple-lines for long lines.
>>
>> To avoid this i defined MaxMessageSize to 64k in "/etc/rsyslog.conf":
>>
>> $MaxMessageSize 32768
>>
>> $ModLoad imudp
>>
>> $UDPServerRun 514
>>
>> $ModLoad imtcp
>>
>> $InputTCPServerRun 514
>>
>> After that i restarted rsyslog.
>> But its not working. Any suggestion?
>>
>
> I strongly suspect that the problem in on the log4j side. I believe that
> the default log4j splits syslog messages at the 1K boundry and sends them
> as multiple messages. You can see this if you do a 'tcpdump -s 0 -A port
> 514'
>
> rsyslog has a log4j replacement up that fixes many of the problems in the
> stock log4j http://www.rsyslog.com/tag/**log4j/<http://www.rsyslog.com/tag/log4j/>
>
> there's also logback http://logback.qos.ch/ which is written by the
> original author of log4j
>
> David Lang
> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>



--
*Thanks & Regards*
*Hanish Bansal*
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


david at lang

May 17, 2013, 7:03 AM

Post #16 of 18 (2066 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

I believe that both of theones I mentioned below are drop-in replacements.

http://www.rsyslog.com/tag/log4j/
http://logback.qos.ch/

David Lang

On Fri, 17 May 2013, Hanish Bansal wrote:

> In log4j.xml i used:
>
> <appender name="syslog" class="org.apache.log4j.net.SyslogAppender">
> <param name="facility" value="local1" />
> <param name="facilityPrinting" value="true" />
> <param name="syslogHost" value="localhost" />
> <param name="threshold" value="debug" />
> <layout class="org.apache.log4j.PatternLayout">
> <param name="ConversionPattern" value="[%p] %c{1}:%L - %m%n" />
> </layout>
> </appender>
>
> Is there any other appender that i can replace here to resolve the problem?
>
>
>
> On Fri, May 17, 2013 at 7:40 AM, David Lang <david [at] lang> wrote:
>
>> On Fri, 17 May 2013, Hanish Bansal wrote:
>>
>> I'm using CentOs 6.3.
>>> rsyslog version : rsyslogd 5.8.10
>>>
>>> I am maintaining Java logs through syslog using a syslog appender in
>>> log4j.
>>> But its creating multiple-lines for long lines.
>>>
>>> To avoid this i defined MaxMessageSize to 64k in "/etc/rsyslog.conf":
>>>
>>> $MaxMessageSize 32768
>>>
>>> $ModLoad imudp
>>>
>>> $UDPServerRun 514
>>>
>>> $ModLoad imtcp
>>>
>>> $InputTCPServerRun 514
>>>
>>> After that i restarted rsyslog.
>>> But its not working. Any suggestion?
>>>
>>
>> I strongly suspect that the problem in on the log4j side. I believe that
>> the default log4j splits syslog messages at the 1K boundry and sends them
>> as multiple messages. You can see this if you do a 'tcpdump -s 0 -A port
>> 514'
>>
>> rsyslog has a log4j replacement up that fixes many of the problems in the
>> stock log4j http://www.rsyslog.com/tag/**log4j/<http://www.rsyslog.com/tag/log4j/>
>>
>> there's also logback http://logback.qos.ch/ which is written by the
>> original author of log4j
>>
>> David Lang
>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


hanish.bansal.agarwal at gmail

May 19, 2013, 9:10 PM

Post #17 of 18 (2051 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

Hi
SyslogTCPAppender is working but as it is based on Syslog4JAppender.
And Syslog4JAppender is under "Lesser GNU Public License" . I can't use it.

So is there any other option for same in apache licensed "syslog" ?



On Fri, May 17, 2013 at 2:03 PM, David Lang <david [at] lang> wrote:

> I believe that both of theones I mentioned below are drop-in replacements.
>
> http://www.rsyslog.com/tag/**log4j/ <http://www.rsyslog.com/tag/log4j/>
> http://logback.qos.ch/
>
> David Lang
>
>
> On Fri, 17 May 2013, Hanish Bansal wrote:
>
> In log4j.xml i used:
>>
>> <appender name="syslog" class="org.apache.log4j.net.**SyslogAppender">
>> <param name="facility" value="local1" />
>> <param name="facilityPrinting" value="true" />
>> <param name="syslogHost" value="localhost" />
>> <param name="threshold" value="debug" />
>> <layout class="org.apache.log4j.**PatternLayout">
>> <param name="ConversionPattern" value="[%p] %c{1}:%L - %m%n" />
>> </layout>
>> </appender>
>>
>> Is there any other appender that i can replace here to resolve the
>> problem?
>>
>>
>>
>> On Fri, May 17, 2013 at 7:40 AM, David Lang <david [at] lang> wrote:
>>
>> On Fri, 17 May 2013, Hanish Bansal wrote:
>>>
>>> I'm using CentOs 6.3.
>>>
>>>> rsyslog version : rsyslogd 5.8.10
>>>>
>>>> I am maintaining Java logs through syslog using a syslog appender in
>>>> log4j.
>>>> But its creating multiple-lines for long lines.
>>>>
>>>> To avoid this i defined MaxMessageSize to 64k in "/etc/rsyslog.conf":
>>>>
>>>> $MaxMessageSize 32768
>>>>
>>>> $ModLoad imudp
>>>>
>>>> $UDPServerRun 514
>>>>
>>>> $ModLoad imtcp
>>>>
>>>> $InputTCPServerRun 514
>>>>
>>>> After that i restarted rsyslog.
>>>> But its not working. Any suggestion?
>>>>
>>>>
>>> I strongly suspect that the problem in on the log4j side. I believe that
>>> the default log4j splits syslog messages at the 1K boundry and sends them
>>> as multiple messages. You can see this if you do a 'tcpdump -s 0 -A port
>>> 514'
>>>
>>> rsyslog has a log4j replacement up that fixes many of the problems in the
>>> stock log4j http://www.rsyslog.com/tag/****log4j/<http://www.rsyslog.com/tag/**log4j/>
>>> <http://www.rsyslog.com/**tag/log4j/ <http://www.rsyslog.com/tag/log4j/>
>>> >
>>>
>>>
>>> there's also logback http://logback.qos.ch/ which is written by the
>>> original author of log4j
>>>
>>> David Lang
>>> ______________________________****_________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> >
>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>> >
>>>
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>>
>>
>>
>>
>> ______________________________**_________________
> rsyslog mailing list
> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>



--
*Thanks & Regards*
*Hanish Bansal*
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.


david at lang

May 20, 2013, 4:09 AM

Post #18 of 18 (2040 views)
Permalink
Re: Rsyslog splitting long-lines into multiple smaller one [In reply to]

On Mon, 20 May 2013, Hanish Bansal wrote:

> Hi
> SyslogTCPAppender is working but as it is based on Syslog4JAppender.
> And Syslog4JAppender is under "Lesser GNU Public License" . I can't use it.

out of curiosity, why is the LGPL not allowed in your organization? it imposes
no licensing restrictions on anything it's linked to.

David Lang

> So is there any other option for same in apache licensed "syslog" ?
>
>
>
> On Fri, May 17, 2013 at 2:03 PM, David Lang <david [at] lang> wrote:
>
>> I believe that both of theones I mentioned below are drop-in replacements.
>>
>> http://www.rsyslog.com/tag/**log4j/ <http://www.rsyslog.com/tag/log4j/>
>> http://logback.qos.ch/
>>
>> David Lang
>>
>>
>> On Fri, 17 May 2013, Hanish Bansal wrote:
>>
>> In log4j.xml i used:
>>>
>>> <appender name="syslog" class="org.apache.log4j.net.**SyslogAppender">
>>> <param name="facility" value="local1" />
>>> <param name="facilityPrinting" value="true" />
>>> <param name="syslogHost" value="localhost" />
>>> <param name="threshold" value="debug" />
>>> <layout class="org.apache.log4j.**PatternLayout">
>>> <param name="ConversionPattern" value="[%p] %c{1}:%L - %m%n" />
>>> </layout>
>>> </appender>
>>>
>>> Is there any other appender that i can replace here to resolve the
>>> problem?
>>>
>>>
>>>
>>> On Fri, May 17, 2013 at 7:40 AM, David Lang <david [at] lang> wrote:
>>>
>>> On Fri, 17 May 2013, Hanish Bansal wrote:
>>>>
>>>> I'm using CentOs 6.3.
>>>>
>>>>> rsyslog version : rsyslogd 5.8.10
>>>>>
>>>>> I am maintaining Java logs through syslog using a syslog appender in
>>>>> log4j.
>>>>> But its creating multiple-lines for long lines.
>>>>>
>>>>> To avoid this i defined MaxMessageSize to 64k in "/etc/rsyslog.conf":
>>>>>
>>>>> $MaxMessageSize 32768
>>>>>
>>>>> $ModLoad imudp
>>>>>
>>>>> $UDPServerRun 514
>>>>>
>>>>> $ModLoad imtcp
>>>>>
>>>>> $InputTCPServerRun 514
>>>>>
>>>>> After that i restarted rsyslog.
>>>>> But its not working. Any suggestion?
>>>>>
>>>>>
>>>> I strongly suspect that the problem in on the log4j side. I believe that
>>>> the default log4j splits syslog messages at the 1K boundry and sends them
>>>> as multiple messages. You can see this if you do a 'tcpdump -s 0 -A port
>>>> 514'
>>>>
>>>> rsyslog has a log4j replacement up that fixes many of the problems in the
>>>> stock log4j http://www.rsyslog.com/tag/****log4j/<http://www.rsyslog.com/tag/**log4j/>
>>>> <http://www.rsyslog.com/**tag/log4j/ <http://www.rsyslog.com/tag/log4j/>
>>>>>
>>>>
>>>>
>>>> there's also logback http://logback.qos.ch/ which is written by the
>>>> original author of log4j
>>>>
>>>> David Lang
>>>> ______________________________****_________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<http://lists.adiscon.net/**mailman/listinfo/rsyslog>
>>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>
>>>> http://www.rsyslog.com/****professional-services/<http://www.rsyslog.com/**professional-services/>
>>>> <http://**www.rsyslog.com/professional-**services/<http://www.rsyslog.com/professional-services/>
>>>>>
>>>>
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>> rsyslog mailing list
>> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
>> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.