Mailing List Archive: RSyslog: users

Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode?

Index | Next | Previous | View Threaded

rgerhards at hq

Apr 19, 2012, 12:02 AM

Post #1 of 4 (205 views)
Permalink

Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode?

> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Chris McCraw
> Sent: Thursday, April 19, 2012 2:18 AM
> To: rsyslog-users
> Subject: [rsyslog] rsyslogd 5.8.5 + heavy message load + compression -
> failure mode?
>
> Hi folks,
>
> I probably missed it, but after awhile searching the docs fruitlessly,
> I decided I'd ask the experts.
>
> We have an rsyslog server that handles a couple million messages (from
> a single remote server) per minute. It logs these to several logfiles
> with a cpu load of about 40% of a single CPU core. logrotate
> currently takes about 12 hours to single-threadedly, consecutively,
> compress and rotate these logs. It's been suggested that we take
> logrotate out of the loop and just have rsyslog write compressed
> files. This seems like a great idea, but I'm curious about how it
> scales, since I don't have a good test environment (just some
> underpowered VM's which don't seem to generate comparable load no
> matter how I try) to work with.
>
> Suppose that rsyslog needs more than a core's worth of CPU to do the
> compression realtime. What happens then? Is rsyslogd multithreaded
> enough (or can it be setup to be multithreaded enough) to spin up more
> threads to handle the compressed writes? Will it ever drop messages?

We'll I can't do the actual lab for you (well, under a support contract...).
But what I can say is that I have the strong feeling this will work for you.
I know at least of one datacenter which has a far higher data rate than you
have and they work very successfully with that feature. But YMMV: you need to
do some testing, which will identify potential bottlenecks, if there are any.

> some more info:
> - The highest traffic logs are in 2 separate files, which have about
> 60% of the load together, the rest is going into a dozen other smaller
> files.
> - we'd be setting OMFileZipLevel to 1
> - we're logging via tcp and splitting based on priority and sending IP
> (though 99.9% of everything comes from one IP)
> - we're willing to change some configuration, but here's the only
> special config we have now:
> - $MainMsgQueueType Direct

Outch - why that? This practically disables all multi-threading and thightly
couples producer and consumer.

Rainer
>
> Thanks for your insight!
>
> --
> Chris McCraw | Operations
> New Relic - http://blog.newrelic.com - @NewRelic on Twitter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

cmccraw at newrelic

Apr 19, 2012, 10:03 AM

Post #2 of 4 (189 views)
Permalink

Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode? [In reply to]

On Thu, Apr 19, 2012 at 12:02 AM, Rainer Gerhards
<rgerhards [at] hq> wrote:
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Chris McCraw
>> Sent: Thursday, April 19, 2012 2:18 AM
>> To: rsyslog-users
>> Subject: [rsyslog] rsyslogd 5.8.5 + heavy message load + compression -
>> failure mode?
>>
>> Hi folks,
>>
>> I probably missed it, but after awhile searching the docs fruitlessly,
>> I decided I'd ask the experts.
>>
>> We have an rsyslog server that handles a couple million messages (from
>> a single remote server) per minute. It logs these to several logfiles
>> with a cpu load of about 40% of a single CPU core. logrotate
>> currently takes about 12 hours to single-threadedly, consecutively,
>> compress and rotate these logs. It's been suggested that we take
>> logrotate out of the loop and just have rsyslog write compressed
>> files. This seems like a great idea, but I'm curious about how it
>> scales, since I don't have a good test environment (just some
>> underpowered VM's which don't seem to generate comparable load no
>> matter how I try) to work with.
>>
>> Suppose that rsyslog needs more than a core's worth of CPU to do the
>> compression realtime. What happens then? Is rsyslogd multithreaded
>> enough (or can it be setup to be multithreaded enough) to spin up more
>> threads to handle the compressed writes? Will it ever drop messages?
>
> We'll I can't do the actual lab for you (well, under a support contract...).
> But what I can say is that I have the strong feeling this will work for you.
> I know at least of one datacenter which has a far higher data rate than you
> have and they work very successfully with that feature. But YMMV: you need to
> do some testing, which will identify potential bottlenecks, if there are any.

I am thrilled to do testing and can even do some in production, but
I'm not sure how to be sure no messages are dropped. Any suggestions
for trackable high-load-generation? I've been using logger and/or nc
in a loop from the command line to log "# rather long message..."
where # increases sequentially to be sure no messages were dropped,
but the production log stream is just a bunch of http requests and I
don't have any gauge of when one doesn't make it through, so
real-world testing might not be informative.

>> - we're willing to change some configuration, but here's the only
>> special config we have now:
>> - $MainMsgQueueType Direct
>
> Outch - why that? This practically disables all multi-threading and thightly
> couples producer and consumer.

Hmm, it was there when I arrived and I never researched it. I suspect
it was put in place for debugging purposes. I'll remove it when I try
this out.

--
Chris McCraw | Operations
New Relic - http://blog.newrelic.com - @NewRelic on Twitter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

radu0gheorghe at gmail

Apr 19, 2012, 11:09 PM

Post #3 of 4 (181 views)
Permalink

Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode? [In reply to]

2012/4/19 Chris McCraw <cmccraw [at] newrelic>:
> I am thrilled to do testing and can even do some in production, but
> I'm not sure how to be sure no messages are dropped. Any suggestions
> for trackable high-load-generation? I've been using logger and/or nc
> in a loop from the command line to log "# rather long message..."
> where # increases sequentially to be sure no messages were dropped,
> but the production log stream is just a bunch of http requests and I
> don't have any gauge of when one doesn't make it through, so
> real-world testing might not be informative.

I use a simple Python script for generating many log messages:

$ cat stresslogger.py
#!/usr/bin/env python
import syslog, sys
syslog.openlog()
for i in xrange(int(sys.argv[1])):
syslog.syslog("This is a stress test log")

The C variant of if was my first C program :D. Here it goes:
$ cat stresslogger.c
#include <syslog.h>
#include <stdlib.h>
int main(int argc, char** argv)
{
openlog ("exampleprog", LOG_PID, LOG_LOCAL5);
int max;
max=atoi(argv[1]);
int i;
for (i = 0; i < max; i++)
{
syslog (LOG_INFO, "This is a stress test log");
}
closelog ();
return 0;
}

Then I use "time" to have a rough check on the performance. And you
can check if you lost a log by just counting them. Send 1M messages,
and if you started with an empty log file, "wc -l" should return 1M.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

Apr 19, 2012, 11:40 PM

Post #4 of 4 (180 views)
Permalink

Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode? [In reply to]

On Thu, 19 Apr 2012, Chris McCraw wrote:

> On Thu, Apr 19, 2012 at 12:02 AM, Rainer Gerhards
> <rgerhards [at] hq> wrote:
>> But what I can say is that I have the strong feeling this will work for you.
>> I know at least of one datacenter which has a far higher data rate than you
>> have and they work very successfully with that feature. But YMMV: you need to
>> do some testing, which will identify potential bottlenecks, if there are any.
>
> I am thrilled to do testing and can even do some in production, but
> I'm not sure how to be sure no messages are dropped. Any suggestions
> for trackable high-load-generation? I've been using logger and/or nc
> in a loop from the command line to log "# rather long message..."
> where # increases sequentially to be sure no messages were dropped,
> but the production log stream is just a bunch of http requests and I
> don't have any gauge of when one doesn't make it through, so
> real-world testing might not be informative.

what I've done in the past is use tcpdump to capture a bunch of real
syslog data, then used tcpreplay to play it back at whatever rate I want
(and since I know how much data was captured, I can compare it to the
resulting logs)

a little better than just a scripted test, and it can work to higher data
rates.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads