rgerhards at hq
Apr 19, 2012, 12:02 AM
Views: 204
Permalink
|
|
Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode?
|
|
> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-
> bounces [at] lists] On Behalf Of Chris McCraw
> Sent: Thursday, April 19, 2012 2:18 AM
> To: rsyslog-users
> Subject: [rsyslog] rsyslogd 5.8.5 + heavy message load + compression -
> failure mode?
>
> Hi folks,
>
> I probably missed it, but after awhile searching the docs fruitlessly,
> I decided I'd ask the experts.
>
> We have an rsyslog server that handles a couple million messages (from
> a single remote server) per minute. It logs these to several logfiles
> with a cpu load of about 40% of a single CPU core. logrotate
> currently takes about 12 hours to single-threadedly, consecutively,
> compress and rotate these logs. It's been suggested that we take
> logrotate out of the loop and just have rsyslog write compressed
> files. This seems like a great idea, but I'm curious about how it
> scales, since I don't have a good test environment (just some
> underpowered VM's which don't seem to generate comparable load no
> matter how I try) to work with.
>
> Suppose that rsyslog needs more than a core's worth of CPU to do the
> compression realtime. What happens then? Is rsyslogd multithreaded
> enough (or can it be setup to be multithreaded enough) to spin up more
> threads to handle the compressed writes? Will it ever drop messages?
We'll I can't do the actual lab for you (well, under a support contract...).
But what I can say is that I have the strong feeling this will work for you.
I know at least of one datacenter which has a far higher data rate than you
have and they work very successfully with that feature. But YMMV: you need to
do some testing, which will identify potential bottlenecks, if there are any.
> some more info:
> - The highest traffic logs are in 2 separate files, which have about
> 60% of the load together, the rest is going into a dozen other smaller
> files.
> - we'd be setting OMFileZipLevel to 1
> - we're logging via tcp and splitting based on priority and sending IP
> (though 99.9% of everything comes from one IP)
> - we're willing to change some configuration, but here's the only
> special config we have now:
> - $MainMsgQueueType Direct
Outch - why that? This practically disables all multi-threading and thightly
couples producer and consumer.
Rainer
>
> Thanks for your insight!
>
> --
> Chris McCraw | Operations
> New Relic - http://blog.newrelic.com - @NewRelic on Twitter
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
|
Re: rsyslogd 5.8.5 + heavy message load + compression -failure mode?
