cmccraw at newrelic
Apr 18, 2012, 5:18 PM
Post #1 of 3
rsyslogd 5.8.5 + heavy message load + compression - failure mode?
I probably missed it, but after awhile searching the docs fruitlessly,
I decided I'd ask the experts.
We have an rsyslog server that handles a couple million messages (from
a single remote server) per minute. It logs these to several logfiles
with a cpu load of about 40% of a single CPU core. logrotate
currently takes about 12 hours to single-threadedly, consecutively,
compress and rotate these logs. It's been suggested that we take
logrotate out of the loop and just have rsyslog write compressed
files. This seems like a great idea, but I'm curious about how it
scales, since I don't have a good test environment (just some
underpowered VM's which don't seem to generate comparable load no
matter how I try) to work with.
Suppose that rsyslog needs more than a core's worth of CPU to do the
compression realtime. What happens then? Is rsyslogd multithreaded
enough (or can it be setup to be multithreaded enough) to spin up more
threads to handle the compressed writes? Will it ever drop messages?
some more info:
- The highest traffic logs are in 2 separate files, which have about
60% of the load together, the rest is going into a dozen other smaller
- we'd be setting OMFileZipLevel to 1
- we're logging via tcp and splitting based on priority and sending IP
(though 99.9% of everything comes from one IP)
- we're willing to change some configuration, but here's the only
special config we have now:
- $MainMsgQueueType Direct
Thanks for your insight!
Chris McCraw | Operations
New Relic - http://blog.newrelic.com - @NewRelic on Twitter
rsyslog mailing list
What's up with rsyslog? Follow https://twitter.com/rgerhards