Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

rsyslogd 5.8.5 + heavy message load + compression - failure mode?

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


cmccraw at newrelic

Apr 18, 2012, 5:18 PM

Post #1 of 3 (333 views)
Permalink
rsyslogd 5.8.5 + heavy message load + compression - failure mode?

Hi folks,

I probably missed it, but after awhile searching the docs fruitlessly,
I decided I'd ask the experts.

We have an rsyslog server that handles a couple million messages (from
a single remote server) per minute. It logs these to several logfiles
with a cpu load of about 40% of a single CPU core. logrotate
currently takes about 12 hours to single-threadedly, consecutively,
compress and rotate these logs. It's been suggested that we take
logrotate out of the loop and just have rsyslog write compressed
files. This seems like a great idea, but I'm curious about how it
scales, since I don't have a good test environment (just some
underpowered VM's which don't seem to generate comparable load no
matter how I try) to work with.

Suppose that rsyslog needs more than a core's worth of CPU to do the
compression realtime. What happens then? Is rsyslogd multithreaded
enough (or can it be setup to be multithreaded enough) to spin up more
threads to handle the compressed writes? Will it ever drop messages?

some more info:
- The highest traffic logs are in 2 separate files, which have about
60% of the load together, the rest is going into a dozen other smaller
files.
- we'd be setting OMFileZipLevel to 1
- we're logging via tcp and splitting based on priority and sending IP
(though 99.9% of everything comes from one IP)
- we're willing to change some configuration, but here's the only
special config we have now:
- $MainMsgQueueType Direct

Thanks for your insight!

--
Chris McCraw | Operations
New Relic - http://blog.newrelic.com - @NewRelic on Twitter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 18, 2012, 5:30 PM

Post #2 of 3 (323 views)
Permalink
Re: rsyslogd 5.8.5 + heavy message load + compression - failure mode? [In reply to]

I can't directly answer your question, but here is what I do on my central
servers (where I've experienced 92K logs in one second). I currently
rotate logs once per minute. rsyslog creates all the files as
<something>-messages

mercury1-p:/var/home/dlang# cat /usr/local/bin/newlogs
#!/bin/sh
#

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
umask 022
year=`date +%Y`
month=`date +%m`
day=`date +%d`
fdate=`date +%Y%m%d.%H%M`

logroot=/var/log

logroll=$logroot/oldlogs

cd $logroot mkdir -p $logroll/$year/$month/$day >/dev/null 2>/dev/null

mv messages messages.$fdate
ls *-messages |while read file; do
mv $file $file.$fdate ;
done
pkill -HUP rsyslogd
gzip -9 *messages.$fdate
mv *messages.$fdate.gz $logroll/$year/$month/$day/


This is not compressing them in parallel (not needed on my system), but to
modify it to do so, replace the final gzip line with

ls *messages.$fdate |while read file; do
gzip -9 $file &
done
wait

and it will do them all in parallel

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


aoz.syn at gmail

Apr 18, 2012, 5:37 PM

Post #3 of 3 (327 views)
Permalink
Re: rsyslogd 5.8.5 + heavy message load + compression - failure mode? [In reply to]

I'm not addressing the "directly compressed" approach since I don't
have any direct experience with it.

On Wed, Apr 18, 2012 at 18:18, Chris McCraw <cmccraw [at] newrelic> wrote:
> with a cpu load of about 40% of a single CPU core.  logrotate
> currently takes about 12 hours to single-threadedly, consecutively,
> compress and rotate these logs.

How many other processor cores are available on the system doing the
compression? Have you looked at pigz or pbzip2? The prior is by one
of the authors of the near-omnipresent zlib, and works quite well.
They both do, actually.

Perhaps less of an option but still potentially a solution, have you
considered compressed filesystems? ZFS-on-Linux is coming along quite
nicely and has runtime-configurable compression; btrfs does
transparent compression as well. Filesystem-level compression doesn't
help if you're moving archives around between filesystems, but in my
uses it's been quite handy.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.