david at lang
Jun 1, 2012, 1:59 PM
Post #6 of 7
(195 views)
Permalink
|
The biggest work is changing the plugin to handle more than one message at
a time. the 'bulkmode' flag is just notifying the core system that the
plugin can handle batches. It's a result, not a cause.
the plugin needs to dequeue multiple messages at a time (I think there is
a global batchsize variable to control how many it will grab). It then
needs to process all the messages, and implement the appropriate error
handling if the output of all the messages at once doesn't work.
I know the high level logic of what should be happening, but I haven't
looked into the details fo exactly what gets done where in the code.
David Lang
On Fri, 1 Jun 2012,
Lars Peterson wrote:
> Looks like a "bulkmode" boolean config variable is availablein the plugin...? Is the number of messages in the batch adjustable?
>
> Hate to ask you to do all the work for me, but is there an example of what this would like in rsyslog.conf?
>
> -Lars
>
> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Friday, June 01, 2012 3:09 PM
> To: rsyslog [at] lists
> Subject: Re: [rsyslog] Output plugin batching
>
> Look at the master-elasticsearch branch.
> Rainer
>
> Lars Peterson hat geschrieben:
> Thanks Rainer.
>
> I grabbed rsyslog-6.3.9 to have a look at the omelasticsearch plugin. Are your changes to support batching included in this release or is it not available to the public? I'm not seeing anything batch related in omelasticsearch.c. Am I missing something?
>
> -Lars
>
> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Rainer Gerhards
> Sent: Friday, June 01, 2012 10:34 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Output plugin batching
>
> Batching must be supported by the plugin, it needs to implement the necessary logic. Most rsyslog-provided plugins are capable of using batch mode. Note that plugin changes can be considerable. For example, in elasticsearch I needed to implement a totally different handler and mode of operation (not due to rsyslog, but the way things work with elasticsearch). So it is best to contact the plugin author.
>
> Rainer
>
>> -----Original Message-----
>> From: rsyslog-bounces [at] lists [mailto:rsyslog-
>> bounces [at] lists] On Behalf Of Lars Peterson
>> Sent: Friday, June 01, 2012 5:28 PM
>> To: rsyslog [at] lists
>> Cc: jamesk [at] okeating
>> Subject: [rsyslog] Output plugin batching
>>
>> Hello again,
>>
>> I'm attempting to use rsyslog to process on average 20k messages
>> /minute. At the moment I'm outputting to both files and James
>> Keating's omsolr plugin (https://github.com/MrJamesK/omsolr/blob/master/omsolr.c).
>>
>> Things were humming along fine until I started throwing more traffic
>> at it. I set up both main message and action queues to try and speed
>> things up, but the omsolr plugin appears to be severely lagging behind.
>>
>> Perhaps there's some general tuning to be done to help, but I have a
>> hunch that the biggest performance gain can be made if the output
>> plugin would send
>> (batch) more than more document to SOLR to index at once. I'm confused
>> on how to configure this or if can even be done given how omsolr is
>> currently implemented. The only example of configurable batching I
>> could find was in the omoracle plugin where I spotted a config option
>> called "omoraclebatchsize". Is there a way to force the batch size in
>> the omsolr plugin using the existing rsyslog framework or would some code changes be required?
>>
>> Here's the snippet of my rsyslog.conf relating to the omsolr plugin:
>>
>> $WorkDirectory /var/spool/rsyslog
>> $ActionQueueFileName solrRuleProd
>> $ActionQueueMaxDiskSpace 4g
>> $ActionQueueDequeueBatchSize 1024
>> $ActionQueueWorkerThreadMinimumMessages 100 $ActionQueueSaveOnShutdown
>> on $ActionQueueType LinkedList $ActionResumeRetryCount -1
>> :fromhost-ip, !isequal, "127.0.0.1" :omsolr:;SolrLogDefault
>>
>> Server information:
>>
>> rsyslog-5.8.7
>> CentOS 6.2
>> 4GB ram
>> 4 x 2GHz CPU (VM)
>>
>> Let me know if you need any more information.
>>
>> Thanks!
>>
>> -Lars
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
|
