Mailing List Archive: ClamAV: devel

libmilter's workers pool in clamav-milter

Index | Next | Previous | View Threaded

ew-reg at mailbox

Dec 15, 2007, 2:37 PM

Post #1 of 6 (989 views)
Permalink

libmilter's workers pool in clamav-milter

Good evening,

version:
libmilter - sendmail-8.14.2
clamav-milter - clamav-0.91.2

If you configure libmilter library with using poll() rather than
select() and use workers pool you might hit problem related with locking
up your clamav-milter.

To reproduce this kind of behavior you should ran clamav-milter in 32
bit mode, lower max-children and set timeout to some rather high value.
When number of simultaneous connections exceed max children then every
new connection will equal one thread. Going step further, when threads
allocate all virtual memory available for clamav-milter workers from
libmilter will not have space to execute (clamav-milter: LAUNCH_WORKER
error: Cannot allocate memory).

Daemon clamav-milter will not answer with temporary error but will
keep going with blocking threads/connections.

How to solve this kind of problem? Does not clamav-milter is prepare for
workers from libmilter?

Most interesing is how to limit clamav-milter's resource based on thre-
ads. In described situation number of threads are not equal number of
connections, limiting it with dont-wait threads is not a good idea.
Maybe there is a way to limit numbers of threads that workers can
consume?

Best regards,
Ernest Wypierowski
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Jose-Marcio.Martins at ensmp

Dec 16, 2007, 12:49 AM

Post #2 of 6 (901 views)
Permalink

Re: libmilter's workers pool in clamav-milter [In reply to]

That's already done inside libmilter :

With sendmail 8.14.0, you shall compile libmilter with :

APPENDDEF(`conf_libmilter_ENVDEF',`-D_FFR_WORKER_MODEL')

This will switch the libmilter thread model to a pool of workers instead
of one thread per sendmail connection. And the number of threads in the
filter will be the number of threads really doing something...

This makes a very big difference on systems handling a big number of
simultaneous connections.

Here are some old slides (2003) from an old presentation I've done at
sendmail about that :

http://j-chkmail.ensmp.fr/papers/workermilter.pdf

Well... I've got an award from sendmail... 8-)

http://www.j-chkmail.org/wiki/doku.php/award

Ernest Wypierowski wrote:
> Good evening,
>
> version:
> libmilter - sendmail-8.14.2
> clamav-milter - clamav-0.91.2
>
>
> If you configure libmilter library with using poll() rather than
> select() and use workers pool you might hit problem related with locking
> up your clamav-milter.
>
> To reproduce this kind of behavior you should ran clamav-milter in 32
> bit mode, lower max-children and set timeout to some rather high value.
> When number of simultaneous connections exceed max children then every
> new connection will equal one thread. Going step further, when threads
> allocate all virtual memory available for clamav-milter workers from
> libmilter will not have space to execute (clamav-milter: LAUNCH_WORKER
> error: Cannot allocate memory).
>
> Daemon clamav-milter will not answer with temporary error but will
> keep going with blocking threads/connections.
>
> How to solve this kind of problem? Does not clamav-milter is prepare for
> workers from libmilter?
>
> Most interesing is how to limit clamav-milter's resource based on thre-
> ads. In described situation number of threads are not equal number of
> connections, limiting it with dont-wait threads is not a good idea.
> Maybe there is a way to limit numbers of threads that workers can
> consume?
>
> Best regards,
> Ernest Wypierowski
> _______________________________________________
> http://lurker.clamav.net/list/clamav-devel.html
> Please submit your patches to our Bugzilla: http://bugs.clamav.net
>

--
---------------------------------------------------------------
Jose Marcio MARTINS DA CRUZ http://j-chkmail.ensmp.fr
Ecole des Mines de Paris
60, bd Saint Michel
75272 - PARIS CEDEX 06 mailto:Jose-Marcio.Martins[at]ensmp.fr
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Jose-Marcio.Martins at ensmp

Dec 16, 2007, 2:25 AM

Post #3 of 6 (901 views)
Permalink

Re: libmilter's workers pool in clamav-milter [In reply to]

Well, I think now I didn't understood your first message.... 8-(

What I can say is that there are a number of servers running this
without problem since 2004. The biggest one I have some feedback handles
almost 30.000 messages an hour.

Surely, this thread model can handle much more traffic than original
threads model, but it to has a limit somewhere.

The problem there is that this new limit shall be evaluated by someone :
libmilter or the filter. The solution I proposed is OS and filter
independent.

So there are things which can be done at libmilter level and others by
the filter.

A filter shall eventually be able to evaluate resources consumption and
behave adequately. So, if the filter says - sorry, I'm not able to
handle any other new connection now, this is better than accepting and
crashing. This is usually what's done in real life with real people.

Limit the number of threads isn't a good idea, unless the filter
(clamav-milter or other) can limit the amount of time allocated to each
task (e.g. message content scanning). If you don't do that, you create a
DoS vulnerability.

If there are new ideas to improve this feature, please let me know.

Regards,

OBS : If you define _FFR_WORKER_MODEL, SM_CONF_POLL is also defined
(libmilter.h) :

#if _FFR_WORKERS_POOL
/* SM_CONF_POLL shall be defined with _FFR_WORKERS_POOL */
# if !SM_CONF_POLL
# define SM_CONF_POLL 1
# endif /* SM_CONF_POLL */
#endif /* _FFR_WORKERS_POOL */

Jose-Marcio Martins da Cruz wrote:
> That's already done inside libmilter :
>
> With sendmail 8.14.0, you shall compile libmilter with :
>
> APPENDDEF(`conf_libmilter_ENVDEF',`-D_FFR_WORKER_MODEL')
>
> This will switch the libmilter thread model to a pool of workers instead
> of one thread per sendmail connection. And the number of threads in the
> filter will be the number of threads really doing something...
>
> This makes a very big difference on systems handling a big number of
> simultaneous connections.
>
> Here are some old slides (2003) from an old presentation I've done at
> sendmail about that :
>
> http://j-chkmail.ensmp.fr/papers/workermilter.pdf
>
> Well... I've got an award from sendmail... 8-)
>
> http://www.j-chkmail.org/wiki/doku.php/award
>
>
> Ernest Wypierowski wrote:
>> Good evening,
>>
>> version:
>> libmilter - sendmail-8.14.2
>> clamav-milter - clamav-0.91.2
>>
>>
>> If you configure libmilter library with using poll() rather than
>> select() and use workers pool you might hit problem related with locking
>> up your clamav-milter.
>>
>> To reproduce this kind of behavior you should ran clamav-milter in 32
>> bit mode, lower max-children and set timeout to some rather high value.
>> When number of simultaneous connections exceed max children then every
>> new connection will equal one thread. Going step further, when threads
>> allocate all virtual memory available for clamav-milter workers from
>> libmilter will not have space to execute (clamav-milter: LAUNCH_WORKER
>> error: Cannot allocate memory).
>>
>> Daemon clamav-milter will not answer with temporary error but will
>> keep going with blocking threads/connections.
>>
>> How to solve this kind of problem? Does not clamav-milter is prepare for
>> workers from libmilter?
>>
>> Most interesing is how to limit clamav-milter's resource based on thre-
>> ads. In described situation number of threads are not equal number of
>> connections, limiting it with dont-wait threads is not a good idea.
>> Maybe there is a way to limit numbers of threads that workers can
>> consume?
>>
>> Best regards,
>> Ernest Wypierowski
>> _______________________________________________
>> http://lurker.clamav.net/list/clamav-devel.html
>> Please submit your patches to our Bugzilla: http://bugs.clamav.net
>>
>
>

--
---------------------------------------------------------------
Jose Marcio MARTINS DA CRUZ http://j-chkmail.ensmp.fr
Ecole des Mines de Paris
60, bd Saint Michel
75272 - PARIS CEDEX 06 mailto:Jose-Marcio.Martins[at]ensmp.fr
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

ew-reg at mailbox

Dec 16, 2007, 3:53 AM

Post #4 of 6 (908 views)
Permalink

Re: libmilter's workers pool in clamav-milter [In reply to]

On Sun, 16.12.2007 at 11:25:47, Jose-Marcio Martins da Cruz wrote:
> A filter shall eventually be able to evaluate resources consumption and
> behave adequately. So, if the filter says - sorry, I'm not able to
> handle any other new connection now, this is better than accepting and
> crashing. This is usually what's done in real life with real people.

I entirely agree. However the filter, clamav-milter, when libmilter/wor-
kers.c could not spawn new thread freeze waiting for answer.

Limiting the connection number to clamav-milter in my opinion is not the
best approach. I can image a event when smtp's client connects to
server and send mail from/rcpt to and wait. Then connection to filter
(clamav- milter) would be set but nothing will not load cpu.

I would like to see the better way to limit the resources, workers from
libmilter inform clamav-milter that they reach the MAX_WORKERS
and clamav-milter sends: 451 4.3.2 AV system temporarily overloaded
to smtp server.

> Limit the number of threads isn't a good idea, unless the filter
> (clamav-milter or other) can limit the amount of time allocated to each
> task (e.g. message content scanning). If you don't do that, you create a
> DoS vulnerability.

So the only limit is to set the max number of connection?

Best regards,
Ernest Wypierowski
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Jose-Marcio.Martins at ensmp

Dec 16, 2007, 5:05 AM

Post #5 of 6 (897 views)
Permalink

Re: libmilter's workers pool in clamav-milter [In reply to]

Ernest Wypierowski wrote:
> On Sun, 16.12.2007 at 11:25:47, Jose-Marcio Martins da Cruz wrote:

>
> Limiting the connection number to clamav-milter in my opinion is not the
> best approach. I can image a event when smtp's client connects to
> server and send mail from/rcpt to and wait. Then connection to filter
> (clamav- milter) would be set but nothing will not load cpu.
>
> I would like to see the better way to limit the resources, workers from
> libmilter inform clamav-milter that they reach the MAX_WORKERS
> and clamav-milter sends: 451 4.3.2 AV system temporarily overloaded
> to smtp server.

I agree with you, but this kind of thing is better done inside the
filter than inside libmilter.

This is what I do inside my filter j-chkmail. Already, it evaluate a
"resource usage" which depends on many things, mainly CPU load and the
number of file descriptors in use.

>
>> Limit the number of threads isn't a good idea, unless the filter
>> (clamav-milter or other) can limit the amount of time allocated to each
>> task (e.g. message content scanning). If you don't do that, you create a
>> DoS vulnerability.
>
> So the only limit is to set the max number of connection?

j-chkmail begins limiting the connection rate per IP address and the
number of connections simultaneously openned by each IP address. At the
same time, it limits the global resource usage.

Some time in the past, j-chkmail could be linked against libclamav. I
removed that as it was doing what was already done by clamav-milter. So
I privilegiate an interface with clamd, which can be, and already is,
better maintained than my interface with libclamav. At the same time, I
can better control resources usage : if clamd refuse connection or
doesn't answer after some delay, I simply tempfail or accept the message.

--
---------------------------------------------------------------
Jose Marcio MARTINS DA CRUZ http://j-chkmail.ensmp.fr
Ecole des Mines de Paris
60, bd Saint Michel
75272 - PARIS CEDEX 06 mailto:Jose-Marcio.Martins[at]ensmp.fr
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Jose-Marcio.Martins at ensmp

Dec 28, 2007, 1:15 PM

Post #6 of 6 (877 views)
Permalink

Re: libmilter's workers pool in clamav-milter [In reply to]

Ernest Wypierowski wrote:
> On Sun, 16.12.2007 at 11:25:47, Jose-Marcio Martins da Cruz wrote:
>> A filter shall eventually be able to evaluate resources consumption and
>> behave adequately. So, if the filter says - sorry, I'm not able to
>> handle any other new connection now, this is better than accepting and
>> crashing. This is usually what's done in real life with real people.
>
> I entirely agree. However the filter, clamav-milter, when libmilter/wor-
> kers.c could not spawn new thread freeze waiting for answer.
>
> Limiting the connection number to clamav-milter in my opinion is not the
> best approach. I can image a event when smtp's client connects to
> server and send mail from/rcpt to and wait. Then connection to filter
> (clamav- milter) would be set but nothing will not load cpu.
>
> I would like to see the better way to limit the resources, workers from
> libmilter inform clamav-milter that they reach the MAX_WORKERS
> and clamav-milter sends: 451 4.3.2 AV system temporarily overloaded
> to smtp server.

Well, I'll propose a patch to handle this. But as long as the problem
comes from pthread_create call, it seems to me that ther's nothing
better than doing an exit (the program will probably be garbaged).

But I wonder why your system is getting frozen... Maybe your system is
underdimensioned, or there is a memory leak in the milter you're
using... or some other problem. I'm seeing this code running for more
than three year on some huge servers without problems.

Original libmilter had some bottlenecks in huge servers. Using poll
instead of select pushed away these limits. Using a pool of workers
instead of one thread per connection pushed the bottleneck very far
away. So, currently, most bottlenecks are more on the filter code than
on libmilter code.

Understanding why your milter is getting frozen can help get things
better (both libmilter and the milter you're using).

--
---------------------------------------------------------------
Jose Marcio MARTINS DA CRUZ http://j-chkmail.ensmp.fr
Ecole des Mines de Paris
60, bd Saint Michel
75272 - PARIS CEDEX 06 mailto:Jose-Marcio.Martins[at]ensmp.fr
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
Please submit your patches to our Bugzilla: http://bugs.clamav.net

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com