Mailing List Archive: ClamAV: devel

the future of clamd

Index | Next | Previous | View Threaded

oz at nixil

Feb 12, 2004, 4:25 PM

Post #1 of 9 (1117 views)
Permalink

the future of clamd

Hello All,

So I've been playing with clamd & clamav-milter on 2 extremely
loaded sendmail servers,
and I think we are pushing clamd over it's limits.. I've tried a
process based (custom) clamd
and the standard thread based clamd, and I feel I've increased the
number of threads higher
than I'm comfortable with to keep things happy (over 150 for a laugh).
At some point clamd
is falling over and sometime after than clamav-milter shuts itself
down. Now I know that this
is an extremely high load environment to place clamd in, but I've been
wondering if improvements
to clamd's design can be made to help it scale better. Would an
apache2 style of multiple
children and multiple threads per child scale better? Can I get some
other people's thoughts
on this?

Phil.

p.s. if I'm missing an option in the config to make things work better
with the current clamd,
then please let me know.

bet at rahul

Feb 12, 2004, 5:53 PM

Post #2 of 9 (1084 views)
Permalink

Re: the future of clamd [In reply to]

We're on it.

clamd is currently based on threads; that's the problem; threads are
a great solution in the context where Blue Screen Of Death is the
expected look-and-feel.

Sometime soon in our future, clamd becomes what's currently called
nclamd, which is processed based (preforking). At that point it
becomes for all intents and purposes bulletproof.

At that point, you're going to be needing to widdle on how you feed
clamd to keep the system from being crushed by the pure load ---
loads that would crash the current hackup won't make it even falter
--- but clamd itself won't keel over, mostly the way you feed it
will become your overall bottleneck.

You'll need a robust scanning module to plug into your MTA of
choice; and it'll need to park the messages for clamd's attention in
files in a ramfs of some sort, passing their names (or workalikes
for names, under Linux at least /proc/$pid/fs/$fileno works a champ)
to clamd for it to ponder.

-Bennett

oz at nixil

Feb 12, 2004, 6:13 PM

Post #3 of 9 (1098 views)
Permalink

Re: the future of clamd [In reply to]

Well.. I kinda did the same thing and reimplemented server-proc.c in cvs
using libevent
and it works.. and works well.. but with the level of mail this thing
is going on I can almost fill
the process table (combined with all the sendmail processes..). Thus, I
was thinking..
that a re-eval is needed.. especially since 0.66 is out of the door
and we can start throwing
code around again instead of patching bugs.

Phil.

Bennett Todd wrote:

>We're on it.
>
>clamd is currently based on threads; that's the problem; threads are
>a great solution in the context where Blue Screen Of Death is the
>expected look-and-feel.
>
>Sometime soon in our future, clamd becomes what's currently called
>nclamd, which is processed based (preforking). At that point it
>becomes for all intents and purposes bulletproof.
>
>At that point, you're going to be needing to widdle on how you feed
>clamd to keep the system from being crushed by the pure load ---
>loads that would crash the current hackup won't make it even falter
>--- but clamd itself won't keel over, mostly the way you feed it
>will become your overall bottleneck.
>
>You'll need a robust scanning module to plug into your MTA of
>choice; and it'll need to park the messages for clamd's attention in
>files in a ramfs of some sort, passing their names (or workalikes
>for names, under Linux at least /proc/$pid/fs/$fileno works a champ)
>to clamd for it to ponder.
>
>-Bennett
>
>

oz at nixil

Feb 13, 2004, 12:54 PM

Post #4 of 9 (1095 views)
Permalink

Re: the future of clamd [In reply to]

Julien Benoist wrote:

>Hi,
>
>On Thu, 12 Feb 2004 17:22:11 -0700
>Phil Oleson <oz [at] nixil> wrote:
>
>
>
>> So I've been playing with clamd & clamav-milter on 2 extremely
>>loaded sendmail servers,
>>and I think we are pushing clamd over it's limits..
>>
>>
>
>Just wondering ... do you have any stats to share ? Scanned mails per sec, avgload, percentage of infected mails, etc ...
>
>Regards,
>
>
>
My Sendmail guy says that the together these 2 boxes process 600,000 to
800,000 e-mails each day.
btw, my libevent patched clamd seems happier than I remembered.. at a
max child of 50. Guess
checking out how the thread version responded in this environment
clouded my memory a bit.

I'm going to check out James's? nclamd again now that I'm more familiar
with clamd/clamav-milter
and can make a better analysis of it. Though I'd like to see where the
project leads want things
to go..

btw, I've found that clamav-milter chokes after a time when --quiet arg
is not provided.. didn't investigate too much.. (mainly because I dont
want that email anyways).

Phil.

erik at midmaine

Feb 13, 2004, 3:21 PM

Post #5 of 9 (1097 views)
Permalink

Re: the future of clamd [In reply to]

Phil Oleson <oz [at] nixil> writes:

> I'm going to check out James's? nclamd again now that I'm more familiar with
> clamd/clamav-milter
> and can make a better analysis of it. Though I'd like to see where the
> project leads want things
> to go..

Does nclamd work with the current clamdscan? I seem to remember it not
working for some reason (looked like protocol differences?).

- Erik

oz at nixil

Feb 13, 2004, 4:14 PM

Post #6 of 9 (1104 views)
Permalink

Re: the future of clamd [In reply to]

Erik Bourget wrote:

>Phil Oleson <oz [at] nixil> writes:
>
>
>
>>I'm going to check out James's? nclamd again now that I'm more familiar with
>>clamd/clamav-milter
>>and can make a better analysis of it. Though I'd like to see where the
>>project leads want things
>>to go..
>>
>>
>
>Does nclamd work with the current clamdscan? I seem to remember it not
>working for some reason (looked like protocol differences?).
>
>- Erik
>
>
>
It looks like nclamd comes with it's own nclamdscan..

Phil.

erik at midmaine

Feb 13, 2004, 4:27 PM

Post #7 of 9 (1093 views)
Permalink

Re: the future of clamd [In reply to]

Phil Oleson <oz [at] nixil> writes:

> Erik Bourget wrote:
>
>>Phil Oleson <oz [at] nixil> writes:
>>
>>
>>
>>>I'm going to check out James's? nclamd again now that I'm more familiar with
>>>clamd/clamav-milter
>>>and can make a better analysis of it. Though I'd like to see where the
>>>project leads want things
>>>to go..
>>>
>>>
>>
>>Does nclamd work with the current clamdscan? I seem to remember it not
>>working for some reason (looked like protocol differences?).
>>
>>- Erik
>>
> It looks like nclamd comes with it's own nclamdscan..

Heh! To be fair, all I did was download it and try to drop it in place of
clamd.

Does nclamd work well in production? I'm unhappy with clamd's crashy
behavior, and if development is going to switch over to nclamd ANYWAY...

- Erik

oz at nixil

Feb 13, 2004, 5:08 PM

Post #8 of 9 (1100 views)
Permalink

Re: the future of clamd [In reply to]

Erik Bourget wrote:

>>It looks like nclamd comes with it's own nclamdscan..
>>
>>
>
>Heh! To be fair, all I did was download it and try to drop it in place of
>clamd.
>
>Does nclamd work well in production? I'm unhappy with clamd's crashy
>behavior, and if development is going to switch over to nclamd ANYWAY...
>
>- Erik
>
>
>
Well I can't really say.. I'm using my own process implementation
that overlays into the existing clamd source.. I dont want to fork
the source.. so I'm waiting to see what the project leads want.

*shrug*

Phil.

tk at lodz

Feb 14, 2004, 1:40 AM

Post #9 of 9 (1092 views)
Permalink

Re: the future of clamd [In reply to]

On Fri, 13 Feb 2004 18:04:28 -0700
Phil Oleson <oz [at] nixil> wrote:

> >Does nclamd work well in production? I'm unhappy with clamd's crashy
> >behavior, and if development is going to switch over to nclamd
> >ANYWAY...

New implementation of clamd (still thread based) will be available in
CVS on Monday. It fixes stability problems on SMP machines. After that
we can integrate clamd with nclamd (e.g. its engine may be enabled with
UseProcesses) - it's not worth to develop two separate similar daemons.

Best regards,
Tomasz Kojm
--
oo ..... tkojm [at] clamav www.ClamAV.net
(\/)\......... http://www.clamav.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Sat Feb 14 09:07:55 CET 2004

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads