Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

ClamAV Memory Usage

  

 
ClamAV users RSS feed   Index | Next | Previous | View Threaded


gordan at bobich

Dec 1, 2009, 4:39 PM

Post #1 of 15 (2070 views)
Permalink
ClamAV Memory Usage
Hi,

Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5 days
of light usage (few hundred emails)? It is the single biggest process on
my mail servers, and I'm not convinced it's size is reasonably
justifiable. The database files under /var/lib/clamav use about 70MB.
So, even assuming this is kept in memory at all times, where does the
other 120MB come from?

I have looked through various mailing list and forum archives, but have
not found a reasonable answer to this question, despite it being raised
a few times.

TIA.

Gordan
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tshaw at oitc

Dec 1, 2009, 4:55 PM

Post #2 of 15 (2016 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
At 12:39 AM +0000 12/2/09, Gordan Bobic wrote:
>Hi,
>
>Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5
>days of light usage (few hundred emails)? It is the single biggest
>process on my mail servers, and I'm not convinced it's size is
>reasonably justifiable. The database files under /var/lib/clamav use
>about 70MB. So, even assuming this is kept in memory at all times,
>where does the other 120MB come from?
>
>I have looked through various mailing list and forum archives, but
>have not found a reasonable answer to this question, despite it
>being raised a few times.

Have your turned on safebrowsing in your config file?

Tom
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


gordan at bobich

Dec 1, 2009, 5:31 PM

Post #3 of 15 (2009 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
Tom Shaw wrote:
> At 12:39 AM +0000 12/2/09, Gordan Bobic wrote:
>> Hi,
>>
>> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5
>> days of light usage (few hundred emails)? It is the single biggest
>> process on my mail servers, and I'm not convinced it's size is
>> reasonably justifiable. The database files under /var/lib/clamav use
>> about 70MB. So, even assuming this is kept in memory at all times,
>> where does the other 120MB come from?
>>
>> I have looked through various mailing list and forum archives, but
>> have not found a reasonable answer to this question, despite it being
>> raised a few times.
>
> Have your turned on safebrowsing in your config file?

No, SafeBrowsing line is commented out. I just tried setting it
explicitly to "no", and it made no significant difference to memory
usage (183MB at startup). Having said that, I just tried changing it to
"yes", and that only increased the resident size to 184MB.

Gordan
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


per at computer

Dec 2, 2009, 12:32 AM

Post #4 of 15 (2014 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
Gordan Bobic wrote:

> Hi,
>
> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5
> days of light usage (few hundred emails)? It is the single biggest
> process on my mail servers, and I'm not convinced it's size is
> reasonably justifiable. The database files under /var/lib/clamav use
> about 70MB. So, even assuming this is kept in memory at all times,
> where does the other 120MB come from?

Maybe when the database is reloaded? I don't know clamd that well, but
I suspect it'll probably have two copies of the database in core during
reload.


/Per Jessen, Zürich

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


ged at jubileegroup

Dec 2, 2009, 5:38 AM

Post #5 of 15 (2014 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
Hi there,


On Wed, 2 Dec 2009 Gordan Bobic wrote:

> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5
> days of light usage (few hundred emails)? It is the single biggest
> process on my mail servers, and I'm not convinced it's size is
> reasonably justifiable.

It's perfectly normal. At the moment, each of my clamd processes uses
a little over 200M. The important thing to watch is how it changes
with time. Every now and then I'll plot a graph, and investigate any
unexplained changes.

http://www.jubileegroup.co.uk/JOS/misc/clamav-milter-0.88.7.jpg
http://www.jubileegroup.co.uk/JOS/misc/clamav-milter-0.91.1.jpg
http://www.jubileegroup.co.uk/JOS/misc/clamav-milter-0.94.1.gif
http://www.jubileegroup.co.uk/JOS/misc/clamd-0.95.3.gif
http://www.jubileegroup.co.uk/JOS/misc/virus_database_count.gif

I doo sometimes see sudden step changes but they're usually artifacts
of the, er, fragile data collection process in my systems rather than
signs of a bug. There were some problems with multiple loading of the
databases and memory leaks a while ago but I think they've been solved
for a quite while. From the behaviour of my clamd daemons, the memory
consumption of the current version seems fine to me.

> The database files under /var/lib/clamav use about 70MB.
> So, even assuming this is kept in memory at all times, where does the
> other 120MB come from?

Your approach is much too simplistic. The database contains regular
expressions. These are not necessarily used verbatim by the scanning
engine, they may have to be compiled first. This can result in much
greater memory usage than the database file sizes might suggest.

If you have a mail server which is exchanging mail, scanning for
viruses and doing very little else I would suggest that a gigabyte of
RAM is a reasonable starting point, and fifteen to twenty percent of
that for malware scanning is also reasonable. Two gigabytes would
be better if the machine is at all busy.

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


acabng at digitalfuture

Dec 2, 2009, 9:04 AM

Post #6 of 15 (2011 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
Gordan Bobic wrote:
> Hi,
>
> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5 days

The figure is "normal". In those 190MB there are likely ~110MB of
database and ~80MB of unused memory which is retained (by either libc or
the kernel) inside the process.
Unfortunately it's not very easy to determine the exact amount of
*really* used memory: you should subtract all unused maps (i.e.
/proc/<PID>/maps) and libc non-returned heaps from the above figure.
You may be able to see a more ram usage figure via clamdtop.

> The database files under /var/lib/clamav use about 70MB.
> So, even assuming this is kept in memory at all times, where does the
> other 120MB come from?

Database files are not stored in ram as they are on disk. In fact, for
performance reasons, signatures are mostly arranged in tries. This
involves lots of pointers, structure alignment and other nasty things.
70MB are roughly equivalent to 90-100MB on 32 bit systems and 110-10MB
on a 64bit system.

-acab
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jvigil at avibank

Dec 2, 2009, 9:14 AM

Post #7 of 15 (2002 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
I find this memory usage troubling. We're using version 0.95 of clamd on AIX; it uses under 5M of RAM. Since memory is tight with our system, I wouldn't want to upgrade if it's going to use that much memory.

- Joe

> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Gordan Bobic
> Sent: Tuesday, December 01, 2009 4:40 PM
> To: clamav-users [at] lists
> Subject: [Clamav-users] ClamAV Memory Usage
>
> Hi,
>
> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5 days
> of light usage (few hundred emails)? It is the single biggest process on
> my mail servers, and I'm not convinced it's size is reasonably
> justifiable. The database files under /var/lib/clamav use about 70MB.
> So, even assuming this is kept in memory at all times, where does the
> other 120MB come from?
>
> I have looked through various mailing list and forum archives, but have
> not found a reasonable answer to this question, despite it being raised
> a few times.
>
> TIA.
>
> Gordan
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Dec 2, 2009, 9:25 AM

Post #8 of 15 (2006 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
Gordan Bobic wrote:
> Hi,
>
> Can anyone explain why clamd 0.95.3 might use 190MB of RAM after 5 days
> of light usage (few hundred emails)? It is the single biggest process on
> my mail servers, and I'm not convinced it's size is reasonably
> justifiable. The database files under /var/lib/clamav use about 70MB.
> So, even assuming this is kept in memory at all times, where does the
> other 120MB come from?

Are you talking about size or resident size? Either way that's not excessive for
my systems. Here's a fairly quiet (smtp) system on Solaris 9 with only 512M
physical memory:

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
7371 clamav 2 2 19 163M 148M sleep 275:20 0.00% clamd

On my very busiest systems it used to spike up over 300M. I had a watch point
set at 350M at which time clamd would be restarted. That was several years ago
and that problem has not come up again.


dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Dec 2, 2009, 9:34 AM

Post #9 of 15 (2005 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
On 2009-12-02 19:14, Vigil, Joe wrote:
> I find this memory usage troubling. We're using version 0.95 of clamd on AIX; it uses under 5M of RAM. Since memory is tight with our system, I wouldn't want to upgrade if it's going to use that much memory.
>
>

5M? Thats not possible, you must be running with an empty database ...

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jvigil at avibank

Dec 2, 2009, 9:47 AM

Post #10 of 15 (2002 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
When I run the ps command on the clamd process id, the VSZ is 4416 and the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ includes the VSZ, or if they're separate. Anyway, does this mean that there's a problem with my install?
Thanks!

- Joe

> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Török Edwin
> Sent: Wednesday, December 02, 2009 9:35 AM
> To: ClamAV users ML
> Subject: Re: [Clamav-users] ClamAV Memory Usage
>
> On 2009-12-02 19:14, Vigil, Joe wrote:
> > I find this memory usage troubling. We're using version 0.95 of clamd on
> AIX; it uses under 5M of RAM. Since memory is tight with our system, I
> wouldn't want to upgrade if it's going to use that much memory.
> >
> >
>
> 5M? Thats not possible, you must be running with an empty database ...
>
> Best regards,
> --Edwin
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Dec 2, 2009, 10:07 AM

Post #11 of 15 (2012 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
On 2009-12-02 19:47, Vigil, Joe wrote:
> When I run the ps command on the clamd process id, the VSZ is 4416 and the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ includes the VSZ, or if they're separate. Anyway, does this mean that there's a problem with my install?
> Thanks!
>
>

How many signatures did clamd load? (should say in its log file)

Also try clamscan /dev/null, it should say "Known viruses: "

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jvigil at avibank

Dec 2, 2009, 10:37 AM

Post #12 of 15 (2001 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Török Edwin
> Sent: Wednesday, December 02, 2009 10:08 AM
> To: ClamAV users ML
> Subject: Re: [Clamav-users] ClamAV Memory Usage
>
> On 2009-12-02 19:47, Vigil, Joe wrote:
> > When I run the ps command on the clamd process id, the VSZ is 4416 and
> the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ
> includes the VSZ, or if they're separate. Anyway, does this mean that
> there's a problem with my install?
> > Thanks!
> >
> >
>
> How many signatures did clamd load? (should say in its log file)
>
> Also try clamscan /dev/null, it should say "Known viruses: "
>
> Best regards,
> --Edwin

Edwin,
It says:
Known viruses: 660971
Engine version: 0.95

- Joe



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Dec 2, 2009, 10:48 AM

Post #13 of 15 (2000 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
On 2009-12-02 20:37, Vigil, Joe wrote:
>> -----Original Message-----
>> From: clamav-users-bounces [at] lists [mailto:clamav-users-
>> bounces [at] lists] On Behalf Of Török Edwin
>> Sent: Wednesday, December 02, 2009 10:08 AM
>> To: ClamAV users ML
>> Subject: Re: [Clamav-users] ClamAV Memory Usage
>>
>> On 2009-12-02 19:47, Vigil, Joe wrote:
>>
>>> When I run the ps command on the clamd process id, the VSZ is 4416 and
>>>
>> the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ
>> includes the VSZ, or if they're separate. Anyway, does this mean that
>> there's a problem with my install?
>>
>>> Thanks!
>>>
>>>
>>>
>> How many signatures did clamd load? (should say in its log file)
>>
>> Also try clamscan /dev/null, it should say "Known viruses: "
>>
>> Best regards,
>> --Edwin
>>
>
> Edwin,
> It says:
> Known viruses: 660971
> Engine version: 0.95
>

That is right, what if you run clamdscan --reload, and then check your
clamd's logfile? How many signatures there?

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jvigil at avibank

Dec 2, 2009, 11:47 AM

Post #14 of 15 (2012 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Török Edwin
> Sent: Wednesday, December 02, 2009 10:49 AM
> To: ClamAV users ML
> Subject: Re: [Clamav-users] ClamAV Memory Usage
>
> On 2009-12-02 20:37, Vigil, Joe wrote:
> >> -----Original Message-----
> >> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> >> bounces [at] lists] On Behalf Of Török Edwin
> >> Sent: Wednesday, December 02, 2009 10:08 AM
> >> To: ClamAV users ML
> >> Subject: Re: [Clamav-users] ClamAV Memory Usage
> >>
> >> On 2009-12-02 19:47, Vigil, Joe wrote:
> >>
> >>> When I run the ps command on the clamd process id, the VSZ is 4416 and
> >>>
> >> the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ
> >> includes the VSZ, or if they're separate. Anyway, does this mean that
> >> there's a problem with my install?
> >>
> >>> Thanks!
> >>>
> >>>
> >>>
> >> How many signatures did clamd load? (should say in its log file)
> >>
> >> Also try clamscan /dev/null, it should say "Known viruses: "
> >>
> >> Best regards,
> >> --Edwin
> >>
> >
> > Edwin,
> > It says:
> > Known viruses: 660971
> > Engine version: 0.95
> >
>
> That is right, what if you run clamdscan --reload, and then check your
> clamd's logfile? How many signatures there?
>
> Best regards,
> --Edwin

Edwin,
Here's the logfile:
Wed Dec 2 11:43:03 2009 -> No stats for Database check - forcing reload
Wed Dec 2 11:43:38 2009 -> Reading databases from /usr/local/share/clamav
Wed Dec 2 11:44:10 2009 -> Database correctly reloaded (660971 signatures)
Wed Dec 2 11:44:10 2009 -> Reading databases from /usr/local/share/clamav
Wed Dec 2 11:44:37 2009 -> Database correctly reloaded (660971 signatures)

Thanks!

- Joe

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Dec 2, 2009, 11:52 AM

Post #15 of 15 (2002 views)
Permalink
Re: ClamAV Memory Usage [In reply to]
On 2009-12-02 21:47, Vigil, Joe wrote:
>> -----Original Message-----
>> From: clamav-users-bounces [at] lists [mailto:clamav-users-
>> bounces [at] lists] On Behalf Of Török Edwin
>> Sent: Wednesday, December 02, 2009 10:49 AM
>> To: ClamAV users ML
>> Subject: Re: [Clamav-users] ClamAV Memory Usage
>>
>> On 2009-12-02 20:37, Vigil, Joe wrote:
>>
>>>> -----Original Message-----
>>>> From: clamav-users-bounces [at] lists [mailto:clamav-users-
>>>> bounces [at] lists] On Behalf Of Török Edwin
>>>> Sent: Wednesday, December 02, 2009 10:08 AM
>>>> To: ClamAV users ML
>>>> Subject: Re: [Clamav-users] ClamAV Memory Usage
>>>>
>>>> On 2009-12-02 19:47, Vigil, Joe wrote:
>>>>
>>>>
>>>>> When I run the ps command on the clamd process id, the VSZ is 4416 and
>>>>>
>>>>>
>>>> the SZ is 5584. The man page on 'ps' doesn't make it clear if the SZ
>>>> includes the VSZ, or if they're separate. Anyway, does this mean that
>>>> there's a problem with my install?
>>>>
>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
>>>> How many signatures did clamd load? (should say in its log file)
>>>>
>>>> Also try clamscan /dev/null, it should say "Known viruses: "
>>>>
>>>> Best regards,
>>>> --Edwin
>>>>
>>>>
>>> Edwin,
>>> It says:
>>> Known viruses: 660971
>>> Engine version: 0.95
>>>
>>>
>> That is right, what if you run clamdscan --reload, and then check your
>> clamd's logfile? How many signatures there?
>>
>> Best regards,
>> --Edwin
>>
>
> Edwin,
> Here's the logfile:
> Wed Dec 2 11:43:03 2009 -> No stats for Database check - forcing reload
> Wed Dec 2 11:43:38 2009 -> Reading databases from /usr/local/share/clamav
> Wed Dec 2 11:44:10 2009 -> Database correctly reloaded (660971 signatures)
> Wed Dec 2 11:44:10 2009 -> Reading databases from /usr/local/share/clamav
> Wed Dec 2 11:44:37 2009 -> Database correctly reloaded (660971 signatures)
>

I don't know what is going on then, but perhaps you should try measuring
memusage with some other tool.

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.