Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

Re: [sanesecurity] Re: Long DB refresh times

 

 

ClamAV users RSS feed   Index | Next | Previous | View Threaded


dehaenp at drever

Apr 24, 2012, 12:11 PM

Post #1 of 13 (1723 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times

On 24 Apr 2012 at 18:11, Steve Basford wrote:

>
> > Has anyone else seen these kinds of delays? Is there any way to get
> > these databases to load faster or to allow ClamAV to continue scanning
> > when the database is being reloaded?
>
> Sorry for the briefness here, as I'm currently sorting out my home
> internet access...
>
> For those having issues:
>
> a) what databases are loaded
> b) what OS are you running
>
> It could be, as someone else suggested a tipping point in memory, but
> we need to get a handle on db's used etc.
>
> Perhaps we can then get a set of test data and create a bugzilla clamav
> entry....
>
>
> Cheers,
>
> Steve
> Sanesecurity


No problem here, a clamscan (which should load all dbs AFAIK) takes 26 sec on an old SPARC. I'm on Solaris, 0.97.3
and I use more or less the same sigs as Robo Kupka but I don't use bofhland sigs.

Pierre

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dehaenp at drever

Apr 25, 2012, 4:33 AM

Post #2 of 13 (1667 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 24 Apr 2012 at 18:11, Steve Basford wrote:

> > Has anyone else seen these kinds of delays? Is there any way to get
> > these databases to load faster or to allow ClamAV to continue scanning
> > when the database is being reloaded?
>
> Sorry for the briefness here, as I'm currently sorting out my home
> internet access...
>
> For those having issues:
>
> a) what databases are loaded
> b) what OS are you running
>
> It could be, as someone else suggested a tipping point in memory, but
> we need to get a handle on db's used etc.
>
> Perhaps we can then get a set of test data and create a bugzilla clamav
> entry....

I don't know if this can help speeding up the process but I collected some statistics on
clamscan of a small file (wallclock duration: ~25sec):

# ./DTraceToolkit-0.96/procsystime -cen clamscan
Elapsed Times for processes clamscan,

SYSCALL TIME (ns)
getuid 3750
gtime 4083
lwp_sigmask 4667
sigpending 4667
systeminfo 5750
times 6167
getpid 7417
sysconfig 11332
fstat 15082
setcontext 16250
getrlimit 17751
fcntl 26000
dup 28750
stat 36251
fsat 44833
mprotect 46000
pread 51416
lstat 53250
schedctl 57500
getdents64 107667
access 154583
ioctl 209083
write 301667
lseek 336754
fstat64 435166
resolvepath 629499
memcntl 749498
llseek 816746
open 1308664
stat64 1714168
brk 1799754
close 1835584
mmap 24644318
munmap 125520181
read 1469157031

Syscall Counts for processes clamscan,

SYSCALL COUNT
fsat 1
getuid 1
gtime 1
lstat 1
lwp_sigmask 1
pread 1
rexit 1
schedctl 1
sigpending 1
systeminfo 1
times 1
fstat 2
getpid 2
getrlimit 2
mprotect 2
setcontext 2
stat 2
sysconfig 3
getdents64 4
dup 6
fcntl 7
access 9
write 10
memcntl 20
ioctl 29
resolvepath 29
open 46
close 52
fstat64 58
lseek 88
stat64 101
brk 170
llseek 175
munmap 612
mmap 674
read 13573

At first glance it looks like the 13573 reads (of 8KB, as seen with another command) take
most of the system time (1.5 sec). Larger reads might help, but it seems that it is in userland
that most of the time is spent. Inside the process, after executing about 19000 the __udivdi3
function, there is a long delay (most of the execution time), and then about 69000 times the
__ashidi3 function. Now, looking at all functions occurences (from binary or libraries), these
are the most often called ones:
FUNCTION COUNT
...
cli_htu32_next 201030
cli_isnumber 202744
atoi 202763
__mul64 206909
cli_caloff 226946
fgets 242252
memccpy 244999
cli_strtok 252988
_realbufend 255751
getxfdat 255774
sprintf 259956
_ndoprnt 259975
ferror 259975
cli_mpool_hex2str 262241
cli_malloc 314546
qcompare 355340
malloc 370694
_malloc_unlocked 372151
free 495212
_free_unlocked 496305
mpool_calloc 558708
strcmp 568989
memset 589919
mutex_lock 869471
mutex_lock_impl 869471
mutex_unlock 869471
cli_mpool_strdup 1211577
hm_addhash 1243854
cli_htu32_find 1243855
strtol 1243908
cli_strtokenize 1407183
strncpy 1413460
cli_chkign 1437106
cli_bm_scanbuff 1437108
cli_chomp 1438745
cli_dbgets 1439239
cli_mpool_virname 1470902
cli_hex2str_to 1506095
strcpy 1525859
hm_sort 1595114
memcmp 1664141
mpool_realloc2 2541350
allocate_aligned 2591715
mpool_realloc 2758207
mpool_malloc 2972354
mpool_free 2989609
to_bits 5564069
strlen 7525133
from_bits 9835704
strchr 13787666
memcpy 25818611


More than 25 000 000 memcpy for 1446134 sigs. It is not due to the perl script I'm scanning
as it is only 2450 bytes long.

HTH
Pierre

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwin+ml-clamav at etorok

Apr 25, 2012, 4:55 AM

Post #3 of 13 (1663 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 04/25/2012 02:33 PM, Pierre Dehaen wrote:
> On 24 Apr 2012 at 18:11, Steve Basford wrote:
>
>>> Has anyone else seen these kinds of delays? Is there any way to get
>>> these databases to load faster or to allow ClamAV to continue scanning
>>> when the database is being reloaded?
>>
>> Sorry for the briefness here, as I'm currently sorting out my home
>> internet access...
>>
>> For those having issues:
>>
>> a) what databases are loaded
>> b) what OS are you running
>>
>> It could be, as someone else suggested a tipping point in memory, but
>> we need to get a handle on db's used etc.
>>
>> Perhaps we can then get a set of test data and create a bugzilla clamav
>> entry....
>
> I don't know if this can help speeding up the process but I collected some statistics on
> clamscan of a small file (wallclock duration: ~25sec):

I think I'm missing some context here: which DB files are slow to load?
The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


steveb_clamav at sanesecurity

Apr 25, 2012, 5:13 AM

Post #4 of 13 (1666 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

> I think I'm missing some context here: which DB files are slow to load?
> The official ones? Just the sanesecurity ones? Any particular DB from the
> sanesecurity ones?

Hi Edwin,

I'm emailed you off-list... but think I've found the issue and work-around.

Sorry for the cross-post to clamav-users.

Cheers,

Steve
Sanesecurity

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwin at clamav

Apr 25, 2012, 5:19 AM

Post #5 of 13 (1663 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 04/25/2012 03:13 PM, Steve Basford wrote:
>
>
>> I think I'm missing some context here: which DB files are slow to load?
>> The official ones? Just the sanesecurity ones? Any particular DB from the
>> sanesecurity ones?
>
> Hi Edwin,
>
> I'm emailed you off-list... but think I've found the issue and work-around.
>
> Sorry for the cross-post to clamav-users.


Most of the time is spent here:

96.19% lt-clamscan libclamav.so.6.1.13 [.] cli_ac_addpatt
2.42% lt-clamscan libc-2.13.so [.] __memcmp_sse2


: if(!ph_add_after && ph->partno <= pattern->partno && (!ph->next || ph->next->partno > pattern->partno)) ▒
47.55 : bc098: movzwl 0x4a(%r12),%eax ▒
2.34 : bc09e: cmp %ax,0x4a(%rbp) ▒
0.09 : bc0a2: ja bbf74 <cli_ac_addpatt+0x294> ▒
0.02 : bc0a8: mov 0x58(%rbp),%rdx ▒
2.03 : bc0ac: test %rdx,%rdx ▒
0.24 : bc0af: je bc127 <cli_ac_addpatt+0x447> ▒
3.94 : bc0b1: cmp 0x4a(%rdx),%ax ▒
5.13 : bc0b5: cmovb %rbp,%r13 ◆
7.47 : bc0b9: jmpq bbf74 <cli_ac_addpatt+0x294>

Thats because all all sigs share a quite long, common prefix as you've found it (in bofhland_malware_URL.ndb).
Perhaps it'd be faster to load these sigs into the BM matcher instead of AC (as they don't use any NDB features).

Best regards,
--Edwin

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dehaenp at drever

Apr 25, 2012, 5:33 AM

Post #6 of 13 (1675 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 25 Apr 2012 at 14:55, Trk Edwin wrote:

> On 04/25/2012 02:33 PM, Pierre Dehaen wrote:
> > On 24 Apr 2012 at 18:11, Steve Basford wrote:
> >
> >>> Has anyone else seen these kinds of delays? Is there any way to get
> >>> these databases to load faster or to allow ClamAV to continue scanning
> >>> when the database is being reloaded?
> >>
> >> Sorry for the briefness here, as I'm currently sorting out my home
> >> internet access...
> >>
> >> For those having issues:
> >>
> >> a) what databases are loaded
> >> b) what OS are you running
> >>
> >> It could be, as someone else suggested a tipping point in memory, but
> >> we need to get a handle on db's used etc.
> >>
> >> Perhaps we can then get a set of test data and create a bugzilla clamav
> >> entry....
> >
> > I don't know if this can help speeding up the process but I collected some statistics on
> > clamscan of a small file (wallclock duration: ~25sec):
>
> I think I'm missing some context here: which DB files are slow to load?
> The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?

$ clamscan --official-db-only=yes afile
afile: OK

----------- SCAN SUMMARY -----------
Known viruses: 1204045
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 14.235 sec (0 m 14 s)


$ clamscan afile
afile: OK

----------- SCAN SUMMARY -----------
Known viruses: 1446134
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 26.130 sec (0 m 26 s)


# This shows time delta between open syscalls (and the DBs used):
$ truss -Dt open clamscan afile
[...]
0.0015 open("/opt/clamav/etc/freshclam.conf", O_RDONLY) = 3
0.0232 open("/opt/clamav/share/clamav/sigwhitelist.ign2", O_RDONLY) = 4
0.0026 open("/opt/clamav/share/clamav/daily.cld", O_RDONLY) = 4
0.0004 open("/opt/clamav/share/clamav/daily.cld", O_RDONLY) = 4
1.4788 open("/opt/clamav/share/clamav/main.cld", O_RDONLY) = 4
11.3070 open("/opt/clamav/share/clamav/winnow_malware.hdb", O_RDONLY) = 4
0.0015 open("/opt/clamav/share/clamav/junk.ndb", O_RDONLY) = 4
1.4502 open("/opt/clamav/share/clamav/jurlbl.ndb", O_RDONLY) = 4
0.2828 open("/opt/clamav/share/clamav/phish.ndb", O_RDONLY) = 4
6.3976 open("/opt/clamav/share/clamav/rogue.hdb", O_RDONLY) = 4
0.0201 open("/opt/clamav/share/clamav/scam.ndb", O_RDONLY) = 4
1.6515 open("/opt/clamav/share/clamav/spamimg.hdb", O_RDONLY) = 4
0.0073 open("/opt/clamav/share/clamav/winnow_malware_links.ndb", O_RDONLY) = 4
0.4164 open("/opt/clamav/share/clamav/MSRBL-Images.hdb", O_RDONLY) = 4
0.0203 open("/opt/clamav/share/clamav/MSRBL-SPAM.ndb", O_RDONLY) = 4
0.2371 open("/opt/clamav/share/clamav/bytecode.cld", O_RDONLY) = 4
0.0609 open("/opt/clamav/share/clamav/pierre.ndb", O_RDONLY) = 4
0.0050 open("/opt/clamav/share/clamav/securiteinfo.hdb", O_RDONLY) = 4
1.0959 open("/opt/clamav/share/clamav/spamattach.hdb", O_RDONLY) = 4
0.0052 open("/opt/clamav/share/clamav/honeynet.hdb", O_RDONLY) = 4
0.0055 open("/opt/clamav/share/clamav/mbl.ndb", O_RDONLY) = 4
0.0512 open("/opt/clamav/share/clamav/sanesecurity.ftm", O_RDONLY) = 4
1.4356 open("afile", O_RDONLY) = 3
afile: OK

----------- SCAN SUMMARY -----------
Known viruses: 1446134
Engine version: 0.97.3
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 26.650 sec (0 m 26 s)


So main.cdl is taking most of the time. Note that I do not complain about the load time: to me,
26sec, it is not a problem. This just delays mail scanning a little bit.

Regards,
Pierre

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


michael at orlitzky

Apr 25, 2012, 7:34 AM

Post #7 of 13 (1678 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 04/25/12 07:55, Trk Edwin wrote:
>>
>> I don't know if this can help speeding up the process but I collected some statistics on
>> clamscan of a small file (wallclock duration: ~25sec):
>
> I think I'm missing some context here: which DB files are slow to load?
> The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?

My problem isn't so much that it takes a while to load the signatures,
but that clamd (and thus the mail server) is effectively down the entire
time.

Assuming clamd has a list of signatures in memory, why not just read the
sigs from disk one-at-a-time, and update the in-memory list atomically?
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Apr 26, 2012, 7:32 AM

Post #8 of 13 (1659 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 4/25/12 7:34 AM, Michael Orlitzky wrote:
> On 04/25/12 07:55, Trk Edwin wrote:
>>>
>>> I don't know if this can help speeding up the process but I collected some statistics on
>>> clamscan of a small file (wallclock duration: ~25sec):
>>
>> I think I'm missing some context here: which DB files are slow to load?
>> The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?
>
> My problem isn't so much that it takes a while to load the signatures,
> but that clamd (and thus the mail server) is effectively down the entire
> time.

This has been a problem on every Sparc system I've ever installed ClamAV on and
that goes back quite a few years. I still use in on several Netra 500 mHz pizza
boxes. It is also quite a memory hole which is more related to the available
memory and number of sigs, so on memory constrained systems I've cut back on the
number of SS signatures. And at my peril, I might add, as they have long been
the most valuable in terms of results. And because of the dead time when
reloading I've cut freshclam to once a day. That has resulted in a net
improvement in detections because of the higher availability time.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


michael at orlitzky

Apr 26, 2012, 10:37 AM

Post #9 of 13 (1668 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 04/26/2012 10:32 AM, Dennis Peterson wrote:
> On 4/25/12 7:34 AM, Michael Orlitzky wrote:
>> On 04/25/12 07:55, Trk Edwin wrote:
>>>>
>>>> I don't know if this can help speeding up the process but I collected some statistics on
>>>> clamscan of a small file (wallclock duration: ~25sec):
>>>
>>> I think I'm missing some context here: which DB files are slow to load?
>>> The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?
>>
>> My problem isn't so much that it takes a while to load the signatures,
>> but that clamd (and thus the mail server) is effectively down the entire
>> time.
>
> This has been a problem on every Sparc system I've ever installed ClamAV on and
> that goes back quite a few years. I still use in on several Netra 500 mHz pizza
> boxes. It is also quite a memory hole which is more related to the available
> memory and number of sigs, so on memory constrained systems I've cut back on the
> number of SS signatures. And at my peril, I might add, as they have long been
> the most valuable in terms of results. And because of the dead time when
> reloading I've cut freshclam to once a day. That has resulted in a net
> improvement in detections because of the higher availability time.
>

The signature databases are created once, and loaded thousands of times.
They should just be sorted, so that lookups are instantaneous.

Then it's trivial to update the databases in the background, because you
can quickly determine if a particular signature was added or deleted.
The wall-time-elapsed would be a bit worse, but nobody would care.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwin at clamav

Apr 26, 2012, 11:18 AM

Post #10 of 13 (1660 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 04/26/2012 08:37 PM, Michael Orlitzky wrote:
> On 04/26/2012 10:32 AM, Dennis Peterson wrote:
>> On 4/25/12 7:34 AM, Michael Orlitzky wrote:
>>> On 04/25/12 07:55, Trk Edwin wrote:
>>>>>
>>>>> I don't know if this can help speeding up the process but I collected some statistics on
>>>>> clamscan of a small file (wallclock duration: ~25sec):
>>>>
>>>> I think I'm missing some context here: which DB files are slow to load?
>>>> The official ones? Just the sanesecurity ones? Any particular DB from the sanesecurity ones?
>>>
>>> My problem isn't so much that it takes a while to load the signatures,
>>> but that clamd (and thus the mail server) is effectively down the entire
>>> time.
>>
>> This has been a problem on every Sparc system I've ever installed ClamAV on and
>> that goes back quite a few years. I still use in on several Netra 500 mHz pizza
>> boxes. It is also quite a memory hole which is more related to the available
>> memory and number of sigs, so on memory constrained systems I've cut back on the
>> number of SS signatures. And at my peril, I might add, as they have long been
>> the most valuable in terms of results. And because of the dead time when
>> reloading I've cut freshclam to once a day. That has resulted in a net
>> improvement in detections because of the higher availability time.
>>
>
> The signature databases are created once, and loaded thousands of times.
> They should just be sorted, so that lookups are instantaneous.
>
> Then it's trivial to update the databases in the background, because you
> can quickly determine if a particular signature was added or deleted.
> The wall-time-elapsed would be a bit worse, but nobody would care.

Its a bit more complicated than that. To ensure fast pattern-matching the signatures are loaded into an Aho-Corasick trie for example.
It would be possible to add to the trie (thats what happens when loading signatures), but removing is more tricky.
And to determine what to remove you need to go through all the signatures in the database anyway.
Also updating the loaded signature database would require the scanning threads to take read locks, which would slow things down
and make updating it harder (right now the loaded signature database is never modified, hence no locks are needed).

It would be easier to just move reload_db to a different thread and allow scanning with the old database during the DB reload.
Then when the DB reload is finished atomically replace the engine pointer and free the old engine.
Downside would be that you get twice the memory usage during reload, but you don't have downtime,
so this should probably be controlled by a flag in clamd.conf.

https://bugzilla.clamav.net/show_bug.cgi?id=790#c14

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jerry at seibercom

Apr 26, 2012, 11:38 AM

Post #11 of 13 (1665 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On Thu, 26 Apr 2012 21:18:41 +0300
Török Edwin articulated:

{snip}

>Its a bit more complicated than that. To ensure fast pattern-matching
>the signatures are loaded into an Aho-Corasick trie for example. It
>would be possible to add to the trie (thats what happens when loading
>signatures), but removing is more tricky. And to determine what to
>remove you need to go through all the signatures in the database
>anyway. Also updating the loaded signature database would require the
>scanning threads to take read locks, which would slow things down and
>make updating it harder (right now the loaded signature database is
>never modified, hence no locks are needed).
>
>It would be easier to just move reload_db to a different thread and
>allow scanning with the old database during the DB reload. Then when
>the DB reload is finished atomically replace the engine pointer and
>free the old engine. Downside would be that you get twice the memory
>usage during reload, but you don't have downtime, so this should
>probably be controlled by a flag in clamd.conf.
>
>https://bugzilla.clamav.net/show_bug.cgi?id=790#c14

Well, with the exception of systems starved for memory to begin with, I
think that would be a perfectly acceptable solution. Adding a flag in
clamd.conf to disable this and use the present method would seem like a
good compromise.

--
Jerry ♔

Disclaimer: off-list followups get on-list replies or get ignored.
Please do not ignore the Reply-To header.
__________________________________________________________________

".reh deknaht neve reven I ,wonk uoy ,dna -- knird ot em evord ohw
namow a saw tI"
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


ged at jubileegroup

Apr 27, 2012, 3:46 AM

Post #12 of 13 (1656 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

Hi there,

In the few days before Fri, 27 Apr 2012, Dennis, Michael, Edwin, Pierre and Jerry wrote:

>On 26 Apr 2012 at 21:18, T?r?k Edwin wrote:
>>On 04/26/2012 08:37 PM, Michael Orlitzky wrote:
>>> On 04/26/2012 10:32 AM, Dennis Peterson wrote:
>>>> On 4/25/12 7:34 AM, Michael Orlitzky wrote:
>>>>> On 04/25/12 07:55, T?r?k Edwin wrote:
>>>>>>>
>>>>>> ... which DB files are slow to load?
>>>>>
>>>>> ... clamd (and thus the mail server) is effectively down the entire time.
>>>>
>>>> ... on memory constrained systems I've cut back on the number of SS signatures.
>>>
>>> The signature databases are created once, and loaded thousands of times.
>>> They should just be sorted, so that lookups are instantaneous.
>>>
>>> Then it's trivial to update the databases in the background, because you
>>> can quickly determine if a particular signature was added or deleted.
>>> The wall-time-elapsed would be a bit worse, but nobody would care.
>>
>> Its a bit more complicated than that. ...
>> It would be easier to just move reload_db to a different thread and
>> allow scanning with the old database during the DB reload.
>> ... should probably be controlled by a flag in clamd.conf.
>
> ... different processes rather than with 2 threads would at least free all
> the initial process memory ... would be more tricky between 2 processes...

+1 to the idea. I don't see that there's a problem with the outlined
approaches, and even the odd memory leak might be mitigated. :)

It does seem odd to me that people appear to be running ClamAV on
memeory constrained systems. I'd suggest that those systems might not
be suitable for the task.

The load times have never been a problem to me, but then I don't rely
on ClamAV for dealing with the vast majority of unwanted mail. That's
a luxury I can afford which perhaps most other cannot. But it has to
be worth mentioning that simple things like the GreetPause feature in
Sendmail, dropping connections from dynamic IPs and known spam sources
(particularly by use of dnsbls) and greylisting all consume virtually
no CPU. The vast majority of mail is unwanted; there's absolutely no
need to scan it all because the vast majority of that vast majority is
easily identified as unwanted - without the use of complex tools which
consume a lot of resources. Let ClamAV (and for that matter other CPU
hungry things like SpamAssassin) burn the cycles only when necessary.

Less than one percent of my mail is scanned by ClamAV. This includes
all mail which is sent or accepted. The rest is rejected by very much
simpler, cheaper, smaller and quicker processes.

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Apr 27, 2012, 7:59 AM

Post #13 of 13 (1649 views)
Permalink
Re: [sanesecurity] Re: Long DB refresh times [In reply to]

On 4/27/12 3:46 AM, G.W. Haywood wrote:

>
> It does seem odd to me that people appear to be running ClamAV on
> memeory constrained systems. I'd suggest that those systems might not
> be suitable for the task.

Adding memory to an older Sparc system does not affect the signature loading
time at all. It only determines how many third-party signatures you can finally
load. An 8G system is the same as a 2G system in this regard. Adding a second
processor doesn't change anything, either. I've built and run it on far faster
Sparc systems and they never keep up with an equivalent Intel system even when
using all the available optimizations in the compiler.

As for why, one of these servers has been running non-stop since 2001,
interrupted only by patching and a colo outage. One 500mHz proc, 2G ram, 72G
mirrored storage in a cheap 1U package and it more than keeps up with the
workload. It is a mail list server that also handles 90 vanity domains.

$ uptime
7:43am up 748 day(s), 22:43, 0 users, load average: 0.47, 0.39, 0.39

Speaking of Oracle systems, I'm running ClamAV on several Oracle Database
Appliances with Oracle's Linux and the load times are measured in seconds for
the core signatures. Will be adding it to some Exadata and ZFS storage appliance
systems shortly and I expect it will be fairly painless to load and scan the
user space. So yeah, throwing hardware at a problem can both solve and mask
problems.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.