Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

Improving Scan Speeds on OS X.4.11

 

 

First page Previous page 1 2 Next page Last page  View All ClamAV users RSS feed   Index | Next | Previous | View Threaded


fitzbew at redshanksoftware

Mar 15, 2011, 12:21 PM

Post #1 of 32 (2244 views)
Permalink
Improving Scan Speeds on OS X.4.11

Hello,

I'm running clamav 0.965 on a G5 (1 processor) with OS X Server 10.4.11. Clamav runs as root. This machine is primarily used as a file server, with a mixture of OS X and Windows clients.

A launchdaemon automatically kicks off an overnight scan by sending a command to clamdscan. Only directories that are used by the Windows machines are scanned.

Because of the huge volume of data being scanned (70 Gb), the scan takes about 6 hours to complete.

Is there a practical way to reduce the scan time?

Thanks.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


alvarnell at mac

Mar 15, 2011, 12:37 PM

Post #2 of 32 (2213 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

Add RAM if you haven't maxed it out yet.

Purchase a faster, Intel Mac. Apple has not supported your OS since 2009
and seems to have removed support for PPC Macs from a software development
standpoint.


-Al-

--
Al Varnell
Mountain View, CA

On 3/15/11 12:21 PM, "Russ Tyndall" <fitzbew [at] redshanksoftware> wrote:

> I'm running clamav 0.965 on a G5 (1 processor) with OS X Server 10.4.11.
> Clamav runs as root. This machine is primarily used as a file server, with a
> mixture of OS X and Windows clients.
>
> A launchdaemon automatically kicks off an overnight scan by sending a command
> to clamdscan. Only directories that are used by the Windows machines are
> scanned.
>
> Because of the huge volume of data being scanned (70 Gb), the scan takes about
> 6 hours to complete.
>
> Is there a practical way to reduce the scan time?


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tshaw at oitc

Mar 15, 2011, 1:48 PM

Post #3 of 32 (2209 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

Russ,

Look at your config file. You don't need to scan all more than probably 200KB of a file. If you're using google; don't. It will help for email but probably will not help finding badness on a file server. Likewise with unofficials. Not all unofficials are appropriate for your application.

Lastly when you complied you clamd what compiler options did you pick?

Tom

On Mar 15, 2011, at 3:21 PM, Russ Tyndall wrote:

> Hello,
>
> I'm running clamav 0.965 on a G5 (1 processor) with OS X Server 10.4.11. Clamav runs as root. This machine is primarily used as a file server, with a mixture of OS X and Windows clients.
>
> A launchdaemon automatically kicks off an overnight scan by sending a command to clamdscan. Only directories that are used by the Windows machines are scanned.
>
> Because of the huge volume of data being scanned (70 Gb), the scan takes about 6 hours to complete.
>
> Is there a practical way to reduce the scan time?
>
> Thanks.
>
> -----------------
> Russ Tyndall
> Wake Forest, NC
>
>
>
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


cswiger at mac

Mar 15, 2011, 1:51 PM

Post #4 of 32 (2211 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 12:21 PM, Russ Tyndall wrote:
> Because of the huge volume of data being scanned (70 Gb), the scan takes about 6 hours to complete.
>
> Is there a practical way to reduce the scan time?

As Al noted, 10.4 is about six years old-- released April 2005, last patch was 10.4.11 in Nov 2007.

One thing you might consider doing is using "find /location -mtime 1" to generate a list of which files have been modified over the past day, and only scanning these via clamdscan -f.

Doing this safely depends on whether files can spoof their last-modified timestamp, which depends on how the fileserver is being accessed by clients. If additional safety is required, you can use tools like tripwire, which create checksums of the content and can thus identify files which have changed regardless of the mtime, and use that to generate the list of changed filed to be re-scanned.

Regards,
--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


steve at greengecko

Mar 15, 2011, 2:14 PM

Post #5 of 32 (2215 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Tue, 2011-03-15 at 13:51 -0700, Chuck Swiger wrote:
> On Mar 15, 2011, at 12:21 PM, Russ Tyndall wrote:
> > Because of the huge volume of data being scanned (70 Gb), the scan takes about 6 hours to complete.
> >
> > Is there a practical way to reduce the scan time?
>
> As Al noted, 10.4 is about six years old-- released April 2005, last patch was 10.4.11 in Nov 2007.
>
> One thing you might consider doing is using "find /location -mtime 1" to generate a list of which files have been modified over the past day, and only scanning these via clamdscan -f.
>
> Doing this safely depends on whether files can spoof their last-modified timestamp, which depends on how the fileserver is being accessed by clients. If additional safety is required, you can use tools like tripwire, which create checksums of the content and can thus identify files which have changed regardless of the mtime, and use that to generate the list of changed filed to be re-scanned.
>
> Regards,

find /location -mtime -1

= modified less than a day ago...

Steve

--
Steve Holdoway BSc(Hons) MNZCS <steve [at] greengecko>
http://www.greengecko.co.nz
MSN: steve [at] greengecko
Skype: sholdowa


fitzbew at redshanksoftware

Mar 15, 2011, 3:39 PM

Post #6 of 32 (2213 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 3:37 PM, Al Varnell wrote:

> Add RAM if you haven't maxed it out yet.
>
> Purchase a faster, Intel Mac. Apple has not supported your OS since 2009
> and seems to have removed support for PPC Macs from a software development
> standpoint.

Shucks, I would be thrilled with an older PowerPC Mac as long as it had dual processors.

In some very unscientific testing with a dual processor G5, when I call clamdscan -m, the scan times improve by 75%.

I tested clamdscan -m on the single processor G5 I am working with it but there was only minor scan time improvement and the CPU spiked at 100% for the duration of the scan.

This environment is stuck with 10.4 indefinitely.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 15, 2011, 3:56 PM

Post #7 of 32 (2207 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 4:48 PM, TR Shaw wrote:

> Look at your config file. You don't need to scan all more than probably 200KB of a file.

So you are suggesting I use the MaxScanSize directive to limit scans to the first 200KB of each file? (i.e., add a line to clamd.conf: MaxScanSize 200KB).

I imagine that would speed things up nicely.... :-)


> If you're using google; don't. It will help for email but probably will not help finding badness on a file server. Likewise with unofficials. Not all unofficials are appropriate for your application.

Sorry, Tom, I don't have the knowledge to understand this.

>
> Lastly when you complied you clamd what compiler options did you pick?

I updated the bzip-related libraries and made sure I was using GCC 3.3.

LDFLAGS="-O3 -L/opt/local/lib"

./configure --prefix=/usr/local --mandir=/usr/local/share/man --sysconfdir=/private/etc/spam/clamav/new --enable-bigstack --with-user=clamav --enable-static --with-group=clamav --with-dbdir=/var/clamav --datadir=/var/clamav

Then, make and install.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tshaw at oitc

Mar 15, 2011, 4:10 PM

Post #8 of 32 (2212 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 6:56 PM, Russ Tyndall wrote:

>
> On Mar 15, 2011, at 4:48 PM, TR Shaw wrote:
>
>> Look at your config file. You don't need to scan all more than probably 200KB of a file.
>
> So you are suggesting I use the MaxScanSize directive to limit scans to the first 200KB of each file? (i.e., add a line to clamd.conf: MaxScanSize 200KB).
>
> I imagine that would speed things up nicely.... :-)
>

Yes. Pick a size you feel comfy with but I believe there are few signatures that span large file sizes. You might want to override this once a week to check large zip/gz files but in general this should be good. Let me know how it helps.

>
>> If you're using google; don't. It will help for email but probably will not help finding badness on a file server. Likewise with unofficials. Not all unofficials are appropriate for your application.
>
> Sorry, Tom, I don't have the knowledge to understand this.

If you haven't enabled this in your config or added other sigs then just ignore me here ;-)

>
>>
>> Lastly when you complied you clamd what compiler options did you pick?
>
> I updated the bzip-related libraries and made sure I was using GCC 3.3.
>
> LDFLAGS="-O3 -L/opt/local/lib"
>
> ./configure --prefix=/usr/local --mandir=/usr/local/share/man --sysconfdir=/private/etc/spam/clamav/new --enable-bigstack --with-user=clamav --enable-static --with-group=clamav --with-dbdir=/var/clamav --datadir=/var/clamav
>
> Then, make and install.
>

That's probably as good as you can do for now. If you can get a 10.5 lics then do it as 10.5 fixes some low level process switch slowdowns that were in Tiger. It isn't a panacea but it should help a bit also.

Tom

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


alvarnell at mac

Mar 15, 2011, 4:42 PM

Post #9 of 32 (2217 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/15/11 1:51 PM, "Chuck Swiger" <cswiger [at] mac> wrote:

> As Al noted, 10.4 is about six years old-- released April 2005, last patch was
> 10.4.11 in Nov 2007.
>
True enough. Apple's rule of thumb is that they only support the current
and one previous release which would have made 10.4 unsupported when 10.6
was release in Aug 2009.

I usually judge OS support by security updates. The last Java update was
Release 9 (Jun 2009) and the last Security update was 2009-005 (Sep 2009),
although I see they continued to update compatible versions of iTunes 9.2.1
(Jul 2010) and Safari 4.1.2 (Sep 2010) after that.


-Al-

--
Al Varnell
Mountain View, CA



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 16, 2011, 7:24 AM

Post #10 of 32 (2190 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 4:51 PM, Chuck Swiger wrote:

> One thing you might consider doing is using "find /location -mtime 1" to generate a list of which files have been modified over the past day, and only scanning these via clamdscan -f.

I experimented with this option last night (also suggested by Steve Holdoway), and it works as expected. (Vastly decreases scan time by reducing the number of files that need to be scanned to a mere pittance.) The risk is obvious that a baddie could be overlooked because it might present a false modification date or simply not be recognized by clamav for some period after it gets dropped onto the computer.

I *think* I ran into one gotcha that I had to work around: I had to filter out directories from the Find results...otherwise, clamav would scan those directories whose contents had already been scanned because those contents were already listed elsewhere in the Find results. Users more experienced with Find may have just thought that requirement was self-evident and didn't need to be stated.

My Find command looks something like this, and is supposed to filter out directories and anything modified more than 60 minutes ago:

find [path to directory] [path to second directory] ! -type d -mmin -60 > [path to output file later read by clamav]

I'm now going to do some testing with the MaxScanSize directive.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Bowie_Bailey at BUC

Mar 16, 2011, 9:11 AM

Post #11 of 32 (2190 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/16/2011 10:24 AM, Russ Tyndall wrote:
> On Mar 15, 2011, at 4:51 PM, Chuck Swiger wrote:
>
>> One thing you might consider doing is using "find /location -mtime 1" to generate a list of which files have been modified over the past day, and only scanning these via clamdscan -f.
> I experimented with this option last night (also suggested by Steve Holdoway), and it works as expected. (Vastly decreases scan time by reducing the number of files that need to be scanned to a mere pittance.) The risk is obvious that a baddie could be overlooked because it might present a false modification date or simply not be recognized by clamav for some period after it gets dropped onto the computer.
>
> I *think* I ran into one gotcha that I had to work around: I had to filter out directories from the Find results...otherwise, clamav would scan those directories whose contents had already been scanned because those contents were already listed elsewhere in the Find results. Users more experienced with Find may have just thought that requirement was self-evident and didn't need to be stated.
>
> My Find command looks something like this, and is supposed to filter out directories and anything modified more than 60 minutes ago:
>
> find [path to directory] [path to second directory] ! -type d -mmin -60 > [path to output file later read by clamav]
>
> I'm now going to do some testing with the MaxScanSize directive.

To minimize the risk of a signature being added after the file gets on
the computer, you could continue to scan for a while, rather than using
the 60 minute limit. Depending on your system usage, a limit of one or
two days might still decrease your scan time far enough while still
allowing files to be rescanned with newer signatures.

--
Bowie
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 16, 2011, 10:14 AM

Post #12 of 32 (2187 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 16, 2011, at 12:11 PM, Bowie Bailey wrote:

> To minimize the risk of a signature being added after the file gets on
> the computer, you could continue to scan for a while, rather than using
> the 60 minute limit. Depending on your system usage, a limit of one or
> two days might still decrease your scan time far enough while still
> allowing files to be rescanned with newer signatures.


Yes, it seems I can expand the modification date out to multiple days with little impact on scan times.

In my unscientific testing, I went 3 days and the scan time still stayed under 10 minutes for a directory with 15GB of data. If I scanned the entire directory, the scan time would be about 2 hours.

I believe I could go out much longer than 3 days and still keep the scan periods down to a "reasonable" time.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 16, 2011, 10:31 AM

Post #13 of 32 (2186 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 15, 2011, at 7:10 PM, TR Shaw wrote:

>> On Mar 15, 2011, at 4:48 PM, TR Shaw wrote:
>>
>>> Look at your config file. You don't need to scan all more than probably 200KB of a file.
>>
>> So you are suggesting I use the MaxScanSize directive to limit scans to the first 200KB of each file? (i.e., add a line to clamd.conf: MaxScanSize 200KB).
>>
>> I imagine that would speed things up nicely.... :-)
>>
>
> Yes. Pick a size you feel comfy with but I believe there are few signatures that span large file sizes. You might want to override this once a week to check large zip/gz files but in general this should be good. Let me know how it helps.

A full scan with default settings (MaxScanSize = 20MB) takes about 2 hours to scan a particular directory.

A full scan with MaxScanSize = 1MB takes about 1 hour.

A full scan with MaxScanSize = 200K takes about 18 minutes.

***

So I now have two tactics to minimize scan time: 1) Partially scan ALL files 2) Fully scan a set of recently modified files.

Which is more likely?: That a partial scan (first 200K) misses a baddie? Or that a baddie fakes a modification date?

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tshaw at oitc

Mar 16, 2011, 10:45 AM

Post #14 of 32 (2185 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 16, 2011, at 1:31 PM, Russ Tyndall wrote:

>
> On Mar 15, 2011, at 7:10 PM, TR Shaw wrote:
>
>>> On Mar 15, 2011, at 4:48 PM, TR Shaw wrote:
>>>
>>>> Look at your config file. You don't need to scan all more than probably 200KB of a file.
>>>
>>> So you are suggesting I use the MaxScanSize directive to limit scans to the first 200KB of each file? (i.e., add a line to clamd.conf: MaxScanSize 200KB).
>>>
>>> I imagine that would speed things up nicely.... :-)
>>>
>>
>> Yes. Pick a size you feel comfy with but I believe there are few signatures that span large file sizes. You might want to override this once a week to check large zip/gz files but in general this should be good. Let me know how it helps.
>
> A full scan with default settings (MaxScanSize = 20MB) takes about 2 hours to scan a particular directory.
>
> A full scan with MaxScanSize = 1MB takes about 1 hour.
>
> A full scan with MaxScanSize = 200K takes about 18 minutes.
>
> ***
>
> So I now have two tactics to minimize scan time: 1) Partially scan ALL files 2) Fully scan a set of recently modified files.
>
> Which is more likely?: That a partial scan (first 200K) misses a baddie? Or that a baddie fakes a modification date?


You play craps?????? LOL

Seriously, as for "faking" mod date... you don't have to fake it just uncompress a archive preserving creation and modification dates and viola.

There are plenty of other approaches (save filenames and mod dates in a DB and only scan additions or changes to it; etc.) but all have diminishing returns. After all there is a window between the time malware is stored and the time you detect it as well and no AV is perfect.

Just do the best you can do. If you feel uncomfortable run full scans on weekends and reduced during the week.

Tom

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


bburke at eecs

Mar 16, 2011, 11:36 AM

Post #15 of 32 (2184 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

> find [path to directory] [path to second directory] ! -type d -mmin -60 > [path to output file later read by clamav]

This might not be too much of an issue, but thought I'd point it out: You might change
"! -type d" to "-type f" (better to be more specific), because I don't think you want to
scan device files, pipes, links, etc.

--
Bryan Burke
IT Administrator
Department of Electrical Engineering and Computer Science
University of Tennessee, Knoxville
bburke [at] eecs
(865) 974-4694
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


ged at jubileegroup

Mar 17, 2011, 4:50 AM

Post #16 of 32 (2173 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

Hi there,

On Thu, 17 Mar 2011 Russ Tyndall wrote:

> So I now have two tactics to minimize scan time:
> 1) Partially scan ALL files
> 2) Fully scan a set of recently modified files.

There might be another option. If you have access to something like
inotify on your OS you could feed incoming data to clamd on the fly,
rather than waiting until the next scan window.

Sorry, I haven't used OSX for a while so I don't know what's available.

--

73,
Ged.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 17, 2011, 7:35 AM

Post #17 of 32 (2173 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 17, 2011, at 7:50 AM, G.W. Haywood wrote:

> On Thu, 17 Mar 2011 Russ Tyndall wrote:
>
>> So I now have two tactics to minimize scan time:
>> 1) Partially scan ALL files
>> 2) Fully scan a set of recently modified files.
>
> There might be another option. If you have access to something like
> inotify on your OS you could feed incoming data to clamd on the fly,
> rather than waiting until the next scan window.
>
> Sorry, I haven't used OSX for a while so I don't know what's available.

It appears that 10.5+ has some technology for monitoring the file system:

<http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/FSEvents_ProgGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40005289-CH1-DontLinkElementID_16>

Since my machine is running 10.4, I did not delve into it very far. But, a cursory scan of Google results suggest that methods exist for kicking off scripts when a file hierarchy changes.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


fitzbew at redshanksoftware

Mar 17, 2011, 8:20 AM

Post #18 of 32 (2170 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 16, 2011, at 2:36 PM, Bryan Burke wrote:

>> find [path to directory] [path to second directory] ! -type d -mmin -60 > [path to output file later read by clamav]
>
> This might not be too much of an issue, but thought I'd point it out: You might change
> "! -type d" to "-type f" (better to be more specific), because I don't think you want to
> scan device files, pipes, links, etc.

Ah, thanks. I did not know whether I should exclude those other types, but I *knew* I did not want directories.

Studying the FIND man page a little, I am wondering whether I should actually be using -cmin instead of -mmin. cmin (according to the man page) returns files that have had a "...change of file status information.." in the results.

A little testing shows that it includes files in the results that have been newly introduced into the file system, in addition to files that have been modified. This would solve the issue of an "old" baddie being copied onto the machine with an "old" modification date.

I'm sure it does not get around the risk of faking file times, though.

-----------------
Russ Tyndall
Wake Forest, NC



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


bburke at eecs

Mar 17, 2011, 9:43 AM

Post #19 of 32 (2171 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

> Studying the FIND man page a little, I am wondering whether I should actually be using -cmin instead of -mmin. cmin (according to the man page) returns files that have had a "...change of file status information.." in the results.
>
> A little testing shows that it includes files in the results that have been newly introduced into the file system, in addition to files that have been modified. This would solve the issue of an "old" baddie being copied onto the machine with an "old" modification date.

If I remember correctly, one of those is the time of file data modification, the other of
file metadata modification. So, you might make your test more inclusive and do something
like "-cmin -# -o -mmin -#" where the #s are the time specs you use; i.e. use both.

> I'm sure it does not get around the risk of faking file times, though.

It doesn't I'm pretty sure. I think all three fields (atime, mtime, and ctime) are all
changeable by the owner of the file.

--
Bryan Burke
IT Administrator
Department of Electrical Engineering and Computer Science
University of Tennessee, Knoxville
bburke [at] eecs
(865) 974-4694
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


alvarnell at mac

Mar 17, 2011, 11:56 AM

Post #20 of 32 (2178 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/17/11 7:35 AM, "Russ Tyndall" <fitzbew [at] redshanksoftware> wrote:

>
> On Mar 17, 2011, at 7:50 AM, G.W. Haywood wrote:
>
>> On Thu, 17 Mar 2011 Russ Tyndall wrote:
>>
>>> So I now have two tactics to minimize scan time:
>>> 1) Partially scan ALL files
>>> 2) Fully scan a set of recently modified files.
>>
>> There might be another option. If you have access to something like
>> inotify on your OS you could feed incoming data to clamd on the fly,
>> rather than waiting until the next scan window.
>>
>> Sorry, I haven't used OSX for a while so I don't know what's available.
>
> It appears that 10.5+ has some technology for monitoring the file system:
>
> <http://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/FSEve
> nts_ProgGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40005289-CH
> 1-DontLinkElementID_16>
>
> Since my machine is running 10.4, I did not delve into it very far. But, a
> cursory scan of Google results suggest that methods exist for kicking off
> scripts when a file hierarchy changes.
>
I believe that ClamXav <http://www.clamxav.com> makes use of this feature in
it's Sentry application to watch selected directories. A small subapp
called gfslogger is used to tap into FSEvents.


-Al-

--
Al Varnell
Mountain View, CA



_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Mar 17, 2011, 6:22 PM

Post #21 of 32 (2154 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/16/11 7:24 AM, Russ Tyndall wrote:
>
> On Mar 15, 2011, at 4:51 PM, Chuck Swiger wrote:
>
>> One thing you might consider doing is using "find /location -mtime 1" to generate a list of which files have been modified over the past day, and only scanning these via clamdscan -f.
>
> I experimented with this option last night (also suggested by Steve Holdoway), and it works as expected. (Vastly decreases scan time by reducing the number of files that need to be scanned to a mere pittance.) The risk is obvious that a baddie could be overlooked because it might present a false modification date or simply not be recognized by clamav for some period after it gets dropped onto the computer.

Since you're thinking in this direction you may discover locate is faster than
find though it has issues of it's own as well as opportunity. See more at man
locate. Locate searches a pre-built database rather than crawling your file system.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


cswiger at mac

Mar 18, 2011, 10:18 AM

Post #22 of 32 (2149 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 17, 2011, at 6:22 PM, Dennis Peterson wrote:
> Since you're thinking in this direction you may discover locate is faster than find though it has issues of it's own as well as opportunity. See more at man locate. Locate searches a pre-built database rather than crawling your file system.

Yes, and while locate is great for older files, is not really intended for detecting files which have appeared over the past day on a fileserver. By default, the locate DB is only rebuilt once a week under OS X....

Regards,
--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Mar 18, 2011, 11:02 AM

Post #23 of 32 (2146 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/18/11 10:18 AM, Chuck Swiger wrote:
> On Mar 17, 2011, at 6:22 PM, Dennis Peterson wrote:
>> Since you're thinking in this direction you may discover locate is faster than find though it has issues of it's own as well as opportunity. See more at man locate. Locate searches a pre-built database rather than crawling your file system.
>
> Yes, and while locate is great for older files, is not really intended for detecting files which have appeared over the past day on a fileserver. By default, the locate DB is only rebuilt once a week under OS X....
>
> Regards,

It is entirely configurable and you can build as many DB's as you like and place
them anywhere you like. It's Unix - you're allowed to think.

"...it has issues of it's own as well as opportunity".

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


cswiger at mac

Mar 18, 2011, 11:12 AM

Post #24 of 32 (2144 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On Mar 18, 2011, at 11:02 AM, Dennis Peterson wrote:
>> Yes, and while locate is great for older files, is not really intended for detecting files which have appeared over the past day on a fileserver. By default, the locate DB is only rebuilt once a week under OS X....
>
> It is entirely configurable and you can build as many DB's as you like and place them anywhere you like. It's Unix - you're allowed to think.

Indeed: so think about the tradeoffs of rebuilding locate databases at least daily versus running find once a day. And then consider that you can point find just at the export point of the OP's fileserver, whereas locate.updatedb will index all local filesystems.

--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

Mar 18, 2011, 11:22 AM

Post #25 of 32 (2147 views)
Permalink
Re: Improving Scan Speeds on OS X.4.11 [In reply to]

On 3/18/11 11:12 AM, Chuck Swiger wrote:
> On Mar 18, 2011, at 11:02 AM, Dennis Peterson wrote:
>>> Yes, and while locate is great for older files, is not really intended for detecting files which have appeared over the past day on a fileserver. By default, the locate DB is only rebuilt once a week under OS X....
>>
>> It is entirely configurable and you can build as many DB's as you like and place them anywhere you like. It's Unix - you're allowed to think.
>
> Indeed: so think about the tradeoffs of rebuilding locate databases at least daily versus running find once a day. And then consider that you can point find just at the export point of the OP's fileserver, whereas locate.updatedb will index all local filesystems.
>

Took a while but you're at least thinking. Don't lose sight of the fact that
locate is multipurpose, for most people the DB is going to be generated no
matter what, locate is extremely fast, retains what it has found, and may well
be worth exploring. One thing seldom considered is it can be used for historical
purposes. There's no reason the newest DB has to result in the most recent DB
being deleted, for example. There are other opportunities as well if you don't
mind thinking outside the box.

I get the feeling your negative reaction is entirely a mental exercise and that
you have not given a great deal of thought to the tool and what it can do for
you. If so then that is the very definition of an unqualified response.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

First page Previous page 1 2 Next page Last page  View All ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.