Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

Re: Deletion of local.ign

 

 

ClamAV users RSS feed   Index | Next | Previous | View Threaded


bill at inetmsg

May 20, 2009, 5:26 AM

Post #1 of 9 (1256 views)
Permalink
Re: Deletion of local.ign

Hi Folks,

I just got the following question off-list, which I would like to
respond to on-list for everyones benefit:

> I just got this on the last update using 3.3:
>
> File 'local.ign' timestamp is older than 24 hours - file deleted
>
> That local.ign I need (it's the whitelist of signatures).
>
> How can I stop the script from deleting this file? (apart from touching it
> every day).

The local.ign file contains signatures that the user would like ClamAV
to bypass when scanning a file due to issues like false-positives. This
is a very short-lived option as the signatures as contained in local.ign
require several fields:

file_name : line_number : signature_name

For example, a local.ign entry might look like the following:

winnow_spam_complete.ndb:24:winnow.spam.ts.xmailer.hc.8

The reason these are short-lived entries is that the actual line
placement of an individual signature within a third-party signature
database can change with each update of the database, thereby nullifying
the local.ign whitelist entry, as the original signature line placement
within the signature database may have changed.

The local.ign entries are really meant to be a very short-term option to
bypass a signature until the signature writer can either modify the
signature or remove it from the particular signature database.

Currently, if the clamav-unofficial-sigs script finds that a local.ign
file exists, and its last timestamp (last change/modification time) is
older than 24 hours, it deletes the file as the entries are very likely
no longer valid.

With that said, if clamav-unofficial-sigs script users would like this
feature in the script to be timeframe configurable, or even to have the
ability to disable it (or both), let me know and I will make this
available with the next update release of the script.

Thanks for any feedback or suggestions.

Bill
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jason at electronet

May 20, 2009, 6:34 AM

Post #2 of 9 (1206 views)
Permalink
Re: Deletion of local.ign [In reply to]

> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Bill Landry
> Sent: Wednesday, May 20, 2009 8:27 AM
> To: sanesecurity [at] freelists; clamav-users [at] lists
> Subject: Re: [Clamav-users] Deletion of local.ign
>
>
> The local.ign file contains signatures that the user would like ClamAV
> to bypass when scanning a file due to issues like false-positives.
> This is a very short-lived option as the signatures as contained in
> local.ign require several fields:
>
> file_name : line_number : signature_name
>
> For example, a local.ign entry might look like the following:
>
> winnow_spam_complete.ndb:24:winnow.spam.ts.xmailer.hc.8
>
> The reason these are short-lived entries is that the actual line
> placement of an individual signature within a third-party signature
> database can change with each update of the database, thereby
> nullifying the local.ign whitelist entry, as the original signature
> line placement within the signature database may have changed.
>
> The local.ign entries are really meant to be a very short-term option
> to bypass a signature until the signature writer can either modify the
> signature or remove it from the particular signature database.
>
> Currently, if the clamav-unofficial-sigs script finds that a local.ign
> file exists, and its last timestamp (last change/modification time) is
> older than 24 hours, it deletes the file as the entries are very likely
> no longer valid.
>
> With that said, if clamav-unofficial-sigs script users would like this
> feature in the script to be timeframe configurable, or even to have the
> ability to disable it (or both), let me know and I will make this
> available with the next update release of the script.
>

The logic makes sense, but it seems that management of that file should be
left to the admin. It may take an unknown amount of time for the bad
signature to be removed. A nice feature to the script might be to add
checks for each entry in the file to see if any are still valid before
deleting.

Jason A. Bertoch
Network Administrator
jason [at] electronet
Electronet Broadband Communications
3411 Capital Medical Blvd.
Tallahassee, FL 32308
(V) 850.222.0229 (F) 850.222.8771


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


bill at inetmsg

May 20, 2009, 6:45 AM

Post #3 of 9 (1207 views)
Permalink
Re: Deletion of local.ign [In reply to]

Jason Bertoch wrote:
>> -----Original Message-----
>> From: clamav-users-bounces [at] lists [mailto:clamav-users-
>> bounces [at] lists] On Behalf Of Bill Landry
>> Sent: Wednesday, May 20, 2009 8:27 AM
>> To: sanesecurity [at] freelists; clamav-users [at] lists
>> Subject: Re: [Clamav-users] Deletion of local.ign
>>
>>
>> The local.ign file contains signatures that the user would like ClamAV
>> to bypass when scanning a file due to issues like false-positives.
>> This is a very short-lived option as the signatures as contained in
>> local.ign require several fields:
>>
>> file_name : line_number : signature_name
>>
>> For example, a local.ign entry might look like the following:
>>
>> winnow_spam_complete.ndb:24:winnow.spam.ts.xmailer.hc.8
>>
>> The reason these are short-lived entries is that the actual line
>> placement of an individual signature within a third-party signature
>> database can change with each update of the database, thereby
>> nullifying the local.ign whitelist entry, as the original signature
>> line placement within the signature database may have changed.
>>
>> The local.ign entries are really meant to be a very short-term option
>> to bypass a signature until the signature writer can either modify the
>> signature or remove it from the particular signature database.
>>
>> Currently, if the clamav-unofficial-sigs script finds that a local.ign
>> file exists, and its last timestamp (last change/modification time) is
>> older than 24 hours, it deletes the file as the entries are very likely
>> no longer valid.
>>
>> With that said, if clamav-unofficial-sigs script users would like this
>> feature in the script to be timeframe configurable, or even to have the
>> ability to disable it (or both), let me know and I will make this
>> available with the next update release of the script.
>>
>
> The logic makes sense, but it seems that management of that file should be
> left to the admin. It may take an unknown amount of time for the bad
> signature to be removed. A nice feature to the script might be to add
> checks for each entry in the file to see if any are still valid before
> deleting.

I actually put this logic in my script but then removed it once I
watched the ClamAV webinar on signature making. In order for a *.ign
entry to be valid, it MUST match the filename, signature name, AND the
signature line placement in the database file.

So, the problem with checking to see if the .ign entry still resides in
the database file or not has a flaw. As a signature writer, if I have a
signature that, for example is called:

Spam.Email.123:25:26f757073

and someone reports this as a false positive and I either modify the
signature (meaning it's still there and the script finds it thus leaves
the whitelist entry in the .ign file - now unnecessarily whitelist), or
I replace it with a new entry of the same name (and again the script
finds it and thus leaves the whitelist entry in the .ign file), we run
into all kinds of potential hassles.

I wish ClamAV had instead opted to whitelist based on the actual
hexadecimal signature instead of the signature file:line:name, as that
would make keeping the .ign files up-to-date for a script a much easier
process.

ClamAV, please consider this a feature request... :-)

Bill
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jason at electronet

May 20, 2009, 7:55 AM

Post #4 of 9 (1194 views)
Permalink
Re: Deletion of local.ign [In reply to]

> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Bill Landry
> Sent: Wednesday, May 20, 2009 9:46 AM
> To: ClamAV users ML; sanesecurity [at] freelists
> Subject: Re: [Clamav-users] Deletion of local.ign
>
>
> I wish ClamAV had instead opted to whitelist based on the actual
> hexadecimal signature instead of the signature file:line:name, as that
> would make keeping the .ign files up-to-date for a script a much easier
> process.
>
> ClamAV, please consider this a feature request... :-)
>

Agreed. The real issue is with the implementation of the .ign file.



Jason A. Bertoch
Network Administrator
jason [at] electronet
Electronet Broadband Communications
3411 Capital Medical Blvd.
Tallahassee, FL 32308
(V) 850.222.0229 (F) 850.222.8771

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


dennispe at inetnw

May 20, 2009, 9:22 AM

Post #5 of 9 (1198 views)
Permalink
Re: Deletion of local.ign [In reply to]

Bill Landry wrote:

>
> So, the problem with checking to see if the .ign entry still resides in
> the database file or not has a flaw. As a signature writer, if I have a
> signature that, for example is called:
>
> Spam.Email.123:25:26f757073
>
> and someone reports this as a false positive and I either modify the
> signature (meaning it's still there and the script finds it thus leaves
> the whitelist entry in the .ign file - now unnecessarily whitelist), or
> I replace it with a new entry of the same name (and again the script
> finds it and thus leaves the whitelist entry in the .ign file), we run
> into all kinds of potential hassles.
>
> I wish ClamAV had instead opted to whitelist based on the actual
> hexadecimal signature instead of the signature file:line:name, as that
> would make keeping the .ign files up-to-date for a script a much easier
> process.
>
> ClamAV, please consider this a feature request... :-)

This looks like a rather trivial awk or perl one-liner to reconstitute the .ign
file with current data rather than deleting it. Compare the array of *.ign
signature names against the changed signature files since the last check and if
the signature still exists, record it's new position (and new file name if it's
been promoted to main, for example). Obviously if the source file for the
signature hasn't changed, or a potential new location for it hasn't change
there's no need to scan.

As a minimum, ClamAV should rename the file to deactivate it, leaving it to the
admin to reconstitute it. Although I'd keep a copy of any .ign files in RCS,
personally.

So why would anyone wish to do this? Because a false positive may not be
entirely false. What is ans is not spam or phishing or what ever is subjective
and each of us finally has to decide that some signatures are simply
inappropriate and the vendor is not obliged to respond to claims of FP status,
anyway. None of us has signed a contract that says any signature vendor will
honor our requests to modify what we deem to be FP signatures.

Not that long ago when the ClamAV team was small and overworked I kept a file of
signature names on file and with each new signature change I ran a script to
delete those signatures from the official or unofficial sig files before
dropping them into the working directory. At the time I worked for a large
company that had an aggressive email marketing team who did not have the common
sense to test email campaigns against our own anti-spam tools let alone those of
others, and the expectation was I would consider any blocked content from our
own spam torrent to be FP's. This is absurd, of course, since our tools
indicated strongly what other sites would do with the spam, er, promotional
material :). So I focused on dealing with 50,000 bounces in 10 minutes time from
30,000 sites (tanks any server farm) instead of them cleaning up the sp^H^H
promo material.

Long winded explanation, but it predated the *.ign file and was persistent and
though for a bad idea, effective and easy to maintain. And entirely under my
control which satisfies my inner control freak.


dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tkojm at clamav

May 20, 2009, 9:39 AM

Post #6 of 9 (1205 views)
Permalink
Re: Deletion of local.ign [In reply to]

On Wed, 20 May 2009 06:45:43 -0700
Bill Landry <bill [at] inetmsg> wrote:

> I wish ClamAV had instead opted to whitelist based on the actual
> hexadecimal signature instead of the signature file:line:name, as that
> would make keeping the .ign files up-to-date for a script a much easier
> process.
>
> ClamAV, please consider this a feature request... :-)

Hi Bill,

the .ign database was designed with the ClamAV db maintainers
and not users in mind. It allows us to disable specific signatures
in main.cvd until a new version is published. It requires this precise
information about target signatures for two reasons: safety
and speed. By requiring the line numbers and signature names
the whitelisting mechanism is more resistant to errors (which
could have really bad consequences) but also doesn't slow down
loading of the databases (because we use the line numbers
as the main filter).

I don't know what your script has to do with the .ign databases
but believe it would be much more effective and easier to implement
any workarounds in the script instead of the clamav engine.

Thanks,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Wed May 20 18:37:04 CEST 2009
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


jason at electronet

May 20, 2009, 9:54 AM

Post #7 of 9 (1197 views)
Permalink
Re: Deletion of local.ign [In reply to]

> -----Original Message-----
> From: clamav-users-bounces [at] lists [mailto:clamav-users-
> bounces [at] lists] On Behalf Of Tomasz Kojm
> Sent: Wednesday, May 20, 2009 12:39 PM
> To: clamav-users [at] lists
> Subject: Re: [Clamav-users] Deletion of local.ign
>
> the .ign database was designed with the ClamAV db maintainers
> and not users in mind. It allows us to disable specific signatures
> in main.cvd until a new version is published. It requires this precise
> information about target signatures for two reasons: safety
> and speed. By requiring the line numbers and signature names
> the whitelisting mechanism is more resistant to errors (which
> could have really bad consequences) but also doesn't slow down
> loading of the databases (because we use the line numbers
> as the main filter).
>
> I don't know what your script has to do with the .ign databases
> but believe it would be much more effective and easier to implement
> any workarounds in the script instead of the clamav engine.
>

Would there be any harm in having the hash as another parameter in the .ign
file? It seems like a minor adjustment in parsing for the clam code would
allow more flexibility and automation on the admin side. Alternatively,
since .ign "was designed with the ClamAV db maintainers and not users in
mind", should a new .ign-type file be created with users/admins in mind?


Jason A. Bertoch
Network Administrator
jason [at] electronet
Electronet Broadband Communications
3411 Capital Medical Blvd.
Tallahassee, FL 32308
(V) 850.222.0229 (F) 850.222.8771

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tkojm at clamav

May 20, 2009, 9:59 AM

Post #8 of 9 (1199 views)
Permalink
Re: Deletion of local.ign [In reply to]

On Wed, 20 May 2009 12:54:04 -0400
"Jason Bertoch" <jason [at] electronet> wrote:

> Would there be any harm in having the hash as another parameter in the .ign
> file? It seems like a minor adjustment in parsing for the clam code would
> allow more flexibility and automation on the admin side. Alternatively,
> since .ign "was designed with the ClamAV db maintainers and not users in
> mind", should a new .ign-type file be created with users/admins in mind?

You're welcome to write a patch and send it to us for a review and
incorporation.

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Wed May 20 18:59:09 CEST 2009
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


bill at inetmsg

May 20, 2009, 10:14 AM

Post #9 of 9 (1198 views)
Permalink
Re: Deletion of local.ign [In reply to]

> On Wed, 20 May 2009 06:45:43 -0700
> Bill Landry <bill [at] inetmsg> wrote:
>
>> I wish ClamAV had instead opted to whitelist based on the actual
>> hexadecimal signature instead of the signature file:line:name, as that
>> would make keeping the .ign files up-to-date for a script a much easier
>> process.
>>
>> ClamAV, please consider this a feature request... :-)
>
> Hi Bill,
>
> the .ign database was designed with the ClamAV db maintainers
> and not users in mind. It allows us to disable specific signatures
> in main.cvd until a new version is published. It requires this precise
> information about target signatures for two reasons: safety
> and speed. By requiring the line numbers and signature names
> the whitelisting mechanism is more resistant to errors (which
> could have really bad consequences) but also doesn't slow down
> loading of the databases (because we use the line numbers
> as the main filter).
>
> I don't know what your script has to do with the .ign databases
> but believe it would be much more effective and easier to implement
> any workarounds in the script instead of the clamav engine.

My script allow users to easily add bypass entries into local.ign based on
the third-party signature name they want to whitelist/bypass (this does
not apply to any 'official" clamav signatures).

However, in its current implementation, there is no easy way to manage
these local.ign entries and determine with any certainty whether a
whitelisted signature has been modified, removed, or replaced. The
complete hex signature would allow for this to be done.


I even tried adding local.ign entries like:

junk.ndb:92:Sanesecurity.Junk.92 #2e706870223e4

With the full hex signature listed after the # sign. And even thought
clamav does not complain about this, it will not use a local.ign file
containing a bypass entry in this format.

It would also be nice if ClamAV would recognize any *.ign file and use it,
but it seems it currently will only support local.ign and daily.ign. Any
change that would allow admins to easily manage bypass entries for
third-party database signatures would be greatly appreciated.

Bill

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.