Mailing List Archive: ClamAV: users

improving ClamAV private mirroring?

Index | Next | Previous | View Threaded

qralston+ml.clamav-users at andrew

Jul 11, 2011, 10:30 AM

Post #1 of 16 (1175 views)
Permalink

improving ClamAV private mirroring?

We are in a situation where we have multiple hosts that need to run
ClamAV, but those hosts are highly restricted in what outbound
Internet access they have. Thus, we need to run a local ClamAV
mirror.

We have been doing this for several months now, using the "Serve .cvd
files from a local web server" (CVD) approach, here:

http://wiki.clamav.net/bin/view/Main/CvdPrivateMirror

This has been extremely problematic for us. And unfortunately, the
"Serve .cvd+.cdiff files from a local web server" (CDIFF) approach
probably won't be any better.

The reason why building a private mirror in this fashion is so
problematic is because both the CVD and CDIFF approaches require your
mirror to download the new CVD file for every database update. For
the CVD approach, this is necessary because freshclam on the clients
is looking only for CVD files (due to disabling ScriptedUpdates). And
for the CDIFF approach, you still need the CVD files, in case a
freshclam client is outdated and needs old CDIFF files that you don't
have anymore.

Downloading the full CVD files for every database update is painful,
particularly for the safebrowsing DB.

For example, we run freshclam on the mirror master every 30 minutes
via cron (as per the comments in freshclam.conf regarding using the
safebrowsing DB), and we have had multiple instances where the next
cron iteration of freshclam starts before the previous instance
completes!

So, here's my main question: is there some way to eliminate the need
to download the latest CVD files for every update?

I was thinking along the lines of adding two new options to freshclam,
as follows:

# The RetainCLDFiles option causes freshclam to save the database
# diff (CDIFF) files it downloads (to [re]build the CLD database
# files) into the DatabaseDirectory. Normally, freshclam deletes
# the CDIFF files after it rebuilds the CLD files, as they are no
# longer needed in that case. Note that freshclam will never
# delete old CDIFF files, so you will need to use some other
# mechanism (e.g., a cron job) to do so. Use this option if you
# want this host to act as a ClamAV private mirror for other
# ClamAV/freshclam hosts on your network.
# Default: disabled
#RetainCLDFiles yes

# The DownloadCLDFiles option affects how freshclam behaves when
# it cannot update a local database via CDIFF files (e.g., because
# it needs an older CDIFF file that is no longer available) and
# must therefore download a fresh copy of the database file.
# Normally, in this case, freshclam will insist on downloading the
# CVD file, so that it can verify the signature. However, if you
# wish to point freshclam at a private mirror, and you trust the
# CLD files from that server, then you can enable this option to
# cause freshclam to additionally try to download CLD database
# files if CVD database files are not available. You should ONLY
# use this option when freshclam is pointed at a private mirror
# that you explicitly control and trust!
# Default: disabled
#DownloadCLDFiles yes

Would this work? I.e., are the CLD files not tied in any way to the
host that builds them? (I.e., they're architecture-independent?)

Is there a better way to run a private mirror avoid having to download
the CVD files for every database update?

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

chris at westnet

Jul 11, 2011, 10:40 AM

Post #2 of 16 (1158 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On Mon, 11 Jul 2011, James Ralston wrote:

> We are in a situation where we have multiple hosts that need to run
> ClamAV, but those hosts are highly restricted in what outbound
> Internet access they have. Thus, we need to run a local ClamAV
> mirror.

I have one machine run freshclam, and use rsync to update all my other
servers with the databases.

The clamav user has to have ssl keys set up so
it can ssh to the other servers without a password.
Then, freshclam.conf has this:
OnUpdateExecute /usr/local/sbin/ClamPush.sh

ClamPush.sh is:
#!/usr/bin/bash

CLAMDIR=/usr/local/share/clamav
RSYNC=/usr/local/bin/rsync

cd $CLAMDIR

for HOST in 'server1' 'server2' 'server3'
do
echo "Updating $HOST:\n"
$RSYNC -avzr --delete . $HOST:$CLAMDIR
echo "Done with $HOST\n\n"
done

==========================================================
Chris Candreva -- chris [at] westnet -- (914) 948-3162
WestNet Internet Services of Westchester
http://www.westnet.com/
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

nathan at cmpublishers

Jul 11, 2011, 11:57 AM

Post #3 of 16 (1159 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 7/11/2011 1:30 PM, James Ralston wrote:
> We are in a situation where we have multiple hosts that need to run
> ClamAV, but those hosts are highly restricted in what outbound
> Internet access they have. Thus, we need to run a local ClamAV
> mirror.
>
> We have been doing this for several months now, using the "Serve .cvd
> files from a local web server" (CVD) approach, here:
>
> http://wiki.clamav.net/bin/view/Main/CvdPrivateMirror
>

Wiki appears to be down.
again. :-)

We use the mirrored system, and it works fairly well.

One freshclam pulling cvd's from outside, several freshclams pulling
from the local mirror.

The only issue we have is the mirror getting behind every once in a
while, not much of a big deal.

--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com

Attachments:

signature.asc (0.25 KB)

luca at clamav

Jul 12, 2011, 3:01 PM

Post #4 of 16 (1142 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

Hello James,

> We are in a situation where we have multiple hosts that need to run
> ClamAV, but those hosts are highly restricted in what outbound
> Internet access they have. Thus, we need to run a local ClamAV
> mirror.

You can install a http proxy server and restrict access to cvd+cdiff files
on db.*.clamav.net.

Best regards

--
Luca Gibelli (luca _at_ clamav.net) ClamAV, a GPL anti-virus toolkit
[Tel] +39 0187 1851862 [Fax] +39 0187 1852252 [IM] nervous/jabber.linux.it
PGP key id 5EFC5582 @ any key-server || http://www.clamav.net/gpg/luca.gpg
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

qralston+ml.clamav-users at andrew

Jul 14, 2011, 9:57 AM

Post #5 of 16 (1132 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 2011-07-11 at 13:40-04 Christopher X Candreva <chris [at] westnet> wrote:

> I have one machine run freshclam, and use rsync to update all my
> other servers with the databases. The clamav user has to have ssl
> keys set up so it can ssh to the other servers without a password.
> Then, freshclam.conf has this: ...

Yeah, we've considered setting up something similar. It wouldn't be
that difficult.

But the thing is, freshclam already has 99% of the code to do this.
All it needs is a "keep the CDIFF files around" option (to enable on
the master) and a "try to grab the CLD file if the CVD file isn't
available" option (to enable on the clients). That's it.

It just seems silly to (essentially) write another version of
freshclam when the current version just needs two new options to do
what is necessary.

On 2011-07-11 14:57:31 -0400 Nathan Gibbs <nathan [at] cmpublishers> wrote:

> We use the mirrored system, and it works fairly well. One freshclam
> pulling cvd's from outside, several freshclams pulling from the
> local mirror. The only issue we have is the mirror getting behind
> every once in a while, not much of a big deal.

Alas, our mirror gets behind much more frequently. I don't know if we
keep landing on overloaded mirrors or what, but it's an issue for us.

On 2011-07-13 00:01:26 +0200 Luca Gibelli <luca [at] clamav> wrote:

> > We are in a situation where we have multiple hosts that need to
> > run ClamAV, but those hosts are highly restricted in what outbound
> > Internet access they have. Thus, we need to run a local ClamAV
> > mirror.
>
> You can install a http proxy server and restrict access to cvd+cdiff
> files on db.*.clamav.net.

I appreciate the suggestion, but to clarify: we are prohibited by
policy from allowing any outbound web access for these hosts, proxied
or not. (The policy is dictated from on high, and will not change.)
We *MUST* use a local private mirror, period.

It would seem that we aren't the only ones who are contemplating
hacking together our own "distribution from a local mirror" mechanism,
because freshclam's solution just isn't adequate/efficient.

If I were to provide a patch that adds the two options I discussed
above to freshclam, would you at least consider accepting it into
trunk?

We need this functionality. But if you won't accept a patch to add
it, then we will have to throw away freshclam and design our own
solution, because I don't want to be put in the situation where I have
to maintain my own local patches and update them after every release.

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

dennispe at inetnw

Jul 14, 2011, 10:30 AM

Post #6 of 16 (1126 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 7/14/11 9:57 AM, James Ralston wrote:
> On 2011-07-11 at 13:40-04 Christopher X Candreva<chris [at] westnet> wrote:
>
>> I have one machine run freshclam, and use rsync to update all my
>> other servers with the databases. The clamav user has to have ssl
>> keys set up so it can ssh to the other servers without a password.
>> Then, freshclam.conf has this: ...
>
> Yeah, we've considered setting up something similar. It wouldn't be
> that difficult.

I've just run into this on a number of systems that we've migrated from McAfee
to ClamAV. One system faces the internet and is not allowed any outbound except
DNS (it is a DNS server). Rsync is the quickest way to get atomic copies and
reliable updates. But it is always a full pull owing to the cdiff issue.

Not that be a deal but esthetically annoying.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

edwintorok at gmail

Jul 14, 2011, 10:33 AM

Post #7 of 16 (1141 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 07/14/2011 07:57 PM, James Ralston wrote:
> On 2011-07-11 at 13:40-04 Christopher X Candreva <chris [at] westnet> wrote:
>
>> I have one machine run freshclam, and use rsync to update all my
>> other servers with the databases. The clamav user has to have ssl
>> keys set up so it can ssh to the other servers without a password.
>> Then, freshclam.conf has this: ...
>
> Yeah, we've considered setting up something similar. It wouldn't be
> that difficult.
>
> But the thing is, freshclam already has 99% of the code to do this.
> All it needs is a "keep the CDIFF files around" option (to enable on
> the master) and a "try to grab the CLD file if the CVD file isn't
> available" option (to enable on the clients). That's it.

I think downloading CLD file is sufficient.
On a LAN it'll probably be faster than downloading&applying all the individual updates.

I think you might be able to configure freshclam to download CLDs with DatabaseCustomURL.
Something like this (untested):
DatabaseMirror <internal-server-with-some-older-version-of-cvds>
DatabaseCustomURL http://<your-internal-webserver>/main.cld
DatabaseCustomURL http://<your-internal-webserver>/main.cvd
DatabaseCustomURL http://<your-internal-webserver>/daily.cld
DatabaseCustomURL http://<your-internal-webserver>/daily.cvd
DatabaseCustomURL http://<your-internal-webserver>/bytecode.cld
DatabaseCustomURL http://<your-internal-webserver>/bytecode.cvd
DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cld
DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cvd

The CLD files are digitally signed too, so you get almost the same integrity checks as with the CVD already.

>
> It just seems silly to (essentially) write another version of
> freshclam when the current version just needs two new options to do
> what is necessary.
>
> On 2011-07-11 14:57:31 -0400 Nathan Gibbs <nathan [at] cmpublishers> wrote:
>
>> We use the mirrored system, and it works fairly well. One freshclam
>> pulling cvd's from outside, several freshclams pulling from the
>> local mirror. The only issue we have is the mirror getting behind
>> every once in a while, not much of a big deal.
>
> Alas, our mirror gets behind much more frequently. I don't know if we
> keep landing on overloaded mirrors or what, but it's an issue for us.
>
> On 2011-07-13 00:01:26 +0200 Luca Gibelli <luca [at] clamav> wrote:
>
>>> We are in a situation where we have multiple hosts that need to
>>> run ClamAV, but those hosts are highly restricted in what outbound
>>> Internet access they have. Thus, we need to run a local ClamAV
>>> mirror.
>>
>> You can install a http proxy server and restrict access to cvd+cdiff
>> files on db.*.clamav.net.
>
> I appreciate the suggestion, but to clarify: we are prohibited by
> policy from allowing any outbound web access for these hosts, proxied
> or not. (The policy is dictated from on high, and will not change.)
> We *MUST* use a local private mirror, period.
>
> It would seem that we aren't the only ones who are contemplating
> hacking together our own "distribution from a local mirror" mechanism,
> because freshclam's solution just isn't adequate/efficient.
>
> If I were to provide a patch that adds the two options I discussed
> above to freshclam, would you at least consider accepting it into
> trunk?
>
> We need this functionality. But if you won't accept a patch to add
> it, then we will have to throw away freshclam and design our own
> solution, because I don't want to be put in the situation where I have
> to maintain my own local patches and update them after every release.

Please open an enhancement request on bugzilla.

Thanks,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

qralston+ml.clamav-users at andrew

Jul 14, 2011, 2:25 PM

Post #8 of 16 (1128 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 2011-07-14 at 20:33+03 T�r�k Edwin <edwintorok [at] gmail> wrote:

> I think you might be able to configure freshclam to download CLDs
> with DatabaseCustomURL.
>
> Something like this (untested):
>
> DatabaseMirror <internal-server-with-some-older-version-of-cvds>
> DatabaseCustomURL http://<your-internal-webserver>/main.cld
> DatabaseCustomURL http://<your-internal-webserver>/main.cvd
> DatabaseCustomURL http://<your-internal-webserver>/daily.cld
> DatabaseCustomURL http://<your-internal-webserver>/daily.cvd
> DatabaseCustomURL http://<your-internal-webserver>/bytecode.cld
> DatabaseCustomURL http://<your-internal-webserver>/bytecode.cvd
> DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cld
> DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cvd

I've just spent some time testing this, and I think this will be a
much better solution. In essence, we can use this to force freshclam
to pull the CLD files from our private mirror, instead of the CVD
files. freshclam appears to use the timestamps of the files on the
web server to determine whether they're more recent than the local
copies, and --quiet suppresses all errors about non-existent files and
duplicate databases.

I'll test this more thoroughly, and report back how well it works for
us.

> I think downloading CLD file is sufficient. On a LAN it'll probably
> be faster than downloading & applying all the individual updates.

It wouldn't surprise me.

The cost of pulling the full CVD/CLD files versus the CDIFF files over
the LAN is negligible for us; we have plenty of bandwidth there. What
we're trying to minimize is the amount of data we have to pull from
the (public) clam mirrors.

> The CLD files are digitally signed too, so you get almost the same
> integrity checks as with the CVD already.

Are you sure about that? Because sigtool says:

$ sigtool -i safebrowsing.cvd
File: safebrowsing.cvd
Build time: 14 Jul 2011 14:45 -0400
Version: 30807
Signatures: 710259
Functionality level: 60
Builder: google
MD5: 2b1b2e868dd74f2aab83bb79c55a68d8
Digital signature: ZstS5RdHytv71PgvErgszQPaVbPqtqgmNrE+w//3lgS0bhP6rrPb87NVfncufL9H2kh/LLx1wwyMPPIJVWsbSYKck4vcwz+ErezX+81gTilryxcrmmEMTWH6WjRvKj24wuqSIF78473JuZWB6Wwi8q2Wgojh1BgBaCB7ghuV/3j
LibClamAV Warning: Detected duplicate databases safebrowsing.cvd and safebrowsing.cld, please manually remove one of them
Verification OK.

$ sigtool -i safebrowsing.cld
File: safebrowsing.cld
Build time: 14 Jul 2011 14:45 -0400
Version: 30807
Signatures: 710259
Functionality level: 60
Builder: google
Verification OK.

The CLD file lacks the "MD5" and "Digital signature" info. Is there
another signature that sigtool isn't displaying?

But even if the CLD files aren't digitally signed, that's not a
dealbreaker, as we can create other mechanisms to verify that the CLD
files weren't tampered with on the wire.

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

tkojm at clamav

Jul 14, 2011, 2:36 PM

Post #9 of 16 (1133 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On Thu Jul 14 2011 23:25:34 GMT+0200 (CET)
James Ralston <qralston+ml.clamav-users [at] andrew> wrote:

>> The CLD files are digitally signed too, so you get almost the same
>> integrity checks as with the CVD already.
>
> Are you sure about that? Because sigtool says:
>
> $ sigtool -i safebrowsing.cvd
> File: safebrowsing.cvd
> Build time: 14 Jul 2011 14:45 -0400
> Version: 30807
> Signatures: 710259
> Functionality level: 60
> Builder: google
> MD5: 2b1b2e868dd74f2aab83bb79c55a68d8
> Digital signature: ZstS5RdHytv71PgvErgszQPaVbPqtqgmNrE+w//3lgS0bhP6rrPb87NVfncufL9H2kh/LLx1wwyMPPIJVWsbSYKck4vcwz+ErezX+81gTilryxcrmmEMTWH6WjRvKj24wuqSIF78473JuZWB6Wwi8q2Wgojh1BgBaCB7ghuV/3j
> LibClamAV Warning: Detected duplicate databases safebrowsing.cvd and safebrowsing.cld, please manually remove one of them
> Verification OK.
>
> $ sigtool -i safebrowsing.cld
> File: safebrowsing.cld
> Build time: 14 Jul 2011 14:45 -0400
> Version: 30807
> Signatures: 710259
> Functionality level: 60
> Builder: google
> Verification OK.
>
> The CLD file lacks the "MD5" and "Digital signature" info. Is there
> another signature that sigtool isn't displaying?

Yes, there's an additional digital signature stored in the .info file
inside the CLD container.

Cheers,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Thu Jul 14 23:33:11 CEST 2011
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

edwin at clamav

Jul 14, 2011, 2:36 PM

Post #10 of 16 (1133 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 07/15/2011 12:25 AM, James Ralston wrote:
> On 2011-07-14 at 20:33+03 T�r�k Edwin <edwintorok [at] gmail> wrote:
>
>> I think you might be able to configure freshclam to download CLDs
>> with DatabaseCustomURL.
>>
>> Something like this (untested):
>>
>> DatabaseMirror <internal-server-with-some-older-version-of-cvds>
>> DatabaseCustomURL http://<your-internal-webserver>/main.cld
>> DatabaseCustomURL http://<your-internal-webserver>/main.cvd
>> DatabaseCustomURL http://<your-internal-webserver>/daily.cld
>> DatabaseCustomURL http://<your-internal-webserver>/daily.cvd
>> DatabaseCustomURL http://<your-internal-webserver>/bytecode.cld
>> DatabaseCustomURL http://<your-internal-webserver>/bytecode.cvd
>> DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cld
>> DatabaseCustomURL http://<your-internal-webserver>/safebrowsing.cvd
>
> I've just spent some time testing this, and I think this will be a
> much better solution. In essence, we can use this to force freshclam
> to pull the CLD files from our private mirror, instead of the CVD
> files. freshclam appears to use the timestamps of the files on the
> web server to determine whether they're more recent than the local
> copies, and --quiet suppresses all errors about non-existent files and
> duplicate databases.
>
> I'll test this more thoroughly, and report back how well it works for
> us.
>
>> I think downloading CLD file is sufficient. On a LAN it'll probably
>> be faster than downloading & applying all the individual updates.
>
> It wouldn't surprise me.
>
> The cost of pulling the full CVD/CLD files versus the CDIFF files over
> the LAN is negligible for us; we have plenty of bandwidth there. What
> we're trying to minimize is the amount of data we have to pull from
> the (public) clam mirrors.
>
>> The CLD files are digitally signed too, so you get almost the same
>> integrity checks as with the CVD already.
>
> Are you sure about that? Because sigtool says:
>
> $ sigtool -i safebrowsing.cvd
> File: safebrowsing.cvd
> Build time: 14 Jul 2011 14:45 -0400
> Version: 30807
> Signatures: 710259
> Functionality level: 60
> Builder: google
> MD5: 2b1b2e868dd74f2aab83bb79c55a68d8
> Digital signature: ZstS5RdHytv71PgvErgszQPaVbPqtqgmNrE+w//3lgS0bhP6rrPb87NVfncufL9H2kh/LLx1wwyMPPIJVWsbSYKck4vcwz+ErezX+81gTilryxcrmmEMTWH6WjRvKj24wuqSIF78473JuZWB6Wwi8q2Wgojh1BgBaCB7ghuV/3j
> LibClamAV Warning: Detected duplicate databases safebrowsing.cvd and safebrowsing.cld, please manually remove one of them
> Verification OK.
>
> $ sigtool -i safebrowsing.cld
> File: safebrowsing.cld
> Build time: 14 Jul 2011 14:45 -0400
> Version: 30807
> Signatures: 710259
> Functionality level: 60
> Builder: google
> Verification OK.
>
> The CLD file lacks the "MD5" and "Digital signature" info. Is there
> another signature that sigtool isn't displaying?

Yes, in the .info file inside the CLD/CVD. You have hashes and the entire file is digitally signed.
A CLD/CVD without a .info file is not valid. IIRC this was introduced in 0.96.

Here is daily.info for example:
ClamAV-VDB:14 Jul 2011 10-06 -0400:13321:151983:60:X:X:ccordes:1310652386
daily.cfg:343:9d6935dd00eeb62fa3f4d23ca12fd2b43b7f0f3ba521389d608d42e131fd028f
daily.ign:4913:62ebdb0bf5ee150585b4f44cab99db98391a97f5d91817b46fc380d8749ea67d
daily.ign2:2831:b3782bcccd110b07b372dff015a36756c379c512c00e53d58bf1b68d93ab208b
daily.ftm:7920:130ff2a2e9c083c62227232fea32465fa7d4ca6ce0f890898e798940825a0d60
daily.db:25622:211deb802bc3ff1ec43b281cb2a0b609f8a945ebcdb9c1371f791c18e7997d48
daily.hdb:55536:13eb748f4abe01859268f7a771b9a8bcbc49e41755f90baf5c5be49b965c3ed5
daily.hdu:2092:0114036b97afef380a8fe44a8118d1a9e935ad0ad8b41f65fd6bca23025fa8bc
daily.mdb:8596584:2cb17cd3c430378de03ae9d4de3a79022d4b3ebfc837bbe76fd2571465a1b953
daily.mdu:63516:3021429cee1050fc9e6ac19ad30aeac581c8e0af7121e7d06a805804f8c909d2
daily.ndb:497846:58d2dad972922453143ee004bf168a53de9ad2ce26fa4445680146eb663a0872
daily.ndu:31220:84a3617710565df41554fdd669c768d9e8f45deb460fba9b2e7d27c7b555ea67
daily.ldb:43618:c40ae7dfdd8a059a121dd9701630bd61650ff03db1755c650f4d968392356699
daily.zmd:8689:87352bb29671c9326bdb6f27c14343acdc9b569fd8e3010bc6acf79883892cd8
daily.idb:30217:8b2ec9ac2e73e81835bb7316db9890c4210de118b1ebd312faa8ec0cfacd76e6
daily.fp:26694:7dfcea433125e62f196c9217abff15d7e2ec21e9f0fa0a345d7bada115414bf2
daily.pdb:4094:e8a573807464b09835948553fdb68414dfbec4c5aa20ef71d2120e9c1514f095
daily.wdb:4533:2076b9a4fa0f47deb9070e6efe6d1dfdddf634418b0ac8787722a0d15ddb7ffe
DSIG:wSkbEX/vqgQ2Xb2hzgJOmJkPbeR8eNdK0dax5IUI28BIFOQsyMhEgk6jwDLhZvv485Sme91Rn6wfh7/pmqLtTGlaDIrGWajn1SPSVDK0kiuFaqJacs5pmqyr0JA1ztu8+tGleHSU3SKpCndxVDjoRNIk9Lrk2ZvpTq5tmBhaWgI0DFluE+8ic2o3x3j7Mwj7JNoye4KVPjLYdeiyUwtGu5igp5p3sgxipDlf5u8rtHVXt6b4/zhncKynkojXhsurP/chZMO0VtFp4m5Ry71RxyFQblSrmhp6NZbk1ILUejdLTg2WioR3BeRPbKaKF+cBegw/7UzqkjKCPRRjZtd9u

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

qralston+ml.clamav-users at andrew

Jul 14, 2011, 8:15 PM

Post #11 of 16 (1172 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 2011-07-14 at 17:25-04 James Ralston <qralston+ml.clamav-users [at] andrew> wrote:

> I've just spent some time testing this, and I think this will be a
> much better solution. In essence, we can use this to force
> freshclam to pull the CLD files from our private mirror, instead of
> the CVD files. freshclam appears to use the timestamps of the files
> on the web server to determine whether they're more recent than the
> local copies, and --quiet suppresses all errors about non-existent
> files and duplicate databases.
>
> I'll test this more thoroughly, and report back how well it works
> for us.

Alas, I was premature: this doesn't work. freshclam will pull the CLD
files, but only if they're not out-of-date. If freshclam detects (via
DNS) that the database is out-of-date, it ignores the
DatabaseCustomURL directive and insists on attempting to grab the CVD
file to update the database:

$ awk -F \# '{print $1}' /etc/freshclam.conf | grep -Ev '^[[:space:]]*$'
LogSyslog yes
LogFacility LOG_DAEMON
DatabaseMirror clam-update.example.org
ScriptedUpdates no
DatabaseCustomURL http://clam-update.example.org/bytecode.cld
DatabaseCustomURL http://clam-update.example.org/daily.cld
DatabaseCustomURL http://clam-update.example.org/main.cld
DatabaseCustomURL http://clam-update.example.org/safebrowsing.cld
SafeBrowsing yes

$ freshclam --verbose
Current working dir is /var/lib/clamav
Max retries == 3
ClamAV update process started at Thu Jul 14 20:09:44 2011
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 900
Software version from DNS: 0.97.1
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.96.5 Recommended version: 0.97.1
DON'T PANIC! Read http://www.clamav.net/support/faq
main.cvd version from DNS: 53
main.cld is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
daily.cvd version from DNS: 13321
daily.cld is up to date (version: 13321, sigs: 151983, f-level: 60, builder: ccordes)
safebrowsing.cvd version from DNS: 30812
Retrieving http://clam-update.example.org/safebrowsing.cvd
Trying to download http://clam-update.example.org/safebrowsing.cvd (IP: 1.2.3.4)
WARNING: getfile: safebrowsing.cvd not found on remote server (IP: 1.2.3.4)
ERROR: Can't download safebrowsing.cvd from clam-update.example.org
Giving up on clam-update.example.org...
Update failed. Your network may be down or none of the mirrors listed in /etc/freshclam.conf is working. Check http://www.clamav.net/support/mirror-problem for possible reasons.

Honestly, the more I think about this, the more I wonder whether the
real problem here is that freshclam (at least currently) is the wrong
tool for this job.

freshclam was designed to efficiently update the clam database on a
single host, using the master clam servers as the source for updates.
It does a very good job of that, whether it hits the update servers
directly, or whether it goes through an HTTP proxy.

But freshclam falls over fairly badly if you try to use it to update
internal clients from a private mirror, even though it has some
options to help adjust its behavior for that purpose.

Now I'm thinking that if freshclam is going to be a good tool for
updating clients behind a private mirror, there really needs to be an
option to specify that directly. Something like:

# If PrivateMirror is set, freshclam assumes that the server it
# names is a private mirror. In this case, freshclam does not
# attempt to use DNS to determine whether its databases are
# out-of-date, but instead downloads the database files from the
# private mirror every time it runs. For each database, freshclam
# first attempts to download the CVD file. If that fails,
# freshclam tries to grab the CLD file. If the attempt to
# download the CVD file and attempt to download the CLD file both
# fail, then that counts as a single attempt against MaxAttempts.
# Default: disabled.
#PrivateMirror clam-update.example.org

But a counterargument would be that rather than turn freshclam into a
glorified wget, if someone really wants to mirror a private mirror,
they should just write a script using wget to do it.

Thoughts?

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

tkojm at clamav

Jul 15, 2011, 4:58 AM

Post #12 of 16 (1130 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On Thu, 14 Jul 2011 23:15:20 -0400 James Ralston
<qralston+ml.clamav-users [at] andrew> wrote:

> But freshclam falls over fairly badly if you try to use it to update
> internal clients from a private mirror, even though it has some
> options to help adjust its behavior for that purpose.
>
> Now I'm thinking that if freshclam is going to be a good tool for
> updating clients behind a private mirror, there really needs to be an
> option to specify that directly. Something like:
>
> # If PrivateMirror is set, freshclam assumes that the server it
> # names is a private mirror. In this case, freshclam does not
> # attempt to use DNS to determine whether its databases are
> # out-of-date, but instead downloads the database files from the
> # private mirror every time it runs. For each database, freshclam
> # first attempts to download the CVD file. If that fails,
> # freshclam tries to grab the CLD file. If the attempt to
> # download the CVD file and attempt to download the CLD file both
> # fail, then that counts as a single attempt against MaxAttempts.
> # Default: disabled.
> #PrivateMirror clam-update.example.org

I see no problem adding such a feature, just open a feature request at
bugs.clamav.net

> But a counterargument would be that rather than turn freshclam into a
> glorified wget, if someone really wants to mirror a private mirror,
> they should just write a script using wget to do it.
>
> Thoughts?

Besides downloading the db files, freshclam performes a number of checks
- it won't install corrupted db file, etc. So it should still be more
effective than wget.

Regards,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Fri Jul 15 13:55:37 CEST 2011
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

luca at clamav

Jul 16, 2011, 5:02 PM

Post #13 of 16 (1112 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

Hello James,

> Alas, I was premature: this doesn't work. freshclam will pull the CLD
> files, but only if they're not out-of-date. If freshclam detects (via
> DNS) that the database is out-of-date, it ignores the

You can use the --no-dns flag.

--
Luca Gibelli (luca _at_ clamav.net) ClamAV, a GPL anti-virus toolkit
[Tel] +39 0187 1851862 [Fax] +39 0187 1852252 [IM] nervous/jabber.linux.it
PGP key id 5EFC5582 @ any key-server || http://www.clamav.net/gpg/luca.gpg
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

nathan at cmpublishers

Jul 17, 2011, 7:34 PM

Post #14 of 16 (1103 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 7/14/2011 12:57 PM, James Ralston wrote:
> If I were to provide a patch that adds the two options I discussed
> above to freshclam, would you at least consider accepting it into
> trunk?
>

If they don't, send them over here.
I'll see about putting them into a future CCEE release.
With attribution of course.

> We need this functionality. But if you won't accept a patch to add
> it, then we will have to throw away freshclam and design our own
> solution, because I don't want to be put in the situation where I have
> to maintain my own local patches and update them after every release.
>

ARRGH!
I hear you there matey.
They won't put my patches in their trunk, so I keep them in mine.
:-)

--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com

Attachments:

signature.asc (0.25 KB)

nathan at cmpublishers

Jul 17, 2011, 8:48 PM

Post #15 of 16 (1104 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On 7/15/2011 7:58 AM, Tomasz Kojm wrote:
> On Thu, 14 Jul 2011 23:15:20 -0400 James Ralston
> <qralston+ml.clamav-users [at] andrew> wrote:
>
>> But freshclam falls over fairly badly if you try to use it to update
>> internal clients from a private mirror, even though it has some
>> options to help adjust its behavior for that purpose.
>>
>> Now I'm thinking that if freshclam is going to be a good tool for
>> updating clients behind a private mirror, there really needs to be an
>> option to specify that directly. Something like:
>>
>> # If PrivateMirror is set, freshclam assumes that the server it
>> # names is a private mirror. In this case, freshclam does not
>> # attempt to use DNS to determine whether its databases are
>> # out-of-date, but instead downloads the database files from the
>> # private mirror every time it runs. For each database, freshclam
>> # first attempts to download the CVD file. If that fails,
>> # freshclam tries to grab the CLD file. If the attempt to
>> # download the CVD file and attempt to download the CLD file both
>> # fail, then that counts as a single attempt against MaxAttempts.
>> # Default: disabled.
>> #PrivateMirror clam-update.example.org

Also have it ignore ScriptedUpdates if Private Mirror is Enabled.

>
> I see no problem adding such a feature, just open a feature request at
> bugs.clamav.net
>
>> But a counterargument would be that rather than turn freshclam into a
>> glorified wget, if someone really wants to mirror a private mirror,
>> they should just write a script using wget to do it.
>>
>> Thoughts?
>

Watch out for bug 2727 if you do.
A zero byte custom DB pushed over your custom mirroring system will
crash all of your clamds.

> Besides downloading the db files, freshclam performes a number of checks
> - it won't install corrupted db file, etc. So it should still be more
> effective than wget.
>
> Regards,
>

Agreed

--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com

Attachments:

signature.asc (0.25 KB)

tkojm at clamav

Sep 12, 2011, 7:33 AM

Post #16 of 16 (813 views)
Permalink

Re: improving ClamAV private mirroring? [In reply to]

On Fri, 15 Jul 2011 13:58:43 +0200 Tomasz Kojm <tkojm [at] clamav> wrote:
> On Thu, 14 Jul 2011 23:15:20 -0400 James Ralston
> <qralston+ml.clamav-users [at] andrew> wrote:
>
>> But freshclam falls over fairly badly if you try to use it to update
>> internal clients from a private mirror, even though it has some
>> options to help adjust its behavior for that purpose.
>>
>> Now I'm thinking that if freshclam is going to be a good tool for
>> updating clients behind a private mirror, there really needs to be an
>> option to specify that directly. Something like:
>>
>> # If PrivateMirror is set, freshclam assumes that the server it
>> # names is a private mirror. In this case, freshclam does not
>> # attempt to use DNS to determine whether its databases are
>> # out-of-date, but instead downloads the database files from the
>> # private mirror every time it runs. For each database, freshclam
>> # first attempts to download the CVD file. If that fails,
>> # freshclam tries to grab the CLD file. If the attempt to
>> # download the CVD file and attempt to download the CLD file both
>> # fail, then that counts as a single attempt against MaxAttempts.
>> # Default: disabled.
>> #PrivateMirror clam-update.example.org
>
> I see no problem adding such a feature, just open a feature request at
> bugs.clamav.net

The option has been implemented in clamav-devel. This is how it works:

# This option allows you to easily point freshclam to private mirrors.
# If PrivateMirror is set, freshclam does not attempt to use DNS
# to determine whether its databases are out-of-date, instead it will
# use the If-Modified-Since request or directly check the headers of the
# remote database files. For each database, freshclam first attempts
# to download the CLD file. If that fails, it tries to download the
# CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo
# and ScriptedUpdates. It can be used multiple times to provide
# fall-back mirrors.
# Default: disabled
#PrivateMirror mirror1.mynetwork.com
#PrivateMirror mirror2.mynetwork.com

Regards,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Mon Sep 12 16:31:07 CEST 2011
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads