Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

HAVP + Linux RAMdisk errors

 

 

ClamAV users RSS feed   Index | Next | Previous | View Threaded


strykar at hotmail

Aug 13, 2009, 6:42 PM

Post #1 of 8 (1330 views)
Permalink
HAVP + Linux RAMdisk errors

Hello list,

I'm using HAVP (http://www.server-side.de) and the ClamAVlib to scan HTTP
streams.
This works fine when using a TEMPDIR on an HDD, but fails mysteriously when
using a RAMdisk.

Hardware and software:
HAVP Version 0.91
ClamAV 0.95.2/9695
Slackware Linux 2.6.24.5-smp i686 Pentium III (Coppermine) GNU/Linux RAM:
380604

clamd is running defaults besides:

# grep -v "#" /etc/clamd.conf | grep -v ^$
LogFile /var/log/clamav/clamd.log
LogFileMaxSize 10M
PidFile /var/run/clamav/clamd.pid
LocalSocket /tmp/clamd.socket
User clamav
AllowSupplementaryGroups yes
ExitOnOOM yes


HAVP running defaults besides:

# grep -v "#" /usr/local/etc/havp/havp.config | grep -v ^$
USER havp
GROUP havp
LOG_OKS false
LOGLEVEL 1
TEMPDIR /mnt/memory
PARENTPROXY 192.168.1.101
PARENTPORT 3128
MAXSCANSIZE 15000000
STREAMUSERAGENT Player Winamp iTunes QuickTime Audio RMA/ MAD/ Foobar2000
XMMS
STREAMSCANSIZE 20000
ENABLECLAMLIB true
CLAMDBDIR /usr/share/clamav

Both ClamAV and HAVP are started as root and then drop privileges to their
respective user/groups.


Here's the RAMdisk issue:
# mount | grep ram0
/dev/ram0 on /mnt/memory type ext3 (rw,mand)
# df -h | grep ram0
/dev/ram0 122M 5.6M 110M 5% /mnt/memory
# ls -alh /mnt/memory/
total 17K
drw-rw-rw- 3 clamav clamav 1.0K 2009-08-14 06:06 ./
drwxr-xr-x 11 root root 4.0K 2009-08-12 05:44 ../
drwx------ 2 clamav clamav 12K 2009-08-14 06:04 lost+found/


# sudo -u clamav clamscan --tempdir=/mnt/memory
LibClamAV Error: cli_cvdload(): Can't create temporary directory
/mnt/memory/clamav-a92c7088eca99a33d3925293fea9f609
LibClamAV Error: Can't load /usr/share/clamav/main.cvd: Can't create
temporary directory
ERROR: Can't create temporary directory

----------- SCAN SUMMARY -----------
Known viruses: 63646
Engine version: 0.95.2
Scanned directories: 0
Scanned files: 0
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 0.565 sec (0 m 0 s)


I assumed this was a RAMdisk size issue, which it does not seem like:
root [at] barg:~# df -h /mnt/memory/
Filesystem Size Used Avail Use% Mounted on
/dev/ram0 122M 5.6M 110M 5% /mnt/memory


So I ran strace, (log available at http://pastebin.com/m64756256) and here's
some of the interesting bits:

~# strace -o /tmp/clam.log sudo -u clamav clamscan --tempdir=/mnt/memory
LibClamAV Error: cli_cvdload(): Can't create temporary directory
/mnt/memory/clamav-df42daba837ac4765e37ff3406ef2682
LibClamAV Error: Can't load /usr/share/clamav/main.cvd: Can't create
temporary directory

..
close(3) = -1 EBADF (Bad file descriptor)
close(4) = -1 EBADF (Bad file descriptor)
close(5) = -1 EBADF (Bad file descriptor)
close(6) = -1 EBADF (Bad file descriptor)
close(1020) = -1 EBADF (Bad file descriptor)
close(1021) = -1 EBADF (Bad file descriptor)
close(1022) = -1 EBADF (Bad file descriptor)

mkdir("/mnt/memory/clamav-df42daba837ac4765e37ff3406ef2682", 0700) = -1
EACCES (Permission denied)
write(2, "LibClamAV Error: cli_cvdload(): "..., 117) = 117
write(2, "LibClamAV Error: Can\'t load /usr"..., 89) = 89
..EOF


Are the "Bad file descriptors" due to file-system errors or bad RAM?
Is mkdir failing because of permissions a bug? As the TEMPDIR (/mnt/memory/)
is owned by clamav:clamav and the same error occurs even with the dir
chmod'ed to 666.

I've also been asking the HAVP developers too
(http://havp.hege.li/forum/viewtopic.php?f=3&t=472&start=0) who think this
is a TEMPDIR space issue as I cannot replicate it when using TEMPDIR on an
HDD like /tmp/ or /var/tmp/



Regards,
S

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


tkojm at clamav

Aug 14, 2009, 12:38 AM

Post #2 of 8 (1242 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

On Fri, 14 Aug 2009 07:12:00 +0530
"Strykar" <strykar [at] hotmail> wrote:

> mkdir("/mnt/memory/clamav-df42daba837ac4765e37ff3406ef2682", 0700) = -1
> EACCES (Permission denied)

Hi,

this is definitely a permission related issue. Please try adding -o uid=xxx,gid=yyy
(where xxx and yyy are the uid and gid numbers of 'clamav' respectively) to your
mount command or to /etc/fstab and let us know if this fixes the problem.

Another solution would be creating a new directory /mnt/memory/clamav with
proper permissions and then pointing clamav to this directory instead of
/mnt/memory.

> Are the "Bad file descriptors" due to file-system errors or bad RAM?

The strace snippet was too short to say what's the reason for them but for
sure it's not a buggy RAM :-)

> Is mkdir failing because of permissions a bug?

Yes, it is

> As the TEMPDIR (/mnt/memory/)
> is owned by clamav:clamav and the same error occurs even with the dir
> chmod'ed to 666.

Usually the proper way to specify the owner and group of the root of the fs
is to use the gid=xxx and uid=yyy options.

> I've also been asking the HAVP developers too
> (http://havp.hege.li/forum/viewtopic.php?f=3&t=472&start=0) who think this
> is a TEMPDIR space issue as I cannot replicate it when using TEMPDIR on an
> HDD like /tmp/ or /var/tmp/

It's definitely not a space issue (empty dirs don't take too much really ;-)),
however you should make the ram tempdir big enough to have clamav
properly handle large archives or multiple files by concurrent threads, etc. but
that's a different story.

HTH,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Fri Aug 14 09:35:58 CEST 2009
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


uhlar at fantomas

Aug 14, 2009, 8:55 AM

Post #3 of 8 (1239 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

On 14.08.09 07:12, Strykar wrote:
> Here's the RAMdisk issue:
> # mount | grep ram0
> /dev/ram0 on /mnt/memory type ext3 (rw,mand)

Is there any reason for not using tmpfs instead of ramdisk? Tmpfs should be
much more effective.

> # ls -alh /mnt/memory/
> total 17K
> drw-rw-rw- 3 clamav clamav 1.0K 2009-08-14 06:06 ./

invalid permissions, execute bit is missing, so it prevents anyone but root
from accessing anything in the directory.

--
Matus UHLAR - fantomas, uhlar [at] fantomas ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Fighting for peace is like fucking for virginity...
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


strykar at hotmail

Aug 14, 2009, 1:04 PM

Post #4 of 8 (1231 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

From: clamav-users-bounces [at] lists
[mailto:clamav-users-bounces [at] lists] On Behalf Of Tomasz Kojm

>this is definitely a permission related issue. Please try adding -o
uid=xxx,gid=yyy
>(where xxx and yyy are the uid and gid numbers of 'clamav' respectively) to
your
>mount command or to /etc/fstab and let us know if this fixes the problem.

The patch for adding UID/GID for ext3 has just been committed recently.

>Another solution would be creating a new directory /mnt/memory/clamav with
>proper permissions and then pointing clamav to this directory instead of
>/mnt/memory.

Mounting to /mnt/memory/clamav and then chown -R /mnt/memory fixed the
permission issue, thank you!!

>The strace snippet was too short to say what's the reason for them but for
>sure it's not a buggy RAM :-)

Right again, memtest shows no error in the RAM chips.

Problem solved, thanks again for your help Tomas.
- S.

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


strykar at hotmail

Aug 14, 2009, 1:09 PM

Post #5 of 8 (1248 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

From: clamav-users-bounces [at] lists
[mailto:clamav-users-bounces [at] lists] On Behalf Of Matus UHLAR -
fantomas


>Is there any reason for not using tmpfs instead of ramdisk? Tmpfs should be
>much more effective.

Good question, could ClamAV developers comment on this?
Would TmpFS be more effective as it would start writing to /swap if the
system runs out of memory instead of stating "Out of memory" and stopping
the process?

What is the recommended FS? TmpFS? RAMdisk?
Obviously, looking for the fastest performance, RAM being no issue, what
should be used if having a lot of users and multiple scanners for HTTP
streams?

>invalid permissions, execute bit is missing, so it prevents anyone but root
>from accessing anything in the directory.

Good eye! I had missed that and changing it fixed the permission issue.

Thanks again Matus.
- S


_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Jason.Haar at trimble

Aug 14, 2009, 3:57 PM

Post #6 of 8 (1238 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

On 08/15/2009 08:09 AM, Strykar wrote:
> Good question, could ClamAV developers comment on this?
> Would TmpFS be more effective as it would start writing to /swap if the
> system runs out of memory instead of stating "Out of memory" and stopping
> the process?
>
These are HAVP questions - not ClamAV. You should have been asking them
on the HAVP list.

Anyway, your choice of filesystem is probably irrelevant. The bottleneck
for AV scanning will be the scanner itself - not the filesystem. I use
the standard (disk) filesystem instead of ramdisk and CPU is my
bottleneck - as expected. (I mean there is always a bottleneck in any
process, it's only a matter of deciding whether it matters or not).

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


acabng at digitalfuture

Aug 17, 2009, 6:38 AM

Post #7 of 8 (1193 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

Strykar wrote:
> Good question, could ClamAV developers comment on this?
> Would TmpFS be more effective as it would start writing to /swap if the
> system runs out of memory instead of stating "Out of memory" and stopping
> the process?

Hi,

My suggestion is that, if you are using sane limits in havp, which is
BTW a good idea, tmpfs is the best approach.
Let's put it this way... If your system is swapping due to a few 5-10
megs tmpfs files, then it's likely that it's going to be swapping
anyway. In fact, in most cases, scanning any file is going to take up
more memory that it's bare size.

Cheers,
-aCaB
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


uhlar at fantomas

Aug 17, 2009, 6:46 AM

Post #8 of 8 (1193 views)
Permalink
Re: HAVP + Linux RAMdisk errors [In reply to]

> On 08/15/2009 08:09 AM, Strykar wrote:
> > Good question, could ClamAV developers comment on this?
> > Would TmpFS be more effective as it would start writing to /swap if the
> > system runs out of memory instead of stating "Out of memory" and stopping
> > the process?

On 15.08.09 10:57, Jason Haar wrote:
> These are HAVP questions - not ClamAV. You should have been asking them
> on the HAVP list.

I think that using temporary directory for clamav is as much ClamAV question
as possible :-)

> Anyway, your choice of filesystem is probably irrelevant. The bottleneck
> for AV scanning will be the scanner itself - not the filesystem. I use
> the standard (disk) filesystem instead of ramdisk and CPU is my
> bottleneck - as expected. (I mean there is always a bottleneck in any
> process, it's only a matter of deciding whether it matters or not).

if the syustem uses much of disk I/O, the temporary filesystem can became
the bottleneck very fast. And while talking about the CPU, tmpfs has smaller
overhead I guess.

I think that due to how tmpfs processes files, it may give us speed benefits
even when system is swapping.

but we can always try some tests to see how much it helps in what
situations...

--
Matus UHLAR - fantomas, uhlar [at] fantomas ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.