Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: ClamAV: users

clamd exits with libclamav error

 

 

ClamAV users RSS feed   Index | Next | Previous | View Threaded


mysqlstudent at gmail

Oct 10, 2011, 12:24 AM

Post #1 of 16 (3003 views)
Permalink
clamd exits with libclamav error

Hi,

I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
for quite a while. Recently, clamd has died with an error similar to
this:

Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)run_av
(ClamAV-clamscan) FAILED - unexpected exit 2, output="LibClamAV
Error:cli_hex2str(): Malformed hexstring: 22|20 (length: 5)\nLibClamAV
Error: cli_parse_add(): Problem adding signature (3).\nLibClamAV
Error: Problem parsing database at line 40734\nLibClamAV Error: Can't
load /var/lib/clamav/INetMsg-SpamDomains-2w.ndb: Malformed
database\nERROR: Malformed database"
Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)ClamAV-clamscan
av-scanner FAILED: /usr/bin/clamscan unexpected exit 2,
output="LibClamAV Error: cli_hex2str(): Malformed hexstring: 22|20
(length: 5)\nLibClamAV Error: cli_parse_add(): Problem adding
signature (3).\nLibClamAV Error: Problem parsing database at line
40734\nLibClamAV Error: Can't load
/var/lib/clamav/INetMsg-SpamDomains-2w.ndb: Malformed database\nERROR:
Malformed database" at (eval 91) line 596.

Is this a corrupt database? I'm using the clamav-unofficial-sigs
script to verify the updates and it hasn't reported a problem.
Restarting clamd apparently resolves the issue temporarily.

It has failed two or three times now over the course of about five
days, so it generally works properly.

The content of INetMsg-SpamDomains-2w.ndb at line 40734 is:

INetMsg.SpamDomain-2w.lakecharmvila_com:4:*:(2e|2f|40|20|3c|5f)6c616b65636861726d76696c612e636f6d(27|22|20|2f|3d|5f|3e|0a|0d)

# md5sum INetMsg-SpamDomains-2w.ndb
06d95496ef6e60fdee63dcf431c06b48  INetMsg-SpamDomains-2w.ndb

# sigtool --find-sigs INetMsg.SpamDomain-2w.lakecharmvila_com |
sigtool --decode-sigs
VIRUS NAME: INetMsg.SpamDomain-2w.lakecharmvila_com
TARGET TYPE: MAIL
OFFSET: *
DECODED SIGNATURE:
{CHAR_ALTERNATIVE:.|/|@| |<|_}lakecharmvila.com{CHAR_ALTERNATIVE:'|"| |/|=|_|>|
}

Thanks for any ideas.
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Oct 10, 2011, 1:09 AM

Post #2 of 16 (2947 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 2011-10-10 10:24, Alex wrote:
> Hi,
>
> I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
> amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
> for quite a while. Recently, clamd has died with an error similar to
> this:
>
> Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)run_av
> (ClamAV-clamscan) FAILED - unexpected exit 2, output="LibClamAV
> Error:cli_hex2str(): Malformed hexstring: 22|20 (length: 5)\nLibClamAV
> Error: cli_parse_add(): Problem adding signature (3).

scanners.c:1667 returns a string split using '|' as delimiter, so I don't see how
hex2str at 1672 can report that it still has a '|'.

Try running memtest86(+) to check that your RAM is fine.

Also what does the clamav-unofficial-sigs log say about the InetMsg database?
Does it report that the integrity test worked when it tested the database with clamscan?

>
> Is this a corrupt database? I'm using the clamav-unofficial-sigs
> script to verify the updates and it hasn't reported a problem.
> Restarting clamd apparently resolves the issue temporarily.
>
> It has failed two or three times now over the course of about five
> days, so it generally works properly.
>
> The content of INetMsg-SpamDomains-2w.ndb at line 40734 is:
>
> INetMsg.SpamDomain-2w.lakecharmvila_com:4:*:(2e|2f|40|20|3c|5f)6c616b65636861726d76696c612e636f6d(27|22|20|2f|3d|5f|3e|0a|0d)

This is a valid database entry, are you sure this is the one causing clamscan to fail with the above message?
Maybe the database got updated in the meantime with a corrected entry.

>
> # md5sum INetMsg-SpamDomains-2w.ndb
> 06d95496ef6e60fdee63dcf431c06b48 INetMsg-SpamDomains-2w.ndb
>
> # sigtool --find-sigs INetMsg.SpamDomain-2w.lakecharmvila_com |
> sigtool --decode-sigs
> VIRUS NAME: INetMsg.SpamDomain-2w.lakecharmvila_com
> TARGET TYPE: MAIL
> OFFSET: *
> DECODED SIGNATURE:
> {CHAR_ALTERNATIVE:.|/|@| |<|_}lakecharmvila.com{CHAR_ALTERNATIVE:'|"| |/|=|_|>|
> }
>
> Thanks for any ideas.
> Alex
> _______________________________________________
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


mysqlstudent at gmail

Oct 10, 2011, 1:25 AM

Post #3 of 16 (2960 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

>> I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
>> amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
>> for quite a while. Recently, clamd has died with an error similar to
>> this:
>>
>> Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)run_av
>> (ClamAV-clamscan) FAILED - unexpected exit 2, output="LibClamAV
>> Error:cli_hex2str(): Malformed hexstring: 22|20 (length: 5)\nLibClamAV
>> Error: cli_parse_add(): Problem adding signature (3).
>
> scanners.c:1667 returns a string split using '|' as delimiter, so I don't see how
> hex2str at 1672 can report that it still has a '|'.
>
> Try running memtest86(+) to check that your RAM is fine.

I ran it before putting the server into production about two weeks
ago, and it has been running fine ever since.

> Also what does the clamav-unofficial-sigs log say about the InetMsg database?
> Does it report that the integrity test worked when it tested the database with clamscan?

There hasn't bee any reports of a failed integrity test in recent
past. Only messages like these:

Oct 10 03:52:33 INFO - Successfully updated Sanesecurity production
database file: INetMsg-SpamDomains-2w.ndb

>> The content of INetMsg-SpamDomains-2w.ndb at line 40734 is:
>>
>> INetMsg.SpamDomain-2w.lakecharmvila_com:4:*:(2e|2f|40|20|3c|5f)6c616b65636861726d76696c612e636f6d(27|22|20|2f|3d|5f|3e|0a|0d)
>
> This is a valid database entry, are you sure this is the one causing clamscan to fail with the above message?
> Maybe the database got updated in the meantime with a corrected entry.

The database was last updated around 02:51:52 and the error was
reported at 02:55:56, so that is the correct database, to the best of
my knowledge. It does look like it was updated one time after that:

Oct 10 03:52:32 INFO - Clamscan reports Sanesecurity
INetMsg-SpamDomains-2w.ndb database integrity tested good

However the timestamp on the file doesn't reflect that:
# ls -la INetMsg-SpamDomains-2w.ndb
-rw-r--r-- 1 amavis amavis 10688391 Oct 10 02:46 INetMsg-SpamDomains-2w.ndb

Is there a way to have it automatically restarted when something like
this happens or be more tolerant of database problems, with
notifications of those problems, in the future?

Thanks,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Oct 10, 2011, 1:42 AM

Post #4 of 16 (2951 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 2011-10-10 11:25, Alex wrote:
> Hi,
>
>>> I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
>>> amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
>>> for quite a while. Recently, clamd has died with an error similar to

Was it clamd that died or both clamd and clamscan?

>>> this:
>>>
>>> Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)run_av
>>> (ClamAV-clamscan) FAILED - unexpected exit 2, output="LibClamAV

The error message refers to clamscan, but maybe because that is the "backup scanner"?

>>> Error:cli_hex2str(): Malformed hexstring: 22|20 (length: 5)\nLibClamAV
>>> Error: cli_parse_add(): Problem adding signature (3).
>>
>> scanners.c:1667 returns a string split using '|' as delimiter, so I don't see how
>> hex2str at 1672 can report that it still has a '|'.
>>
>> Try running memtest86(+) to check that your RAM is fine.
>
> I ran it before putting the server into production about two weeks
> ago, and it has been running fine ever since.
>
>> Also what does the clamav-unofficial-sigs log say about the InetMsg database?
>> Does it report that the integrity test worked when it tested the database with clamscan?
>
> There hasn't bee any reports of a failed integrity test in recent
> past. Only messages like these:
>
> Oct 10 03:52:33 INFO - Successfully updated Sanesecurity production
> database file: INetMsg-SpamDomains-2w.ndb

Was there an "integrity tested good" message before that?

>
>>> The content of INetMsg-SpamDomains-2w.ndb at line 40734 is:
>>>
>>> INetMsg.SpamDomain-2w.lakecharmvila_com:4:*:(2e|2f|40|20|3c|5f)6c616b65636861726d76696c612e636f6d(27|22|20|2f|3d|5f|3e|0a|0d)
>>
>> This is a valid database entry, are you sure this is the one causing clamscan to fail with the above message?
>> Maybe the database got updated in the meantime with a corrected entry.
>
> The database was last updated around 02:51:52 and the error was
> reported at 02:55:56, so that is the correct database, to the best of
> my knowledge. It does look like it was updated one time after that:
>
> Oct 10 03:52:32 INFO - Clamscan reports Sanesecurity
> INetMsg-SpamDomains-2w.ndb database integrity tested good
>
> However the timestamp on the file doesn't reflect that:
> # ls -la INetMsg-SpamDomains-2w.ndb
> -rw-r--r-- 1 amavis amavis 10688391 Oct 10 02:46 INetMsg-SpamDomains-2w.ndb
>
> Is there a way to have it automatically restarted when something like
> this happens or be more tolerant of database problems, with
> notifications of those problems, in the future?

Restarting won't help if the database is corrupted, or is there is some problem parsing the database.

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


nathan at cmpublishers

Oct 10, 2011, 8:00 AM

Post #5 of 16 (2946 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 10/10/2011 4:42 AM, Török Edwin wrote:
> On 2011-10-10 11:25, Alex wrote:
>> Is there a way to have it automatically restarted when something like
>> this happens or be more tolerant of database problems, with
>> notifications of those problems, in the future?

If bug 2727 is any indication, don't bet on it.

> Restarting won't help if the database is corrupted, or is there is some problem parsing the database.
>

Correct, if you mess up a sig DB on a system, you've messed up the
ClamAV on the system.
And most of the time it doesn't log a thing, it just dies.
Lots of fun. :-)

--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com
Attachments: signature.asc (0.25 KB)


mysqlstudent at gmail

Oct 10, 2011, 9:15 AM

Post #6 of 16 (2980 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

>>>> I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
>>>> amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
>>>> for quite a while. Recently, clamd has died with an error similar to
>
> Was it clamd that died or both clamd and clamscan?

It looks like both:

Oct 10 01:11:02 mail02 amavis[31956]: (31956-07-4) ClamAV-clamd: Can't
send to socket /var/spool/amavisd/clamd.sock: Transport endpoint is
not connected, retrying (1)

And here is clamd failing:

Oct 10 12:03:29 mail02 amavis[14313]: (14313-03-6) (!)ClamAV-clamscan
av-scanner FAILED: /usr/bin/clamscan unexpected exit 2,
output="LibClamAV Error: cli_loadhash: Problem parsing database at
line 662180\nLibClamAV Error: Can't load main.mdb: Malformed
database\nLibClamAV Error: cli_tgzload: Can't load main.mdb\nLibClamAV
Error: Can't load /var/lib/clamav/main.cvd: Malformed database\nERROR:
Malformed database" at (eval 91) line 596.

I notice that it's not always the same database or line number that it
is failing on, and it's now just happened again, so it's now more
frequent.

I suppose it could be a hardware problem, but it's a kvm virtual
machine running on new x86_64 Xeon hardware that was stress tested
before putting into production. It ran without any difficulties for
probably a week prior to the first occurrence of the problem.

>>>> Oct 10 02:55:56 mail02 amavis[25696]: (25696-18) (!)run_av
>>>> (ClamAV-clamscan) FAILED - unexpected exit 2, output="LibClamAV
>
> The error message refers to clamscan, but maybe because that is the "backup scanner"?

It looks like clamd is the primary and clamscan is set up as a backup
with amavisd.


>> Oct 10 03:52:33 INFO - Successfully updated Sanesecurity production
>> database file: INetMsg-SpamDomains-2w.ndb
>
> Was there an "integrity tested good" message before that?

Yes, it always reported that afterwards. I've just run freshclam
manually, and the output is interesting:

# freshclam
ClamAV update process started at Mon Oct 10 12:04:16 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
connect_error: getsockopt(SO_ERROR): fd=5 error=111: Connection refused
Can't connect to port 80 of host db.local.clamav.net (IP: 69.12.162.28)
Downloading daily-13777.cdiff [100%]
daily.cld updated (version: 13777, sigs: 206679, f-level: 60, builder: ccordes)
Empty script safebrowsing-32883.cdiff, need to download entire database
Downloading safebrowsing.cvd [100%]
WARNING: Mirror 194.186.47.19 is not synchronized.
Trying again in 5 secs...
ClamAV update process started at Mon Oct 10 12:04:39 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
daily.cld is up to date (version: 13777, sigs: 206679, f-level: 60,
builder: ccordes)
Empty script safebrowsing-32883.cdiff, need to download entire database
Downloading safebrowsing.cvd [100%]
WARNING: Mirror 150.214.142.197 is not synchronized.
Trying again in 5 secs...
ClamAV update process started at Mon Oct 10 12:05:00 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
daily.cld is up to date (version: 13777, sigs: 206679, f-level: 60,
builder: ccordes)
Empty script safebrowsing-32883.cdiff, need to download entire database
Downloading safebrowsing.cvd [100%]
WARNING: Mirror 200.236.31.1 is not synchronized.
Giving up on db.local.clamav.net...
Update failed. Your network may be down or none of the mirrors listed
in /etc/freshclam.conf is working. Check
http://www.clamav.net/support/mirror-problem for possible reasons.

After it finished, I ran it again and it completed successfully. I'm
also sure there isn't anything wrong with the network.

I'm really stuck here. I hope someone has some ideas.

Thanks again,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


mysqlstudent at gmail

Oct 10, 2011, 9:29 AM

Post #7 of 16 (2943 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

>>> Is there a way to have it automatically restarted when something like
>>> this happens or be more tolerant of database problems, with
>>> notifications of those problems, in the future?
>
> If bug 2727 is any indication, don't bet on it.

I don't think it's that bug, since I have a version greater than
0.97.0.2, and this bug was resolved in April.

>> Restarting won't help if the database is corrupted, or is there is some problem parsing the database.
>
> Correct, if you mess up a sig DB on a system, you've messed up the
> ClamAV on the system.
> And most of the time it doesn't log a thing, it just dies.
> Lots of fun. :-)

In my case, restarting does fix the problem.

Is there anything I should watch for, or do when it happens again? How
can I manually check the integrity of all the databases when it fails?

Thanks,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Oct 10, 2011, 9:46 AM

Post #8 of 16 (2956 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 2011-10-10 19:15, Alex wrote:
> Hi,
>
>>>>> I have a fedora15 x86_64 box with clamav-0.97.2, postfix-2.8.4, and
>>>>> amavisd-new-2.6.6 with spamassassin-3.3.2 that has been running fine
>>>>> for quite a while. Recently, clamd has died with an error similar to
>>
>> Was it clamd that died or both clamd and clamscan?
>
> It looks like both:
>
> Oct 10 01:11:02 mail02 amavis[31956]: (31956-07-4) ClamAV-clamd: Can't
> send to socket /var/spool/amavisd/clamd.sock: Transport endpoint is
> not connected, retrying (1)
>
> And here is clamd failing:
>
> Oct 10 12:03:29 mail02 amavis[14313]: (14313-03-6) (!)ClamAV-clamscan
> av-scanner FAILED: /usr/bin/clamscan unexpected exit 2,
> output="LibClamAV Error: cli_loadhash: Problem parsing database at
> line 662180\nLibClamAV Error: Can't load main.mdb: Malformed
> database\nLibClamAV Error: cli_tgzload: Can't load main.mdb\nLibClamAV
> Error: Can't load /var/lib/clamav/main.cvd: Malformed database\nERROR:
> Malformed database" at (eval 91) line 596.

main.cvd was last updated in 2010, and it is definitely not broken.
So this random database parsing failure can be 2 things:
- hardware issue
- memory corruption bug in libclamav

For the 1st all I can suggest is to run memtest again, but you probably can't afford
to take down a production server just to do that.
There is another one, memtester which can be run from userspace without rebooting, you can try that.
Of course it could be some other HW problem, but RAM is the one that fails most often.

For the 2nd you can try running clamscan under valgrind and see if it reports any warnings, i.e.
valgrind clamscan /dev/null.

>
> I notice that it's not always the same database or line number that it
> is failing on, and it's now just happened again, so it's now more
> frequent.
>
> I suppose it could be a hardware problem, but it's a kvm virtual
> machine running on new x86_64 Xeon hardware that was stress tested
> before putting into production. It ran without any difficulties for
> probably a week prior to the first occurrence of the problem.

The next time this happens (or if you can still reproduce the problem)
take a backup of the database directory (cp -a), upload it somewhere,
open a bug and put the link there, will take a look if there's anything wrong with the parsing code.

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwin at clamav

Oct 10, 2011, 9:49 AM

Post #9 of 16 (2946 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 2011-10-10 19:29, Alex wrote:
> Hi,
>
>>>> Is there a way to have it automatically restarted when something like
>>>> this happens or be more tolerant of database problems, with
>>>> notifications of those problems, in the future?
>>
>> If bug 2727 is any indication, don't bet on it.
>
> I don't think it's that bug, since I have a version greater than
> 0.97.0.2, and this bug was resolved in April.
>
>>> Restarting won't help if the database is corrupted, or is there is some problem parsing the database.
>>
>> Correct, if you mess up a sig DB on a system, you've messed up the
>> ClamAV on the system.
>> And most of the time it doesn't log a thing, it just dies.
>> Lots of fun. :-)
>
> In my case, restarting does fix the problem.
>
> Is there anything I should watch for, or do when it happens again? How
> can I manually check the integrity of all the databases when it fails?

Run clamscan /dev/null (or any file), and it will print an error if any database is wrong.
Official databases have digital signatures, and clamscan (and freshclam) checks it.

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


nathan at cmpublishers

Oct 10, 2011, 10:44 AM

Post #10 of 16 (2946 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 10/10/2011 12:29 PM, Alex wrote:
> Hi,
>
>>>> Is there a way to have it automatically restarted when something like
>>>> this happens or be more tolerant of database problems, with
>>>> notifications of those problems, in the future?
>>
>> If bug 2727 is any indication, don't bet on it.
>
> I don't think it's that bug, since I have a version greater than
> 0.97.0.2, and this bug was resolved in April.
>
What I meant was that the ClamAV Team isn't likely to make the software
more tolerant of database problems.

RE: 2727, if "WORKSFORME" is a resolution, then I'll run Win95 on
everything, because it obviously isn't broken.
;-)

--
Sincerely,

Nathan Gibbs

Systems Administrator
Christ Media
http://www.cmpublishers.com
Attachments: signature.asc (0.25 KB)


tkojm at clamav

Oct 10, 2011, 12:14 PM

Post #11 of 16 (2947 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On Mon Oct 10 2011 18:49:33 GMT+0200 (CEST)
Török Edwin <edwin [at] clamav> wrote:

>> Is there anything I should watch for, or do when it happens again? How
>> can I manually check the integrity of all the databases when it fails?
>
> Run clamscan /dev/null (or any file), and it will print an error if any database is wrong.
> Official databases have digital signatures, and clamscan (and freshclam) checks it.

Also freshclam makes sure that a new database gets parsed properly by
the version of libclamav installed in the system before installing the
database and removing the old one.

Together with ClamAV 0.97.2 we announced a 3rd Party Signature portal:
http://lurker.clamav.net/message/20110725.160953.ad286fef.en.html

This portal allows 3rd party signature creators to use our systems to
develop, test and publish their signatures on our public mirrors. With
this solution the end users would only need to add a single line to
freshclam.conf to benefit from automatic and *safe* updates. Hopefully,
more vendors will decide to join this initiative soon!

Regards,

--
oo ..... Tomasz Kojm <tkojm [at] clamav>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Mon Oct 10 20:54:47 CEST 2011
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


mysqlstudent at gmail

Oct 18, 2011, 10:43 AM

Post #12 of 16 (2894 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

A few days ago we were discussing a problem I was having with clamd
reporting a corrupt database.

>>>> Restarting won't help if the database is corrupted, or is there is some problem parsing the database.
>>>
>>> Correct, if you mess up a sig DB on a system, you've messed up the
>>> ClamAV on the system.
>>> And most of the time it doesn't log a thing, it just dies.
>>> Lots of fun. :-)
>>
>> In my case, restarting does fix the problem.
>>
>> Is there anything I should watch for, or do when it happens again? How
>> can I manually check the integrity of all the databases when it fails?
>
> Run clamscan /dev/null (or any file), and it will print an error if any database is wrong.
> Official databases have digital signatures, and clamscan (and freshclam) checks it.

I've since learned there was a cooling problem with the processor and
started receiving events like these on the host:

kernel: [73788.355981] [Hardware Error]: Machine check events logged
kernel: [73914.635576] CPU4: Package temperature above threshold, cpu
clock throttled (total events = 5538406)
kernel: [73914.635581] CPU0: Package temperature above threshold, cpu
clock throttled (total events = 5538398)

I've since corrected the cooling issue, but am still receiving
messages such as these on the kvm guest:

[169245.360511] clamscan[27448] general protection ip:7f125f2e6ffb
sp:7fff117566f0 error:0 in libclamav.so.6.1.11[7f125f229000+9ce000]

[29016.445470] clamd[1110] general protection ip:30df2c3981
sp:7fffa08f4fe0 error:0 in libclamav.so.6.1
.11[30df200000+9ce000]

I realize this may be a problem with the system and not specifically
clamd or clamscan, but since that is how it is currently presenting
itself, I thought someone may have some ideas of what is happening
here?

Is there any way to determine if the processor is permanently damaged
due to overheating?

Could this be related to library or system corruption caused by the
processor overheating? I've reinstalled all the clam applications, and
there haven't been any further messages in the logs except for this
one clamscan line.

I'm hoping there are some smart people here that can tell me with
certainty that it is indeed the processor and I should replace it as
quickly as possible.

Thanks for any ideas.
Best,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


cswiger at mac

Oct 18, 2011, 11:01 AM

Post #13 of 16 (2891 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On Oct 18, 2011, at 10:43 AM, Alex wrote:
> I've since learned there was a cooling problem with the processor and
> started receiving events like these on the host:
>
> kernel: [73788.355981] [Hardware Error]: Machine check events logged
> kernel: [73914.635576] CPU4: Package temperature above threshold, cpu
> clock throttled (total events = 5538406)
> kernel: [73914.635581] CPU0: Package temperature above threshold, cpu
> clock throttled (total events = 5538398)

Since your CPU had thermal protection, it's supposed to take effect before the hardware is permanently damaged, but the thermal stress might have affected it, or other components like memory or the PSU.

> I've since corrected the cooling issue, but am still receiving
> messages such as these on the kvm guest:
>
> [169245.360511] clamscan[27448] general protection ip:7f125f2e6ffb
> sp:7fff117566f0 error:0 in libclamav.so.6.1.11[7f125f229000+9ce000]
>
> [29016.445470] clamd[1110] general protection ip:30df2c3981
> sp:7fffa08f4fe0 error:0 in libclamav.so.6.1
> .11[30df200000+9ce000]

What OS? Looks to be Unix-style library names, but a Windows-style error messages? I'd expect a Unix platform to be logging a SIGSEGV or SIGBUS....

> I realize this may be a problem with the system and not specifically
> clamd or clamscan, but since that is how it is currently presenting
> itself, I thought someone may have some ideas of what is happening
> here?
>
> Is there any way to determine if the processor is permanently damaged
> due to overheating?
>
> Could this be related to library or system corruption caused by the
> processor overheating? I've reinstalled all the clam applications, and
> there haven't been any further messages in the logs except for this
> one clamscan line.

Your hardware manufacturer should have hardware diagnostics available which will identify anything obviously wrong. But running prime95 in test mode or memtest86 for 24 hours is a decent sanity check.

Regards,
--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


mysqlstudent at gmail

Oct 18, 2011, 11:12 AM

Post #14 of 16 (2889 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

>> kernel: [73788.355981] [Hardware Error]: Machine check events logged
>> kernel: [73914.635576] CPU4: Package temperature above threshold, cpu
>> clock throttled (total events = 5538406)
>> kernel: [73914.635581] CPU0: Package temperature above threshold, cpu
>> clock throttled (total events = 5538398)
>
> Since your CPU had thermal protection, it's supposed to take effect before the hardware is permanently damaged, but the thermal stress might have affected it, or other components like memory or the PSU.

There was an "event" listed in the BIOS (Asus P8B-M Xeon E3-1240)

EFI 03058003 Major

This is probably just detailing the overheating, though.

>> [169245.360511] clamscan[27448] general protection ip:7f125f2e6ffb
>> sp:7fff117566f0 error:0 in libclamav.so.6.1.11[7f125f229000+9ce000]
>>
>> [29016.445470] clamd[1110] general protection ip:30df2c3981
>> sp:7fffa08f4fe0 error:0 in libclamav.so.6.1
>> .11[30df200000+9ce000]
>
> What OS?  Looks to be Unix-style library names, but a Windows-style error messages?  I'd expect a Unix platform to be logging a SIGSEGV or SIGBUS....

It's a fedora15 x86_64 guest on a fedora15 x86_86 host.

> Your hardware manufacturer should have hardware diagnostics available which will identify anything obviously wrong.  But running prime95 in test mode or memtest86 for 24 hours is a decent sanity check.

I think I'm going to put the disks in another server for a day or so
and see if the problem persists, and test this hardware at the same
time.

Thanks again,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


mysqlstudent at gmail

Oct 19, 2011, 11:53 AM

Post #15 of 16 (2878 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

Hi,

>> kernel: [73788.355981] [Hardware Error]: Machine check events logged
>> kernel: [73914.635576] CPU4: Package temperature above threshold, cpu
>> clock throttled (total events = 5538406)
>> kernel: [73914.635581] CPU0: Package temperature above threshold, cpu
>> clock throttled (total events = 5538398)
>
> Since your CPU had thermal protection, it's supposed to take effect before the hardware is permanently damaged, but the thermal stress might have affected it, or other components like memory or the PSU.

>> [29016.445470] clamd[1110] general protection ip:30df2c3981
>> sp:7fffa08f4fe0 error:0 in libclamav.so.6.1
>> .11[30df200000+9ce000]

I've now switched the hard disks to the old server (also an x86_64
arch) and it has been running fine with no 'general protection' errors
for more than twelve hours. I think it's safe to assume there is no
software bug causing these errors?

I've also been stress testing the new hardware separately. It
succeeded through two full passes of memtest86 without any errors.
It's now been running mprime for more than twelve hours and has not
failed.

When these 'general protection' errors were produced, the system was
typically under high load and high IO.

I realize this may be a hardware issue, but does anyone have any ideas
how to determine what is really going on?

Is there a way to stress-test clamav on the new hardware, to try and
induce an error through high IO?

Thanks,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


edwintorok at gmail

Oct 19, 2011, 11:59 AM

Post #16 of 16 (2883 views)
Permalink
Re: clamd exits with libclamav error [In reply to]

On 2011-10-19 21:53, Alex wrote:
> Hi,
>
>>> kernel: [73788.355981] [Hardware Error]: Machine check events logged
>>> kernel: [73914.635576] CPU4: Package temperature above threshold, cpu
>>> clock throttled (total events = 5538406)
>>> kernel: [73914.635581] CPU0: Package temperature above threshold, cpu
>>> clock throttled (total events = 5538398)
>>
>> Since your CPU had thermal protection, it's supposed to take effect before the hardware is permanently damaged, but the thermal stress might have affected it, or other components like memory or the PSU.
>
>>> [29016.445470] clamd[1110] general protection ip:30df2c3981
>>> sp:7fffa08f4fe0 error:0 in libclamav.so.6.1
>>> .11[30df200000+9ce000]
>
> I've now switched the hard disks to the old server (also an x86_64
> arch) and it has been running fine with no 'general protection' errors
> for more than twelve hours. I think it's safe to assume there is no
> software bug causing these errors?
>
> I've also been stress testing the new hardware separately. It
> succeeded through two full passes of memtest86 without any errors.
> It's now been running mprime for more than twelve hours and has not
> failed.
>
> When these 'general protection' errors were produced, the system was
> typically under high load and high IO.
>
> I realize this may be a hardware issue, but does anyone have any ideas
> how to determine what is really going on?

There are some packages for stress-testing, like cpuburn.
cpuburn in MMX mode is quite good at raising your CPU temperature, I suggest you keep
an eye on the CPU sensors (sensors -l) if you do run it.
Try running one cpuburn on each CPU core for a while.

Of course its also possible that your hardware was fine before and you'll damage it by running
the stress tests (if you have inadequate cooling for example), so you do so on your own risk!

>
> Is there a way to stress-test clamav on the new hardware, to try and
> induce an error through high IO?

For high I/O try this: run updatedb to update your locate database,
and at the same time launch a clamd multiscan:
clamdscan -m /

Another test that you can do is to compile some large pieces of software (Linux kernel, OpenOffice, etc.)
with make -j N, where N = nr_cores * 2. GCC uses a _lot_ of pointer manipulation and will randomly
crash on faulty hardware, although in that case memtest usually detects the errors too.

Best regards,
--Edwin
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

ClamAV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.