Mailing List Archive: ClamAV: users

Thoughts on software QA Testing (or lack thereof...)

Index | Next | Previous | View Threaded

georgek at netwrx1

Nov 5, 2009, 8:27 AM

Post #1 of 7 (276 views)
Permalink

Thoughts on software QA Testing (or lack thereof...)

>On Mon, 2 Nov 2009 16:32:57 +0000 (GMT), you wrote:

>Hi there,
>
>On Thu, 29 Oct 2009 aCaB wrote:
>
>> On our side we do a lot of QA
>> ...
>> I really believe something needs to happen here so that these type of
>> bugs can be caught quickly before they affect a number of users.
>>
>> Thoughts?
>
>I suspect that rather than QA, what you do is just a lot of hap-hazard
>testing. That's why, whenever I see a new release of ClamAV, first I
>will suppress a groan and then, before I risk it on any of my servers,
>I'll wait a while and watch the users' list to see how much trouble it
>causes. This approach serves me well, although I can't say I'm proud
>of the fact that I'm letting a lot of poor innocents do my acceptance
>testing for me.

OK...I'm probably going to PO a few folks here but you asked for
opinions so I'll share one....

We run about 50 Unix/Linux servers here...almost all are 32-bit RHEL4
not an unknown distro by any means and what I'd suspect is easy to
test on. Several more are Solaris 9, again a well known OS, and
several more are Fedora Core 10, again a well known common OS build.

This latest release had basic make issues in ALL THREE OS versions
that should have been caught by simply typing make as it wouldn't even
finish the make step without various patches....in my mind that
doesn't qualify it as even ALPHA software much less BETA or a Release
Candidate and surely not a Production Release version...please don't
embarrass yourselves by saying you do alot of QA....that obviously
didn't happen in this release and has been a problem in the last few
releases....

I expect better from a commercial company like SourceFire now that
they own the product as a commercial concern...its the same codebase
for free or paid users as far as I know, the differences is paying
users get to call for support.

George
--
===[George R. Kasica]=== +1 262 677 0766
President +1 206 374 6482 FAX
Netwrx Consulting Inc. Jackson, WI USA
http://www.netwrx1.com
georgek[at]netwrx1.com
ICQ #12862186
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

edwintorok at gmail

Nov 5, 2009, 10:44 AM

Post #2 of 7 (266 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

On 2009-11-05 18:27, George R. Kasica wrote:
>> On Mon, 2 Nov 2009 16:32:57 +0000 (GMT), you wrote:
>>
>
>
>> Hi there,
>>
>> On Thu, 29 Oct 2009 aCaB wrote:
>>
>>
>>> On our side we do a lot of QA
>>> ...
>>> I really believe something needs to happen here so that these type of
>>> bugs can be caught quickly before they affect a number of users.
>>>
>>> Thoughts?
>>>
>> I suspect that rather than QA, what you do is just a lot of hap-hazard
>> testing. That's why, whenever I see a new release of ClamAV, first I
>> will suppress a groan and then, before I risk it on any of my servers,
>> I'll wait a while and watch the users' list to see how much trouble it
>> causes. This approach serves me well, although I can't say I'm proud
>> of the fact that I'm letting a lot of poor innocents do my acceptance
>> testing for me.
>>
>
> OK...I'm probably going to PO a few folks here but you asked for
> opinions so I'll share one....
>
> We run about 50 Unix/Linux servers here...almost all are 32-bit RHEL4
> not an unknown distro by any means and what I'd suspect is easy to
> test on. Several more are Solaris 9, again a well known OS, and
> several more are Fedora Core 10, again a well known common OS build.
>

Hi,

We test ClamAV on several architectures and OSes:
Debian (etch, lenny, unstable) on various architectures (x86_64,
powerpc, sparc64, armv5tel, ...),
RHEL5.2 (x86-64), Solaris 10 (sparc), and Mac OS X (ppc).
Some of these tests are running on the GCC compile farm, some on our own
boxes, some on machines we have been kindly offered access for testing.
This of course can always be improved.

One of the build problems was an error when 'git' was not installed.
Unfortunately all of the machines we test ClamAV on have git in $PATH
(it is how it checks out the latest version).
For the next release we'll setup a buildhost that doesn't have git
installed.
The other problem (missing include unistd.h) only occurs on old
distributions, which we lack in our buildfarm.

We're currently investigating the possibility of using the OpenSUSE
build service to test the next ClamAV release on multiple Linux
distributions, including many old ones:
openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

> This latest release had basic make issues in ALL THREE OS versions
> that should have been caught by simply typing make as it wouldn't even
> finish the make step without various patches....in my mind that
> doesn't qualify it as even ALPHA software much less BETA or a Release
> Candidate and surely not a Production Release version...please don't
> embarrass yourselves by saying you do alot of QA....that obviously
> didn't happen in this release and has been a problem in the last few
> releases....
>

We do more than just build tests for QA, but as you've realized the
build tests for old(er) OSes are currently insufficient.

> I expect better from a commercial company like SourceFire now that
> they own the product as a commercial concern...its the same codebase
> for free or paid users as far as I know, the differences is paying
> users get to call for support.
>

There are also certified binary packages available.

Best regards,
--Edwin

_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

georgek at netwrx1

Nov 5, 2009, 3:29 PM

Post #3 of 7 (265 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

>We're currently investigating the possibility of using the OpenSUSE
>build service to test the next ClamAV release on multiple Linux
>distributions, including many old ones:
>openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
>Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

OK...I'm not going to debate "old" vs. "new" here but I'm fairly sure
the installed base of Fedora Core 10 and RHEL4 and Solaris 9 which are
all actively supported by the various Vendors/groups would disagree
with your assessment as would alot of businesses that are running them
in a day to day production setting for front line work.

Frankly, Don't think I would be able to get a "new" OS such as FC-11
or RHEL5 OK 'd to go into production at our company due to lack of
sufficient background from a security standpoint etc.

In any case, if you're looking for a test spot for FC10, Solaris 9,
RHEL4 I'd be happy to try to run some stuff here on a box - I'm not a
programmer but I can do basic things if given clear steps or test the
ability to at least get it to make etc in our QA/Test environment.

Let me know.

George
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

acabng at digitalfuture

Nov 6, 2009, 1:57 AM

Post #4 of 7 (257 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

George R. Kasica wrote:
> In any case, if you're looking for a test spot for FC10, Solaris 9,
> RHEL4 I'd be happy to try to run some stuff here on a box - I'm not a
> programmer but I can do basic things if given clear steps or test the
> ability to at least get it to make etc in our QA/Test environment.

Hi George,
That would be cool!

There are basically two options.
The least intrusive is a small shell script to be run daily or so from
cron which posts resuts available here: http://farm.0xacab.net/
This only requires git, a compatible compiler and an ftp client.

The other one is to run a buildbot slave. Results are available at
http://www.0xacab.net:8010/waterfall

If you want to help with either, please mail Edwin or me off list.

Thanks,
-acab
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

fajar at fajar

Nov 6, 2009, 2:12 AM

Post #5 of 7 (257 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

2009/11/6 T�r�k Edwin <edwintorok[at]gmail.com>:
> One of the build problems was an error when 'git' was not installed.
> Unfortunately all of the machines we test ClamAV on have git in $PATH
> (it is how it checks out the latest version).

I usually do software compilation/packaging on specially-prepared
chroot environment, which I can tune according to that software's
special needs. For most packages it's simply a new directory created
with "yum --installroot=... install ..."

Fedora has Mock, which does the similar thing.

> For the next release we'll setup a buildhost that doesn't have git
> installed.
> The other problem (missing include unistd.h) only occurs on old
> distributions, which we lack in our buildfarm.
>
> We're currently investigating the possibility of using the OpenSUSE
> build service to test the next ClamAV release on multiple Linux
> distributions, including many old ones:
> openSUSE 11.x, SLES/SLED 9/10/11, Fedora 10/11, RHEL 4/5, CentOS 5,
> Mandriva 2009, xUbuntu 6.06/8.04/8.10/9.04

My build host is actually just a Xen domU running RHEL4 64bit, with
chroot environments for RHEL4 and 5 32 and 64bit (which is what we
use). It saves the number of host and resources needed for testing.

For (open)Solaris, you could probably make use of branded zones which
can give you a Linux 2.4 (e.g. Centos3), Solaris 8, Solaris 9, and
Solaris 10/Opensolaris build environment.

--
Fajar
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

iane at sussex

Nov 6, 2009, 6:58 AM

Post #6 of 7 (255 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

--On 5 November 2009 20:44:51 +0200 Török Edwin <edwintorok[at]gmail.com>
wrote:

> and Mac OS X (ppc).

But, not Intel? I'm surprised, but not complaining since we still use PPC
mail servers - but not for much longer - they're a bit long in the tooth
now.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

tshaw at oitc

Nov 6, 2009, 8:26 AM

Post #7 of 7 (254 views)
Permalink

Re: Thoughts on software QA Testing (or lack thereof...) [In reply to]

At 2:58 PM +0000 11/6/09, Ian Eiloart wrote:
>--On 5 November 2009 20:44:51 +0200 T�r�k Edwin <edwintorok[at]gmail.com> wrote:
>
>>and Mac OS X (ppc).
>
>But, not Intel? I'm surprised, but not
>complaining since we still use PPC mail servers
>- but not for much longer - they're a bit long
>in the tooth now.

Works fine on intel and 64 bit as well under OSX.
Have universal build and installer ready shortly.

Tom
_______________________________________________
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com