Mailing List Archive: Maemo: Developers

Extras QA checklist

Index | Next | Previous | View Threaded

andrew at bleb

Oct 27, 2009, 2:20 AM

Post #1 of 19 (1389 views)
Permalink

Extras QA checklist

Hi,

Quim's done a sterling job producing a first draft of the Extras QA
checklist. This is the list of things which need to be checked before
an application should get a "thumbs up" in the packages UI[1]:

http://wiki.maemo.org/Extras-testing/QA_Checklist

There are various threads on talk.maemo.org, and there's the Talk:
page in the wiki, however I think this is something which developers
should be involved, as we're the ones who will have to live up to
these standards (and I'm not sure that Talk:QA_Checklist is the best
forum for this discussion).

It can be summarised as:

* MUST have bug tracker URL in XSBC-Bugtracker control field.
* MUST NOT violate licences or copyright; MUST respect trademarks.
* MUST have all core features which are advertised (and they MUST work)
* MUST NOT promote illegal or dubious activities
* MUST NOT have reproducible crashes
* MUST NOT compromise system performance
* MUST NOT waste space in the rootfs and SHOULD use /opt as much as
possible.
* MUST NOT waste battery life when in background or in "normal" use.
* MUST NOT introduce security risks.
* SHOULD have good, complete user-facing meta-data such as package
description, XB-Maemo-Icon-26 etc.
* DON'T CARE about UI optimisation.
* DON'T CARE about visual style.

Having reviewed the list to write the above, I have three issues:

BUG TRACKER
~~~~~~~~~~~
My own issue is the XSBC-Bugtracker requirement. Some applications
really are trivial and having this as a blocker doesn't make sense to
me; after all, testers can leave comments against a specific version
in the packages UI and most end-users don't raise bugs IME:

http://wiki.maemo.org/Talk:Extras-testing/QA_Checklist#Bug_tracker

"DUBIOUS" CONTENT
~~~~~~~~~~~~~~~~~
The phrases here include "promote or endorse the misuse of alcohol,
tobacco, illegal drugs or other addictive substances" and "promote
gambling". These are weasel words. What about "Dope Wars"[2] or a
simulated gambiling poker application? What about an actual online
gaming application?

I think this should be a SHOULD NOT and contain "excessive" and/or
"likely to cause offense".

META-DATA
~~~~~~~~~
* I think a good package description should be a MUST.
* I think XB-Maemo-Icon-26 should be a SHOULD.
* I think XB-Maemo-Display-Name should be a SHOULD (there are apps,
like vim, which don't require any capitalisation or spacing
changes over and above the package name).

Comments welcome.

Cheers,

Andrew

[1] http://maemo.org/packages/repository/qa/fremantle_extras-testing/
[2] http://en.wikipedia.org/wiki/Dope_Wars

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 27, 2009, 4:53 AM

Post #2 of 19 (1346 views)
Permalink

Re: Extras QA checklist [In reply to]

A few random thoughts, not pushing for anything.

On Tuesday 27 October 2009 10:20:39 Andrew Flegg wrote:
> * MUST have bug tracker URL in XSBC-Bugtracker control field.

This is a machine controllable thing, so if it's a MUST, it has nothing to
with QA -> it should be part of the autobuilder/promoter mechanism.

> * MUST NOT violate licences or copyright; MUST respect trademarks.

This one is quite tricky. Currently, packages on Maemo often do not include
license files, the package page also does not list a license so it's actually
an effort to find out how a package/project is licensed and whether it's in
compliance with it.

> * MUST NOT have reproducible crashes

This is basically impossible to enforce, you need to be more lax than this.
For example, imagine an app has 2 reproducible crash scenarios, both
discovered only after it got to Extras. Now, the author says one is trivial
and one very difficult to solve. If you follow the above MUST NOT to he
letter he will not be able to push the fix for the trivial bug until both are
fixed as the difficult bug will serve as a blocker.

> * MUST NOT introduce security risks.

Again, the formulation is too broad here. For example installing SSH or apache
arguably *is* a security risk.

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

bundyo at gmail

Oct 27, 2009, 5:18 AM

Post #3 of 19 (1356 views)
Permalink

Re: Extras QA checklist [In reply to]

Hi,

+2c from me:

On Tue, Oct 27, 2009 at 11:20 AM, Andrew Flegg <andrew [at] bleb> wrote:

> * MUST NOT waste battery life when in background or in "normal" use.
>

What about applications that can't control what they are showing and such
that shouldn't be suspended when in background?

Regards:
Bundyo

andrew at bleb

Oct 27, 2009, 5:21 AM

Post #4 of 19 (1354 views)
Permalink

Re: Extras QA checklist [In reply to]

On Tue, Oct 27, 2009 at 12:18, Kamen Bundev <bundyo [at] gmail> wrote:
> On Tue, Oct 27, 2009 at 11:20 AM, Andrew Flegg <andrew [at] bleb> wrote:
>>
>> * MUST NOT waste battery life when in background or in "normal" use.
>
> What about applications that can't control what they are showing and such
> that shouldn't be suspended when in background?

I think the key here is "waste" and what the definition of "normal" is
for each application. Some applications will use lots of power; some
are intended to continue working in the background; however a text
editor shouldn't usually be doing any work when not in the foreground,
and this is what that rule is trying to deal with.

Does that help?

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

quim.gil at nokia

Oct 27, 2009, 5:54 AM

Post #5 of 19 (1346 views)
Permalink

Re: Extras QA checklist [In reply to]

Hi,

ext Andrew Flegg wrote:
> Hi,
>
> Quim's done a sterling job producing a first draft of the Extras QA
> checklist. This is the list of things which need to be checked before
> an application should get a "thumbs up" in the packages UI[1]:
>
> http://wiki.maemo.org/Extras-testing/QA_Checklist

Thanks Andrew. So... with this I interpret my task

9.08-10 Draft quality guidelines for extras-testing to extras promotion

is DONE and you continue with

9.08-11 Document & communicate packages interface to testers

> BUG TRACKER
> ~~~~~~~~~~~
> My own issue is the XSBC-Bugtracker requirement. Some applications
> really are trivial and having this as a blocker doesn't make sense to
> me; after all, testers can leave comments against a specific version
> in the packages UI and most end-users don't raise bugs IME:
>
> http://wiki.maemo.org/Talk:Extras-testing/QA_Checklist#Bug_tracker

So this could be an automatic check from extras-devel to extras-testing
making sure the field exists and is filled with a value starting with
"http://"

For starters and for simple apps is good enough to add
"http://maemo.org/packages" as bug tracker. But if things get more
complex then testers should encourage the developers to have a propoer
bug tracker in bugs.maemo.org or elsewhere.

If we all agree then we can implement this in the extras-devel automated
tests and remove it from the Extras-testing QA checklist once it's
proven to work.

> "DUBIOUS" CONTENT
> ~~~~~~~~~~~~~~~~~
> The phrases here include "promote or endorse the misuse of alcohol,
> tobacco, illegal drugs or other addictive substances" and "promote
> gambling". These are weasel words. What about "Dope Wars"[2] or a
> simulated gambiling poker application? What about an actual online
> gaming application?
>
> I think this should be a SHOULD NOT and contain "excessive" and/or
> "likely to cause offense".

Yes, agreed. I didn't know exactly where to start so I took the Ovi
guidelines almost literally.

> META-DATA
> ~~~~~~~~~
> * I think a good package description should be a MUST.
> * I think XB-Maemo-Icon-26 should be a SHOULD.
> * I think XB-Maemo-Display-Name should be a SHOULD (there are apps,
> like vim, which don't require any capitalisation or spacing
> changes over and above the package name).

Sounds good.

About the responses, it looks like we are trying to find corner cases
that according to the written word would be approved/blocked. Let's not
forget that ultimately such guidelines need to convince at least 10 very
human testers. Common sense is expected to prevail over the literal
written word.

It's difficult to describe beauty but it's easy to recognize it when you
see it. We can fine tune the ugly corner cases as they come.

--
Quim Gil
open source advocate
Maemo Devices @ Nokia
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Oct 27, 2009, 8:25 AM

Post #6 of 19 (1349 views)
Permalink

Re: Extras QA checklist [In reply to]

On Tue, Oct 27, 2009 at 12:54, Quim Gil <quim.gil [at] nokia> wrote:
>
> 9.08-10 Draft quality guidelines for extras-testing to extras promotion
> is DONE and you continue with
> 9.08-11 Document & communicate packages interface to testers

Yup. There is pretty much a week left in this sprint. Hmm. Might be
tricky, but I'll have a go.

> About the responses, it looks like we are trying to find corner cases
> that according to the written word would be approved/blocked. Let's not
> forget that ultimately such guidelines need to convince at least 10 very
> human testers. Common sense is expected to prevail over the literal
> written word.

Except the written word becomes the point of last resort in a dispute.
For example, if something's got a rubbish description which is going
to confuse a user, I'd give it a thumbs down - and have only today
noticed that the QA guidelines say I should let it pass.

> It's difficult to describe beauty but it's easy to recognize it when you
> see it. We can fine tune the ugly corner cases as they come.

Continually changing the goal posts doesn't seem very useful to
application authors, or fair on testers; so I think it's worth the
effort. Testers shouldn't have to keep getting emails saying "we've
tweaked the rules again", and regular testers should be able to do it
without referring to the rules each time.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Oct 28, 2009, 7:23 AM

Post #7 of 19 (1327 views)
Permalink

Re: Extras QA checklist [In reply to]

On Oct 27, 2009, at 12:53, Attila Csipa wrote:

> A few random thoughts, not pushing for anything.
>
> On Tuesday 27 October 2009 10:20:39 Andrew Flegg wrote:
>> * MUST have bug tracker URL in XSBC-Bugtracker control field.
>
> This is a machine controllable thing, so if it's a MUST, it has
> nothing to
> with QA -> it should be part of the autobuilder/promoter mechanism.
>
>> * MUST NOT violate licences or copyright; MUST respect trademarks.
>
> This one is quite tricky. Currently, packages on Maemo often do not
> include
> license files, the package page also does not list a license so it's
> actually
> an effort to find out how a package/project is licensed and whether
> it's in
> compliance with it.

Actually, this is not that hard. The license information has to be in
the debian/copyright file. If the package comes from debian, you can
be pretty sure that the license (i.e. the copyright file) is correct.
In the debian perl group we go to significant lengths to get
permissions from the copyright holder when the copyright is not
obvious and we don't package something if we cannot determine the
source of copyright.

In maemo we have to be strict about this because no one wants a
copyright violation so packagers are going to be asked to be diligent
and follow this as a MUST.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 28, 2009, 8:45 AM

Post #8 of 19 (1332 views)
Permalink

Re: Extras QA checklist [In reply to]

On Wednesday 28 October 2009 15:23:14 Jeremiah Foster wrote:
> Actually, this is not that hard. The license information has to be in
> the debian/copyright file. If the package comes from debian, you can
> be pretty sure that the license (i.e. the copyright file) is correct.

I was mostly referring to packages that are not straight Debian ports but
rather 'native' (or significantly altered) Maemo applications. But we do not
need to go that far. Often even really basic building blocks lack proper
license/attribution information (for example Qt itself has no
debian/copyright or license files included. Probably not commercial edition,
but is it GPL or LGPL ? Maybe both ?).

Anyway, I'm among the first ones to check for licensing info, and the current
approach (leaving it out from the binary packages because of space
considerations and not displaying it anywhere on the maemo.org web interface)
does not make it any easier. Surely burling through the debian dir of source
packages is not the best/easiest long-term way of doing this, especially if
it's something *QA testers* are supposed to do ?

Regards,
Attila

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 28, 2009, 9:05 AM

Post #9 of 19 (1327 views)
Permalink

Re: Extras QA checklist [In reply to]

On Wednesday 28 October 2009 16:45:56 Attila Csipa wrote:
> proper license/attribution information (for example Qt itself has no
> debian/copyright or license files included. Probably not commercial

To make this a bit more precise before someone misunderstands - the binary
package is missing the license files, the source package has three different
license files in there (GPL, LGPL, and commercial preview), and figuring out
which one(s) apply requires going through debian/rules and analyzing/running
the ./configure script).

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

avs at iki

Oct 28, 2009, 11:28 AM

Post #10 of 19 (1328 views)
Permalink

Re: Extras QA checklist [In reply to]

> * MUST NOT introduce security risks.

I'd rephrase "MUST NOT contain known security vulnerabilities" and
"MUST specify a security vulnerability reporting contact point".

This would take the ambiguity out of a security *risk* (almost nothing
is risk-free). Vulnerabilities, however, are more tangible. There is,
of course, still a class of vulnerabilities that could result in a
debate, but much less so than when talking about risk.

"Known" is also tricky - known by whom? - but it could suffice, as if
anyone who is actually involved in this QA checking "knows", it would
trigger this.

The contact point would usually be an email address and perhaps an
associated GPG key, but the bug tracker could also suffice if the
project is really keen on full disclosure.

- Antti
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Oct 28, 2009, 12:00 PM

Post #11 of 19 (1329 views)
Permalink

Re: Extras QA checklist [In reply to]

On Oct 28, 2009, at 16:45, Attila Csipa wrote:

> On Wednesday 28 October 2009 15:23:14 Jeremiah Foster wrote:
>> Actually, this is not that hard. The license information has to be in
>> the debian/copyright file. If the package comes from debian, you can
>> be pretty sure that the license (i.e. the copyright file) is correct.
>
> I was mostly referring to packages that are not straight Debian
> ports but
> rather 'native' (or significantly altered) Maemo applications. But
> we do not
> need to go that far.

No - I think you are right, packages that are not straight ports need
to be scrutinized. :)

> Often even really basic building blocks lack proper
> license/attribution information (for example Qt itself has no
> debian/copyright or license files included. Probably not commercial
> edition,
> but is it GPL or LGPL ? Maybe both ?).
>
> Anyway, I'm among the first ones to check for licensing info, and
> the current
> approach (leaving it out from the binary packages because of space
> considerations and not displaying it anywhere on the maemo.org web
> interface)
> does not make it any easier. Surely burling through the debian dir
> of source
> packages is not the best/easiest long-term way of doing this,
> especially if
> it's something *QA testers* are supposed to do ?

Yeah, this is kind of detailed work and hard to automate, but I think
we ought to try and do some type of automated check and raise a flag
if we see copyright issues.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Oct 28, 2009, 12:02 PM

Post #12 of 19 (1325 views)
Permalink

Re: Extras QA checklist [In reply to]

On Oct 28, 2009, at 19:28, Antti V�h�-Sipil� wrote:

>> * MUST NOT introduce security risks.
>
> I'd rephrase "MUST NOT contain known security vulnerabilities" and
> "MUST specify a security vulnerability reporting contact point".

This makes sense to me.
>
> This would take the ambiguity out of a security *risk* (almost nothing
> is risk-free). Vulnerabilities, however, are more tangible. There is,
> of course, still a class of vulnerabilities that could result in a
> debate, but much less so than when talking about risk.
>
> "Known" is also tricky - known by whom? - but it could suffice, as if
> anyone who is actually involved in this QA checking "knows", it would
> trigger this.

Perhaps a check against the CVE database?
>
> The contact point would usually be an email address and perhaps an
> associated GPG key, but the bug tracker could also suffice if the
> project is really keen on full disclosure.

Seems reasonable.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

avs at iki

Oct 28, 2009, 12:28 PM

Post #13 of 19 (1319 views)
Permalink

Re: Extras QA checklist [In reply to]

>> "Known" is also tricky - known by whom? - but it could suffice, as if
>> anyone who is actually involved in this QA checking "knows", it would
>> trigger this.
>
> Perhaps a check against the CVE database?

That could be a plus, but many vulnerabilities never get CVE entries
(for various reasons). So I'd still say "known" - if it's in the CVE
database, it's definitely "known" but it would also cover those known
only internally to the project.

- Antti

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Oct 28, 2009, 3:49 PM

Post #14 of 19 (1318 views)
Permalink

Re: Extras QA checklist [In reply to]

On Wednesday 28 October 2009 18:28:24 Antti V�h�-Sipil� wrote:
> > * MUST NOT introduce security risks.
>
> I'd rephrase "MUST NOT contain known security vulnerabilities" and
> "MUST specify a security vulnerability reporting contact point".

The second requirement is not reasonable. Many small programs, particularly
one-person projects, don't need "a security vulnerability reporting contact
point". There is already a maintainer field (mandatory) and the maintainer
is the contact point. In fact, I am not even keen to allow an optional
security vulnerability reporting contact point as that will mean creating yet
another Maemo-specific package control field.

And "known" means known by the developer -- no more and no less. Of course,
once a tester has found a security bug and reported it, it is known by the
developer so that means it cannot proceed until the bug is fixed.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Oct 29, 2009, 5:17 PM

Post #15 of 19 (1300 views)
Permalink

Re: Extras QA checklist [In reply to]

On Tuesday 27 October 2009 13:54:33 Quim Gil wrote:
> It's difficult to describe beauty but it's easy to recognize it when you
> see it. We can fine tune the ugly corner cases as they come.

Fine-tune case 1. Allow no packages to put data with imagery or sound files in
(non-hidden dirs inside) MyDocs unless it has very good reasons to do so.
These will invariably wreak havoc on the trackerd-MAFW-media player/photos
line. It was a 'dont worry about it' last time I asked, but today I had to
manually kill stuff to make my N900 useable for photos (I won't say project
names as the ones I encountered are all in -devel, so caveat emptor, but I
would like this to be perhaps a bit more clearly defined in the testing QA)

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

eero.tamminen at nokia

Jul 22, 2010, 6:10 AM

Post #16 of 19 (642 views)
Permalink

Re: Extras QA checklist [In reply to]

Hi,

ext Kamen Bundev wrote:
> On Tue, Oct 27, 2009 at 11:20 AM, Andrew Flegg
>> * MUST NOT waste battery life when in background or in "normal" use.
>
> What about applications that can't control what they are showing
> and such that shouldn't be suspended when in background?

They don't belong to a battery powered mobile device.

There are very few valid application use-cases for waking up on
the background:
- downloading data user has requested
- playing music (or other audio)
- tracking a route

However, even while doing these, application should never update its
window when on background as window updates are a much heavier operation
and can slow down the foreground application noticeably[1].

This is easy to test with xresponse from the SDK tools repository, just
do in SSH console:
xresponse -w '0' -a '*'

And see whether it reports anything when the app is on background.

To check for wakeups in general, one can do this in console:
strace -f -p <application PID>

E.g. Games should automatically pause when they go to background
for obvious usability reasons, not just because of battery consumption.

ext Graham Cobb wrote:
> On Wednesday 28 October 2009 18:28:24 Antti V�h�-Sipil� wrote:
>>> * MUST NOT introduce security risks.
>> I'd rephrase "MUST NOT contain known security vulnerabilities" and

So a release that fixes only some of the issues won't get promoted
until all (known) issues are fixed (and potentially new unknown ones
added)?

I would rephrase this as "MUST NOT introduce (additional) security
risks."

Then one thing missing from the QA checklist completely is checking
for resource leakage. A leaky home applet can slow down home a lot
and the caused increased swapping probably slows also rest of the device
to some extent.

Any tested application should be used for couple of days to see
that it doesn't cause problems for the device.

- Eero

[1] For example popular gPodder application is buggy because it
apparently listens to orientation changes when it's not visible and
does e.g. lots of operations when user tries to answer a call.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

th.perl at gmail

Jul 23, 2010, 4:54 AM

Post #17 of 19 (636 views)
Permalink

Re: Extras QA checklist [In reply to]

2010/7/22 Eero Tamminen <eero.tamminen [at] nokia>:
> [1] For example popular gPodder application is buggy because it
> apparently listens to orientation changes when it's not visible and
> does e.g. lots of operations when user tries to answer a call.

If you reported a bug against said app, the developer would know about it ;)

Thanks!
Thomas
(who just accidentally stepped over this thread..)
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

eero.tamminen at nokia

Jul 27, 2010, 8:30 AM

Post #18 of 19 (621 views)
Permalink

Re: Extras QA checklist [In reply to]

Hi,

ext Thomas Perl wrote:
> 2010/7/22 Eero Tamminen <eero.tamminen [at] nokia>:
>> [1] For example popular gPodder application is buggy because it
>> apparently listens to orientation changes when it's not visible and
>> does e.g. lots of operations when user tries to answer a call.
>
> If you reported a bug against said app, the developer would know about it ;)

Sure, but such bugs should be done by somebody using gPodder,
I've never tried it myself (sorry).

I just bumped in bugs.maemo.org into an issue resulting from
the bg activity by gPodder and some other apps.

- Eero
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Jul 27, 2010, 9:08 AM

Post #19 of 19 (621 views)
Permalink

Re: Extras QA checklist [In reply to]

On Tuesday 27 July 2010 17:30:13 Eero Tamminen wrote:
> I just bumped in bugs.maemo.org into an issue resulting from
> the bg activity by gPodder and some other apps.

Still, what Extras QA, as a crowdsourced effort can do in that regard is rather
limited. The number of people who are able to do such checks is very very low.
I know it sounds scary to QA engineer ears, but the best we can offer is a sort
of "it's not apparent it's leaking". This again leads back the the extras-
testing-is-not-a-proper-QA-just-a-quick-check-for-obvious-showstoppers
discussion, but I digress...

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads