Mailing List Archive: Maemo: Developers

Quality Assurance and Extras-testing discussion on IRC

Index | Next | Previous | View Threaded

jeremiah at jeremiahfoster

Nov 4, 2009, 6:47 AM

Post #1 of 8 (637 views)
Permalink

Quality Assurance and Extras-testing discussion on IRC

Hello,

During the most recent Maemo.org Sprint meeting, Valerio suggested we
get together on IRC and discuss the Quality Assurance process, and
Extras-testing. There are a number of things to discuss, both
technical and non-technical, and it would be great if we could come
together and create a list of actionable items so we can make the QA
process as efficient and usable as possible.

Note that we as a community can drive this process, it is we who can
shape it, make it great. Furthermore, if we can increase quality and
can demonstrate a responsible and effective way of testing
applications, we'll get more confidence from users and Nokia.
Hopefully this confidence will result in the ability to also shape the
entire development process - the toolchain, the licensing, scratchbox,
etc. This is why I think it is paramount that we act responsibly and
get as much feedback from developers as possible.

We have a great opportunity here. We are establishing a method for
doing QA and testing that is community based, on the most advanced
mobile platform in the world. We have participation from people inside
the Nokia firewall, from the community, and some excellent developers.
Soon the device will be in the hands of users and all our hard work
will have paid off.

I encourage us as a community to be open and honest about the process
- we have to fix what doesn't work and promote that which does. We
can't do that without your participation.

Meeting details:

IRC: irc.freenode.org
Channel: #maemo-meeting
Time: Tuesday, November 10th, 14:30 UTC

I hope to collect a list of issues, fixes, complaints, and maybe even
some praise, which I will further send out to our existing
communication channels so that we can execute the changes that this
process needs. Please note that Niels is away in November and his
feedback and views are crucial to our discussion since he has been
involved in maemo and the infrastructure for such a long time. But
make no mistake, this is a community process and we have Valerio to
thank for taking a leadership role in this latest phase of testing
Extras-testing.

Best regards,

Jeremiah Foster
Maemo.org debmaster
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 4, 2009, 9:36 AM

Post #2 of 8 (617 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

On Wed, Nov 4, 2009 at 14:47, Jeremiah Foster
<jeremiah [at] jeremiahfoster> wrote:
>
> There are a number of things to discuss, both technical and
> non-technical, and it would be great if we could come together and
> create a list of actionable items so we can make the QA process as
> efficient and usable as possible.

Agreed. However, IME (from other IRC meetings and the /opt BOF etc.)
no matter how much consensus is got during the meeting, there'll be
further discussion on the mailing list. This needs to be chaired as
well :-)

> Meeting details:
>
> IRC: irc.freenode.org
> Channel: #maemo-meeting
> Time: Tuesday, November 10th, 14:30 UTC

I won't be able to make this as I will be onsite at a client, however
my views can be summarised as:

* QA is good.
* The criteria are a good start, but need tweaks (see thread)
* The packages UI needs some streamlining for testers.
* As a tester, a better reminder of the checklist when checking
would be good. I also like the ease with which I can give feedback.
* As a developer, I don't *think* I want to be constrained with
"release early, release often" when fixing bugs or introducing new
small features (i.e. karma resets to zero).

I'm happy, from what I've seen in the discussion to date, that people
share enough views with me that I don't need to be there :-)

Cheers,

Andrew

> Jeremiah Foster
> Maemo.org debmaster

This should be "maemo.org debmaster" BTW ;-)

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 4, 2009, 10:56 AM

Post #3 of 8 (616 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

On Nov 4, 2009, at 18:36, Andrew Flegg wrote:

> On Wed, Nov 4, 2009 at 14:47, Jeremiah Foster
> <jeremiah [at] jeremiahfoster> wrote:
>>
>> There are a number of things to discuss, both technical and
>> non-technical, and it would be great if we could come together and
>> create a list of actionable items so we can make the QA process as
>> efficient and usable as possible.
>
> Agreed. However, IME (from other IRC meetings and the /opt BOF etc.)
> no matter how much consensus is got during the meeting, there'll be
> further discussion on the mailing list.

Yeah, I agree. Mailing list discussion is the most focused and can be
the most detailed, in my experience.

> This needs to be chaired as
> well :-)

I think Valerio will do that. I was supposed to forward this mail to
him for approval first, I failed. :-/

>
>> Meeting details:
>>
>> IRC: irc.freenode.org
>> Channel: #maemo-meeting
>> Time: Tuesday, November 10th, 14:30 UTC
>
> I won't be able to make this as I will be onsite at a client, however
> my views can be summarised as:
>
> * QA is good.
> * The criteria are a good start, but need tweaks (see thread)
> * The packages UI needs some streamlining for testers.
> * As a tester, a better reminder of the checklist when checking
> would be good. I also like the ease with which I can give
> feedback.
> * As a developer, I don't *think* I want to be constrained with
> "release early, release often" when fixing bugs or introducing new
> small features (i.e. karma resets to zero).

I've scooped this up and put it on the wiki page for the meeting as an
agenda.
http://wiki.maemo.org/QAMeeting

>
>> Jeremiah Foster
>> Maemo.org debmaster
>
> This should be "maemo.org debmaster" BTW ;-)

maemo.org debslave
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Nov 5, 2009, 1:21 PM

Post #4 of 8 (595 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

On Wed, Nov 04, 2009 at 05:36:26PM +0000, Andrew Flegg wrote:
> I won't be able to make this as I will be onsite at a client, however
> my views can be summarised as:
>
> * QA is good.
> * The criteria are a good start, but need tweaks (see thread)
> * The packages UI needs some streamlining for testers.
> * As a tester, a better reminder of the checklist when checking
> would be good. I also like the ease with which I can give feedback.
> * As a developer, I don't *think* I want to be constrained with
> "release early, release often" when fixing bugs or introducing new
> small features (i.e. karma resets to zero).

Unfortunately I also am unable to make it -- I am travelling at the moment
(and for another couple of weeks).

My views are generally in line with Andrew's except I would add the point
that I think this process has to be focused on preventing serious issues
from escaping, not on wider quality issues such as whether applications are usable, pretty,
etc. We have a user rating system to allow the good applications to rise
above the poor ones -- that is not the purpose of **this** process. Reporting
all bugs found is great. But only "dangerous" issues should cause
blockages. Those could include issues such as poor quality descriptions (or
upgrade-descriptions) because they may mislead or confuse people into
installing the application. But they don't include UX issues except ones
which "break" things (for example, an unresponsive screen).

We have to be aware that every thumbs-down is a message that we are rejecting
the application. We need to be able to point to clear rules which are
violated whenever that occurs. To be honest that already causes me some discomfort
with the "illegal or immoral" rule -- those are both judged differently in
different places, at diferent times and within different communities.

>
> I'm happy, from what I've seen in the discussion to date, that people
> share enough views with me that I don't need to be there :-)

I hope some of them can make it!

Sorry I can't be there.

Graham
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 5, 2009, 2:09 PM

Post #5 of 8 (599 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

Graham wrote:
> On Wed, Nov 04, 2009 at 05:36:26PM +0000, Andrew Flegg wrote:
> >
> >Â Â Â * QA is good.
> >Â Â Â * The criteria are a good start, but need tweaks (see thread)
> >Â Â Â * The packages UI needs some streamlining for testers.
> >Â Â Â * As a tester, a better reminder of the checklist when checking
> >Â Â Â Â Â would be good. I also like the ease with which I can give feedback.
> >Â Â Â * As a developer, I don't *think* I want to be constrained with
> >Â Â Â Â Â "release early, release often" when fixing bugs or introducing new
> >Â Â Â Â Â small features (i.e. karma resets to zero).
>
> My views are generally in line with Andrew's

Cool. Hopefully these views are fairly inline with the other opinions which have been offered.

> except I would add the point
> that I think this process has to be focused on preventing serious issues
> from escaping, not on wider quality issues such as whether applications are
> usable, pretty, etc.Â We have a user rating system to allow the good
> applications to rise above the poor ones -- that is not the purpose of **this**
> process.Â Reporting all bugs found is great.Â But only "dangerous" issues should
> cause blockages.Â Those could include issues such as poor quality descriptions
> (or upgrade-descriptions) because they may mislead or confuse people into
> installing the application.Â But they don't include UX issues except ones
> which "break" things (for example, an unresponsive screen).

You've got me convinced, 100% behind that.

> We have to be aware that every thumbs-down is a message that we are rejecting
> the application.

Indeed. A point which should be made clear to testers (and the corollary, every thumbs up is an endorsement that this piece of software won't cause a user unexpected problems).

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] blebÂ |Â http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Nov 6, 2009, 1:21 AM

Post #6 of 8 (585 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

ext Graham Cobb <g+770 [at] cobb> writes:

> Reporting all bugs found is great. But only "dangerous" issues should
> cause blockages.

My gut feeling is that a specific bug report should always be required
to block a package from entering Extras.

I.e, the logic would be "A package can pass into Extras when there is no
critical issue reported for this package, and we have been looking hard
enough to be confident that no critical issue is hiding."

The "enough" qualification in "looking hard enough" could be controlled
with karma, but input from the developer is always needed, if only to
reset the packagw karma at the appropriate time.

We can modify that to be "A package can pass into Extras when it has
less critical bugs reported than the version currently in Extras (or
both have zero), and we have been looking hard enough to be confident
that no further critical issues are hiding."

This is what Debian does, and we might need to do it too, eventually,
since it will happen that critical issues will be found in Extras and
then we have to weigh the ciritical issues in extras-testing somehow
against those in extras and decide which version we want to have in
extras. We can also leave this open and require a manual decision in
these cases.

In any case, this puts a lot of weight on the bug reports, and there
will be cases of "is not a bug!", "is, too", "ok, is fixed", "is not!",
"yeah, but is not critical", "mofo, it is! for me!" fights and we have
to settle them. Maybe that is something where karma can help, too.

So, hmm, I think this all means that I want developers and testers to
have karma, not packages. A developer with high karma would be able to
push packages through the process faster, and a high rolling tester
would be able keep bugs open and classified as critical in case of
disagreement.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

henrik.hedberg at innologies

Nov 6, 2009, 4:54 AM

Post #7 of 8 (586 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

Marius Vollmer wrote:
> ext Graham Cobb <g+770 [at] cobb> writes:
>
>> Reporting all bugs found is great. But only "dangerous" issues should
>> cause blockages.
>
> My gut feeling is that a specific bug report should always be required
> to block a package from entering Extras.

Finally someone got it! ;)

> We can modify that to be "A package can pass into Extras when it has
> less critical bugs reported than the version currently in Extras (or
> both have zero), and we have been looking hard enough to be confident
> that no further critical issues are hiding."

Yes, yes. Yes, please. I second to that.

> So, hmm, I think this all means that I want developers and testers to
> have karma, not packages. A developer with high karma would be able to
> push packages through the process faster, and a high rolling tester
> would be able keep bugs open and classified as critical in case of
> disagreement.

Very good addition.

BR,

Henrik

--
Henrik Hedberg - http://www.henrikhedberg.net/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 6, 2009, 5:10 AM

Post #8 of 8 (583 views)
Permalink

Re: Quality Assurance and Extras-testing discussion on IRC [In reply to]

On Nov 6, 2009, at 13:54, Henrik Hedberg wrote:

> Marius Vollmer wrote:
>> ext Graham Cobb <g+770 [at] cobb> writes:
>>
>>> Reporting all bugs found is great. But only "dangerous" issues
>>> should
>>> cause blockages.
>>
>> My gut feeling is that a specific bug report should always be
>> required
>> to block a package from entering Extras.
>
> Finally someone got it! ;)
>
>> We can modify that to be "A package can pass into Extras when it has
>> less critical bugs reported than the version currently in Extras (or
>> both have zero), and we have been looking hard enough to be confident
>> that no further critical issues are hiding."
>
> Yes, yes. Yes, please. I second to that.

This is actually quite similar to how debian works. Debian is not
released until all "release critical bugs" are at zero.

>
>> So, hmm, I think this all means that I want developers and testers to
>> have karma, not packages. A developer with high karma would be
>> able to
>> push packages through the process faster, and a high rolling tester
>> would be able keep bugs open and classified as critical in case of
>> disagreement.
>
> Very good addition.

Added these points to the agenda for the IRC testing meeting.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads