Mailing List Archive: Maemo: Developers

Testing marathon & Q&A Feedback

Index | Next | Previous | View Threaded

vdv100 at gmail

Nov 1, 2009, 6:20 AM

Post #1 of 11 (195 views)
Permalink

Testing marathon & Q&A Feedback

Hey,

as some know yesterday we did a Extras-testing marathon, in order to
get more applications ready to Extras but also to test the Q&A process
elaborated by the community. We had more than 60 testers (around 40
all the time), but not all actively participated, was good to see some
newcomers participating and learning more about Maemo :).
The testing was more slow than I expected, In about 5 hours, we tested
around 45 apps, but we write a lot of comments and bug reports that
will help to improve the tested applications for sure.
The major difficult that we faced was our maemo.org infrastructure,
most of the time was impossible to vote or comment a app, this was
very frustrating.

A lot of feedback was gathered around the Q&A process and the packages
interface, I'll name some of the improvements discussed below, if you
attended the marathon please report your experience as well, and the
help us improve the processes.

Package Interface:

- Developers should be able to remove their own apps from testing (at
lest the entry in the package interface) - We found some apps where
the developer explicit says to not vote the app, this helps to waste
more time.
- Voters should be able to change their votes - Mistakes happen,
sometimes we voted down a app, then the developer replies to our
comments, and we see that we did a mistake, but we can't change our
vote anymore.
- Only allow one vote per person - I think this is partially fixed,
but due to the lack of response from the servers, I saw a lot of
repeated votes and comments in some apps.
- Keep app karma - already discussed in other thread.
- Keep old comments - Put some kind of separator in the comments or
add a link to the old comments. It's easier to test based in the old
comments, we can test the major blockers first.
- Subscribe to comments option - Is really annoying to keep checking
the app comments page, seeking for a reply to our comments.

Q&A:

- THe Bug tracker shouldn't be a blocker, if their's a way to contact
the maintainer in the packages interface - A lot of developers don't
want to have a bug tracker for very small applications and they prefer
to receive the bug reports by e-mail, of course we should encourage
them to add their "bug contact" in the correct field, but almost
everyone agreed that shouldn't be a blocker.
- What to do with command line applications ? - I saw the reaction of
the testers when they installed some command line apps (Where's the
app icon ? Is this thing broken ? How to use it ?). IMO those are
'hackers' apps and an hacker will know how to install them, do we
really want these tool under /utilities or /network mixed with other
end users apps ? (IMO they should have their own category, if
/development doesn't fit here)

In general I strongly believe that we're in the right path, if we want
quality applications, we need to follow some criteria and we need to
respect the rules, same criteria for everyone, this is not a
popularity contest.
The only blocker that I see right now is the miss features in the
packages interface, as some already reported in other thread, is
really annoying start from zero everytime that a small fix is applied
to a app.
Improvements to the package interface is for me the highest priority
task that we've right now, this really need to be improved.

Best regards,

--
Valério Valério

http://www.valeriovalerio.org
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrea at borgia

Nov 1, 2009, 7:22 AM

Post #2 of 11 (187 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

Valerio Valerio ha scritto:

> as some know yesterday we did a Extras-testing marathon, in order to
> get more applications ready to Extras but also to test the Q&A process
> elaborated by the community. We had more than 60 testers (around 40
> all the time), but not all actively participated, was good to see some
> newcomers participating and learning more about Maemo :).

As one of those newcomers, I thank you again for the great job you did
in conducting the marathon.

I share your comments on the process, but I want to add a tiny thought:
the all-together testing style left me wondering whether everybody else
was checking the same issues over and over (lots of people checking /opt
for a package) instead of splitting tasks to cover more ground.

I guess for the future I'll adopt a few packages and keep an eye on
them, makes more sense to me, though the idea of participating in the
marathon was indeed a good motivator to start testing.

A.
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Nov 1, 2009, 7:31 AM

Post #3 of 11 (187 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

On Sunday 01 November 2009 15:20:07 Valerio Valerio wrote:
> as some know yesterday we did a Extras-testing marathon, in order to
> get more applications ready to Extras but also to test the Q&A process

Just to chime in, thanks to all the folks taking the trouble to test apps they
might personally not even be interested in. Special thanks to Valerio for
spearheading/moderating the effort. He also listed most points that caused
delays and grief during testing, I'd like to add only a few minor points:

- The icon-cache (aka free space) leak bug make testing really difficult for me.
I don't have too much free space so had to reboot every 15 minutes. Not fun. I
see this as a serious problem for end users, those who agree please vote bug
up. https://bugs.maemo.org/show_bug.cgi?id=5450

- Too many applications compared to time and available testers. The effort was
valiant, but even so we hardly went through the first page, and even there,
plenty of apps did not get 10 votes (regardless of up or down). We need more
people doing this, so the effort could be split (e.g. tester group A checks one
app while tester group B checks another).

- The approval process could have been more interactive - often the developers
were right there, but because of the time of getting through the autobuilder,
tester feedback response resulted only in a bug report, even though we could
have had 'fixed' applications on the spot (online hackfest if you wish) - and
appropriate votes to confirm the changes.

- The discussed fix-or-promote dilemma. I promoted the latest version of my
mini-app to testing in anticipation of larger tester response (previous
version already had 7 karma at that point). Now the app has 4 karma and no new
feedback. It's hard to recommend to people to upload newest versions for
testing when that means they go to the end of the test queue.

- Does this bring karma ? I sure think serious testing is worth a LOT and do
not see that adequately reflected in karma. Could also be motivating in
bringing more testers to the table. Ways/ideas of making this abuse-proof are
welcome.

Regards,
Attila
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

vdv100 at gmail

Nov 1, 2009, 7:45 AM

Post #4 of 11 (187 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

HI,

On Sun, Nov 1, 2009 at 3:31 PM, Attila Csipa <maemo[at]csipa.in.rs> wrote:
> On Sunday 01 November 2009 15:20:07 Valerio Valerio wrote:
>> as some know yesterday we did a Extras-testing marathon, in order to
>> get more applications ready to Extras but also to test the Q&A process
>
> Just to chime in, thanks to all the folks taking the trouble to test apps they
> might personally not even be interested in. Special thanks to Valerio for
> spearheading/moderating the effort. He also listed most points that caused
> delays and grief during testing, I'd like to add only a few minor points:
>
> - The icon-cache (aka free space) leak bug make testing really difficult for me.
> I don't have too much free space so had to reboot every 15 minutes. Not fun. I
> see this as a serious problem for end users, those who agree please vote bug
> up. https://bugs.maemo.org/show_bug.cgi?id=5450
>
> - Too many applications compared to time and available testers. The effort was
> valiant, but even so we hardly went through the first page, and even there,
> plenty of apps did not get 10 votes (regardless of up or down). We need more
> people doing this, so the effort could be split (e.g. tester group A checks one
> app while tester group B checks another).
>
> - The approval process could have been more interactive - often the developers
> were right there, but because of the time of getting through the autobuilder,
> tester feedback response resulted only in a bug report, even though we could
> have had 'fixed' applications on the spot (online hackfest if you wish) - and
> appropriate votes to confirm the changes.
>
> - The discussed fix-or-promote dilemma. I promoted the latest version of my
> mini-app to testing in anticipation of larger tester response (previous
> version already had 7 karma at that point). Now the app has 4 karma and no new
> feedback. It's hard to recommend to people to upload newest versions for
> testing when that means they go to the end of the test queue.
>
> - Does this bring karma ?

Yes, two points per comment, 0.25 per thumb.

Best regards,

--
Valério Valério
Maemo Community Council Chair

http://www.valeriovalerio.org

I sure think serious testing is worth a LOT and do
> not see that adequately reflected in karma. Could also be motivating in
> bringing more testers to the table. Ways/ideas of making this abuse-proof are
> welcome.
>
> Regards,
> Attila
> _______________________________________________
> maemo-developers mailing list
> maemo-developers[at]maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

vilre at cs

Nov 1, 2009, 11:46 AM

Post #5 of 11 (178 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

Hi,

just a bit more fuel on the fire :)

> Just to chime in, thanks to all the folks taking the trouble to test apps they
> might personally not even be interested in.

Is the approval/karma process going to be actually a popularity contest?
Popular titles get votes fast, a niche software will not. Unless there
is regular testing marathons, I think this will be an issue.

> We need more people doing this, so the effort could be split (e.g.
> tester group A checks one app while tester group B checks another).

One can not test everything on virtual machine. For good testing, one
should have a device with a default setup so the effects can be observed
- ie. energy usage, system configuration changes, compability, etc. Good
testing takes quite a lot of time.

Additionally, I would think that one does not want to just put any
packages for testing on personal device with personal data. You might
accept the risk for software you like (and trust for some reason), but
not for all random packages -> popularity contest. As comparison,
nothing should never be tested on a server in production.

Who could have more than a one device, so one could be used for testing?
Probably only somebody who is developing on the device for a company.
I'm not, I could assume that there is many other hobbyists too..

> - Does this bring karma ? I sure think serious testing is worth a LOT and do
> not see that adequately reflected in karma. Could also be motivating in
> bringing more testers to the table. Ways/ideas of making this abuse-proof are
> welcome.

Good QA is worth a lot. The testing karma system at worst is just a gate
where the entrace criteria is "looks good or I want the new version".
Why the testing karma is even labeled as karma, when people have karma
too.. confusing. Should be acceptance points, manna, or something else.

Maybe this manna/karma thing has been though out, but somehow if feel
that the research for similar systems was not done before rolling it
out. There seems to be too many holes, and I just though a while. Every
serious linux distribution has some kind of QA system. Most of the
software makers have QA systems. Do not invent the wheel again..

There is no separate queue for the security fixes as mentioned elsewhere
in the thread. How is the security process run anyways?
-> learn and imitate debian/suse/redhat..

How and what the person tested before giving thumb?
-> Checklist which has to be filled before vote, all QA checklist points
has to be checked before acceptance, some by recognized persons.

How to motivate to do the testing? How are the testers rewarded?
-> Karma, priviledges, titles (devel -> tester -> senior tester->...)

The testing and package history is not visible, just current comments.
There should be a combined view with bugs etc.
-> Hmm, Launchpad or some other system?

Looking current queue first page, "libeet1" in queue - any takers? How
one does test a library? There is just procedure for application. A
library could be a can of worms.. or the next version could be..
-> What are others doing..

Popular packages are voted more than others - popularity contest.
-> Each application area such as command line, tools, games etc. should
have their own group of people who take care of them.

Until there is testers groups, upload a package, wait 10 days and
register 10 accounts and upvote.. but luckily it going just to extras :)
-> Upvotes should be accepted from known persons only. Otherwise there
might be trust issues. Testing is already a priviledge. Nokia could
support testers, for example with personal device and testing device.

--
VRe :: http://iki.fi/vre :: +358 40 5775 456
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

vdv100 at gmail

Nov 1, 2009, 12:50 PM

Post #6 of 11 (174 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

HI,

some good points, but this isn't easy as it seems, some of the
suggestions here involve a lot of resources that maemo.org can't
provide right now, in my opinion, lets try to concentrate in the basic
improvements, following the actual testing criteria and adopting some
improvements.

On Sun, Nov 1, 2009 at 7:46 PM, Ville Reijonen <vilre[at]cs.tut.fi> wrote:
>
> Hi,
>
> just a bit more fuel on the fire :)
>
>> Just to chime in, thanks to all the folks taking the trouble to test apps they
>> might personally not even be interested in.
>
> Is the approval/karma process going to be actually a popularity contest?
> Popular titles get votes fast, a niche software will not. Unless there
> is regular testing marathons, I think this will be an issue.

Of course, I expect that people vote for the applications they use.
One idea already suggested in the past is the creation of testing
teams as you mention below.

>
>
>> We need more people doing this, so the effort could be split (e.g.
>> tester group A checks one app while tester group B checks another).
>
> One can not test everything on virtual machine. For good testing, one
> should have a device with a default setup so the effects can be observed
> - ie. energy usage, system configuration changes, compability, etc. Good
> testing takes quite a lot of time.
>
> Additionally, I would think that one does not want to just put any
> packages for testing on personal device with personal data. You might
> accept the risk for software you like (and trust for some reason), but
> not for all random packages -> popularity contest. As comparison,
> nothing should never be tested on a server in production.
>
> Who could have more than a one device, so one could be used for testing?
> Probably only somebody who is developing on the device for a company.
> I'm not, I could assume that there is many other hobbyists too..
>
>
>> - Does this bring karma ? I sure think serious testing is worth a LOT and do
>> not see that adequately reflected in karma. Could also be motivating in
>> bringing more testers to the table. Ways/ideas of making this abuse-proof are
>> welcome.
>
> Good QA is worth a lot. The testing karma system at worst is just a gate
> where the entrace criteria is "looks good or I want the new version".
> Why the testing karma is even labeled as karma, when people have karma
> too.. confusing. Should be acceptance points, manna, or something else.

You get karma for commenting and rating the applications, like in
other 'divisions' of maemo.org (Applications comments, news,
brainstorming,..), it's a form of reward for your collaboration, of
course it isn't perfect and can be abused.

>
> Maybe this manna/karma thing has been though out, but somehow if feel
> that the research for similar systems was not done before rolling it
> out. There seems to be too many holes, and I just though a while. Every
> serious linux distribution has some kind of QA system. Most of the
> software makers have QA systems. Do not invent the wheel again..

Well, if you can suggest a QA system already implemented that fit our
needs, we're listen.

>
>
> There is no separate queue for the security fixes as mentioned elsewhere
> in the thread. How is the security process run anyways?
> -> learn and imitate debian/suse/redhat..

That's one of the miss feature of the package interface among others.

>
> How and what the person tested before giving thumb?
> -> Checklist which has to be filled before vote, all QA checklist points
> has to be checked before acceptance, some by recognized persons.
>
> How to motivate to do the testing? How are the testers rewarded?
> -> Karma, priviledges, titles (devel -> tester -> senior tester->...)

Karma is already there ;).
>
> The testing and package history is not visible, just current comments.
> There should be a combined view with bugs etc.
> -> Hmm, Launchpad or some other system?

Another miss feature...

>
> Looking current queue first page, "libeet1" in queue - any takers? How
> one does test a library? There is just procedure for application. A
> library could be a can of worms.. or the next version could be..
> -> What are others doing..

Errr, wrong queue, try this one:
http://maemo.org/packages/repository/qa/fremantle_extras-testing/
>
> Popular packages are voted more than others - popularity contest.
> -> Each application area such as command line, tools, games etc. should
> have their own group of people who take care of them.

Teams of testers, right.

>
> Until there is testers groups, upload a package, wait 10 days and
> register 10 accounts and upvote.. but luckily it going just to extras :)
> -> Upvotes should be accepted from known persons only. Otherwise there
> might be trust issues. Testing is already a priviledge. Nokia could
> support testers, for example with personal device and testing device.

Not that easy to trick the system, our community is very active, and
believe me, we're not sleeping here.

Best regards,

--
Valério Valério
Maemo Community Council Chair

http://www.valeriovalerio.org

>
> --
> VRe :: http://iki.fi/vre :: +358 40 5775 456
> _______________________________________________
> maemo-developers mailing list
> maemo-developers[at]maemo.org
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 1, 2009, 3:41 PM

Post #7 of 11 (172 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

On Nov 1, 2009, at 16:22, Andrea Borgia wrote:

> Valerio Valerio ha scritto:
>
>> as some know yesterday we did a Extras-testing marathon, in order to
>> get more applications ready to Extras but also to test the Q&A
>> process
>> elaborated by the community. We had more than 60 testers (around 40
>> all the time), but not all actively participated, was good to see
>> some
>> newcomers participating and learning more about Maemo :).
>
> As one of those newcomers, I thank you again for the great job you did
> in conducting the marathon.

I agree - this was a productive initiative, well done Valerio!

I think we should do this again.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 1, 2009, 4:00 PM

Post #8 of 11 (172 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

On Nov 1, 2009, at 21:50, Valerio Valerio wrote:

>>
>> Maybe this manna/karma thing has been though out, but somehow if feel
>> that the research for similar systems was not done before rolling it
>> out. There seems to be too many holes, and I just though a while.
>> Every
>> serious linux distribution has some kind of QA system. Most of the
>> software makers have QA systems. Do not invent the wheel again..

In the automated QA process you'll be happy to know that we are using
tools from debian. Specifically, we are adapting Lintian, the debian
policy checker, to Maemo and calling it Maemian because it will
eventually check different things. All additions will be passed
upstream for merging if debian so wishes. We definitely don't want to
re-invent the wheel. :)

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

quim.gil at nokia

Nov 2, 2009, 2:08 AM

Post #9 of 11 (163 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

ext Ville Reijonen wrote:
> Is the approval/karma process going to be actually a popularity contest?
> Popular titles get votes fast, a niche software will not. Unless there
> is regular testing marathons, I think this will be an issue.

I believe this problem will be solved as soon as we have the first
thousands of real users.

Very popular apps will get a critical mass of testers ready to process
new releases.

Very specialized apps... too. You only need a dozen of committed users.

Then it ight be that new fresh apps take a while to get enough testers
and it might be that the first jump to Extras is difficult. Having a
brigade of mercenary testers might help, sure.

But if after some time an app still hasn't got enough interested to
atract a dozen of regular testers... well, it might be that such project
has other deeper problems due to lack of user interest.

>> We need more people doing this, so the effort could be split (e.g.
>> tester group A checks one app while tester group B checks another).
>
> One can not test everything on virtual machine. For good testing, one
> should have a device with a default setup so the effects can be observed
> - ie. energy usage, system configuration changes, compability, etc. Good
> testing takes quite a lot of time.

But the time should be better split. The basic feedback I got from busy
Nokia employees with devices, professional experience and willingness to
help is that the thumbs up/down should be applied to each QA criteria.

If you are asking me to give thumbs up / down on 10 criteria the
realistic options are:

- I won't rate.
- I will rate ignoring some QA criteria.
- I will rate after investing a lot of time and skills.

If instead the rating would be done by QA criteria then it would be
easier to see whet are the parts missing testing. Someone might find
interesting to test optification because it's simple for him, and this
will save the testing to another one more interested on crashes, or
legal aspects, usability, etc.

I believe this idea needs further thinking. This might be a big part of
the solution.

> Additionally, I would think that one does not want to just put any
> packages for testing on personal device with personal data. You might
> accept the risk for software you like (and trust for some reason), but
> not for all random packages -> popularity contest. As comparison,
> nothing should never be tested on a server in production.

The N900 is not a server. There is not lack of people out there ready to
install testing versions of Debian, Ubuntu, Windows, Firefox and long
etc for their primary use. For instance, that has been my case for many
years and for something like my laptop = 100 times more critical than my
mobile device.

> Maybe this manna/karma thing has been though out, but somehow if feel
> that the research for similar systems was not done before rolling it
> out. There seems to be too many holes, and I just though a while. Every
> serious linux distribution has some kind of QA system. Most of the
> software makers have QA systems. Do not invent the wheel again..

URLs proving your point, please?

Not every Linux distribution has the consumer focus of Maemo 5 / N900
and definitely not every Linux distribution needs to handle specific
problems of mobile devices. Besides, not every Linux distribution pay
actually much attention to the application packages being pushed by its
contributors and it takes only 10 random installs from your distro to
find out.

So no, there are not that many references for community QA models. I
believe the Extras QA process is highly innovative and it has its
chances of serving a useful purpose to the rest of the free software
community.

--
Quim Gil
open source advocate
Maemo Devices @ Nokia
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

vilre at cs

Nov 2, 2009, 12:46 PM

Post #10 of 11 (153 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

Hi,

> But the time should be better split. The basic feedback I got from busy
> Nokia employees with devices, professional experience and willingness to
> help is that the thumbs up/down should be applied to each QA criteria.

Yap, this I suggested. Good that this is verified. So, it should be
possible to give thumbs up for single item on the QA list. This of
course means that current 10 upvotes system does not work as is.
Proposal below (feel free to hack):

-Every package has minimum 10 days of quarantee.
-Package overstaying over a month is booted back to devel.
(If it is not accepted by then, it will never be)
-Package retains its history for later viewing.
(Nice to see what has happened before)
-Every package has QA list.
(Based the current wiki list - the testing criteria)
-Every item on the QA list has to be examined before acceptance.
(The procedure should be the same for all)
-A person can check one or multiple QA items.
(Do what you are good at or what you have time for)
-Same QA list item can be checked by multiple persons.
(Never trust a single person, or do we? High karma -> power of two)
-Some QA items can be checked automatically.
(optification for example)
-Bugfix will reset the quarantee timer.
(The quarantine was clearly working, some more is therefore needed)
-Bugfix will remove the second upvote for every QA item.
(Bugfix can introduce other bugs, better to be sure)
-A qualified application is released to extras.

As said, feel free to modify. First is always rough.. :)

..I have seen something quite similar, a familiar basic structure..
- Queues
- Quarantine
- Checklist
- History
- Check and cross check procedures
- Automation
Anyone else?
A hint: ever travelled with a pet?

>> Maybe this manna/karma thing has been though out, but somehow if feel
>> that the research for similar systems was not done before rolling it
>> out. There seems to be too many holes, and I just though a while.
>> Every serious linux distribution has some kind of QA system. Most of
>> the software makers have QA systems. Do not invent the wheel again..
>
>URLs proving your point, please?

I have limited free time, sorry. I'll just give yet another example. How
extras security is supposed to be handled? How to report a security
problem? Currently there is no way. It can not go thru voting, it'll be
stalled in queue (note from earlier discussion). How Debian does it? Ah,
separate security team which can bypass the queues. How about that..

Actually it is not just QA, it's the whole process.

> Not every Linux distribution has the consumer focus of Maemo 5 / N900
> and definitely not every Linux distribution needs to handle specific
> problems of mobile devices. Besides, not every Linux distribution pay
> actually much attention to the application packages being pushed by its
> contributors and it takes only 10 random installs from your distro to
> find out.

But some (distributions) have consumer focus? And some (distributions)
handle specific problems (for mobile devices)? And some pay attention to
what they ship? Of course there is no single one which is exact match.
One needs to take the best parts, and add some own sauce. Building up
from zero because none is similar is just plain silly. We could be using
tools which already work and spice them a bit!

Having people voting for packages to go through queue does not make us
any special yet - that will come later on.

> So no, there are not that many references for community QA models.

But there are some? If there is need to go deeper, maybe it is not QA
models but some other community based structures what should be looked
at. Maybe some friendly sociologist could help. I know only hairy
computer scientists and business yuppies so no go for me :)

> I believe the Extras QA process is highly innovative and it has its
> chances of serving a useful purpose to the rest of the free software
> community.

Definitely, it will be.

--
VRe :: http://iki.fi/vre :: +358 40 5775 456
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

quim.gil at nokia

Nov 3, 2009, 12:20 AM

Post #11 of 11 (127 views)
Permalink

Re: Testing marathon & Q&A Feedback [In reply to]

Hi,

In general I think that new apps should tend to get all ratings reseted
if they go back to extras-devel because of a blocker, while app updates
would keep their current positive ratings through new extras-testing
iterations. This way we are conservative with new apps but more liberal
with updates. Makes sense?

ext Ville Reijonen wrote:
> Hi,
>
>> But the time should be better split. The basic feedback I got from busy
>> Nokia employees with devices, professional experience and willingness to
>> help is that the thumbs up/down should be applied to each QA criteria.
>
> Yap, this I suggested. Good that this is verified. So, it should be
> possible to give thumbs up for single item on the QA list. This of
> course means that current 10 upvotes system does not work as is.
> Proposal below (feel free to hack):
>
> -Every package has minimum 10 days of quarantee.
> -Package overstaying over a month is booted back to devel.
> (If it is not accepted by then, it will never be)
> -Package retains its history for later viewing.
> (Nice to see what has happened before)
> -Every package has QA list.
> (Based the current wiki list - the testing criteria)
> -Every item on the QA list has to be examined before acceptance.
> (The procedure should be the same for all)
> -A person can check one or multiple QA items.
> (Do what you are good at or what you have time for)
> -Same QA list item can be checked by multiple persons.
> (Never trust a single person, or do we? High karma -> power of two)
> -Some QA items can be checked automatically.
> (optification for example)
> -Bugfix will reset the quarantee timer.
> (The quarantine was clearly working, some more is therefore needed)
> -Bugfix will remove the second upvote for every QA item.
> (Bugfix can introduce other bugs, better to be sure)
> -A qualified application is released to extras.

Looks good overall.

- If something can be checked automatically then it must be moved to the
jump between extras-devel to extras-testing, where the automated tests
are done.

- Maybe it would be good to present user karma next to their thumbs? It
would be useful to have an approximate idea of the "experience" put in
an evaluation, or at least it could be useful when someone is trying to
game the system creating a bunch of virtual users.

- Maybe 3 thumbs up in each criteria is enough? I mean, if someone with
high karma (read community reputation) says in the Power Management
entry "I have run powertop and this app is fine" next to his thumbs
up... most of us will give thumbs up only after checking that indeed our
battery is not drained after installing the app.

- If an app is dumped back to extras-devel because of a blocker in one
row, the karma there should be reseted but the other rows could just stay.

> As said, feel free to modify. First is always rough.. :)

ditto

--
Quim Gil
open source advocate
Maemo Devices @ Nokia
_______________________________________________
maemo-developers mailing list
maemo-developers[at]maemo.org
https://lists.maemo.org/mailman/listinfo/maemo-developers

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com