Mailing List Archive: Maemo: Developers

QA process = bug fixing disincentive?

1 2 3

View All

Index | Next | Previous | View Threaded

marius.vollmer at nokia

Nov 3, 2009, 12:43 AM

Post #26 of 54 (1956 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

ext Jeremiah Foster <jeremiah [at] jeremiahfoster> writes:

> To beat the horse dead;
>
> foo_1.0-1maemo0 -> bug fix -> foo_1.0-1maemo1 = All karma retained
> foo_1.0-1maemo0 -> feature -> foo_1.1-1maemo0 = Karma set to zero

Nitpick: 1.0 -> 1.1 might well be a bug fix release as well. Also, I
think that many packages in Extras are native and don't use a "Maemo
revision" in their version. Or did we redefine the meaning of the part
of a version after the dash?
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 3, 2009, 3:16 AM

Post #27 of 54 (1957 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Tue, Nov 3, 2009 at 08:43, Marius Vollmer <marius.vollmer [at] nokia> wrote:
> ext Jeremiah Foster <jeremiah [at] jeremiahfoster> writes:
>
>> To beat the horse dead;
>>
>> foo_1.0-1maemo0 -> bug fix -> foo_1.0-1maemo1 = All karma retained
>> foo_1.0-1maemo0 -> feature -> foo_1.1-1maemo0 = Karma set to zero
>
> Nitpick: 1.0 -> 1.1 might well be a bug fix release as well. Also, I
> think that many packages in Extras are native and don't use a "Maemo
> revision" in their version. Or did we redefine the meaning of the part
> of a version after the dash?

Agreed. -maemo0 to -maemo1 is supposed to be a Maemo-specific change
or a packaging change (AIUI). Native packages (such as Hermes,
Attitude etc.) don't have that suffix and use a traditional x.y.z
numbering scheme.

Minor bug fixes increment 'z', new features increment 'y' and major
milestones increment 'x'.

So, according to this the packages interface would need a heuristic to
detect a change in just the least significant part of the number.

Something like, in Perl:

my ($base, $lsb) = $oldVersion =~ /^(.*?)(\d+)$/;
my $minor = $newVersion eq $base.($lsb + 1);
&resetPackageKarma() unless $minor;

This'd handle:
2.0 -> 2.1
2.0.0 -> 2.0.1 (but not 2.0.0 -> 2.1.0)
2.0.0-maemo1 -> 2.0.0-maemo2 (but not -> 2.0.1-maemo1)

Thoughts?

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 5:35 AM

Post #28 of 54 (1964 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 3, 2009, at 12:16, Andrew Flegg wrote:

> On Tue, Nov 3, 2009 at 08:43, Marius Vollmer
> <marius.vollmer [at] nokia> wrote:
>> ext Jeremiah Foster <jeremiah [at] jeremiahfoster> writes:
>>
>>> To beat the horse dead;
>>>
>>> foo_1.0-1maemo0 -> bug fix -> foo_1.0-1maemo1 = All karma
>>> retained
>>> foo_1.0-1maemo0 -> feature -> foo_1.1-1maemo0 = Karma set
>>> to zero
>>
>> Nitpick: 1.0 -> 1.1 might well be a bug fix release as well. Also, I
>> think that many packages in Extras are native and don't use a "Maemo
>> revision" in their version. Or did we redefine the meaning of the
>> part
>> of a version after the dash?
>
> Agreed. -maemo0 to -maemo1 is supposed to be a Maemo-specific change
> or a packaging change (AIUI). Native packages (such as Hermes,
> Attitude etc.) don't have that suffix and use a traditional x.y.z
> numbering scheme.
>
> Minor bug fixes increment 'z', new features increment 'y' and major
> milestones increment 'x'.
>
> So, according to this the packages interface would need a heuristic to
> detect a change in just the least significant part of the number.
>
> Something like, in Perl:
>
> my ($base, $lsb) = $oldVersion =~ /^(.*?)(\d+)$/;
> my $minor = $newVersion eq $base.($lsb + 1);
> &resetPackageKarma() unless $minor;
>
> This'd handle:
> 2.0 -> 2.1
> 2.0.0 -> 2.0.1 (but not 2.0.0 -> 2.1.0)
> 2.0.0-maemo1 -> 2.0.0-maemo2 (but not -> 2.0.1-maemo1)
>
> Thoughts?

This is what I had in mind but skipped on elaborating in an effort to
keep things as clear as possible. I think this solution is excellent
and perhaps we can implement it if there is consensus?

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

henrik.hedberg at innologies

Nov 3, 2009, 5:58 AM

Post #29 of 54 (1960 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

> On Nov 3, 2009, at 12:16, Andrew Flegg wrote:

>> Agreed. -maemo0 to -maemo1 is supposed to be a Maemo-specific change
>> or a packaging change (AIUI). Native packages (such as Hermes,
>> Attitude etc.) don't have that suffix and use a traditional x.y.z
>> numbering scheme.

Not necessarily. There is no official version numbering sceme for
native Maemo packages. For example, some packages are using date, like
20091019.

Jeremiah Foster wrote:

> This is what I had in mind but skipped on elaborating in an effort to
> keep things as clear as possible. I think this solution is excellent
> and perhaps we can implement it if there is consensus?

It is nice to see that there is a strong push to make changes into
the way karma is handled in QA. However, I suggest that you do not try
to guess anything from a version number - unless you want to make a new
requirement for a package as a side effect. Forcing developers to use a
specific version numbering scheme would make Maemo even more different
than other Linux distributions (see, for example [1]).

Packages are promoted with the web interface. Simple checkbox there
is enough to implement this feature.

BR,

Henrik

[1]
http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version

--
Henrik Hedberg - http://www.henrikhedberg.net/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 3, 2009, 6:06 AM

Post #30 of 54 (1958 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Tue, Nov 3, 2009 at 13:58, Henrik Hedberg
<henrik.hedberg [at] innologies> wrote:
>> On Nov 3, 2009, at 12:16, Andrew Flegg wrote:
>
>>> Agreed. -maemo0 to -maemo1 is supposed to be a Maemo-specific change
>>> or a packaging change (AIUI). Native packages (such as Hermes,
>>> Attitude etc.) don't have that suffix and use a traditional x.y.z
>>> numbering scheme.
>
> Not necessarily. There is no official version numbering sceme for
> native Maemo packages. For example, some packages are using date, like
> 20091019.

True.

> Packages are promoted with the web interface. Simple checkbox there
> is enough to implement this feature.

Except how do you try to prevent abuse (whether intentional or
accidental)? At least with the version number you've got some safety
check (although it is in no way comprehensive). It also requires more
changes at more levels (I bet), so harder to implement.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

frank.banul at gmail

Nov 3, 2009, 6:34 AM

Post #31 of 54 (1954 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Tue, Nov 3, 2009 at 8:06 AM, Andrew Flegg <andrew [at] bleb> wrote:
> On Tue, Nov 3, 2009 at 13:58, Henrik Hedberg
> <henrik.hedberg [at] innologies> wrote:
>>> On Nov 3, 2009, at 12:16, Andrew Flegg wrote:
>>
>>>> Agreed. -maemo0 to -maemo1 is supposed to be a Maemo-specific change
>>>> or a packaging change (AIUI). Native packages (such as Hermes,
>>>> Attitude etc.) don't have that suffix and use a traditional x.y.z
>>>> numbering scheme.
>>
>> Not necessarily. There is no official version numbering sceme for
>> native Maemo packages. For example, some packages are using date, like
>> 20091019.
>
> True.
>
>> Packages are promoted with the web interface. Simple checkbox there
>> is enough to implement this feature.
>
> Except how do you try to prevent abuse (whether intentional or
> accidental)? At least with the version number you've got some safety
> check (although it is in no way comprehensive). It also requires more
> changes at more levels (I bet), so harder to implement.

I don't see the difference between the safety of version numbers or
check boxes on the promotion page since I control both.

Frank
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 3, 2009, 6:38 AM

Post #32 of 54 (1951 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Tue, Nov 3, 2009 at 14:34, Frank Banul <frank.banul [at] gmail> wrote:
> On Tue, Nov 3, 2009 at 8:06 AM, Andrew Flegg <andrew [at] bleb> wrote:
>>
>> Except how do you try to prevent abuse (whether intentional or
>> accidental)? At least with the version number you've got some safety
>> check (although it is in no way comprehensive). It also requires more
>> changes at more levels (I bet), so harder to implement.
>
> I don't see the difference between the safety of version numbers or
> check boxes on the promotion page since I control both.

Version numbers are more visible? It's easier to lie when no-one will
know about it (unless we percolate "this is a minor upgrade over
0.2.2" all through the website).

It's easier to check a checkbox for a semi-major release whereas
version numbers are something more sacrosanct (at least to me).

However, as I said, bigger reason is that it'll be easier to implement :-)

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

rael at edge

Nov 3, 2009, 10:25 AM

Post #33 of 54 (1960 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

Hello!

> Except how do you try to prevent abuse (whether intentional or
> accidental)? At least with the version number you've got some safety
> check (although it is in no way comprehensive). It also requires more
> changes at more levels (I bet), so harder to implement.

I think it is time to decide (again?) if we trust developers in their
atempt to get their software/package into extras or not.

There are some points in the decussion about handling of extras-testing
extras propagation for which we should find a simple solution for - once we
have finally (for some time ;-)) decided about trust.

It looks to me that currently we are randomly hopping form the "trust" to
the "as stable as possible" road back and forth and have to discuss/find
(implicitely) the road we are on for every sub problem again and again.

We likely agree that either extreme is not good for different reasons, but
we havn't a clear definition for the middle way either (it is dark out
there even in the night on the highway). We defined some criteria for
extras which I find astonishly lax in some parts (but since I likely will
have advantages for my software because of this I will not complain ;-))
and on the other hand sometimes it looks like we are defining fort knox.

Perhaps we have different definitions for the trust and stability of one
package and
trust and stability of the repository in whole (breaking dependencies
etc...) but then lets clearify that. Breaking a package is not nice but not
really a drama for the community, breaking the repository or large parts of
it will be a far bigger problem.

(Perhaps we should also define what we expect the average user to be able
to handle. Since such assumptions have consequences, too).

Again: What is our vision for extras?

P.S.: Don't trust my version numbers! Trust my checkbox choice!

--
GruÃŸ...
Tim
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

henrik.hedberg at innologies

Nov 3, 2009, 11:36 AM

Post #34 of 54 (1956 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

Tim Teulings wrote:

>> Except how do you try to prevent abuse (whether intentional or
>> accidental)? At least with the version number you've got some safety
>> check (although it is in no way comprehensive). It also requires more
>> changes at more levels (I bet), so harder to implement.
>
> I think it is time to decide (again?) if we trust developers in their
> atempt to get their software/package into extras or not.

Currently, we trust ten random testers rather than one well-known
developer.

Why could not we trust well-known developers who have track record
of producing high-enough quality software? They may have their own
methods for testing, like couple of active and skilful dedicated testers
for the application domain. I see that more trustful than those random
testers who vote subjectively based on their opinions of an user interface.

We could have a group of accredited developers who have access to
the Extras. They are committed to release only validated and verified
packages. When a new developer wants to upload a package of his own, an
accredited developer could sponsor him, i.e. act like a gatekeeper.

Sounds familiar? See Debian New Maintainers Process [1].

Another possibility is to have the team of accredited testers, who
can make the final decision of their own. To become an accredited tester
required commiting to write sane bug reports and to make decisions on
generally agreed checks, among others. Then there would not be a need
for ten random strangers and ten day delays.

BR,

Henrik

[1] http://www.debian.org/devel/join/newmaint.en.html

--
Henrik Hedberg - http://www.henrikhedberg.net/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 1:40 PM

Post #35 of 54 (1946 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 3, 2009, at 19:25, Tim Teulings wrote:

> P.S.: Don't trust my version numbers! Trust my checkbox choice!

That is totally fine with me. I thought a version number was less
intrusive, developers didn't have to do anything, just remember which
part of their version to change. But as version numbers are so
invariant an excellent argument against this has been presented.

Shall we put a checkbox by the package promotion page, or somewhere
where we remember, which keeps all accrued karma?

Jeremiah

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 3, 2009, 1:52 PM

Post #36 of 54 (1945 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 3, 2009, at 20:36, Henrik Hedberg wrote:

> Tim Teulings wrote:
>
>>> Except how do you try to prevent abuse (whether intentional or
>>> accidental)? At least with the version number you've got some safety
>>> check (although it is in no way comprehensive). It also requires
>>> more
>>> changes at more levels (I bet), so harder to implement.
>>
>> I think it is time to decide (again?) if we trust developers in their
>> atempt to get their software/package into extras or not.
>
> Currently, we trust ten random testers rather than one well-known
> developer.

Please note that many of the people who have tested your app are
developers, maemo staff, or longtime members of the community. They
have a stake in your success since they use your application. No one
needs to be rude or harsh, but the criticism they pass along is only
meant to make Mauku better, it is not directed at you personally.

> Why could not we trust well-known developers who have track record
> of producing high-enough quality software?

I don't think anyone does not want to trust the developers. I most
adamantly have stated that we should, in fact we must, trust devs
implicitly.

> They may have their own
> methods for testing, like couple of active and skilful dedicated
> testers
> for the application domain. I see that more trustful than those random
> testers who vote subjectively based on their opinions of an user
> interface.

One of the lessons of any QA process is that testing and development
should be separate. This leads to greater ability in finding bugs and
often shortens the development cycle because developers are developing
based on user requests and doing more Test Driven Design.

> We could have a group of accredited developers who have access to
> the Extras. They are committed to release only validated and verified
> packages. When a new developer wants to upload a package of his own,
> an
> accredited developer could sponsor him, i.e. act like a gatekeeper.

But now it is you who doesn't trust people.

> Sounds familiar? See Debian New Maintainers Process [1].

This is unfortunately not an example I would like to follow. Becoming
a debian developer takes at least a year, and many developers go MIA.
Debian's founder Ian Murdoch calls debian 'process run amok'. I don't
think we need to replicate that here. Furthermore, I think Nokia would
loathe to participate in a program that excludes anyone who paid good
money to buy a device. I see no reason to do so and I think any
barrier to entry for testers, power users, users and devs is a non-
starter.
>
> Another possibility is to have the team of accredited testers, who
> can make the final decision of their own. To become an accredited
> tester
> required commiting to write sane bug reports and to make decisions on
> generally agreed checks, among others. Then there would not be a need
> for ten random strangers and ten day delays.

Again - no accreditation, no background checks, no process. Rather, we
adjust the process so that trust is granted to everyone and we put up
with occasional misuse. I think this is an excellent community and I
trust the developers here to do the right thing.

Let's not forget this is designed to be a QA process - to make
software better. That is the goal. If we are not reaching that goal,
we have to change, I think everyone is aware from the maemo.org side
that if this isn't working, it will get fixed to the satisfaction of
all.

And despite various complaints, many are saying that the process will
in fact produce better software. So we are in the right area anyway.

Jeremiah

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

liquid at gmail

Nov 3, 2009, 2:09 PM

Post #37 of 54 (1948 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Tue, Nov 3, 2009 at 9:52 PM, Jeremiah Foster <jeremiah [at] jeremiahfoster
> wrote:

>
> <snip>
>

> And despite various complaints, many are saying that the process will
> in fact produce better software. So we are in the right area anyway.
>

here here.
teething troubles and getting used to a different stance.

> Jeremiah
>
>
>
> _______________________________________________
> maemo-developers mailing list
> maemo-developers [at] maemo
> https://lists.maemo.org/mailman/listinfo/maemo-developers
>

andrew at bleb

Nov 3, 2009, 11:29 PM

Post #38 of 54 (1920 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

Jeremiah wrote:
>
> On Nov 3, 2009, at 19:25, Tim Teulings wrote:
>
> > P.S.: Don't trust my version numbers! Trust my checkbox choice!
>
> That is totally fine with me. I thought a version number was lessÂ
> intrusive, developers didn't have to do anything, just remember whichÂ
> part of their version to change. But as version numbers are soÂ
> invariant an excellent argument against this has been presented.

Agreed. It seemed like a nice hack, but isn't going to work in practice.

> Shall we put a checkbox by the package promotion page, or somewhereÂ
> where we remember, which keeps all accrued karma?

That's probably a good first step, however I wonder if long term something like Marius suggested might be better: remaining karma for an app is...

* proportional to current app karma
* proportional to developer's karma
* proportional to testers' karma
* inversely proportional to the time between
last build and this build.

This'd mean that if I released an app and had it voted up by Ryan, Tim, Daniel, Quim and a few others on the first karma page; and I released a new version the next day (short time => probable bug fix); my app might only lose one or two points.

Maybe this works for time too (or vote by high roller reduces time?) Or maybe it's just too complicated?

Thoughts welcome.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] blebÂ |Â http://www.bleb.org/

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

tero.kojo at nokia

Nov 4, 2009, 1:03 AM

Post #39 of 54 (1913 views)
Permalink

RE: QA process = bug fixing disincentive? [In reply to]

Andrew Flegg wrote.
>
> Jeremiah wrote:
> >
> > Shall we put a checkbox by the package promotion page, or somewhere
> > where we remember, which keeps all accrued karma?
>
> That's probably a good first step, however I wonder if long
> term something like Marius suggested might be better:
> remaining karma for an app is...
>
> * proportional to current app karma
> * proportional to developer's karma
> * proportional to testers' karma
> * inversely proportional to the time between
> last build and this build.
>
> This'd mean that if I released an app and had it voted up by
> Ryan, Tim, Daniel, Quim and a few others on the first karma
> page; and I released a new version the next day (short time
> => probable bug fix); my app might only lose one or two points.
>
> Maybe this works for time too (or vote by high roller reduces
> time?) Or maybe it's just too complicated?

Too complicated.

I start testing an application that I pick from extras-testing (a repository that is enabled on my device). I do my tests, am happy and go press a thumbs up.

Two days later I notice a blinking orange light in my status bar. I see a new version of the application. I install, I check what has changed (minor or major?), I run my tests and thumb it up again.

I didn't loose too many minutes or hours of my life, no one got killed or injured, and the punitive damages to all parties appear to be zero. Life's good and I get a cup of coffee.

In other words, I test applications that I use or leave on device. It's not something I do randomly. I believe that the author of the app has committed to the app, so in return I commit to it as well. The rules and process are a good thing, but in the end it is people who do the things. No matter if it is the developer, the tester or the user, it's about the people.

Tero

> Thoughts welcome.
>
> Cheers,
>
> Andrew
>
> --
> Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
>
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

andrew at bleb

Nov 4, 2009, 1:28 AM

Post #40 of 54 (1917 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Wed, Nov 4, 2009 at 09:03, <tero.kojo [at] nokia> wrote:
>
> Two days later I notice a blinking orange light in my status bar. I see
> a new version of the application. I install, I check what has changed
> (minor or major?), I run my tests and thumb it up again.

Aside: how do you check what has changed?

> I didn't loose too many minutes or hours of my life, no one got killed
> or injured, and the punitive damages to all parties appear to be zero.
> Life's good and I get a cup of coffee.

Trouble is, developers (myself and others) are saying the current
process of resetting to zero on every single promotion is incompatible
with "release early and release often". The theory about resetting to
zero is that testers will do the full QA process again (check it still
uses /opt, check a bug hasn't been introduced which introduces battery
management issues, check that it both installs, uninstalls and
upgrades cleanly).

> In other words, I test applications that I use or leave on device.
> It's not something I do randomly. I believe that the author of the app
> has committed to the app, so in return I commit to it as well. The
> rules and process are a good thing, but in the end it is people who
> do the things. No matter if it is the developer, the tester or the
> user, it's about the people.

Yep, however I now disagree with the logical conclusion of your
statement (which has been expressed elsewhere) which is "when we have
lots of users it'll all work itself out".

[and your original statement]
> Too complicated.

We seem(ed ;-)) to have a consensus between this thread, and the QA
marathon feedback, between both developers and testers that resetting
to zero was causing more problems than it solved. We've now moved on
to ways of maintaining some app karma from one release to another, but
it sounds like you disagree with that consensus? It may be that there
are two kind of testers:

* Dedicated QA browsers. Will install any app from the QA queue
to evaluate and score it. Will be done whenever they have time.

* Dedicated application users. Will install upgrades of apps they've
installed and evaluate/score them as pushed to.

Cheers,

Andrew

--
Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

tero.kojo at nokia

Nov 4, 2009, 1:56 AM

Post #41 of 54 (1923 views)
Permalink

RE: QA process = bug fixing disincentive? [In reply to]

Andrew Flegg wrote:
> On Wed, Nov 4, 2009 at 09:03, <tero.kojo [at] nokia> wrote:
> >
> > Two days later I notice a blinking orange light in my status bar. I
> > see a new version of the application. I install, I check what has
> > changed (minor or major?), I run my tests and thumb it up again.
>
> Aside: how do you check what has changed?

For the stuff in garage it's simple; skim the source, and for others see what the package installed, and see if everything is still there.

> > I didn't loose too many minutes or hours of my life, no one
> got killed
> > or injured, and the punitive damages to all parties appear
> to be zero.
> > Life's good and I get a cup of coffee.
>
> Trouble is, developers (myself and others) are saying the
> current process of resetting to zero on every single
> promotion is incompatible with "release early and release
> often". The theory about resetting to zero is that testers
> will do the full QA process again (check it still uses /opt,
> check a bug hasn't been introduced which introduces battery
> management issues, check that it both installs, uninstalls
> and upgrades cleanly).

Yes, none of those are so hard that I wouldn't do it. (Ok, I sometimes skimp on the uninstall testing for apps that will never leave my device. I'm a bad person)

> > In other words, I test applications that I use or leave on device.
> > It's not something I do randomly. I believe that the author
> of the app
> > has committed to the app, so in return I commit to it as well. The
> > rules and process are a good thing, but in the end it is
> people who do
> > the things. No matter if it is the developer, the tester or
> the user,
> > it's about the people.
>
> Yep, however I now disagree with the logical conclusion of
> your statement (which has been expressed elsewhere) which is
> "when we have lots of users it'll all work itself out".

I still think that. 10 votes from an installed base of fifty, a few hundred or say ten thousand is a different thing.

> [and your original statement]
> > Too complicated.

Referring to the complex schemes of saving the karma points. Sounds too complicated to start counting things like time, rate times or things like that. It might be cool to implement, but does it make sense?

Just do something really simple like new version get's the old votes if the app is still in testing. The tester will see the new version and change her/his mind if necessary.

> We seem(ed ;-)) to have a consensus between this thread, and
> the QA marathon feedback, between both developers and testers
> that resetting to zero was causing more problems than it
> solved. We've now moved on to ways of maintaining some app
> karma from one release to another, but it sounds like you
> disagree with that consensus?

I see issues in saving the karma, but they can most likely be worked out. Either dump the votes or keep them. Just don't make it too complicated.

> It may be that there are two
> kind of testers:
>
> * Dedicated QA browsers. Will install any app from the QA queue
> to evaluate and score it. Will be done whenever they have time.
>
> * Dedicated application users. Will install upgrades of
> apps they've
> installed and evaluate/score them as pushed to.

This most likely is the case. I do admit to random testing once in a while, but it is more to see whether there is something I could start using occasionally.

Tero

> Cheers,
>
> Andrew
>
> --
> Andrew Flegg -- mailto:andrew [at] bleb | http://www.bleb.org/
>
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

maemo at csipa

Nov 4, 2009, 3:49 AM

Post #42 of 54 (1907 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Wednesday 04 November 2009 10:28:58 Andrew Flegg wrote:
> On Wed, Nov 4, 2009 at 09:03, <tero.kojo [at] nokia> wrote:
> > Two days later I notice a blinking orange light in my status bar. I see
> > a new version of the application. I install, I check what has changed
> > (minor or major?), I run my tests and thumb it up again.
>
> Aside: how do you check what has changed?

This is one of the things I miss sorely from the Application Manager. Sure,
you can see the description and version, but not things like free/non-free,
the repo and last, but not least, the changelog (maybe based on the
debian/changelog most packages have, anyway).

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

riku.voipio at nokia

Nov 4, 2009, 4:46 AM

Post #43 of 54 (1907 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

ext Henrik Hedberg wrote:
> Tim Teulings wrote:
>
>
>>> Except how do you try to prevent abuse (whether intentional or
>>> accidental)? At least with the version number you've got some safety
>>> check (although it is in no way comprehensive). It also requires more
>>> changes at more levels (I bet), so harder to implement.
>>>
>> I think it is time to decide (again?) if we trust developers in their
>> atempt to get their software/package into extras or not.
>>
>
> Currently, we trust ten random testers rather than one well-known
> developer.
>
> Why could not we trust well-known developers who have track record
> of producing high-enough quality software? They may have their own
> methods for testing, like couple of active and skilful dedicated testers
> for the application domain. I see that more trustful than those random
> testers who vote subjectively based on their opinions of an user interface.

Every company has software testers, yet it doesn't mean they dont trust
their developers :) But even highly skilled, well-known developers are
just humans and make mistakes every now and then. Therefor, testing and
QA process is needed.

The problem with "just trust the developer" and "skillful dedicated
testers" is exactly that they become rapidly used to their own UI
mistakes and stop caring about them. 10 random users is a bit extreme,
but in principle it is a very good idea. If random extras-testing users
don't like the Use Experience, chances are very high that actual
"extras" endusers don't like it either! Certainly there are cases where
testers will thumb down unfairly, but if we have enough testers it
doesn't matter.

QA process is always going to somewhat painful. Still, the pain needs to
minimized and the QA process must be presented in a way that developers
can see the advantages of it rather than consider it as arbitrary
hurdle. Currently it appears the package promotion website is simply too
crude for both developers and testers. At the minimum we need a better
communication channel between developers and testers so that any
misunderstandings can be cleared directly without resorting to ranting
here :)

> Sounds familiar? See Debian New Maintainers Process [1].

Being a longterm Debian Developer, and a for a while a Application
Manager for the New Maintainer process, I think applying such model for
maemo extras would be a very bad idea.

1. It is a massive hurdle of contribution, discouraging people from
joining (I don't see you in debian or NM although you create Debian
packages;)
2. "Accredited" Debian Developers still upload crap all the time
3. It leads to complex workarounds such as sponsoring uploads for others

(In fact I hope we can some day dismantle NM it in Debian as well, and
move to a model where more people can upload directly, but _all_ uploads
are reviewed before accepting them in.)

ps. I also agree that guessing from version number is bad.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Nov 4, 2009, 5:05 AM

Post #44 of 54 (1899 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

"Voipio Riku (Nokia-D/Helsinki)" <riku.voipio [at] nokia> writes:

> Every company has software testers, yet it doesn't mean they dont trust
> their developers :)

I think there are two kinds of trust on the table here: trust in
developers not to make mistakes, and trust in developers not to abuse
the process malevolently.

We don't need to trust developers not to make mistakes. A developer
doesn't even trust himself/herself not to make mistakes, of course.

But I think wa want to trust developers not to be malevolent. There
will be exceptions, they will be found out and punished, the malecreants
will create another account, and life goes on.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 4, 2009, 5:47 AM

Post #45 of 54 (1903 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 4, 2009, at 12:49, Attila Csipa wrote:

> On Wednesday 04 November 2009 10:28:58 Andrew Flegg wrote:
>> On Wed, Nov 4, 2009 at 09:03, <tero.kojo [at] nokia> wrote:
>>> Two days later I notice a blinking orange light in my status bar.
>>> I see
>>> a new version of the application. I install, I check what has
>>> changed
>>> (minor or major?), I run my tests and thumb it up again.
>>
>> Aside: how do you check what has changed?

Technically there could be a way to use debdiff as well as parse the
changelog. I wouldn't want to do to much more that say;

> 10 Lines of code changed
> Changelog says: Added bugfix for frobulation

Jeremiah

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

g+770 at cobb

Nov 4, 2009, 6:21 AM

Post #46 of 54 (1902 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

Attila said...
> On Wednesday 04 November 2009 10:28:58 Andrew Flegg wrote:
> > On Wed, Nov 4, 2009 at 09:03,Â <tero.kojo [at] nokia> wrote:
> > > Two days later I notice a blinking orange light in my status bar. I see
> > > a new version of the application. I install, I check what has changed
> > > (minor or major?), I run my tests and thumb it up again.
> >
> > Aside: how do you check what has changed?
>
> This is one of the things I miss sorely from the Application Manager. Sure,
> you can see the description and version, but not things like free/non-free,
> the repo and last, but not least, the changelog (maybe based on the
> debian/changelog most packages have, anyway).

For users AppMgr has a user-friendly alternative: you specify an update description in the control file. Now that I think of it this needs to go into the extras testing rules: any package which is already in Extras must specify an update description to give people information on why they should upgrade.

But the update description does not help with testing: (a) it is user friendly text, not a developer changelog and (b) the description is vs. the version already in Extras not vs. the last extras-testing version.

It sounds like it would be useful to start including changelog in Maemo packages and make it visible from the package page.

Graham

_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

rabelg5 at gmail

Nov 4, 2009, 10:05 AM

Post #47 of 54 (1893 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 4, 2009, at 9:21 AM, Graham Cobb wrote:

> But the update description does not help with testing: (a) it is
> user friendly text, not a developer changelog and (b) the
> description is vs. the version already in Extras not vs. the last
> extras-testing version.

I'd still love to see a Testing Mode in h-a-m. Show Debian changelogs,
show repository origin, give you an interface to packages, etc.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

jeremiah at jeremiahfoster

Nov 10, 2009, 4:31 AM

Post #48 of 54 (1695 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

On Nov 4, 2009, at 8:29, Andrew Flegg wrote:

> Jeremiah wrote:
>>
>> On Nov 3, 2009, at 19:25, Tim Teulings wrote:
>>
>>> P.S.: Don't trust my version numbers! Trust my checkbox choice!
>>
>> That is totally fine with me. I thought a version number was less
>> intrusive, developers didn't have to do anything, just remember which
>> part of their version to change. But as version numbers are so
>> invariant an excellent argument against this has been presented.
>
> Agreed. It seemed like a nice hack, but isn't going to work in
> practice.
>
>> Shall we put a checkbox by the package promotion page, or somewhere
>> where we remember, which keeps all accrued karma?
>
> That's probably a good first step, however I wonder if long term
> something like Marius suggested might be better: remaining karma for
> an app is...
>
> * proportional to current app karma
> * proportional to developer's karma
> * proportional to testers' karma
> * inversely proportional to the time between
> last build and this build.
>
> This'd mean that if I released an app and had it voted up by Ryan,
> Tim, Daniel, Quim and a few others on the first karma page; and I
> released a new version the next day (short time => probable bug
> fix); my app might only lose one or two points.
>
> Maybe this works for time too (or vote by high roller reduces time?)
> Or maybe it's just too complicated?

This is cool, and I think it could be done, but the karma calculation
algos are starting to be a little complicated.

Jeremiah
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

th.perl at gmail

Nov 11, 2009, 6:19 AM

Post #49 of 54 (1657 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

2009/11/4 Graham Cobb <g+770 [at] cobb>:
> Attila said...
>> On Wednesday 04 November 2009 10:28:58 Andrew Flegg wrote:
>> > On Wed, Nov 4, 2009 at 09:03, <tero.kojo [at] nokia> wrote:
>> > > Two days later I notice a blinking orange light in my status bar. I see
>> > > a new version of the application. I install, I check what has changed
>> > > (minor or major?), I run my tests and thumb it up again.
>> >
>> > Aside: how do you check what has changed?
>>
>> This is one of the things I miss sorely from the Application Manager. Sure,
>> you can see the description and version, but not things like free/non-free,
>> the repo and last, but not least, the changelog (maybe based on the
>> debian/changelog most packages have, anyway).
>
> For users AppMgr has a user-friendly alternative: you specify an update description in the control file. Now that I think of it this needs to go into the extras testing rules: any package which is already in Extras must specify an update description to give people information on why they should upgrade.
>
> But the update description does not help with testing: (a) it is user friendly text, not a developer changelog and (b) the description is vs. the version already in Extras not vs. the last extras-testing version.

The following is a rant about XB-Maemo-Upgrade-Description
with some suggestions for improvement...

I personally used this field in my packages, but the problem is that
the developer might forget to update the description, as normal
changes are logged in debian/changelog.

Also, imagine this row of releases (with the update text in brackets):

1.0-0 ("Initial release")
1.0-1 ("Fixed bug 1234")
1.0-2 ("Added a new icon")
2.0-0 ("Complete rewrite, everything changed")
2.0-1 ("Increased the contrast of the application icon")

Now, when a user has e.g. version "1.0-1" installed, and the
repository already contains version "2.0-1", the user is offered
to upgrade it with the description "Increased the contrast of the
application icon", when in reality, the upgrade from 1.0-1 to 2.0-1
does a lot more in reality (after all, 2.0 was a complete rewrite).

Not every user upgrades from every point release to another,
especially since the new Extras-Testing process. This field
("XB-Maemo-Upgrade-Description") is bad for these reasons,
as the upgrade description really depends on which version you
upgrade from.

My suggestion is to either use the Debian changelog, or if this
sounds too "technical" for the end user, agree on some way to
mark "user-relevant" changes in the Debian changelog (by using
"USER:" as a prefix for a one-line summary or by having a
convention of having the first entry in the Debian changelog be
a user-friendly summary of all changes) and then parse the
changelog and display all user-relevant changes in the AppMgr.

The one-line summary in the AppManager UI (that is now taken
from XB-Maemo-Upgrade-Description) can then be taken from
these user-friendly changelog entries, and a upgrade that jumps
multiple versions can then display like this:

"Increased the contrast of the application icon (+ 2 more changes)"

Clicking on the details should then reveal all changes from the
installed version to the version that the package is being upgraded
to.

Thomas
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

marius.vollmer at nokia

Nov 11, 2009, 6:29 AM

Post #50 of 54 (1645 views)
Permalink

Re: QA process = bug fixing disincentive? [In reply to]

ext Thomas Perl <th.perl [at] gmail> writes:

> The following is a rant about XB-Maemo-Upgrade-Description
> with some suggestions for improvement...

Yeah, as soon as I 'invented' it, I could see how it is not going to
work very well. I actually think it is best to ignore this field.

> My suggestion is to either use the Debian changelog, or if this sounds
> too "technical" for the end user, agree on some way to mark
> "user-relevant" changes in the Debian changelog (by using "USER:" as a
> prefix for a one-line summary or by having a convention of having the
> first entry in the Debian changelog be a user-friendly summary of all
> changes) and then parse the changelog and display all user-relevant
> changes in the AppMgr.

Yes, we pretty much have to have a full list of changes and the
Application manager then can display the relevant ones. The
apt-listchanges program does this for Debian.
_______________________________________________
maemo-developers mailing list
maemo-developers [at] maemo
https://lists.maemo.org/mailman/listinfo/maemo-developers

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads