glyph at divmod
Jun 26, 2008, 12:32 PM
Post #4 of 15
(980 views)
Permalink
|
|
Re: Community buildbots and Python release quality metrics
[In reply to]
|
|
I do tend to ramble on, so here's an executive summary of my response:
I want python developers to pay attention to the community buildbots and
to treat breakages of existing projects as a serious issue. However, I
don't think that maintaining those projects is the core team's job, so
all I'm asking for is for core developers to:
* treat breakages of 3rd party packages as a potentially serious issue,
* if possible (i.e. if they find out about the breakage soon enough,
which should be the case in any pybots failure) revert the change that
caused the problem until the problem can be fixed, and
* notify 3rd party maintainers when it's decided that the breakage will
not be fixed.
This only applies to breakages that the core developers find out about,
which for all practical purposes means the ones on the community
builders page.
Those of you looking for point-by-point responses and some repetition of
the above points, enjoy :).
On 05:03 pm, guido [at] python wrote:
>On Thu, Jun 26, 2008 at 9:21 AM, <glyph [at] divmod> wrote:
>>On 03:33 pm, guido [at] python wrote:
>>>It needs to be decided case-by-case.
>>(misunderstanding)
>No, I just meant that we need to figure out for each 3rd party test
>that fails whether the failure is our fault (too incompatibile) or
>theirs (relied on undefined behavior) and what the best fix is (change
>our code or theirs -- note that even if it's there fault there are
>cases where the best fix is to change our code.
This is basically fine, as far as I'm concerned.
I would like to suggest, however, that these issues be dealt with as
soon as possible, rather than waiting for the release process to begin.
A lot of decisions are made on this mailing list about the supposed
properties of "average" python code, without any actual survey of said
code. Sometimes the results of that survey can be really surprising.
The end goal of any particular compatibility policy, of a distinction
between "public" and "private" APIs, and so on, is to keep code working.
>I'm sorry if your interpretation of the terminology is different, but
>this is mine and this is what we've always used, and it's not likely
>to change. (At least not for the 2.6/3.0 release.)
I have no problem with your definitions of these terms. I think that
they should probably be in PEP 101 though. Would you accept a patch
that added an edited / expanded version of this paragraph?
>>Still, I'm bringing this up now because it _is_ a beta,
>Absolutely correct. The RCs are hoped to be as good as the final
>release. *Now* is the time to bring up issue.
Well, that's good, at least :)
>But please bring up specific issues -- I don't want to have an
>extended discussion about process or quality or expectations. I just
>want the code to be fixed.
Well, one specific issue has been bumped in priority as a result of this
thread, and others are under discussion. The code is getting fixed.
>>(I just care that I stop having problems with incompatibility.)
>
>And here we seem to be parting our ways. We have a large amount of
>process already. I don't want more.
Looking at it from my perspective, I'm proposing a reduction in process.
Under the current process, if a buildbot goes red, the developer makes a
judgment call, the release manager makes a judgment call, there's
discussion on a ticket, a ticket gets filed, it gets promoted, it gets
demoted, the RM forgets to re-promote it...
My suggestion is that the process be, simply: if a buildbot (community
or otherwise) goes red, the change that broke it gets reverted. No
questions asked! It's still there in the revision history, ready to be
re-applied once the issues get worked out. Discussion can then take
place and case-by-case judgments can be applied.
>If you're talking about community buildbots (which I presume are
>testing 3rd party packages against the core release) being red, that's
>out of scope for the core developers.
I don't necessarily think that keeping the community buildbots red is
the core developers' responsibility, but I don't think it should be
entirely out of scope, either. The python test suite is, frankly, poor
- and I hope I'm not surprising you by saying that. It's full of race
conditions, tends to fail intermittently, and is frequently ignored.
Not only that, but it is quite often changed, so tests for issues that
affect real code are quite often removed. So, the community buildbots
are not just making sure that 3rd-party code still works, they are an
expanded, independently developed test suite to make sure that *python
itself* still works. Sometimes they will not fill that role
particularly well, but they are worth paying attention to.
If python had a good, exhaustive regression test suite that was
immutable between major versions, I'd probably feel differently. But
that's not the world we live in.
Right now, apparently, the *default* policy is that if the community
buildbots go red, later, before a release, someone will maybe take a
look at it. I'd suggest that the *default* policy ought to be that if a
particular changeset breaks a community buildbot, it needs further
examination before being merged to trunk.
However, this is just the way I prefer to do development; if you think
that would slow things down too much, the only thing I'm _really_ asking
for is a clear statement that says "there should be no test failures on
community buildbots that have not been explicitly accepted before a
final release". I'm not even sure what "explicitly accepted" means -
you have to sign off? the release manager, maybe? A discussion on this
list? I don't really care, as long as _somebody_ does.
Right now, my impression of the process is this:
* The community buildbot goes red; no core developer looks at it.
* If the project is Twisted, JP fixes the bug on Twisted's end.
* If the project is Django, nobody notices.
* Months later, a beta goes out. A few people try it out and report
some bugs, but don't really understand the output. A good number go un-
triaged.
* A little while later, a final release comes out. Many projects are
broken as a result.
This is not a hypothetical concern. This is what happened with 2.5;
Twisted was broken for months, and Zope *to this day* does not support
Python 2.5. 2.6 looks like it's headed for the same trajectory. To be
clear: this is with all of Python's _own_ tests passing, so it is
specific to paying attention to community buildbots. (And the community
buildbots only build django and twisted right now. I'm not talking
about a massive pan-galactic survey of all possible python projects.
I'm only talking about those popular enough to make this select list.
Which should still be a slightly longer list, but I digress...)
>Some of the
>core buildbots are red because, well, frankly, they run on a cranky
>old version of an OS that few people care about.
On Twisted, we have a distinction between "supported" and "unsupported"
platforms, to provide the ability to run on platforms which aren't
really supported and don't really run the whole suite, but we are
nevertheless interested in. I don't believe the setup is too hard and
we'll definitely help out with that if you want to do it. (I believe
Thomas Herve volunteered to do this at PyCon...)
>I hope the community buildbots can be used the same way: a red bot
>means someone needs to look into an issue. The issue could be with the
>core or with the 3rd party package being tested. I don't think a
>policy like "no community buildbots should be red" makes any sense.
These bots have been red for months. The issues exist, but have not
been looked into. As a result, Barry made a specific commitment on a
ticket (i.e. "this should block beta 1") which was not met. I think
_something_ has to be changed to encourage people to do this more
immediately or more seriously.
>Whoever made what change? You can't seriously expect core developers
>investigating issues with 3rd party packages, no matter what the
>cause. The correct process is that someone who cares about the 3rd
>party package (could be an end user, a developer of that package, or a
>core developer who happens to care) looks into the issue enough to
>diagnose it, and then either proposes a fix or files a bug with the
>likely culprit, which could be the core or the 3rd party package. If
>nobody cares, well, that's open source too.
If the breakage is calculated and expected, and the benefits clearly
oughtweigh the costs... oh well, too bad for the 3rd party people. It
would be nice if the core developers would notify the third party if
they find out about it so that it can be verified that the change in
question wasn't obscuring some *other* problem, but from what I've seen,
the breakages that I have been concerned about have not been
intentional, calculated changes, but side-effects of other things.
I'm talking about the case where the breakage reveals either a bug in
Python, or an unintentional / side-effect change in behavior, which is
surprisingly frequent.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com
|
