Mailing List Archive: Python: Dev

Community buildbots and Python release quality metrics

1 2

View All

Index | Next | Previous | View Threaded

glyph at divmod

Jun 26, 2008, 8:18 AM

Post #1 of 32 (1333 views)
Permalink

Community buildbots and Python release quality metrics

Today on planetpython.org, Doug Hellman announced the June issue of
Python magazine. The cover story this month is about Pybots, "the
fantastic automation system that has been put in place to make sure new
releases of Python software are as robust and stable as possible".

Last week, there was a "beta" release of Python which, according to the
community buildbots, cannot run any existing python software. Normally
I'd be complaining here about Twisted, but in fact Twisted is doing
relatively well right now; only 80 failing tests. Django apparently
cannot even be imported.

The community buildbots have been in a broken state for months now[1].
I've been restraining myself from whinging about this, but now that it's
getting close to release, it's time to get these into shape, or it's
time to get rid of them.

There are a few separate issues here. One is, to what extent should
Python actually be compatible between releases? There's not much point
in saying "this release is compatible with python 2.5" if all python 2.5
programs need to be modified in order to run on python 2.6. Of course,
it's quite likely that there are some 3rd-party Python programs that
continue to work, but python.org provides links to 2 projects that don't
and none that do.

Another way to phrase this question is, "whose responsibility is it to
make Python 2.5 programs run on Python 2.6"? Or, "what happens when the
core team finds out that a change they have made has broken some python
software 'in the wild'"?

Here are a couple of ways to answer these questions:

1) It's the python core team's responsibility. There should be Python
buildbots for lots of different projects and they should all be green
before the code can be considered [one of: allowed in trunk, alpha,
beta, RC].
2) It's the python core team's responsibility as long as the major
projects are using public APIs. The code should not be considered
[alpha, beta, RC] if there are any known breakages in public APIs.
3) It's only the python core team's responsibility to notify projects
of incompatible changes of which they are aware, which may occur in
public or private APIs. Major projects with breakages will be given a
grace period before a [beta, RC, final] to release a forward-compatible
version.
4) The status quo: every new Python release can (and will, at least
speaking in terms of 2.6) break lots of popular software, with no
warning aside from the beta period. There are no guarantees.

For obvious reasons, I'd prefer #1, but any of the above is better than
#4. I don't think #4 is an intentional choice, either; I think it's the
result of there being no clear consequence for community buildbots
failing. Or, for that matter, of core buildbots failing. Investigating
the history is tricky (I see a bunch of emails from Barry here, but I'm
not sure which is the final state) but I believe the beta went out with
a bunch of buildbots offline or failing.

A related but independent issue is: should the community buildbots
continue to exist? An automated test is only as good as the
consequences of its failure. As it stands, the community buildbots seem
to be nothing but an embarrassment. I don't mean this to slam Grig, or
the Python team - I mean, literally, if there is no consequence of their
failure then the only use I can see for the data the buildbots currently
provide (i.e. months and months of failures) is to embarrass the Python
core developers. python.org's purpose should not be to provide negative
press for Python ;), so if this continues to be the case, they probably
shouldn't be linked. This isn't just an issue with changes to Python
breaking stuff; the pybots' configuration is apparently not well
maintained, since the buildbots which say "2.5" aren't passing either,
and that's not a problem with the code, it's just that the slaves are
actually running against trunk[at]HEAD.

Ultimately these tools only exist to ensure the quality of Python
releases. The really critical question here is, what does it mean to
have a Python release that is high-quality? There are some obvious
things: it shouldn't crash, it should conform to its own documentation.
Personally I think "it passes all of its own tests" and "it runs
existing Python code" are important metrics too, but the most important
thing I'm trying to get across here is that there should be a clear
understanding of which goals the release / QA / buildbot / test process
is trying to accomplish. For me, personally, it really needs to be
clear when I can say "you guys screwed up and broke stuff", and when I
just have to suck it up and deal with the consequences of a new version
of Python in Twisted.

It's definitely bad for all of us if the result is that new releases of
Python just break everything. Users don't care where the responsibility
lies, they just know that stuff doesn't work, so we should decide on a
process which allows Twisted (and other popular projects, like Django,
Plone, pytz, PyGTK, pylons, ...) to (mostly) run when new versions of
Python are released.

[1]: http://mail.python.org/pipermail/python-dev/2008-May/079212.html
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 8:33 AM

Post #2 of 32 (1295 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

Too verbose, Glyph. :-)

It needs to be decided case-by-case. The beta is called beta because,
well, it may break stuff and we may want to fix it. But there are also
likely many frameworks that use undefined behavior. I'm not
particularly impressed by statistics like "all tests are red" -- this
may all be caused by a single issue. For example, I'd much rather read
an explanation about *why* Django cannot even be imported than a
blanket complaint that this is a disgrace. So why is it?

--Guido

On Thu, Jun 26, 2008 at 8:18 AM, <glyph[at]divmod.com> wrote:
> Today on planetpython.org, Doug Hellman announced the June issue of Python
> magazine. The cover story this month is about Pybots, "the fantastic
> automation system that has been put in place to make sure new releases of
> Python software are as robust and stable as possible".
>
> Last week, there was a "beta" release of Python which, according to the
> community buildbots, cannot run any existing python software. Normally I'd
> be complaining here about Twisted, but in fact Twisted is doing relatively
> well right now; only 80 failing tests. Django apparently cannot even be
> imported.
>
> The community buildbots have been in a broken state for months now[1]. I've
> been restraining myself from whinging about this, but now that it's getting
> close to release, it's time to get these into shape, or it's time to get rid
> of them.
>
> There are a few separate issues here. One is, to what extent should Python
> actually be compatible between releases? There's not much point in saying
> "this release is compatible with python 2.5" if all python 2.5 programs need
> to be modified in order to run on python 2.6. Of course, it's quite likely
> that there are some 3rd-party Python programs that continue to work, but
> python.org provides links to 2 projects that don't and none that do.
>
> Another way to phrase this question is, "whose responsibility is it to make
> Python 2.5 programs run on Python 2.6"? Or, "what happens when the core
> team finds out that a change they have made has broken some python software
> 'in the wild'"?
>
> Here are a couple of ways to answer these questions:
>
> 1) It's the python core team's responsibility. There should be Python
> buildbots for lots of different projects and they should all be green before
> the code can be considered [one of: allowed in trunk, alpha, beta, RC].
> 2) It's the python core team's responsibility as long as the major projects
> are using public APIs. The code should not be considered [alpha, beta, RC]
> if there are any known breakages in public APIs.
> 3) It's only the python core team's responsibility to notify projects of
> incompatible changes of which they are aware, which may occur in public or
> private APIs. Major projects with breakages will be given a grace period
> before a [beta, RC, final] to release a forward-compatible version.
> 4) The status quo: every new Python release can (and will, at least
> speaking in terms of 2.6) break lots of popular software, with no warning
> aside from the beta period. There are no guarantees.
>
> For obvious reasons, I'd prefer #1, but any of the above is better than #4.
> I don't think #4 is an intentional choice, either; I think it's the result
> of there being no clear consequence for community buildbots failing. Or,
> for that matter, of core buildbots failing. Investigating the history is
> tricky (I see a bunch of emails from Barry here, but I'm not sure which is
> the final state) but I believe the beta went out with a bunch of buildbots
> offline or failing.
>
> A related but independent issue is: should the community buildbots continue
> to exist? An automated test is only as good as the consequences of its
> failure. As it stands, the community buildbots seem to be nothing but an
> embarrassment. I don't mean this to slam Grig, or the Python team - I mean,
> literally, if there is no consequence of their failure then the only use I
> can see for the data the buildbots currently provide (i.e. months and months
> of failures) is to embarrass the Python core developers. python.org's
> purpose should not be to provide negative press for Python ;), so if this
> continues to be the case, they probably shouldn't be linked. This isn't
> just an issue with changes to Python breaking stuff; the pybots'
> configuration is apparently not well maintained, since the buildbots which
> say "2.5" aren't passing either, and that's not a problem with the code,
> it's just that the slaves are actually running against trunk[at]HEAD.
>
> Ultimately these tools only exist to ensure the quality of Python releases.
> The really critical question here is, what does it mean to have a Python
> release that is high-quality? There are some obvious things: it shouldn't
> crash, it should conform to its own documentation. Personally I think "it
> passes all of its own tests" and "it runs existing Python code" are
> important metrics too, but the most important thing I'm trying to get across
> here is that there should be a clear understanding of which goals the
> release / QA / buildbot / test process is trying to accomplish. For me,
> personally, it really needs to be clear when I can say "you guys screwed up
> and broke stuff", and when I just have to suck it up and deal with the
> consequences of a new version of Python in Twisted.
>
> It's definitely bad for all of us if the result is that new releases of
> Python just break everything. Users don't care where the responsibility
> lies, they just know that stuff doesn't work, so we should decide on a
> process which allows Twisted (and other popular projects, like Django,
> Plone, pytz, PyGTK, pylons, ...) to (mostly) run when new versions of Python
> are released.
>
> [1]: http://mail.python.org/pipermail/python-dev/2008-May/079212.html
> _______________________________________________
> Python-Dev mailing list
> Python-Dev[at]python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jun 26, 2008, 8:42 AM

Post #3 of 32 (1296 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

glyph[at]divmod.com wrote:
> Today on planetpython.org, Doug Hellman announced the June issue of
> Python magazine. The cover story this month is about Pybots, "the
> fantastic automation system that has been put in place to make sure new
> releases of Python software are as robust and stable as possible".
>
> Last week, there was a "beta" release of Python which, according to the
> community buildbots, cannot run any existing python software. Normally
> I'd be complaining here about Twisted, but in fact Twisted is doing
> relatively well right now; only 80 failing tests. Django apparently
> cannot even be imported.

beta 1 has some trouble running *our* test suite - I'd be fairly
surprised if the community buildbots were in significantly better shape.

> The community buildbots have been in a broken state for months now[1].

Continuously running community buildbots on the maintenance trees makes
sense, since those trees should always be in a releasable state. For the
trunk, they're really only interesting when the Python core buildbots
are reporting all green, but some of the community buildbots are
reporting red.

One of the problems is what the term "beta" means to different groups -
for us, this first beta was really about saying "zero new features from
here on, focus on making what we have now work properly". The relatively
late landing of a couple of major PEPs (371, 3108) also didn't do any
favours for trunk stability.

If the community buildbots aren't largely green by the time beta 2 comes
out, that's when I'll agree we have a problem - they should definitely
be green by the time first release candidate comes out.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan[at]gmail.com | Brisbane, Australia
---------------------------------------------------------------
http://www.boredomandlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

glyph at divmod

Jun 26, 2008, 9:33 AM

Post #4 of 32 (1292 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On 03:42 pm, ncoghlan[at]gmail.com wrote:
>glyph[at]divmod.com wrote:

>beta 1 has some trouble running *our* test suite - I'd be fairly
>surprised if the community buildbots were in significantly better
>shape.

That's another problem, yes :)
>>The community buildbots have been in a broken state for months now[1].

>If the community buildbots aren't largely green by the time beta 2
>comes out, that's when I'll agree we have a problem - they should
>definitely be green by the time first release candidate comes out.

I have mixed feelings about this: on the one hand, if this were a matter
of public record, I would have known that this was too early to start
complaining, and we could have saved everyone a lot of time reading
these messages. That would be good; I would prefer to just let everyone
work without bothering about process.

On the other hand, it's much easier to deal with test failures as they
arise than in one giant chunk of work at the end of the development
process. I spoke to a few core developers at PyCon who thought that the
buildbots should always be green and any change that broke them should
be reverted. They may remain nameless unless they wish to raise their
hands :-) but that's definitely how I feel about it.
>Continuously running community buildbots on the maintenance trees makes
>sense, since those trees should always be in a releasable state. For
>the trunk, they're really only interesting when the Python core
>buildbots are reporting all green, but some of the community buildbots
>are reporting red.

(... which should be all the time ...)
>One of the problems is what the term "beta" means to different groups -
>for us, this first beta was really about saying "zero new features from
>here on, focus on making what we have now work properly". The
>relatively late landing of a couple of major PEPs (371, 3108) also
>didn't do any favours for trunk stability.

A big part of why I wrote this message is that I wanted a clear
understanding of the relationship between the definition of "alpha",
"beta" and "RC" and the state of various buildbots. If that
relationship exists already, just linking to it from
http://python.org/download/releases/2.6/ would be good. By the way,
that page still says these are "alpha" releases.
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

schmir at gmail

Jun 26, 2008, 9:42 AM

Post #5 of 32 (1295 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <guido[at]python.org> wrote:

> an explanation about *why* Django cannot even be imported than a
> blanket complaint that this is a disgrace. So why is it?
>

File "/home/ralf/pediapress/appserver/django/db/models/options.py",
line 198, in _many_to_many
self._fill_m2m_cache()
File "/home/ralf/pediapress/appserver/django/db/models/options.py",
line 221, in _fill_m2m_cache
cache[field] = None
File "/home/ralf/pediapress/appserver/django/utils/datastructures.py",
line 75, in __setitem__
super(SortedDict, self).__setitem__(key, value)
TypeError: unhashable type: 'ManyToManyField'

same for sqlalchemy:
egg/sqlalchemy/schema.py", line 113, in __call__
return type.__call__(self, name, metadata, *args, **kwargs)
File "/home/ralf/py26/lib/python2.6/site-packages/SQLAlchemy-0.5.0beta1-py2.6.egg/sqlalchemy/schema.py",
line 207, in __init__
self.primary_key = PrimaryKeyConstraint()
File "/home/ralf/py26/lib/python2.6/site-packages/SQLAlchemy-0.5.0beta1-py2.6.egg/sqlalchemy/schema.py",
line 294, in _set_primary_key
self.constraints.add(pk)
TypeError: unhashable type: 'PrimaryKeyConstraint'

and already discussed:
http://mail.python.org/pipermail/python-dev/2008-April/078421.html

- Ralf
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 10:03 AM

Post #6 of 32 (1287 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 9:21 AM, <glyph[at]divmod.com> wrote:
> On 03:33 pm, guido[at]python.org wrote:
>> It needs to be decided case-by-case.
>
> If certain tests are to be ignored on a case-by-case basis, why not record
> that decision by disabling the test in the code? Otherwise, the decision
> inevitably gets boiled down to "it's okay to release the code with a bunch
> of tests failing, but I don't know which ones". As it was on Twisted when
> we used to make case-by-case decisions about failures, and as it is here
> now.

No, I just meant that we need to figure out for each 3rd party test
that fails whether the failure is our fault (too incompatibile) or
theirs (relied on undefined behavior) and what the best fix is (change
our code or theirs -- note that even if it's there fault there are
cases where the best fix is to change our code.

>> The beta is called beta because, well, it may break stuff and we may want
>> to fix it.
>
> That's also why the alpha is called an alpha. My informal understanding is
> that a beta should have no (or at least very few) known issues, with a
> medium level of confidence that no further ones will be found. An RC would
> have absolutely no known issues with a fairly high level of confidence that
> no further ones will be found. This would include the community buildbots
> basically working for the most part; I would not be surprised if there were
> a few tests that still had issues.

I guess we differ in interpretation. I see alphas as early outreach to
the community. I see betas as a line in the sand where we stop
adding/changing functionality -- but certainly not the moment where we
expect everything to be perfect. During the beta period we fix bugs
only -- those could be problems with new features, but those could
also be problems where we accidentally introduced a backwards
incompatibility.

I'm sorry if your interpretation of the terminology is different, but
this is mine and this is what we've always used, and it's not likely
to change. (At least not for the 2.6/3.0 release.)

> But clearly the reality does not meet my informal expectations, so it would
> be nice to have something written down to check against. Still, I'm
> bringing this up now because it _is_ a beta, and I think it will be too late
> to talk about dealing with persistent test failures after the RCs start
> coming out.

Absolutely correct. The RCs are hoped to be as good as the final
release. *Now* is the time to bring up issue.

But please bring up specific issues -- I don't want to have an
extended discussion about process or quality or expectations. I just
want the code to be fixed.

> (Of course, I'm just being sneaky. I don't actually care if it's clearly
> documented, I just care that I stop having problems with incompatibility.
> But I believe having it clearly spelled out would actually prevent a lot of
> the problems that I've been having, since I don't think anyone would
> *intentionally* select a policy where every release breaks at least one
> major dependent project.)

And here we seem to be parting our ways. We have a large amount of
process already. I don't want more.

>> I'm not particularly impressed by statistics like "all tests are red" --
>> this
>> may all be caused by a single issue.
>
> The issue, for me, is not specifically that tests are red. It's that
> there's no clear policy about what to do about that. Can a release go out
> with some of the tests being red? If so, what are the extenuating
> circumstances? Does this have to be fixed? If not, why not? Why are we
> talking about this now? Shouldn't the change which caused Django to become
> unimportable have been examined at the time, rather than months later? (In
> other words, if it *is* a single issue, great, it's easy to fix: revert that
> single issue.) If not, then shouldn't someone in Django-land have been
> notified so they could cope with the change?
>
> Sorry that there are so many questions here; if I had fewer, I'd use fewer
> words to ask them.

If you're talking about community buildbots (which I presume are
testing 3rd party packages against the core release) being red, that's
out of scope for the core developers. We have enough trouble keeping
our own (core) buildbots green, as you may have noticed. Some of the
core buildbots are red because, well, frankly, they run on a cranky
old version of an OS that few people care about. And then sometimes we
leave them red, or turn them off.

I see the core buildbots as a tool for developers to have their
changes tested on a wide variety of platforms. They are an internal
tool that gives us guidance about whether we're ready.

I hope the community buildbots can be used the same way: a red bot
means someone needs to look into an issue. The issue could be with the
core or with the 3rd party package being tested. I don't think a
policy like "no community buildbots should be red" makes any sense.

>> For example, I'd much rather read an explanation about *why* Django cannot
>> even be imported than a blanket complaint that this is a disgrace. So why is
>> it?
>
> I don't know. JP is already addressing the issues affecting Twisted in
> another thread (incompatible changes in the private-but-necessary-to-
> get-any-testing-done API of the warnings module). But I really think that
> whoever made the change which broke it should be the one investigating it,
> not me.

Whoever made what change? You can't seriously expect core developers
investigating issues with 3rd party packages, no matter what the
cause. The correct process is that someone who cares about the 3rd
party package (could be an end user, a developer of that package, or a
core developer who happens to care) looks into the issue enough to
diagnose it, and then either proposes a fix or files a bug with the
likely culprit, which could be the core or the 3rd party package. If
nobody cares, well, that's open source too.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 10:07 AM

Post #7 of 32 (1294 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

And just to make my position perfectly clear, I've unassigned it,
since I don't foresee to be able to give this issue the quality time
it clearly needs. Mind you, I agree it's a release blocker. But I
don't have time to personally investigate it. Sorry.

On Thu, Jun 26, 2008 at 9:54 AM, <glyph[at]divmod.com> wrote:
> On 04:42 pm, schmir[at]gmail.com wrote:
>>
>> On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <guido[at]python.org>
>> wrote:
>>>
>>> an explanation about *why* Django cannot even be imported than a
>>> blanket complaint that this is a disgrace. So why is it?
>
>> and already discussed:
>> http://mail.python.org/pipermail/python-dev/2008-April/078421.html
>
> Following that trail of breadcrumbs, I ended up here:
>
> http://bugs.python.org/issue2235
>
> with a message from some guy named "Barry Warsaw" (anyone know him?) that
> says:
>
> """
> Guido, can you comment on Amaury's latest patch? I'm going to bump this
> back to critical so as not to hold up 2.6 alpha, but it should be marked
> as a release blocker for the first beta.
> """
>
> I don't know if this "Barry" guy has the appropriate permissions on the
> bugtracker to increase priorities, so I've taken the liberty of upgrading it
> as a release blocker for the _second_ beta ... ;-). So, at least there's
> been one productive consequence of this discussion.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev[at]python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

schmir at gmail

Jun 26, 2008, 10:40 AM

Post #8 of 32 (1294 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 6:54 PM, <glyph[at]divmod.com> wrote:
> On 04:42 pm, schmir[at]gmail.com wrote:
>>
>> On Thu, Jun 26, 2008 at 5:33 PM, Guido van Rossum <guido[at]python.org>
>> wrote:
>>>
>>> an explanation about *why* Django cannot even be imported than a
>>> blanket complaint that this is a disgrace. So why is it?
>
>> and already discussed:
>> http://mail.python.org/pipermail/python-dev/2008-April/078421.html
>
> Following that trail of breadcrumbs, I ended up here:
>
> http://bugs.python.org/issue2235
>
> with a message from some guy named "Barry Warsaw" (anyone know him?) that
> says:
>
> """
> Guido, can you comment on Amaury's latest patch? I'm going to bump this
> back to critical so as not to hold up 2.6 alpha, but it should be marked
> as a release blocker for the first beta.
> """
>
> I don't know if this "Barry" guy has the appropriate permissions on the
> bugtracker to increase priorities, so I've taken the liberty of upgrading it
> as a release blocker for the _second_ beta ... ;-). So, at least there's
> been one productive consequence of this discussion.

this patch even make things worse for me. now py.test also dies.
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 10:55 AM

Post #9 of 32 (1294 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <schmir[at]gmail.com> wrote:
> this patch even make things worse for me. now py.test also dies.

Please add details to the tracker.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

schmir at gmail

Jun 26, 2008, 11:07 AM

Post #10 of 32 (1298 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 7:55 PM, Guido van Rossum <guido[at]python.org> wrote:
> On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <schmir[at]gmail.com> wrote:
>> this patch even make things worse for me. now py.test also dies.
>
> Please add details to the tracker.
>

Well, I think most probably the patch is correct and it just catches
another class which defines __eq__ but does not define __hash__. So
nothing to add there.
The question is if this behaviour should be kept or not. I guess
there's no bug tracking that...
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 11:45 AM

Post #11 of 32 (1296 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 11:07 AM, Ralf Schmitt <schmir[at]gmail.com> wrote:
> On Thu, Jun 26, 2008 at 7:55 PM, Guido van Rossum <guido[at]python.org> wrote:
>> On Thu, Jun 26, 2008 at 10:40 AM, Ralf Schmitt <schmir[at]gmail.com> wrote:
>>> this patch even make things worse for me. now py.test also dies.
>>
>> Please add details to the tracker.

> Well, I think most probably the patch is correct and it just catches
> another class which defines __eq__ but does not define __hash__. So
> nothing to add there.

So you're saying py.test needs to be fixed? Fine with me, but then I
don't understand why you bothered bringing it up here instead of
fixing it (or reporting to the py.test maintainers, I don't know if
you're one or not).

> The question is if this behaviour should be kept or not. I guess
> there's no bug tracking that...

The best place is that very same issue. If this feature is really too
annoying something needs to be done. Detailed reports about real code
that gets broken without a good cause help making that case.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

g.brandl at gmx

Jun 26, 2008, 12:44 PM

Post #12 of 32 (1278 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

glyph[at]divmod.com schrieb:

> Another way to phrase this question is, "whose responsibility is it to
> make Python 2.5 programs run on Python 2.6"? Or, "what happens when the
> core team finds out that a change they have made has broken some python
> software 'in the wild'"?
>
> Here are a couple of ways to answer these questions:
>
> 1) It's the python core team's responsibility. There should be Python
> buildbots for lots of different projects and they should all be green
> before the code can be considered [one of: allowed in trunk, alpha,
> beta, RC].
> 2) It's the python core team's responsibility as long as the major
> projects are using public APIs. The code should not be considered
> [alpha, beta, RC] if there are any known breakages in public APIs.
>
> 3) It's only the python core team's responsibility to notify projects
> of incompatible changes of which they are aware, which may occur in
> public or private APIs. Major projects with breakages will be given a
> grace period before a [beta, RC, final] to release a forward-compatible
> version.
> 4) The status quo: every new Python release can (and will, at least
> speaking in terms of 2.6) break lots of popular software, with no
> warning aside from the beta period. There are no guarantees.

Python's backwards compatibility has never extended to non-public APIs.
That rules out 1). 2) would be nice, but to be honest, 2) and 3) are
unrealistic given the number of Python core developers and the tasks
already at hand. 4) is not acceptable either.

However, you seem to be overlooking that there's a 3.5) in there: it's
the Python core team's responsibility as long as the major projects are
using public APIs; but to reduce the workload every project should watch
its own buildbots and notify the core team using the issue tracker in
order to get the issue resolved.

At no time will a policy "the community buildbots must be green" be
useful: the tests that run on these buildbots are not under our control,
so if the tests test things we deem non-public we can't do anything
about it. (And we may have a hard time convincing project authors to
change the tests if we promised to make them run anyway.)

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

g.brandl at gmx

Jun 26, 2008, 12:46 PM

Post #13 of 32 (1285 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

glyph[at]divmod.com schrieb:
> On 03:42 pm, ncoghlan[at]gmail.com wrote:
>>glyph[at]divmod.com wrote:
>
>>beta 1 has some trouble running *our* test suite - I'd be fairly
>>surprised if the community buildbots were in significantly better
>>shape.
>
> That's another problem, yes :)
>>>The community buildbots have been in a broken state for months now[1].
>
>>If the community buildbots aren't largely green by the time beta 2
>>comes out, that's when I'll agree we have a problem - they should
>>definitely be green by the time first release candidate comes out.
>
> I have mixed feelings about this: on the one hand, if this were a matter
> of public record, I would have known that this was too early to start
> complaining, and we could have saved everyone a lot of time reading
> these messages. That would be good; I would prefer to just let everyone
> work without bothering about process.

The problem is that not enough people care about the community buildbots,
and this isn't likely to improve since we have not enough manpower to do
it.

> On the other hand, it's much easier to deal with test failures as they
> arise than in one giant chunk of work at the end of the development
> process. I spoke to a few core developers at PyCon who thought that the
> buildbots should always be green and any change that broke them should
> be reverted. They may remain nameless unless they wish to raise their
> hands :-) but that's definitely how I feel about it.

I think no core developer will not tell you that the (core) buildbots should be
green all the time. As for reverting changes that break, I'd support this
only for changes that break *all* of them. For example, I only use one
platform to develop on (and I guess it's the same for many others), having
the buildbots go red on another platform means I can try to fix the issue.

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

hall.jeff at gmail

Jun 26, 2008, 12:48 PM

Post #14 of 32 (1279 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

To me (and I'm an outsider not a direct developer), it's Python developers
responsibility to handle the Python problems and the Python build bots. The
community build bots are the responsibility of their authors. If JP is
handling the Twisted one then great. It's got a gatekeeper. If no one is
handling the Django build bot that's not the Python Dev Team's problem but
is a problem on the Django side and someone involved in that should be
tasked with monitoring.

Having said all that, I agree that the community bots ought to at least be
looked at. If I check in a patch in and I notice that a community bot went
from green to red then I probably should double check my code. The problem
is that, as you said, the community bots haven't been well maintained...
They've been red for awhile. So asking the developers to then go do the leg
work to find the original error, fix it and then back check everything
between then and now that might have ALSO caused a problem is alot of
effort.

It's not the Bank's responsibility to balance my check book. They give me
the tools to do it and then I have to check. A similar principle applies
here, I believe.

exarkun at divmod

Jun 26, 2008, 12:48 PM

Post #15 of 32 (1279 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, 26 Jun 2008 21:46:48 +0200, Georg Brandl <g.brandl[at]gmx.net> wrote:
> [snip]
>
>As for reverting changes that break, I'd support this
>only for changes that break *all* of them. For example, I only use one
>platform to develop on (and I guess it's the same for many others), having
>the buildbots go red on another platform means I can try to fix the issue.

BuildBot has two ways to let you run your code on all builders before you
commit it to trunk. You can force a build on a branch or you can try a
build with a patch. I don't know if these options are enabled on Python's
buildmaster. If they are, then if you want, you can use them to make sure
your code works on all platforms before you put it into trunk, where it may
cause problems for someone else.

Jean-Paul
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

tjreedy at udel

Jun 26, 2008, 1:02 PM

Post #16 of 32 (1285 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

glyph[at]divmod.com wrote:

> to what extent should Python actually be compatible between releases?

As I understand things from years of observation, the following are fair
game to changed in ways possibly backward-incompatible for specific
code: bugs, detailed float behavior (which may be system-specific
anyway), error messages, private interfaces, non-enforcement of
documented rules, and defined implementation-dependent behavior. But
Guido has been somewhat conservative about this (least so for bug fixes,
I think) and sometimes put off changes even when the fault lies with
questionable usages.

One fly in the ointment is that bugs (a discrepancy between doc and
code) may be fixed either by changing the doc or the code, according to
which embodies the design intention. So neither can be taken as gospel.

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steve at holdenweb

Jun 26, 2008, 1:04 PM

Post #17 of 32 (1287 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

Georg Brandl wrote:
> glyph[at]divmod.com schrieb:
>
>> Another way to phrase this question is, "whose responsibility is it to
>> make Python 2.5 programs run on Python 2.6"? Or, "what happens when
>> the core team finds out that a change they have made has broken some
>> python software 'in the wild'"?
>>
>> Here are a couple of ways to answer these questions:
>>
>> 1) It's the python core team's responsibility. There should be
>> Python buildbots for lots of different projects and they should all be
>> green before the code can be considered [one of: allowed in trunk,
>> alpha, beta, RC].
>> 2) It's the python core team's responsibility as long as the major
>> projects are using public APIs. The code should not be considered
>> [alpha, beta, RC] if there are any known breakages in public APIs.
> >
>> 3) It's only the python core team's responsibility to notify
>> projects of incompatible changes of which they are aware, which may
>> occur in public or private APIs. Major projects with breakages will
>> be given a grace period before a [beta, RC, final] to release a
>> forward-compatible version.
>> 4) The status quo: every new Python release can (and will, at least
>> speaking in terms of 2.6) break lots of popular software, with no
>> warning aside from the beta period. There are no guarantees.
>
> Python's backwards compatibility has never extended to non-public APIs.
> That rules out 1). 2) would be nice, but to be honest, 2) and 3) are
> unrealistic given the number of Python core developers and the tasks
> already at hand. 4) is not acceptable either.
>
> However, you seem to be overlooking that there's a 3.5) in there: it's
> the Python core team's responsibility as long as the major projects are
> using public APIs; but to reduce the workload every project should watch
> its own buildbots and notify the core team using the issue tracker in
> order to get the issue resolved.
>
> At no time will a policy "the community buildbots must be green" be
> useful: the tests that run on these buildbots are not under our control,
> so if the tests test things we deem non-public we can't do anything
> about it. (And we may have a hard time convincing project authors to
> change the tests if we promised to make them run anyway.)
>
If, however, the community buildbot goes red because of a regression or
backwards incompatibility the developers should, when notified, at least
undertake to verify that the breakage is intentional.

Unfortunately having a buildbot is like installing a firewall. Some
people assume that the very presence of the buildbot is a guard against
version incompatibilities, whereas in truth there is always the messy
business of communication required to ensure the information gets to
where it is useful.

The bottom line, of course, is that if project X can't be bothered to
tell the core development team when changes break their build it's
nobody's fault but their own. If the report it and the core developers
ignore the reports, that's something that should be rectified.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steve at holdenweb

Jun 26, 2008, 1:04 PM

Post #18 of 32 (1283 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

martin at v

Jun 26, 2008, 1:05 PM

Post #19 of 32 (1277 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

> BuildBot has two ways to let you run your code on all builders before you
> commit it to trunk. You can force a build on a branch or you can try a
> build with a patch. I don't know if these options are enabled on Python's
> buildmaster. If they are, then if you want, you can use them to make sure
> your code works on all platforms before you put it into trunk, where it may
> cause problems for someone else.

Even if that's possible, it adds a burden. As likely not all people
agree that such a procedure would be a good thing, it's necessary that
somebody establishes a policy. Guido has already indicated that he is
not interested in further policies; Barry (the canonical other person
to set policies) has indicated in the past that he would like such a
policy, but deems it unrealistic at this point. He gives other
procedural changes higher priority, so that realistically, such a policy
might not be established within the next two years.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

glyph at divmod

Jun 26, 2008, 1:06 PM

Post #20 of 32 (1281 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On 07:44 pm, g.brandl[at]gmx.net wrote:
>At no time will a policy "the community buildbots must be green" be
>useful: the tests that run on these buildbots are not under our
>control,
>so if the tests test things we deem non-public we can't do anything
>about it. (And we may have a hard time convincing project authors to
>change the tests if we promised to make them run anyway.)

That's not what I'm suggesting.

If there is a legitimate disagreement between Python developers and
developers of a project about whether an API should continue to be
supported, then clearly, the Python developers get to win. Welcome to
open source.

However, I believe that the core team is not paying attention to other
projects breaking. I'm not saying that you want to make breaking
changes, or that bug reports are not dealt with, but the problem is that
dealing with these problems after the fact makes it _much_ more painful.
When you get to the release, and you have 30 bug reports due to other
projects breaking, they get triaged, some get left in, and some features
of lots of different projects are left broken. And many projects do not
bother to test with the beta.

If developers were simply presented with the results of their changes
immediately ("you broke twisted, django doesn't import, zope segfaults
and mercurial destroys your repository") then perhaps they would weigh
the benefits of compatibility differently, and do the trivial, obvious
fix immediately, rather than waiting 6 months and having to figure out
what the heck change caused the bug. I accept the fact that python core
development is done differently than i.e. Java development, and some
backward compatibility may simply be broken.

Case in point: changes to the warnings module being discussed in a
separate thread break a significant number of Twisted's tests, because
they remove functionality which is not public but is required to test
code that uses the warnings module. Would Brett have taken this into
account if he knew about it immediately when revision 62303 was
committed? Maybe not, but now that the code is written and done,
there's significantly less motivation to go back and fix it. At the
time it might have only been a few minutes' work to continue supporting
the use-case which Twisted needs. Brett wouldn't even have necessarily
needed to do the work: it would have been easier to convince a Twisted
developer to do the work it to keep the community buildbot green rather
than to make it a bit less red. Now, though, it's much easier to say
"oh well, that's private, you lose". I don't ascribe this to malice -
it really *would* be much harder to fix it now, for us as well as for
him.
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

g.brandl at gmx

Jun 26, 2008, 1:36 PM

Post #21 of 32 (1282 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

glyph[at]divmod.com schrieb:
> On 07:44 pm, g.brandl[at]gmx.net wrote:
>>At no time will a policy "the community buildbots must be green" be
>>useful: the tests that run on these buildbots are not under our
>>control,
>>so if the tests test things we deem non-public we can't do anything
>>about it. (And we may have a hard time convincing project authors to
>>change the tests if we promised to make them run anyway.)
>
> That's not what I'm suggesting.

Sure, but I wanted to say that nevertheless :)

> If there is a legitimate disagreement between Python developers and
> developers of a project about whether an API should continue to be
> supported, then clearly, the Python developers get to win. Welcome to
> open source.
>
> However, I believe that the core team is not paying attention to other
> projects breaking.

Quite true. It is their duty to bring the breakage to our attention, if
they want to see results.

> I'm not saying that you want to make breaking
> changes, or that bug reports are not dealt with, but the problem is that
> dealing with these problems after the fact makes it _much_ more painful.
> When you get to the release, and you have 30 bug reports due to other
> projects breaking, they get triaged, some get left in, and some features
> of lots of different projects are left broken. And many projects do not
> bother to test with the beta.

I thought we're talking about the projects that *do* use the buildbots?

> If developers were simply presented with the results of their changes
> immediately ("you broke twisted, django doesn't import, zope segfaults
> and mercurial destroys your repository") then perhaps they would weigh
> the benefits of compatibility differently, and do the trivial, obvious
> fix immediately, rather than waiting 6 months and having to figure out
> what the heck change caused the bug.

This sounds very nice in theory, but it must be implemented in a way that
does not add a burden to the developer, as Martin said.

> Case in point: changes to the warnings module being discussed in a
> separate thread break a significant number of Twisted's tests, because
> they remove functionality which is not public but is required to test
> code that uses the warnings module. Would Brett have taken this into
> account if he knew about it immediately when revision 62303 was
> committed? Maybe not, but now that the code is written and done,
> there's significantly less motivation to go back and fix it. At the
> time it might have only been a few minutes' work to continue supporting
> the use-case which Twisted needs. Brett wouldn't even have necessarily
> needed to do the work: it would have been easier to convince a Twisted
> developer to do the work it to keep the community buildbot green rather
> than to make it a bit less red. Now, though, it's much easier to say
> "oh well, that's private, you lose". I don't ascribe this to malice -
> it really *would* be much harder to fix it now, for us as well as for
> him.

I must say that I'm increasingly surprised by this discussion since it seems
that not only the Python core devs neglect the community buildbots.
Shouldn't the project authors have at least the same interest that their
code runs on the next major Python version? And while they don't necessarily
have more time to watch the buildbots than we have, the amount of what they
need to keep an eye on.

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

g.brandl at gmx

Jun 26, 2008, 1:39 PM

Post #22 of 32 (1276 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

Terry Reedy schrieb:
>
> glyph[at]divmod.com wrote:
>
>> to what extent should Python actually be compatible between releases?
>
> As I understand things from years of observation, the following are fair
> game to changed in ways possibly backward-incompatible for specific
> code: bugs, detailed float behavior (which may be system-specific
> anyway), error messages, private interfaces, non-enforcement of
> documented rules, and defined implementation-dependent behavior. But
> Guido has been somewhat conservative about this (least so for bug fixes,
> I think) and sometimes put off changes even when the fault lies with
> questionable usages.

Which is the source of many unresolved open issues in the tracker -- everyone
agrees that "it" is a bug and should be fixed, perhaps there's even a patch,
but people may be using it this way, so nothing is done forever.

(And these bugs usually are the kind of things we don't want to change in
3.0 either since they are subtle things.)

However, I don't have a solution here, so I won't complain about it anymore.

Georg

_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Jun 26, 2008, 2:06 PM

Post #23 of 32 (1284 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jun 26, 2008, at 11:42 AM, Nick Coghlan wrote:

> If the community buildbots aren't largely green by the time beta 2
> comes out, that's when I'll agree we have a problem - they should
> definitely be green by the time first release candidate comes out.

Just FTR,

I'll be looking at the Python buildbots and Roundup tracker to decide
whether I think beta2 is in shape to be released or not. I definitely
won't be tracking the non-core buildbots, so if something troublesome
comes up there, you will have to submit a bug report to the Python
tracker.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSGQEu3EjvBPtnXfVAQIp6QP/bOU+8vImlRkfKCXazYhi3yEXcNwCPpWI
e3AoiOlgKbOhSPraNTyUBjC+OjuqHQttyMJz8TTpexSwhpG/erDIqUjRbkW0Yaas
attvFZBYbJhF3qvHyLzmSp9U3oXY6VyWayL8ZdzIGcukEW7g8DgQNQyoGOE4kEeX
i6+4RIRf+7A=
=X1lh
-----END PGP SIGNATURE-----
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 2:17 PM

Post #24 of 32 (1283 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 12:35 PM, <glyph[at]divmod.com> wrote:
>
> On 05:14 pm, martin[at]v.loewis.de wrote:
>>>
>>> I don't know. JP is already addressing the issues affecting Twisted in
>>> another thread (incompatible changes in the private-but-necessary-to-
>>> get-any-testing-done API of the warnings module). But I really think
>>> that whoever made the change which broke it should be the one
>>> investigating it, not me.
>>
>> How could they have known that they broke it?
>
> Because the relevant community buildbot turned red with that revision of
> trunk. Keep in mind I'm not talking about every piece of Python code in the
> universe; just the ones selected for the community buildbots. It would be
> nice if there were a dozen or so projects on there, but paying attention to
> the two builders that do actually run right now would be a good start.

Sorry, this is an unacceptable burden on the core developers. The core
developers aren't going to look at the community buildbots (unless
voluntarily) and they aren't going to roll back changes just because
some community buildbot goes red.

In general someone outside the core developer group needs to bring the
issue to the core developers' attention and then a fix will be created
if appropriate. Rollbacks are generally reserved for accidental
checkins or checkins against the process rules (e.g. during a code
freeze). Heck, we don't even roll back if one of our own buildbots
goes red.

I'm fine with requesting that the core developers pay serious
attention to reports about 3rd party code being broken. The community
buildbots are a fine tool to find out about this. But any policy that
requires an automatic rollback because a buildbot (community or core)
goes red is unacceptable.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Jun 26, 2008, 2:24 PM

Post #25 of 32 (1265 views)
Permalink

Re: Community buildbots and Python release quality metrics [In reply to]

On Thu, Jun 26, 2008 at 1:06 PM, <glyph[at]divmod.com> wrote:
> On 07:44 pm, g.brandl[at]gmx.net wrote:
>>
>> At no time will a policy "the community buildbots must be green" be
>> useful: the tests that run on these buildbots are not under our control,
>> so if the tests test things we deem non-public we can't do anything
>> about it. (And we may have a hard time convincing project authors to
>> change the tests if we promised to make them run anyway.)
>
> That's not what I'm suggesting.
>
> If there is a legitimate disagreement between Python developers and
> developers of a project about whether an API should continue to be
> supported, then clearly, the Python developers get to win. Welcome to open
> source.
>
> However, I believe that the core team is not paying attention to other
> projects breaking. I'm not saying that you want to make breaking changes,
> or that bug reports are not dealt with, but the problem is that dealing with
> these problems after the fact makes it _much_ more painful. When you get to
> the release, and you have 30 bug reports due to other projects breaking,
> they get triaged, some get left in, and some features of lots of different
> projects are left broken. And many projects do not bother to test with the
> beta.

Well, sorry, that's life. We're not going to deal with breakage in 3rd
party code on a "drop all other work" basis.

You seem to have a different idea about how to do development than the
core developers. Tough luck. You can't tell us how to do our work.
We're doing the best we can, and we're happy to listen to suggestions
you're making, but the set of suggestions you're making in this thread
are an unacceptable burden, and it's not going to happen.

> If developers were simply presented with the results of their changes
> immediately ("you broke twisted, django doesn't import, zope segfaults and
> mercurial destroys your repository") then perhaps they would weigh the
> benefits of compatibility differently, and do the trivial, obvious fix
> immediately, rather than waiting 6 months and having to figure out what the
> heck change caused the bug. I accept the fact that python core development
> is done differently than i.e. Java development, and some backward
> compatibility may simply be broken.
>
> Case in point: changes to the warnings module being discussed in a separate
> thread break a significant number of Twisted's tests, because they remove
> functionality which is not public but is required to test code that uses the
> warnings module. Would Brett have taken this into account if he knew about
> it immediately when revision 62303 was committed? Maybe not, but now that
> the code is written and done, there's significantly less motivation to go
> back and fix it.

I disagree. It's broken and should be fixed. Beta 1 just came out so
this is the perfect time to file a bug.

> At the time it might have only been a few minutes' work to
> continue supporting the use-case which Twisted needs. Brett wouldn't even
> have necessarily needed to do the work: it would have been easier to
> convince a Twisted developer to do the work it to keep the community
> buildbot green rather than to make it a bit less red.

Why? That sounds like it's a problem with the psychology of the
Twisted developers.

> Now, though, it's
> much easier to say "oh well, that's private, you lose". I don't ascribe
> this to malice - it really *would* be much harder to fix it now, for us as
> well as for him.

I don't believe this.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________
Python-Dev mailing list
Python-Dev[at]python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com