Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Gentoo: Dev

Disabling auto-bumping of active Python version

 

 

Gentoo dev RSS feed   Index | Next | Previous | View Threaded


sping at gentoo

Nov 14, 2010, 8:28 AM

Post #1 of 24 (2778 views)
Permalink
Disabling auto-bumping of active Python version

Hello,

thanks for your interest. This thread is not about Python 3.x in
particular, in case you wonder.


In this mail
============
- Typical GCC update (for comparison)
- Python 2.7 update simulation (and how it fails)
- The scenario
- What happens
- How it happens
- Conclusion


Typical GCC update
==================
Before we look at Python, let's have a look at how it works with GCC:

When a new version of GCC appears in Gentoo, it is installed into
a new slot. To activate the new version, you run gcc-config.
In the meantime your system is in working state and continues to
use the old version of GCC.


Python 2.7 update simulation
============================

The scenario
------------
With Python it's very different. Let's assume a system with Python 2.6
as the only Python around. Now we would like to run this command:

# sudo emerge dev-lang/python:2.7 dev-python/pyinotify

Assume that dev-lang/python:2.7 is unmasked already.
The resulting system state now depends on these two factors:

- Has pyinotify been installed to the system before our invocation of
emerge? Depending on that, pyinotify may be installed first, or
python 2.7 may. Let's assume pyinotify comes last. The difference
is marginal.

- Were we using USE_PYTHON in /etc/make.conf (e.g. USE_PYTHON="2.6")
or not? I'll assume USE_PYTHON is not set manually and implied
during installation. The last three devs I spoke to had not heard
of variable USE_PYTHON, so that assumption may work.


What happens
------------
Steps taken by emerge:
- Python 2.7 installed
- Python 2.7 is made the active version of Python automatically
- pyinotify is installed for Python ABI 2.7 (neither 2.6 nor both)

Resulting system state:
- Python 2.7 now is the main active version of Python
- All Python packages except pyinotify are installed for Python ABI 2.6
- pyinotify is the only package installed for Python ABI 2.7
- before running python-updater larger parts of the system may be
unusable


How it happens
--------------
All dev-lang/python ebuilds run a Bash function called
"eselect_python_update" at the beginning pkg_postinst stage.
Former function wraps a call to

eselect python update ${eselect_python_options}


Conclusion
==========
When you update GCC your system stays at a healthy state.
You trigger the switch of active versions.

With Python, when you install a newer version if gets activated, leaving
your system in broken state. An update of Python always involves
running python-updater. If the user/admin has to run that anyway, why
should we take the call to eselect python off his shoulders, especially
we break the system doing so?

So I plan to remove the call to eselect_python_update from all Python
ebuilds in two weeks from now. Besides Arfrever everybody I spoke to on
this topic so far agreed to this approach. The two weeks window are to
give people room to think and discuss about it.

Please try to keep this thread as low-heat as possible.
Thanks for reading.



Sebastian


neurogeek at gentoo

Nov 14, 2010, 10:04 AM

Post #2 of 24 (2731 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

+1

I totally agree with the conclusion. I believe that choice, "select
the active version of Python", should not be implied by us, but taken
by the user even if he explicitly asked for a new version to be
installed.

I've been in this situation before, and just because I wanted to try
some new cool features, sometimes I ended up with a semi broken system
just because I wouldn't take extra care with it.

Plus I think users are going to miss that feature much.

Best regards,

Jesus

On 11/14/10, Sebastian Pipping <sping [at] gentoo> wrote:
> Hello,
>
> thanks for your interest. This thread is not about Python 3.x in
> particular, in case you wonder.
>
>
> In this mail
> ============
> - Typical GCC update (for comparison)
> - Python 2.7 update simulation (and how it fails)
> - The scenario
> - What happens
> - How it happens
> - Conclusion
>
>
> Typical GCC update
> ==================
> Before we look at Python, let's have a look at how it works with GCC:
>
> When a new version of GCC appears in Gentoo, it is installed into
> a new slot. To activate the new version, you run gcc-config.
> In the meantime your system is in working state and continues to
> use the old version of GCC.
>
>
> Python 2.7 update simulation
> ============================
>
> The scenario
> ------------
> With Python it's very different. Let's assume a system with Python 2.6
> as the only Python around. Now we would like to run this command:
>
> # sudo emerge dev-lang/python:2.7 dev-python/pyinotify
>
> Assume that dev-lang/python:2.7 is unmasked already.
> The resulting system state now depends on these two factors:
>
> - Has pyinotify been installed to the system before our invocation of
> emerge? Depending on that, pyinotify may be installed first, or
> python 2.7 may. Let's assume pyinotify comes last. The difference
> is marginal.
>
> - Were we using USE_PYTHON in /etc/make.conf (e.g. USE_PYTHON="2.6")
> or not? I'll assume USE_PYTHON is not set manually and implied
> during installation. The last three devs I spoke to had not heard
> of variable USE_PYTHON, so that assumption may work.
>
>
> What happens
> ------------
> Steps taken by emerge:
> - Python 2.7 installed
> - Python 2.7 is made the active version of Python automatically
> - pyinotify is installed for Python ABI 2.7 (neither 2.6 nor both)
>
> Resulting system state:
> - Python 2.7 now is the main active version of Python
> - All Python packages except pyinotify are installed for Python ABI 2.6
> - pyinotify is the only package installed for Python ABI 2.7
> - before running python-updater larger parts of the system may be
> unusable
>
>
> How it happens
> --------------
> All dev-lang/python ebuilds run a Bash function called
> "eselect_python_update" at the beginning pkg_postinst stage.
> Former function wraps a call to
>
> eselect python update ${eselect_python_options}
>
>
> Conclusion
> ==========
> When you update GCC your system stays at a healthy state.
> You trigger the switch of active versions.
>
> With Python, when you install a newer version if gets activated, leaving
> your system in broken state. An update of Python always involves
> running python-updater. If the user/admin has to run that anyway, why
> should we take the call to eselect python off his shoulders, especially
> we break the system doing so?
>
> So I plan to remove the call to eselect_python_update from all Python
> ebuilds in two weeks from now. Besides Arfrever everybody I spoke to on
> this topic so far agreed to this approach. The two weeks window are to
> give people room to think and discuss about it.
>
> Please try to keep this thread as low-heat as possible.
> Thanks for reading.
>
>
>
> Sebastian
>
>

--
Sent from my mobile device

Jesus Rivero
Gentoo Python Team


lists at f_philipp

Nov 14, 2010, 1:42 PM

Post #3 of 24 (2737 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

Am 14.11.2010 17:28, schrieb Sebastian Pipping:
>
> With Python, when you install a newer version if gets activated, leaving
> your system in broken state. An update of Python always involves
> running python-updater. If the user/admin has to run that anyway, why
> should we take the call to eselect python off his shoulders, especially
> we break the system doing so?
>
> So I plan to remove the call to eselect_python_update from all Python
> ebuilds in two weeks from now.
[...]

Is there a chance to do the same for perl and perl-cleaner?

I understand perl is not slotted at the moment and there are probably
good technical reasons for not doing so but I still wanted to ask. :)

There are similar issues with ocaml and ocaml-rebuild.sh.

Best regards,
Florian Philipp
Attachments: signature.asc (0.26 KB)


sping at gentoo

Nov 15, 2010, 6:37 AM

Post #4 of 24 (2735 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 11/14/10 22:42, Florian Philipp wrote:
> Is there a chance to do the same for perl and perl-cleaner?
>
> I understand perl is not slotted at the moment and there are probably
> good technical reasons for not doing so but I still wanted to ask. :)

I'm not sure if the Perl team is watching this thread.
Maybe ask on gentoo-perl directly.

http://www.gentoo.org/main/en/lists.xml


> There are similar issues with ocaml and ocaml-rebuild.sh.

Maybe try contacting ml [at] g on that one. I don't see any Ocaml project
pages or mailing lists in Gentoo.



Sebastian


aballier at gentoo

Nov 15, 2010, 2:02 PM

Post #5 of 24 (2738 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On Monday 15 November 2010 11:37:08 Sebastian Pipping wrote:
> On 11/14/10 22:42, Florian Philipp wrote:
> > Is there a chance to do the same for perl and perl-cleaner?
> >
> > I understand perl is not slotted at the moment and there are probably
> > good technical reasons for not doing so but I still wanted to ask. :)
>
> I'm not sure if the Perl team is watching this thread.
> Maybe ask on gentoo-perl directly.
>
> http://www.gentoo.org/main/en/lists.xml
>
> > There are similar issues with ocaml and ocaml-rebuild.sh.
>
> Maybe try contacting ml [at] g on that one. I don't see any Ocaml project
> pages or mailing lists in Gentoo.


Since none are slotted, I don't understand what outcome you expect by
contacting the maintainers.
At best, perl-cleaner and ocaml-rebuild.sh shall be merged into portage in a
preserve-libs like manner.

A.


zmedico at gentoo

Nov 15, 2010, 7:22 PM

Post #6 of 24 (2727 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 11/15/2010 02:02 PM, Alexis Ballier wrote:
> Since none are slotted, I don't understand what outcome you expect by
> contacting the maintainers.
> At best, perl-cleaner and ocaml-rebuild.sh shall be merged into portage in a
> preserve-libs like manner.

In case some aren't aware, I'll mention that abi-slot dependencies
(if/when implemented) can potentially serve as a generic solution that
covers all such un-slotted cases (as discussed in bug #192319 [1]). For
slotted packages, the rebuild logic will be slightly different, but it
will be nice to devise a generic solution for those too.

[1] https://bugs.gentoo.org/show_bug.cgi?id=192319
--
Thanks,
Zac


sping at gentoo

Nov 27, 2010, 5:00 AM

Post #7 of 24 (2695 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 11/14/10 17:28, Sebastian Pipping wrote:
> So I plan to remove the call to eselect_python_update from all Python
> ebuilds in two weeks from now.

Now fixed in both the main tree as well as the Python overlay.
Please let me know in case I broke anything.

Best,



Sebastian


Arfrever at gentoo

Nov 28, 2010, 10:04 AM

Post #8 of 24 (2692 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

2010-11-27 14:00:28 Sebastian Pipping napisał(a):
> On 11/14/10 17:28, Sebastian Pipping wrote:
> > So I plan to remove the call to eselect_python_update from all Python
> > ebuilds in two weeks from now.
>
> Now fixed in both the main tree as well as the Python overlay.
> Please let me know in case I broke anything.

You probably broke generation of stages :) .
(I have restored a minimal version of eselect_python_update() in python overlay.)

--
Arfrever Frehtes Taifersar Arahesis
Attachments: signature.asc (0.82 KB)


sping at gentoo

Nov 28, 2010, 11:39 PM

Post #9 of 24 (2675 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 11/28/10 19:04, Arfrever Frehtes Taifersar Arahesis wrote:
> You probably broke generation of stages :) .
> (I have restored a minimal version of eselect_python_update() in python overlay.)

Could you elaborate on that, please? How did I break stages? Which
stages? Future stage tarballs?



Sebastian


Arfrever at gentoo

Nov 29, 2010, 12:35 AM

Post #10 of 24 (2674 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

2010-11-29 08:39:46 Sebastian Pipping napisał(a):
> On 11/28/10 19:04, Arfrever Frehtes Taifersar Arahesis wrote:
> > You probably broke generation of stages :) .
> > (I have restored a minimal version of eselect_python_update() in python overlay.)
>
> Could you elaborate on that, please? How did I break stages? Which
> stages? Future stage tarballs?

There will probably be no active version of Python set.

--
Arfrever Frehtes Taifersar Arahesis
Attachments: signature.asc (0.82 KB)


sping at gentoo

Nov 29, 2010, 3:34 AM

Post #11 of 24 (2678 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
> There will probably be no active version of Python set.

You had two weeks to come up with this.

Please find my on IRC to team up on an agreed fix.



Sebastian


fauli at gentoo

Nov 29, 2010, 4:10 AM

Post #12 of 24 (2684 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

Hi,

Sebastian Pipping <sping [at] gentoo>:

> On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
> > There will probably be no active version of Python set.
>
> You had two weeks to come up with this.
>
> Please find my on IRC to team up on an agreed fix.

$ eselect python --help
Manage Python symlinks
Usage: eselect python <action> <options>

[...]
update Switch to the most recent CPython interpreter
--if-unset Do not override existing implementation
--ignore SLOT Ignore SLOT when setting symlinks
--python2 Set active Python 2 interpreter without
setting of main active Python interpreter if it is not set to Python 2
--python3 Set active Python 3 interpreter without
setting of main active Python interpreter if it is not set to Python 3

See the --if-unset option.

V-Li

--
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>
Attachments: signature.asc (0.19 KB)


sping at gentoo

Nov 29, 2010, 4:31 AM

Post #13 of 24 (2683 views)
Permalink
Re: Re: Disabling auto-bumping of active Python version [In reply to]

On 11/29/10 13:10, Christian Faulhammer wrote:
> $ eselect python --help
> Manage Python symlinks
> Usage: eselect python <action> <options>
>
> [...]
> update Switch to the most recent CPython interpreter
> --if-unset Do not override existing implementation
> --ignore SLOT Ignore SLOT when setting symlinks
> --python2 Set active Python 2 interpreter without
> setting of main active Python interpreter if it is not set to Python 2
> --python3 Set active Python 3 interpreter without
> setting of main active Python interpreter if it is not set to Python 3
>
> See the --if-unset option.

I see, thanks.

What I would now like to call from the ebuild is

eselect python set --if-unset ${SLOT}

Problem is ..
* action "set" wants and index, not a slot
* --if-unset works with update only

If there's a cleaner way than duplicating an ugly (and probably
error-prone) future wrapper to all ebuilds of dev-lang/python, I would
rather try that.

Any ideas?



Sebastian


fauli at gentoo

Nov 29, 2010, 5:12 AM

Post #14 of 24 (2686 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

Hi,

Sebastian Pipping <sping [at] gentoo>:
> On 11/29/10 13:10, Christian Faulhammer wrote:
> > $ eselect python --help
> > Manage Python symlinks
> > Usage: eselect python <action> <options>
> >
> > [...]
> > update Switch to the most recent CPython
> > interpreter --if-unset Do not override existing
> > implementation --ignore SLOT Ignore SLOT when setting
> > symlinks --python2 Set active Python 2 interpreter
> > without setting of main active Python interpreter if it is not set
> > to Python 2 --python3 Set active Python 3
> > interpreter without setting of main active Python interpreter if it
> > is not set to Python 3
> >
> > See the --if-unset option.
>
> I see, thanks.
>
> What I would now like to call from the ebuild is
>
> eselect python set --if-unset ${SLOT}
>
> Problem is ..
> * action "set" wants and index, not a slot

The Python eselect module can take python${SLOT} as argument for set.

> * --if-unset works with update only

At least for the Emacs eselect module update works like set if
--if-unset is given. So Python 2.6 has to be merged first to set it to
python2.6 executable, then all subsequent calls from newer Python
ebuilds (like 3.1) will not change the setting.

V-Li

--
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>
Attachments: signature.asc (0.19 KB)


jmbsvicetto at gentoo

Nov 30, 2010, 8:02 PM

Post #15 of 24 (2656 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29-11-2010 10:34, Sebastian Pipping wrote:
> On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
>> There will probably be no active version of Python set.
>
> You had two weeks to come up with this.
>
> Please find my on IRC to team up on an agreed fix.

As Arfrever noted, this is likely the cause of the broken automated
weekly stages for this past week. By not having a python symlink /
wrapper, stages generation failed on stage2 run.
I'd like to take this chance to recall this is the 2nd time on the last
few months where stage generation was broken by python changes. Also,
we've been unable to create hardened stages for over 8 weeks because of
a sandbox issue.
The weekly stages generation depends on the quality and stability of the
"stable" tree. Therefore, the RelEng team kindly asks all maintainers to
pay attention to the stable ebuilds in the system set and to please fix
any failures asap as they may / can prevent stage generation. Be sure to
think carefully about changes that can impact the stage generation, in
particular when they involve python.

> Sebastian
>

- --
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJM9cjoAAoJEC8ZTXQF1qEPdC0QAK0HLSse/T3XUL9+vjp3VHro
xXED/VsiXLlJtwlEsfWLf5kkMhejEXTO5gbLWZmqaJynOCFBk55eqQz04YXZQqWv
XmtEnXzaf+394chTaWV3hPIzuVayZzJRtVJhEj1imbgLIa3Qyx7XJjpC4NwH2MVo
1Odef7Eh8vhpE2bD48BxJfp9KGIf+MQf0TPex/4TACirYzJ60J4aGZ627gbzyaym
3o9DoLD1DRRrURO66buLyV2eXvnMrNrO3UYKPP1M93uW1hq4RXZRGJ4oGePbBfwQ
JuoGNc227lixeaivlLA83AOQbfYM3OW8zuM1l2kVMNSvzeVyh/wEx9U9ptvLhhLR
nd0FJNt8RsIH8Yzi5NUfv4JMJoAQK5km2kNVFH1bE8vVp+RwyTVFPnuCtdNL1uLJ
rVl+ptrF/Aiey5u/gVXpYSZGjU/lYtp73EebZhO+Bn1mymGMNVr2/UU9HWgCowmN
so0AIcKa5NkT5L4Y3kI+bSokcrm/5bLbjZQ5MayWsRuEwJkWyt1vxfW2B2DaQbjQ
+hRezNy/GwnfLM21yEYn46h5RnArdtV3vT6J6vksG+4aepa6WNAFnw47ABJW63z0
jAL8n/qjrsi2PJlUdrbZ230iqIFV1RmPtP5Z7M9gAM4VAtb8dNPWxB2ooEYOqg6u
8yAYnSVjKv/dTK7nPhSk
=gX34
-----END PGP SIGNATURE-----


jmbsvicetto at gentoo

Nov 30, 2010, 8:22 PM

Post #16 of 24 (2660 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01-12-2010 03:02, Jorge Manuel B. S. Vicetto wrote:
> On 29-11-2010 10:34, Sebastian Pipping wrote:
> As Arfrever noted, this is likely the cause of the broken automated
> weekly stages for this past week. By not having a python symlink /
> wrapper, stages generation failed on stage2 run.

I just tested the stages creation locally and as expected this must be
the source of the failure has there's no python symlink when building
stage2:

atlantis [at] atl4cor /home/release $ ls -la
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/py*
- -rwxr-xr-x 1 root root 78 Dez 1 02:57
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/pydoc2.6
- -rwxr-xr-x 1 root root 78 Dez 1 02:59
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/pydoc3.1
- -rwxr-xr-x 1 root root 6104 Dez 1 02:58
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/python2.6
- -rwxr-xr-x 1 root root 10272 Dez 1 03:00
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/python3.1
- -rwxr-xr-x 1 root root 1200 Dez 1 02:57
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/python-config-2.6
- -rwxr-xr-x 1 root root 1177 Dez 1 02:59
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/python-config-3.1
- -rwxr-xr-x 1 root root 10328 Dez 1 02:52
buildroot/amd64-dev/tmp/default/stage2-amd64-20101201/usr/bin/python-wrapper

As such the RelEng team would "appreciate" a quick fix of the broken
python stable ebuilds.

- --
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJM9c1rAAoJEC8ZTXQF1qEPlygP/2wib7n4YOXvXRfBCrYoKuDi
VTHLPpXHIhOxqWvAqIczyHfJhsIwAzeVm5wehmj+NDUSs7HXeO4Why4H6X9FY530
0KQqAsnjBQJU4xqfE8kofcRt8TUY7jmToObmEnGDM8raqouvpHjUlpp/2CC/eNCz
xOVuJ66+DJC3LIjmCcMQIAhbrhZZ63s/3J9O3D7XHJtLGdWBmNvAfRELTdxNM+Zw
UZRntctOWaPnyB6aSTvfK3SvC8cgPyBwvQFGWioZozWn9W+0J97/aux+aJRSCKJy
f+BzqVCGfeEq8j4yUzyQUzkXRV7fbIXMHhbGq6wUuMgMvZo/tAa64x77nebPBtCm
ZH49HAnKRRzy8O72AE3BVKVT3hCAxEBU/len309Dc2Cbwznb17WYm18Ihl5ShACG
/UZ+TkYwyctuh/ICmtFE0DsUIgcXHsMaCllpLF1iNxIEX6yOaxWvPoE1Pv4cnyIv
BWGHHL4sHsybyRNkiGYbeQQarYaXKS78N6wOeBhjXMi+T7QqanrJ76897l7LsRr2
3+RaPA0KHCvexixI7EuEUjfrIcYNFpZJoLLGci8ZxtjDe9MRRpnY1J6540B1sTi9
1Amb8ZwrXib4ngId7oZPN//E9umbB1FFOixGRNGfn9E2Ovmc9D+BFm5/+xCxr2/0
B6bpK4SXkB5tedu+D4a6
=uqe1
-----END PGP SIGNATURE-----


sping at gentoo

Dec 1, 2010, 9:28 AM

Post #17 of 24 (2653 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 12/01/10 05:02, Jorge Manuel B. S. Vicetto wrote:
> As Arfrever noted, this is likely the cause of the broken automated
> weekly stages for this past week. By not having a python symlink /
> wrapper, stages generation failed on stage2 run.

Yes, a fellow dev already reported that issue to me.
I have proposed a patch in the python 2.7 status thread.
Before mass-applying that I wanted to have review and/or more testing by
myself. I'm afraid of causing worse damage than that. That's why it
isn't fixed yet.


> I'd like to take this chance to recall this is the 2nd time on the last
> few months where stage generation was broken by python changes. Also,
> we've been unable to create hardened stages for over 8 weeks because of
> a sandbox issue.

Can you give us a pointer on details?


> The weekly stages generation depends on the quality and stability of the
> "stable" tree. Therefore, the RelEng team kindly asks all maintainers to
> pay attention to the stable ebuilds in the system set and to please fix
> any failures asap as they may / can prevent stage generation. Be sure to
> think carefully about changes that can impact the stage generation, in
> particular when they involve python.

Agreed.



Sebastian


antarus at gentoo

Dec 1, 2010, 12:13 PM

Post #18 of 24 (2650 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On Tue, Nov 30, 2010 at 8:02 PM, Jorge Manuel B. S. Vicetto
<jmbsvicetto [at] gentoo> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 29-11-2010 10:34, Sebastian Pipping wrote:
>> On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
>>> There will probably be no active version of Python set.
>>
>> You had two weeks to come up with this.
>>
>> Please find my on IRC to team up on an agreed fix.
>
> As Arfrever noted, this is likely the cause of the broken automated
> weekly stages for this past week. By not having a python symlink /
> wrapper, stages generation failed on stage2 run.
> I'd like to take this chance to recall this is the 2nd time on the last
> few months where stage generation was broken by python changes. Also,
> we've been unable to create hardened stages for over 8 weeks because of
> a sandbox issue.
> The weekly stages generation depends on the quality and stability of the
> "stable" tree. Therefore, the RelEng team kindly asks all maintainers to
> pay attention to the stable ebuilds in the system set and to please fix
> any failures asap as they may / can prevent stage generation. Be sure to
> think carefully about changes that can impact the stage generation, in
> particular when they involve python.

Two issues:

proj/en/releng is old as hell and doesn't even mention stage generation.

How does a developer know when the stage generation is broken? Is
there a dashboard? At work we have a guy who is basically a build cop
and checks our build dashboard once a day or so and if it is broken he
goes and finds the guy who broke it and punches him in the face until
he fixes it. I imagine we do not have staff for this (and no one has
invented punching over the internet.)

I am curious how often stage builds fail (how long can they be broken
until we actually care?)

>
>> Sebastian
>>
>
> - --
> Regards,
>
> Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
> Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.16 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQIcBAEBAgAGBQJM9cjoAAoJEC8ZTXQF1qEPdC0QAK0HLSse/T3XUL9+vjp3VHro
> xXED/VsiXLlJtwlEsfWLf5kkMhejEXTO5gbLWZmqaJynOCFBk55eqQz04YXZQqWv
> XmtEnXzaf+394chTaWV3hPIzuVayZzJRtVJhEj1imbgLIa3Qyx7XJjpC4NwH2MVo
> 1Odef7Eh8vhpE2bD48BxJfp9KGIf+MQf0TPex/4TACirYzJ60J4aGZ627gbzyaym
> 3o9DoLD1DRRrURO66buLyV2eXvnMrNrO3UYKPP1M93uW1hq4RXZRGJ4oGePbBfwQ
> JuoGNc227lixeaivlLA83AOQbfYM3OW8zuM1l2kVMNSvzeVyh/wEx9U9ptvLhhLR
> nd0FJNt8RsIH8Yzi5NUfv4JMJoAQK5km2kNVFH1bE8vVp+RwyTVFPnuCtdNL1uLJ
> rVl+ptrF/Aiey5u/gVXpYSZGjU/lYtp73EebZhO+Bn1mymGMNVr2/UU9HWgCowmN
> so0AIcKa5NkT5L4Y3kI+bSokcrm/5bLbjZQ5MayWsRuEwJkWyt1vxfW2B2DaQbjQ
> +hRezNy/GwnfLM21yEYn46h5RnArdtV3vT6J6vksG+4aepa6WNAFnw47ABJW63z0
> jAL8n/qjrsi2PJlUdrbZ230iqIFV1RmPtP5Z7M9gAM4VAtb8dNPWxB2ooEYOqg6u
> 8yAYnSVjKv/dTK7nPhSk
> =gX34
> -----END PGP SIGNATURE-----
>
>


nightmorph at gentoo

Dec 1, 2010, 12:34 PM

Post #19 of 24 (2652 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On Wed, 1 Dec 2010 12:13:03 -0800
Alec Warner <antarus [at] gentoo> wrote:

> On Tue, Nov 30, 2010 at 8:02 PM, Jorge Manuel B. S. Vicetto
> <jmbsvicetto [at] gentoo> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On 29-11-2010 10:34, Sebastian Pipping wrote:
> >> On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
> >>> There will probably be no active version of Python set.
> >>
> >> You had two weeks to come up with this.
> >>
> >> Please find my on IRC to team up on an agreed fix.
> >
> > As Arfrever noted, this is likely the cause of the broken
> > automated weekly stages for this past week. By not having a
> > python symlink / wrapper, stages generation failed on stage2 run.
> > I'd like to take this chance to recall this is the 2nd time on
> > the last few months where stage generation was broken by python
> > changes. Also, we've been unable to create hardened stages for
> > over 8 weeks because of a sandbox issue.
> > The weekly stages generation depends on the quality and stability
> > of the "stable" tree. Therefore, the RelEng team kindly asks all
> > maintainers to pay attention to the stable ebuilds in the system
> > set and to please fix any failures asap as they may / can prevent
> > stage generation. Be sure to think carefully about changes that
> > can impact the stage generation, in particular when they involve
> > python.
>
> Two issues:
>
> proj/en/releng is old as hell and doesn't even mention stage
> generation.
>
> How does a developer know when the stage generation is broken? Is
> there a dashboard? At work we have a guy who is basically a build
> cop and checks our build dashboard once a day or so and if it is
> broken he goes and finds the guy who broke it and punches him in
> the face until he fixes it. I imagine we do not have staff for
> this (and no one has invented punching over the internet.)

Catalyst sends automated emails to releng [at] gentoo from the
various build boxes: dolphin, poseidon, other dev.g.o machines.

> I am curious how often stage builds fail (how long can they be
> broken until we actually care?)

Fairly often, especially in the last couple of months or so. There
were some arches that, last I checked, hadn't had
any new media in several months. Python is the usual cause.
Remember the last huge Python debacle that resulted in suspension?
Yeah, that was one of the reasons for continually broken media.

Python issues are pretty much the only reason why general stage builds
fail (hardened is its own set of problems.)

Here's part of a typical message from one of the boxes, minus a whole
bunch of "bad interpreter" errors:

---------------------------------------------------------------------
[[ (1/3) Configuring environment ]]
/usr/portage/scripts/bootstrap.sh: line 307: python: command not found
---------------------------------------------------------------------
[[ (2/3) Updating portage ]]
env: emerge: No such file or directory

!!! catalyst: run script failed.

Traceback (most recent call last):
File "modules/generic_stage_target.py", line 1207, in run_local
"run script failed.",env=self.env)
File "/usr/lib64/catalyst/modules/catalyst_support.py", line 542,
in cmd
raise CatalystError,myexc
CatalystError
None

I see messages like this pretty much every day. Releng is
understaffed on a few arches, which is why no one has time to track
down the errors, fix them, and get the builds completed.
Attachments: signature.asc (0.19 KB)


phajdan.jr at gentoo

Dec 1, 2010, 12:39 PM

Post #20 of 24 (2651 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 12/1/10 9:13 PM, Alec Warner wrote:
>> How does a developer know when the stage generation is broken? Is
>> there a dashboard? At work we have a guy who is basically a build cop
>> and checks our build dashboard once a day or so and if it is broken he
>> goes and finds the guy who broke it and punches him in the face until
>> he fixes it. I imagine we do not have staff for this (and no one has
>> invented punching over the internet.)

It should be possible to create a public dashboard for the stage build
process.

Also, we have an automated warning about the bug queue size, we can have
a similar warning about stage generation.
Attachments: signature.asc (0.19 KB)


phajdan.jr at gentoo

Dec 1, 2010, 12:58 PM

Post #21 of 24 (2653 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On 12/1/10 9:34 PM, Joshua Saddler wrote:
> Catalyst sends automated emails to releng [at] gentoo from the
> various build boxes: dolphin, poseidon, other dev.g.o machines.

So we have some automatic reporting. Can we have a webpage for that, or
a mailing list that people can subscribe to?

Idea: make python team receive those emails. ;-)

>> I am curious how often stage builds fail (how long can they be
>> broken until we actually care?)
>
> Fairly often, especially in the last couple of months or so. There
> were some arches that, last I checked, hadn't had
> any new media in several months.

Do we have some bugs or other actions towards fixing those problems?

> I see messages like this pretty much every day. Releng is
> understaffed on a few arches, which is why no one has time to track
> down the errors, fix them, and get the builds completed.

Is it possible to bisect the portage tree and identify a breaking commit?

By the way, it would be great to have some kind of continuous
integration for the portage tree, even something very basic, and visible
to the public.
Attachments: signature.asc (0.19 KB)


darkside at gentoo

Dec 1, 2010, 1:08 PM

Post #22 of 24 (2646 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On Wed, 01 Dec 2010 21:58:20 +0100, Paweł Hajdan, Jr. wrote:
> On 12/1/10 9:34 PM, Joshua Saddler wrote:
>> Catalyst sends automated emails to releng [at] gentoo from the
>> various build boxes: dolphin, poseidon, other dev.g.o machines.
>
> So we have some automatic reporting. Can we have a webpage for that,
> or
> a mailing list that people can subscribe to?

Before anyone creates a duplicate bug: http://bugs.gentoo.org/329165 ;)
-Jeremy

> Idea: make python team receive those emails. ;-)
>
>>> I am curious how often stage builds fail (how long can they be
>>> broken until we actually care?)
>>
>> Fairly often, especially in the last couple of months or so. There
>> were some arches that, last I checked, hadn't had
>> any new media in several months.
>
> Do we have some bugs or other actions towards fixing those problems?
>
>> I see messages like this pretty much every day. Releng is
>> understaffed on a few arches, which is why no one has time to track
>> down the errors, fix them, and get the builds completed.
>
> Is it possible to bisect the portage tree and identify a breaking
> commit?
>
> By the way, it would be great to have some kind of continuous
> integration for the portage tree, even something very basic, and
> visible
> to the public.


nightmorph at gentoo

Dec 1, 2010, 1:23 PM

Post #23 of 24 (2648 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

On Wed, 01 Dec 2010 21:58:20 +0100
"Paweł Hajdan, Jr." <phajdan.jr [at] gentoo> wrote:
> So we have some automatic reporting. Can we have a webpage for
> that, or a mailing list that people can subscribe to?

Mailing list: http://bugs.gentoo.org/329165

I have no idea how to go about doing an automated webpage or other
integration with other projects.

> Do we have some bugs or other actions towards fixing those problems?

Not that I know of. AFAIK problems are dealt with on the basis of
"whoever sees the email AND has time to fix it." The Releng team
members in charge of minor arches generally do a pretty good job of
fixing stuff on a timely basis. Our major arches seem to break more
frequently, but receive less TLC. Again, manpower and time issues.

> Is it possible to bisect the portage tree and identify a breaking
> commit?

Beats me. Git would make that easier. From what I can see, anything
that's ever been done to Python has resulted in breakage.
Attachments: signature.asc (0.19 KB)


jmbsvicetto at gentoo

Dec 1, 2010, 4:19 PM

Post #24 of 24 (2650 views)
Permalink
Re: Disabling auto-bumping of active Python version [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01-12-2010 19:13, Alec Warner wrote:
> On Tue, Nov 30, 2010 at 8:02 PM, Jorge Manuel B. S. Vicetto
> <jmbsvicetto [at] gentoo> wrote:
> On 29-11-2010 10:34, Sebastian Pipping wrote:
>>>> On 11/29/10 09:35, Arfrever Frehtes Taifersar Arahesis wrote:
>>>>> There will probably be no active version of Python set.
>>>>
>>>> You had two weeks to come up with this.
>>>>
>>>> Please find my on IRC to team up on an agreed fix.
>
> As Arfrever noted, this is likely the cause of the broken automated
> weekly stages for this past week. By not having a python symlink /
> wrapper, stages generation failed on stage2 run.
> I'd like to take this chance to recall this is the 2nd time on the last
> few months where stage generation was broken by python changes. Also,
> we've been unable to create hardened stages for over 8 weeks because of
> a sandbox issue.
> The weekly stages generation depends on the quality and stability of the
> "stable" tree. Therefore, the RelEng team kindly asks all maintainers to
> pay attention to the stable ebuilds in the system set and to please fix
> any failures asap as they may / can prevent stage generation. Be sure to
> think carefully about changes that can impact the stage generation, in
> particular when they involve python.
>
>> Two issues:
>
>> proj/en/releng is old as hell and doesn't even mention stage generation.

Yeah, as everywhere else in Gentoo we could use more man power in RelEng.

>> How does a developer know when the stage generation is broken? Is
>> there a dashboard? At work we have a guy who is basically a build cop
>> and checks our build dashboard once a day or so and if it is broken he
>> goes and finds the guy who broke it and punches him in the face until
>> he fixes it. I imagine we do not have staff for this (and no one has
>> invented punching over the internet.)

Lately I've become that guy for the amd64 and x86 builds. I've been
going after maintainers to get the issues that are breaking stages
fixed. I haven't tried "punching" anyone yet, though ;-)

>> I am curious how often stage builds fail (how long can they be broken
>> until we actually care?)

They've been failing a bit in the past 6 months or so. We have some
issues that affect all arches and others that affect specific arches.
The worst and longest issue in the recent past was the mess with
python-3.1 that lead to catalyst to die when python3 was becoming the
main interpreter in stage1. We lost stages for a few months before we
acted on that one.
The previous issue that still affected this weeks stages build was the
issue with sandbox that prevented any stages from being built for the
past 2 months.
This last issue with python was first reported for the ppc builds on
20101128. I only found out yesterday when I was poked by Raúl (armin76)
as I missed all Gentoo emails for the previous 10 days.
For random issues I generally start worrying on the 2nd email.

- --
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJM9uYXAAoJEC8ZTXQF1qEPJkcP/jMau8sJ5UObjIGrGHeXODgB
k76hNPm5j64jjqiQnqByevukIYQ0VfZhEbCcvRRdlMRzVfDoukmDx26372bZVO8d
VkkS5JUmWxVR392W9flZqJw85DlrOB4p7HlxfjamgcsCxzrKVkKAqVDQcQkDGQOB
qCOeDfLlijWOuc1bLMDUsQo1dYCr1XSKvsuY37lC6oQf5oCHr5a1+4xdhto1EbGr
ASq3O9aKu/J+YLGI7rZtet5Lm8kAS803j6DLz0t15/I8obHzx+CeIzFLyr9Bly72
yL9DqU3aQ7xGFEI/GvCcS0vEOiile1VAT/YDLJAE3e24fs6r7Bf6PD1bUXoTiuAR
tOGEJNwpj0Pa5lQoEs7Xe/xaY7W3YkoHe5QmyA8MNa2VyDpoMEm9HJkNH8IumyIn
to68EOc4lHLHCYnm3CFa3C83jP6l3oq04VRlrs/D+geDP2rdTCCXvU8eg8O+c1NV
buca8gK7iuA1sqHZWmInKSKp679Suw82rh7AxPJgXPc/YOzinhtBcMad3XmGmU4g
XdoAXm5VWmO7/fj7ssCK9lzAWGn9ZaDKqGSl3rgpDkSTn7p7/Pih3OCcdcwoI/Gp
uoJ/QD0eI8KAn6OTMN6UWr/i6P/Ys58m7dStKODsyboLejjtZMANbu1KiUKuumix
AYLQ86te5Evz78MaFxeb
=Yo6Y
-----END PGP SIGNATURE-----

Gentoo dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.