Mailing List Archive: Python: Dev

Re: sharing stdlib across python implementations

Index | Next | Previous | View Threaded

chris at simplistix

Sep 30, 2009, 7:28 AM

Post #1 of 6 (621 views)
Permalink

Re: sharing stdlib across python implementations

Frank Wierzbicki wrote:
> Talk has started up again on the stdlib-sig list about finding a core
> stdlib + tests that can be shared by all implementations, potentially
> living apart from CPython. I have volunteered to put together a PEP
> on the subject, with Jessie Noller and Brett Canon are helping me out.
> When I have something worth showing, I'll start the real PEP process.

I'm on on stdlib-sig and I'm afraid I don't have the bandwidth to start
on it, but I'd just like to throw in (yet again) that it would be great
if the stdlib was actually a set of separate python packages with their
own version metadata so that packaging tools could manage them, and
upgrade them independently of python packages when there are bug fixes.
If that were the case, then pure python packages in the stdlib, of which
there are many, *really* could be used across python implementations
with no changes whatsoever...

The big changes I can see from here would be moving the tests to the
packages from the central tests directory, and adding a setup.py file or
some other form of metadata providion for each package. Not that big now
that I've written it ;-)

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Sep 30, 2009, 7:43 AM

Post #2 of 6 (581 views)
Permalink

Re: sharing stdlib across python implementations [In reply to]

Chris Withers <chris <at> simplistix.co.uk> writes:
>
> I'm on on stdlib-sig and I'm afraid I don't have the bandwidth to start
> on it, but I'd just like to throw in (yet again) that it would be great
> if the stdlib was actually a set of separate python packages with their
> own version metadata so that packaging tools could manage them, and
> upgrade them independently of python packages when there are bug fixes.

This sounds like a bad idea to me. Each Python release is tested and debugged as
a whole. If you have a lot of possible combinations (module A version 1.1 with
module B version 1.2, etc.), it becomes impossible for us to ensure proper QA
for the whole and as a result the quality might become lower, rather than higher.

(of course, if we rigorously enforce APIs and preserve compatibility, this might
not be a real issue; but our compatibility story is a bit irregular, IMHO)

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

chris at simplistix

Sep 30, 2009, 7:45 AM

Post #3 of 6 (575 views)
Permalink

Re: sharing stdlib across python implementations [In reply to]

Antoine Pitrou wrote:
> This sounds like a bad idea to me. Each Python release is tested and debugged as
> a whole. If you have a lot of possible combinations (module A version 1.1 with
> module B version 1.2, etc.), it becomes impossible for us to ensure proper QA
> for the whole and as a result the quality might become lower, rather than higher.

I'd certainly still see there being "blessed" python releases.
(The Zope community does this in the form of "known good sets" and
these, for me, would be "the python releases")

However, it's pretty frustrating to not be able to upgrade a pure-python
package with a critical bug (as happened recently to me with httplib)
because it's part of the stdlib.

It's also frustrating to have to if-then-else about a package or
project's dependencies when using dependency management tools like
buildout when libraries such as elementtree and argparse become part of
the stdlib when they weren't before...

> (of course, if we rigorously enforce APIs and preserve compatibility, this might
> not be a real issue; but our compatibility story is a bit irregular, IMHO)

Breaking it apart like this would *make* the compatibility story better...

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

olemis at gmail

Sep 30, 2009, 8:18 AM

Post #4 of 6 (572 views)
Permalink

Re: sharing stdlib across python implementations [In reply to]

On Wed, Sep 30, 2009 at 9:28 AM, Chris Withers <chris [at] simplistix> wrote:
> Frank Wierzbicki wrote:
>>
>> Talk has started up again on the stdlib-sig list about finding a core
>> stdlib + tests that can be shared by all implementations, potentially
>> living apart from CPython.
>
[...]
>
> if the
> stdlib was actually a set of separate python packages with their own version
> metadata so that packaging tools could manage them, and upgrade them
> independently of python packages when there are bug fixes. If that were the
> case, then pure python packages in the stdlib, of which there are many,
> *really* could be used across python implementations with no changes
> whatsoever...
>

nice ! That's something I really liked about Python.NET

:)

BTW ... is there something like that for Java ? I mean to use J2[SE]E
classes using CPython ?

This could also be useful to have personalized distributions. I mean,
if I want to implement a Py app that will run in devices with limited
capabilities, and let's say that it only imports `sockets` module (or
a few more ;o), then it will be easier to prepare a subset of stdlib
in order to deploy just what is needed in such devices, and save some
space ;o).

Projects like py2exe or others, could use something like that in order
to extract relevant stdlib (modules | packages) and make them
available to Windows apps distributed as exe files (e.g. Hg )

CMIIW anyway, perhaps I'm just dreaming.

however ...

> The big changes I can see from here would be moving the tests to the
> packages from the central tests directory, and adding a setup.py file or
> some other form of metadata providion for each package. Not that big now
> that I've written it ;-)
>

In this case I envision the following issues if one setup.py file is
generated for every module or top-level package (... which is
-considering the previous message- how u plan to do it, isn't it ? )

- the maintenance effort might increase
- what about dependencies between stdlib modules ?
- there are many attributes which will take the same values for each
and every
packages (e.g. version info, issue tracker, ...) and some that
will be specific
(e,g, maintainer, author, contact info, dependencies ...)
- Overhead when a new package is included in stdlib (need to create and
maintain `setup.py` script, and so on ...)

So my $0.02 here are :

- to have a single `setup.py` file (in the end it's a script, isn't it ? )
- provide an argument in order to select module(s) to be included
(full stdlib if missing) in source distribution. In general any other
parameterization of the `setup.py` may be just ok, the goal is
to have only one
- use a mechanism in order to specify config options for specific pkgs
modules, and make it available to the global `setup.py`. For example :
* Specify metadata using top-level fields in modules (e.g. __author__,
__maintainer__, ...)
* Specify metadata using separate INI files for each target

What d'u think ?

There may be some issues with sdist anyway
:-/

PS: Will those packages be submitted to PyPI too ? I mean if not
sdists, at least meta-data ?

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Sobrepasa las 300 descargas el m�dulo dutest -
http://feedproxy.google.com/~r/simelo-es/~3/U5rff5iTQPI/sobrepasa-las-300-descargas-el-modulo.html
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

olemis at gmail

Sep 30, 2009, 8:37 AM

Post #5 of 6 (572 views)
Permalink

Re: sharing stdlib across python implementations [In reply to]

On Wed, Sep 30, 2009 at 9:43 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
> Chris Withers <chris <at> simplistix.co.uk> writes:
>>
>> I'm on on stdlib-sig and I'm afraid I don't have the bandwidth to start
>> on it, but I'd just like to throw in (yet again) that it would be great
>> if the stdlib was actually a set of separate python packages with their
>> own version metadata so that packaging tools could manage them, and
>> upgrade them independently of python packages when there are bug fixes.
>
> This sounds like a bad idea to me. Each Python release is tested and debugged as
> a whole. If you have a lot of possible combinations (module A version 1.1 with
> module B version 1.2, etc.), it becomes impossible for us to ensure proper QA
> for the whole and as a result the quality might become lower, rather than higher.
>

You are right here !

-1 for splitting the test code and test suite

but I still think it could be a good idea ...

> (of course, if we rigorously enforce APIs and preserve compatibility,

-1

> this might
> not be a real issue; but our compatibility story is a bit irregular, IMHO)
>

mainly because of this ;o)

But it's still possible to use a parameterized `setup.py` and leave
things just like they are right now.

For instance, I have started something like that has been dome by the
FLiOOPS project [1]_ (and possibly others, it's just an example ;o).
In the SVN repository there's a single trunk containing the whole
PyOOP package (which is not complete BTW). Inside of it there are
separate `setup.py` files for other distributions too :

- `PyOOP` (which should contain them all) employs default `setup.py`
- `dutest` is a single file (there are more in there) under
`oop.utils` package and
is distributed separately too, so you can find in there `setup_dutest.py`
script. All other packages rely on it to build test suites ;o)
- CASPy (Comprehensive Assertion Support for Python) should be distributed
separately too, so it should have its own `setup_dbc.py` script
once it's ready.

When I create (SVN) tags for each package, I have to rename it
(extra-overhead), and I once global metadata changes, then I have to
change them all

PS: Another alternative for stdlib could be to have common data in
`setup.cfg` and separate `setup.py` scripts, but I don't really like
this one ...

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Sobrepasa las 300 descargas el m�dulo dutest -
http://feedproxy.google.com/~r/simelo-es/~3/U5rff5iTQPI/sobrepasa-las-300-descargas-el-modulo.html
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

olemis at gmail

Sep 30, 2009, 8:51 AM

Post #6 of 6 (572 views)
Permalink

Re: sharing stdlib across python implementations [In reply to]

On Wed, Sep 30, 2009 at 10:37 AM, Olemis Lang <olemis [at] gmail> wrote:
> On Wed, Sep 30, 2009 at 9:43 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
>> Chris Withers <chris <at> simplistix.co.uk> writes:
>>>
[...]
>
> For instance, I have started something like that has been dome by the
> FLiOOPS �project [1]_

Sorry for the missing reference and typos, should be

For instance, something like that has been done by the FLiOOPS �project [1]_

.. [1] py trunk @ FLiOOPS @ sf.net
(http://sourceforge.net/apps/trac/flioops/browser/py/trunk)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Sobrepasa las 300 descargas el m�dulo dutest -
http://feedproxy.google.com/~r/simelo-es/~3/U5rff5iTQPI/sobrepasa-las-300-descargas-el-modulo.html
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads