Mailing List Archive: Python: Dev

Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

1 2

View All

Index | Next | Previous | View Threaded

ziade.tarek at gmail

Oct 8, 2009, 1:31 AM

Post #1 of 44 (1409 views)
Permalink

Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

Here's a quick summary of the main things that are going to happen in
Distutils, and Distribute, and a few words on virtualenv and pip.
(there is much much more work going on, but I don't want to drown
people with details)

= Distutils =

Distutils is a package manager and competes with OS package managers.
This is a good thing because, unless you are developing a library or
an application that will only run one specific system that has its own
packaging system like Debian, you will be able to reach much more
people. Of course the goal is to avoid making the work of a Debian
packager (or any other OS that has a package manager) too hard. In
other words, re-packaging a Distutils-based project should be easy and
Distutils should not get in their way (or as less as possible).

But right now Distutils is incomplete in many ways and we are trying to fix'em.

== What's installed ? what's the installation format ? how to uninstall ? ==

First, it's an incomplete package manager : you can install a
distribution using it, but there's no way to list installed
distributions. Worst, you can't uninstall a distribution.

PEP 376 resolves this, and once it's finished, the goal is to include
the APIs described there into Distutils itself and into the pkgutil
module in stdlib. Notice that there's an implementation at
http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376
so people can see what we are talking about.

Another problem that popped during the last years is the fact that, in
the same site-packages, depending on the tool that was used to install
a Distribution, and depending if this distribution uses Distutils or
Setuptools, you can have different installation formats.

End-users end up with zipped eggs (one file), unzipped eggs (one
self-contained format in a directory) and regular Distutils (packages
and modules in site-packages). And the Metadata are also located in
many different places depending on the installation format used.

That can't be. there's no point to keep various installation format in
the *same* site-packages directory.

PEP 376 also resolves this by describing a *unique* format that works
for all. Once this is finished, Distutils will implement it by
changing the install command accordingly.

- Work left to do in PEP 376 : restrict its scope to a disk-based,
file-based site-packages.
- Goal: 2.7 / 3.2

== Dependencies ==

The other feature that makes a packaging system nice is dependencies.
e.g. a way to list in a distribution, the distributions it requires to
run. As a matter of fact, PEP 314 has introduced in the Metadata new
fields for this purpose ("Requires", "Provides and "Obsoletes"). So,
you can write things like "Requires: lxml >= 2.2.1", meaning that your
distribution requires lxml 2.2.1 or a newer version to run. But this
was just description fields and Distutils was not providing any
feature based on these new fields.

In fact, no third-party tool either provided a feature based on those
fields. Setuptools provided "easy_install" a script that looks for the
dependencies and install them, by querying the Python Package Index
(PyPI). But this feature was implemented with its own metadata: you
can add an "install_requires" option in the setup() call in setup.py,
and it will end up in a "requires.txt" file at installation time that
is located alongside the Metadata for you distribution.

So the goal is to review PEP 314 and update the Metadata w.r.t. the
setuptools feedback and community usage. Once it's done, Distutils
will implement this new metadata version and promote its usage.
Promoting its usage means that Distutils will provide some APIs to
work with these APIs, like a version comparison algorithm.

And while we're at it, we need to work out some inconsistency with the
"Author" and "Maintainer" fields. (The latter doesn't exists in the
Metadata but exists on setup.py side).

- Work left to do in PEP 314 : finish PEP 386, finish the discussion
on the "maintainer" field.
- Goal: 2.7 / 3.2

== Version comparison ==

Once you provide dependency fields in the metadata, you need to
provide a version scheme: a way to compare two versions. Distutils has
two version comparison algorithms that are not used in its code and in
only one place in the stdlib where it could be removed with no pain.
One version scheme is "strict" and one is "loose". And Setuptools has
another one, which is more heuristic (it will deal with any version
string and compare it, wether it's wrong or not).

PEP 386 goal is to describe a version scheme that can be used by all
and if we can meet a consensus there, we can move on and add
it as a reference in the update done in PEP 314, besides the
dependencies fields. Then, in Distutils we can deprecate the existing
version
comparison algorithms and provide a new one based on PEP 386 and
promote its usage.

One very important point: we will not force the community to use the
scheme described in PEP 386, but *there is* already a
de-facto convention on version schemes at PyPI if you use Pip or
easy_install, so let's have a documented standard for this,
and a reference implementation in Distutils.

There's an implementation at
http://bitbucket.org/tarek/distutilsversion that is kept up-to-date
with PEP 386.

- Work left to do in PEP 386 : another round with the community
- Goal: 2.7 / 3.2

== The fate of setup.py, and static metadata ==

Setup.py is a CLI to create distribution, install them etc. You can
also use it to retrieve the metadata of a distribution. For
example you can call "python setup.py --name" and the name will be
displayed. That's fine. That's great for developers.

But there's a major flaw: it's Python code. It's a problem because,
depending on the complexity of this file, an OS packager that just
wants to get the metadata for the platform he's working on, will run
arbitrary code that mught do unwanted things (or even that light not
work)

So we are going to separate the metadata description from setup.py, in
a static configuration file, that can be open and read by anyone
without
running any code. The only problem with this is the fact that some
metadata fields might depend on the execution environment. For
instance, once "Requires" is re-defined and re-introduced via PEP 314,
we will have cases where "pywin32" will be a dependency to have only
on win32 systems.

So we've worked on that lately in Distutils-SIG and came up with a
micro-language, based on a ConfigParser file, that allows
writing metadata fields that depends on sys.platform etc. I won't
detail the syntax here but the idea is that the interpretation
of this file can be done with a vanilla Python without running arbitrary code.

In other words : we will be able to get the metadata for a
distribution without having to install it or to run any setup.py
command.
One use case is the ability to list all dependencies a distribution
requires for a given platform, just by querying PyPI.

So I am adding this in Distutils for 2.7.

Of course setup.py stays, and this is backward compatible.

- Work left to do : publish the final syntax, and do the implementation
- Goal: 2.7 / 3.2

== The fate of bdist_* commands ==

During last Pycon summit we said that we would remove commands like
bdist_rpm because Python is unable, due to its release cycle,
to do a good work there. Here's an example: I have from time to time
cryptic issues in the issue tracker from people from Fedora (or any
rpm-based system), and I have all the pain in the world for these very
specific problems to do the proper fix unless some RPM expert helps
around. And by the time it's detected then fixed, it can be year(s)
before it's available on their side. That's why, depending on the
communities, commands like bdist_rpm are just totally ignored, and OS
packager have their own tools.

So the best way to handle this is to ask these communities to build
their own tool and to encourage them to use Distutils as a basis for
that.

This does not concern bdist_* commands for win32 because those are
very stable and don't change too much: Windows doesn't have a package
manager that would require these commands to evolve with it.

Anyways, when we said that we would remove bdist_rpm, this was very
controversial because some people use it and love it.

So what is going to happen is a status-quo: no bdist_* command will be
removed but no new bdist_* command wil be added. That's why I've
encouraged Andrew and Garry, that are working on a bdist_deb command,
to keep it in the "stdeb" project, and eventually we will
refer to it in the Distutils documentation if this bdist_deb comply
with Distutils standard. It doesn't right now because it uses a
custom version of the Distribution class (through Setuptools) that
doesn't behave like Distutils' one anymore.

For Distutils, I'll add some documentation explaining this, and a
section that will list community-driven commands.

- Work left to do : write the documentation
- Goal: 2.7 / 3.2

= Distribute =

I won't explain here again why we have forked, I think it's obvious to
anyone here now. I'll rather explain what
we are planning in Distribute and how it will interact with Distutils.

Distribute has two branches:

- 0.6.x : provides a Setuptools-0.6c9 compatible version
- 0.7.x : will provide a refactoring

== 0.6.x ==

Not "much" is going to happen here, we want this branch to be helpful
to the community *today* by addressing the 40-or-so bugs
that were found in Setuptools and never fixed. This is eventually
happen soon because its development is
fast : there are up to 5 commiters that are working on it very often
(and the number grows weekly.)

The biggest issue with this branch is that it is providing the same
packages and modules setuptools does, and this
requires some bootstrapping work where we make sure once Distribute is
installed, all Distribution that requires Setuptools
will continue to work. This is done by faking the metadata of
Setuptools 0.6c9. That's the only way we found to do this.

There's one major thing though: thanks to the work of Lennart, Alex,
Martin, this branch supports Python 3,
which is great to have to speed up Py3 adoption.

The goal of the 0.6.x is to remove as much bugs as we can, and try if
possible to remove the patches done
on Distutils. We will support 0.6.x maintenance for years and we will
promote its usage everywhere instead of
Setuptools.

Some new commands are added there, when they are helpful and don't
interact with the rest. I am thinking
about "upload_docs" that let you upload documentation to PyPI. The
goal is to move it to Distutils
at some point, if the documentation feature of PyPI stays and starts to be used.

== 0.7.x ==

We've started to refactor Distribute with this roadmap in mind (and
no, as someone said, it's not vaporware,
we've done a lot already)

- 0.7.x can be installed and used with 0.6.x

- easy_install is going to be deprecated ! use Pip !

- the version system will be deprecated, in favor of the one in Distutils

- no more Distutils monkey-patch that happens once you use the code
(things like 'from distutils import cmd; cmd.Command = CustomCommand')

- no more custom site.py (that is: if something misses in Python's
site.py we'll add it there instead of patching it)

- no more namespaced packages system, if PEP 381 (namespaces package
support) makes it to 2.7

- The code is splitted in many packages and might be distributed under
several distributions.

- distribute.resources: that's the old pkg_resources, but
reorganized in clean, pep-8 modules. This package will
only contain the query APIs and will focus on being PEP 376
compatible. We will promote its usage and see if Pip wants
to use it as a basis. And maybe PyPM once it's open source ?
(<hint> <hint>).
It will probably shrink a lot though, once the stdlib provides PEP 376 support.

- distribute.entrypoints: that's the old pkg_resources entry points
system, but on its own. it uses distribute.resources

- distribute.index: that's package_index and a few other things.
everything required to interact with PyPI. We will promote
its usage and see if Pip wants to use it as a basis.

- distribute.core (might be renamed to main): that's everything
else, and uses the other packages.

Goal: A first release before (or when) Python 2.7 / 3.2 is out.

= Virtualenv and the multiple version support in Distribute =

(I am not saying "We" here because this part was not discussed yet
with everyone)

Virtualenv allows you to create an isolated environment to install
some distribution without polluting the
main site-packages, a bit like a user site-packages.

My opinion is that this tool exists only because Python doesn't
support the installation of multiple versions for the same
distributions.
But if PEP 376 and PEP 386 support are added in Python, we're not far
from being able to provide multiple version support with
the help of importlib.

Setuptools provided a multiple version support but I don't like its
implementation and the way its works.
I would like to create a new site-packages format that can contains
several versions of the same distribution, and :

- a special import system using importlib that would automatically
pick the latest version, thanks to PEP 376.
- an API to force at runtime a specific version (that would be located
at the beginning of all imports, like __future__)
- a layout that is compatible with the way OS packagers works with
python packages

Goal: a prototype asap (one was started under the "VSP" name (virtual
site-packages) but not finished yet)

Regards
Tarek

--
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Oct 8, 2009, 1:56 AM

Post #2 of 44 (1367 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

2009/10/8 Tarek Ziad� <ziade.tarek [at] gmail>:
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)

Thanks for this summary. The overview was getting lost in all the details.

> == What's installed ? what's the installation format ? how to uninstall ? ==
>
> First, it's an incomplete package manager : you can install a
> distribution using it, but there's no way to list installed
> distributions. Worst, you can't uninstall a distribution.
>
> PEP 376 resolves this, and once it's finished, the goal is to include
> the APIs described there into Distutils itself and into the pkgutil
> module in stdlib. Notice that there's an implementation at
> http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376
> so people can see what we are talking about.
>
> Another problem that popped during the last years is the fact that, in
> the same site-packages, depending on the tool that was used to install
> a Distribution, and depending if this distribution uses Distutils or
> Setuptools, you can have different installation formats.
>
> End-users end up with zipped eggs (one file), unzipped eggs (one
> self-contained format in a directory) and regular Distutils (packages
> and modules in site-packages). And the Metadata are also located in
> many different places depending on the installation format used.
>
> That can't be. there's no point to keep various installation format in
> the *same* site-packages directory.
>
> PEP 376 also resolves this by describing a *unique* format that works
> for all. Once this is finished, Distutils will implement it by
> changing the install command accordingly.
>
> - Work left to do in PEP 376 : restrict its scope to a disk-based,
> file-based site-packages.
> - Goal: 2.7 / 3.2

+1

This addresses my biggest concern with the way PEP 376 was going
(largely prompted by my suggestions, so I'll take the blame for this!)
which was that PEP 376 was trying to define list/uninstall methods for
package formats (i.e. zip files/eggs) that it couldn't install itself.

I presume that when you refer to the restriction in scope, you do mean
that PEP 375 will lose all support for zip files, as well as the
generalised PEP 302 support I proposed? I still believe that
special-casing zip files is wrong - PEP 302 deliberately abstracted
the import mechanisms and built zipfile support on top of that, and I
still believe that's a good thing. But expanding PEP 302 to cover the
requirements of PEP 376 looked more and more inappropriate the more I
wrote code to do it - what would be needed would be a *new* protocol
on top of which zip-based formats could be layered. I'm quite happy
for that to be outside the scope of PEP 376, though, as distutils
itself doesn't support any such formats.

The egg format

One thing missing from your roadmap (unless I missed it) is the fate
of the egg (zipfile) format. If it's to remain a valid option
(bdist_egg and the like) then I'd assume that Distribute needs to be
the place to develop it. One thing it will need is an equivalent to
PEP 376 introspection and uninstall features. And *that* is where I
believe a standard protocol belongs - developed based on the real use
requirements for egg files, with a suitable file-based layer which can
be added into distutils (somewhat like importlib does for PEP 302)
once the protocol is agreed and standardised.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ziade.tarek at gmail

Oct 8, 2009, 2:11 AM

Post #3 of 44 (1366 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 10:56 AM, Paul Moore <p.f.moore [at] gmail> wrote:
>
> +1
>
> This addresses my biggest concern with the way PEP 376 was going
> (largely prompted by my suggestions, so I'll take the blame for this!)
> which was that PEP 376 was trying to define list/uninstall methods for
> package formats (i.e. zip files/eggs) that it couldn't install itself.
>
> I presume that when you refer to the restriction in scope, you do mean
> that PEP 375 will lose all support for zip files, as well as the
> generalised PEP 302 support I proposed? I still believe that
> special-casing zip files is wrong - PEP 302 deliberately abstracted
> the import mechanisms and built zipfile support on top of that, and I
> still believe that's a good thing. But expanding PEP 302 to cover the
> requirements of PEP 376 looked more and more inappropriate the more I
> wrote code to do it - what would be needed would be a *new* protocol
> on top of which zip-based formats could be layered. I'm quite happy
> for that to be outside the scope of PEP 376, though, as distutils
> itself doesn't support any such formats.

Yes exactly so.

>
> The egg format
>
> One thing missing from your roadmap (unless I missed it) is the fate
> of the egg (zipfile) format. If it's to remain a valid option
> (bdist_egg and the like) then I'd assume that Distribute needs to be
> the place to develop it.
> One thing it will need is an equivalent to
> PEP 376 introspection and uninstall features. And *that* is where I
> believe a standard protocol belongs - developed based on the real use
> requirements for egg files, with a suitable file-based layer which can
> be added into distutils (somewhat like importlib does for PEP 302)
> once the protocol is agreed and standardised.

Distribute is planning to continue supporting the egg format, but will
probably drop the zipped version.
This is still under discussions though so don't take this as the final word.

In any case, distribute.resources will probably add the missing bits
on the top of a PEP 376 -compliant Distutils
to provide support for any file querying Distutils will not provide
(for installation and uninstallation)

Tarek
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

cournape at gmail

Oct 8, 2009, 2:42 AM

Post #4 of 44 (1368 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 5:31 PM, Tarek Ziadé <ziade.tarek [at] gmail> wrote:

> = Virtualenv and the multiple version support in Distribute =
>
> (I am not saying "We" here because this part was not discussed yet
> with everyone)
>
> Virtualenv allows you to create an isolated environment to install
> some distribution without polluting the
> main site-packages, a bit like a user site-packages.
>
> My opinion is that this tool exists only because Python doesn't
> support the installation of multiple versions for the same
> distributions.

I am really worried about this, because it may encourage people to use
multiple versions as a bandaid to maintaining backward compatibility.
At least with virtual-env, the problem is restricted to the user.

Generalized multiple, side by side installation has been tried in
many different contexts, and I have never seen a single one working
and not bringing more problems that it solved. One core problem being
the exponential number of combination (package A depends on B and C, B
depends on one version of D, C on another version of D). Being able to
install *some* libraries in multiple versions is OK, but generalizing
is very dangerous IMHO.

David
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ziade.tarek at gmail

Oct 8, 2009, 2:59 AM

Post #5 of 44 (1363 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 11:42 AM, David Cournapeau <cournape [at] gmail> wrote:
> On Thu, Oct 8, 2009 at 5:31 PM, Tarek Ziad� <ziade.tarek [at] gmail> wrote:
>
>> = Virtualenv and the multiple version support in Distribute =
>>
>> (I am not saying "We" here because this part was not discussed yet
>> with everyone)
>>
>> Virtualenv allows you to create an isolated environment to install
>> some distribution without polluting the
>> main site-packages, a bit like a user site-packages.
>>
>> My opinion is that this tool exists only because Python doesn't
>> support the installation of multiple versions for the same
>> distributions.
>
> I am really worried about this, because it may encourage people to use
> multiple versions as a bandaid to maintaining backward compatibility.
> At least with virtual-env, the problem is restricted to the user.
>
> Generalized multiple, side by side installation �has been tried in
> many different contexts, and I have never seen a single one working
> and not bringing more problems that it solved. One core problem being
> the exponential number of combination (package A depends on B and C, B
> depends on one version of D, C on another version of D). Being able to
> install *some* libraries in multiple versions is OK, but generalizing
> is very dangerous IMHO.

The goal here is to be able to have a *single* instance of "Foo 1.2" on your
system, managed by the admin sys. that can be shared and used for
several python
apps. And yes, I don't expect to have so many different versions of
many distributions
on a system, but just to provide a mechanism to allow it.

But that's just an idea we had with some people, and it still requires
a lot of work and brainstroming (even if the prototype kinda work)

I should've mentioned that I've added it at the end of the roadmap
because I had the feeling that something has to be done to reunite
zc.buildout, virtualenv and Python one day.

But consider this as the topic for the "fun sprint" we would have at
Pycon, after we will have issued other points. If you are interested
in this particular topic, I propose that we continue this discussion
on distutils-SIG.

Regards
Tarek
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

hodgestar+pythondev at gmail

Oct 8, 2009, 5:52 AM

Post #6 of 44 (1357 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziadé <ziade.tarek [at] gmail> wrote:
> = Virtualenv and the multiple version support in Distribute =
...
> My opinion is that this tool exists only because Python doesn't
> support the installation of multiple versions for the same
> distributions.

This is not at all how I use virtualenv. For me virtualenv is a
sandbox so that I don't have to become root whenever I need to install
a Python package for testing purposes and to allow me to hop between
sets of installed Python packages while developing on multiple Python
projects. I also use it as a sandbox for build bots so that multiple
bots on the same machine can each build their own projects with just
the known dependencies installed.

Schiavo
Simon
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Oct 8, 2009, 5:56 AM

Post #7 of 44 (1349 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

2009/10/8 Tarek Ziad� <ziade.tarek [at] gmail>:
>> The egg format
>>
>> One thing missing from your roadmap (unless I missed it) is the fate
>> of the egg (zipfile) format. If it's to remain a valid option
>> (bdist_egg and the like) then I'd assume that Distribute needs to be
>> the place to develop it.
>> One thing it will need is an equivalent to
>> PEP 376 introspection and uninstall features. And *that* is where I
>> believe a standard protocol belongs - developed based on the real use
>> requirements for egg files, with a suitable file-based layer which can
>> be added into distutils (somewhat like importlib does for PEP 302)
>> once the protocol is agreed and standardised.
>
> Distribute is planning to continue supporting the egg format, but will
> probably drop the zipped version.
> This is still under discussions though so don't take this as the final word.
>
> In any case, distribute.resources will probably add the missing bits
> on the top of a PEP 376 -compliant Distutils
> to provide support for any file querying Distutils will not provide
> (for installation and uninstallation)

[.Note: For the purposes of below I use "egg format" to only refer to
zipped eggs. I see no value AT ALL in having an unzipped "egg format"
which is almost the same as, but subtly different from, the
distutils-supported standard filesystem format. If the distutils
standard doesn't provide for all the requirements, fix that, don't add
another variation!]

I'm not sure I follow this. I think it's important to provide for
non-filesystem based packages (that's the whole point of things like
PEP 302 and importlib!). The fact that core distutils doesn't support
installing, querying or removing such packages is fine. They can be
supplied by 3rd party systems like Distribute.

However, it is *not* fine (in my view) for each independent format to
have to implement its own set of interfaces. PEP 302 is precisely the
solution for this in terms of import - every importer has a
standardised set of protocols to follow, and if it does so, then it
will (should :-)) work seamlessly with any Python code. PEP 302 is not
perfect, in part because it aims to be minimal rather than
comprehensive, hence the fact that not all packages are "zip safe".
But that's a failing in PEP 302 rather than in the concept.

Multiple package formats should be supported in the same way - with a
set of well-defined (via a PEP) interfaces that all package formats
must provide, which enable core and user code to be written in a
format-independent way. Once such a PEP is written, distutils' PEP 376
support should be rewritten to use those interfaces, and to implement
the necessary filesystem-format support layer. (In other words,
whereas I support reducing PEP 376's scope to filesystem only for now,
I still strongly advocate that the longer-term solution should be
based on a user extensible format-independent approach).

In this context, eggs are "merely" the first (and most important)
example of a format extension, and so should drive the development of
a standard.

To summarise:

I believe that we need a statement of direction on the (zipped) egg
format. I see a number of options:

1. Retain it as-is in Distribute, but deprecated and no further development.
2. Deprecate egg support in Distribute, and ultimately drop it.
3. Develop into a reference example of an extensible architecture for
adding to PEP 376 support.
4. Extend PEP 376 support to eggs by means of egg-specific approaches
hooking into distutils.

For the record, I dislike (1), I see (2) as a lost opportunity, I see
(3) as the right way to go, but appreciate that it's the most work,
and I strongly oppose (4) (not least because it effectively makes a
latrge part of PEP 302 and importlib wasted effort).

It looks like you're proposing (2), but aren't entirely clear because
you're still allowing for non-zipped "egg" formats (which I see no
value in, as I noted above).

I'm willing to help work on option (3). Once it's decoupled from
questions around OS package managers (which are more the province of
PEP 376) then I'd hope that my lack of knowledge about package
managers is less of a problem.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

fuzzyman at voidspace

Oct 8, 2009, 6:52 AM

Post #8 of 44 (1356 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Simon Cross wrote:
> On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziadé <ziade.tarek [at] gmail> wrote:
>
>> = Virtualenv and the multiple version support in Distribute =
>>
> ...
>
>> My opinion is that this tool exists only because Python doesn't
>> support the installation of multiple versions for the same
>> distributions.
>>
>
> This is not at all how I use virtualenv. For me virtualenv is a
> sandbox so that I don't have to become root whenever I need to install
> a Python package for testing purposes and to allow me to hop between
> sets of installed Python packages while developing on multiple Python
> projects. I also use it as a sandbox for build bots so that multiple
> bots on the same machine can each build their own projects with just
> the known dependencies installed.
>
>

This is exactly why I use virtualenv as well. I don't recall ever having
wanted / needed to install multiple versions of the same library -
whilst I can appreciate that it *can* be a real issue it has never been
a problem for me.

Michael

> Schiavo
> Simon
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>

--
http://www.ironpythoninaction.com/

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Oct 8, 2009, 7:30 AM

Post #9 of 44 (1358 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Oct 8, 2009, at 4:31 AM, Tarek Ziad� wrote:

> - no more namespaced packages system, if PEP 381 (namespaces package
> support) makes it to 2.7

Make that PEP 382:

http://www.python.org/dev/peps/pep-0382/

-Barry

Attachments:

PGP.sig (0.81 KB)

mal at egenix

Oct 8, 2009, 7:55 AM

Post #10 of 44 (1366 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Tarek Ziad� wrote:
> On Thu, Oct 8, 2009 at 1:27 PM, M.-A. Lemburg <mal [at] egenix> wrote:
>>
>>>> == The fate of setup.py, and static metadata ==
>>>>
>>>> So we are going to separate the metadata description from setup.py, in
>>>> a static configuration file, that can be open and read by anyone
>>>> without
>>>> running any code. The only problem with this is the fact that some
>>>> metadata fields might depend on the execution environment. For
>>>> instance, once "Requires" is re-defined and re-introduced via PEP 314,
>>>> we will have cases where "pywin32" will be a dependency to have only
>>>> on win32 systems.
>>
>> How are you planning to deal with dynamic version strings, e.g.
>> say you want to build a snapshot release the current date and
>> revision number included in the version string ?
>>
>> I suppose this file is going to be uploaded to PyPI, so the
>> file would have to be formatted at build time and include
>> markers to be replaced with values taken from the distribution
>> meta-data.
>
> The metadata will always be customizable from setup.py somehow.
>
> We won't treat those cases : if the metadata is provided in the static file,
> you can get it without any further processing, if not you will get an
> "UNKOWN" value
> for it and you will have to grab the whole archive and run setup.py to get it.
>
> Now the example you've shown is in my opinion not a problem : you can
> create the static value for "version" that you put in your static file
> with some pre-processing that
> occurs when you build your release. (ala autoconf)

Hmm, then I'm not really sure I understand the proposal for a
new static meta-data file...

distutils already creates the static meta-data file PKG-INFO
when building sdist distributions and that's done based on the
data read from the setup() call. It would be easy to use that
file as static meta-data file in other distribution formats
as well (e.g. like you proposed in PEP 376).

OTOH, the register command sends the meta-data directly to
the PyPI database, so it doesn't even need another file
for storing away the meta data.

Unless I'm missing something important (which I probably am),
the only added value of the new format over PKG-INFO is the
micro-language.

Couldn't we just stick that into the values of the various
meta-data fields, e.g. to have the Requires field be different
depending on platform ?

We'd then only have one meta-data format in distutils -
the PKG-INFO one.

>>>> - Work left to do : publish the final syntax, and do the implementation
>>
>> Is there a PEP for this ?
>
> No, but if you ask for it I can write it.

Please do. The format will need documentation anyway and the
motivation should be made clear as well.

>>>> == The fate of bdist_* commands ==
>>>>
>>>> So the best way to handle this is to ask these communities to build
>>>> their own tool and to encourage them to use Distutils as a basis for
>>>> that.
>>
>> Rather than having external communities build their own
>> tools and then basically fork distutils in order to get
>> these implemented, I think it'd be better to make the
>> set of distutils commands extensible via namespace packages.
>
> I am not sure, why they would fork distutils.

Not really a fork, but a copy of distutils in order to give
the user the complete tool set in one download, rather than
having to configure a system installed distutils to load
the new command.

> Distutils is already extensible, you can configure in distutils.cfg a
> list of namespace packages,
> that contains commands and they'll get loaded by dist.py.

Right, but dist.py could load all distutils.command modules
it can find to make the installation of extensions more user
friendly (like in a plugin system).

Just a minor nit, though.

>>>> So what is going to happen is a status-quo: no bdist_* command will be
>>>> removed but no new bdist_* command will be added.
>>
>> I hope that comment means "no new bdist_* command will be added
>> *right now*". It's well possible to create new bdist_* commands
>> that don't rely on external tool chains or only on ones that
>> don't change often.
>>
>> I'm thinking of bdist_inno (Windows installer using InnoSetup)
>> and bdist_nsis (Windows installer using NSIS) here.
>
> As I said, windows don't have any packaging system so I don't think
> it's a problem to have more installers for this platform.

Ok.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Oct 08 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jnoller at gmail

Oct 8, 2009, 9:04 AM

Post #11 of 44 (1352 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 9:52 AM, Michael Foord <fuzzyman [at] voidspace> wrote:
> Simon Cross wrote:
>>
>> �On Thu, Oct 8, 2009 at 10:31 AM, Tarek Ziad� <ziade.tarek [at] gmail>
>> wrote:
>>
>>>
>>> = Virtualenv and the multiple version support in Distribute =
>>>
>>
>> ...
>>
>>>
>>> My opinion is that this tool exists only because Python doesn't
>>> support the installation of multiple versions for the same
>>> distributions.
>>>
>>
>> This is not at all how I use virtualenv. For me virtualenv is a
>> sandbox so that I don't have to become root whenever I need to install
>> a Python package for testing purposes and to allow me to hop between
>> sets of installed Python packages while developing on multiple Python
>> projects. I also use it as a sandbox for build bots so that multiple
>> bots on the same machine can each build their own projects with just
>> the known dependencies installed.
>>
>>
>
> This is exactly why I use virtualenv as well. I don't recall ever having
> wanted / needed to install multiple versions of the same library - whilst I
> can appreciate that it *can* be a real issue it has never been a problem for
> me.
>
> Michael

+1 - virtualenv, AFAIK is used for sandboxing/isolation of various
environments, not dealing with multiple versions within the *same*
environment. Of course, it does solve the "being dependent on a
specific version" of a dependency because it *is* sandboxed from
everything else.

Adding multiple version support doesn't remove the need for sandboxing.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ziade.tarek at gmail

Oct 8, 2009, 9:44 AM

Post #12 of 44 (1351 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 4:55 PM, M.-A. Lemburg <mal [at] egenix> wrote:
> OTOH, the register command sends the meta-data directly to
> the PyPI database, so it doesn't even need another file
> for storing away the meta data.
>
> Unless I'm missing something important (which I probably am),
> the only added value of the new format over PKG-INFO is the
> micro-language.

That's one. The other one is to avoid to deal with yet another
configuration file
alongside setup.py, and just deal with a [metadata]�section in setup.cfg.

And PKG-INFO stays a static, platform-specific file that exists only when the
distribution is installed on the target platform.

>> No, but if you ask for it I can write it.
>
> Please do. The format will need documentation anyway and the
> motivation should be made clear as well.

Ok I will

Tarek
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

masklinn at masklinn

Oct 8, 2009, 10:41 AM

Post #13 of 44 (1352 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On 8 Oct 2009, at 19:22 , David Cournapeau wrote:
>
> This is not true - stow solves the problem in a more general way (in
> the sense that it is not restricted to python), at least on platforms
> which support softlink.
I was, of course, talking about "pure" Python solutions (but I should
indeed have qualified my statement thus), in the general cases there
are several solutions to handle that including but not limited to stow
you already mentioned, BSD jails or full-blown virtual machines.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

mal at egenix

Oct 8, 2009, 11:02 AM

Post #14 of 44 (1350 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Tarek Ziad� wrote:
> On Thu, Oct 8, 2009 at 4:55 PM, M.-A. Lemburg <mal [at] egenix> wrote:
>> OTOH, the register command sends the meta-data directly to
>> the PyPI database, so it doesn't even need another file
>> for storing away the meta data.
>>
>> Unless I'm missing something important (which I probably am),
>> the only added value of the new format over PKG-INFO is the
>> micro-language.
>
> That's one. The other one is to avoid to deal with yet another
> configuration file
> alongside setup.py, and just deal with a [metadata] section in setup.cfg.
>
> And PKG-INFO stays a static, platform-specific file that exists only when the
> distribution is installed on the target platform.

Hmm, PKG-INFO is generated from the parameters you pass to setup()
in setup.py, so the package author will not have to deal
with another config file and only has to write down the meta-data
in setup.py.

Being able to add some meta-data via setup.cfg would probably be an
extra feature which some may find handy.

However, your summary made also made it look like the meta-data would
be required in setup.cfg and that the whole file would get uploaded
(including all the options and settings for other distutils commands).

Was that intended ?

>>> No, but if you ask for it I can write it.
>>
>> Please do. The format will need documentation anyway and the
>> motivation should be made clear as well.
>
> Ok I will

Thanks, I'll wait for the PEP.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Oct 08 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

sridharr at activestate

Oct 8, 2009, 12:35 PM

Post #15 of 44 (1354 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, 08 Oct 2009 06:52:57 -0700, Michael Foord
<fuzzyman [at] voidspace> wrote:

> I don't recall ever having wanted / needed to install multiple versions
> of the same library - whilst I can appreciate that it *can* be a real
> issue it has never been a problem for me.

Multiple versions is going to be a mess. It is a pathetic workaround for
packages that do not care about backward compatibility (eg: CherryPy-2.x
vs CherryPy-3.x). Drop support for multiple versions to force package
authors to deal with it.

I applaud the Jinja team for doing this:

ARMIN: The final version of the Jinja2 Django-inspired template
engine was just released. It comes with tons of improvements over
the older Jinja1 engine and breaks backwards compatibility to some
point over the old Jinja1 engine. It's packaged as a separate
package so that you can use both right next to each other for an
easier transition.

http://lucumr.pocoo.org/2008/7/17/jinja2-final-aka-jinjavitus-released

This is something the Python community can also learn from the Perl
world.

-srid

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

nad at acm

Oct 8, 2009, 2:28 PM

Post #16 of 44 (1346 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

In article
<94bdd2610910080131j323b98d9i871bce43465f237a [at] mail>,
Tarek Ziad� <ziade.tarek [at] gmail> wrote:
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)

Thanks for the summary! Unfortunately, as the saying goes, the devil
is in the details. Here are a few comments.

> = Distribute =
>
> I won't explain here again why we have forked, I think it's obvious to
> anyone here now. I'll rather explain what
> we are planning in Distribute and how it will interact with Distutils.
>
> Distribute has two branches:
>
> - 0.6.x : provides a Setuptools-0.6c9 compatible version
> - 0.7.x : will provide a refactoring
>
> == 0.6.x ==
>
> Not "much" is going to happen here, we want this branch to be helpful
> to the community *today* by addressing the 40-or-so bugs
> that were found in Setuptools and never fixed. This is eventually
> happen soon because its development is
> fast : there are up to 5 commiters that are working on it very often
> (and the number grows weekly.)
>
> The biggest issue with this branch is that it is providing the same
> packages and modules setuptools does, and this
> requires some bootstrapping work where we make sure once Distribute is
> installed, all Distribution that requires Setuptools
> will continue to work. This is done by faking the metadata of
> Setuptools 0.6c9. That's the only way we found to do this.
>
> There's one major thing though: thanks to the work of Lennart, Alex,
> Martin, this branch supports Python 3,
> which is great to have to speed up Py3 adoption.
>
> The goal of the 0.6.x is to remove as much bugs as we can, and try if
> possible to remove the patches done
> on Distutils. We will support 0.6.x maintenance for years and we will
> promote its usage everywhere instead of
> Setuptools.

This is a key and, I think, difficult issue that's going to need a lot
of thought and work to be successful. setuptools / easy_install filled
several unmet needs of both package developers and end users and became
a de-facto standard because there was essentially no competition. So,
in a sense, there was never a migration to setuptools from something.
But now, especially given GvR's blessing of Distribute, there needs to
be a migration from setuptools to Distribute, although neither are part
of the standard library. There are no references to setuptools or
easy_install in any of the standard Python documentation.

So how do end users know about easy_install? Their first encounter
is/was probably through some recipe on a web site, most likely the
installation instructions for some package they want to use. And I
suspect pretty much all of those installation instructions direct the
user to one of two places, either the PyPi page for setuptools or the
peakcommunity wiki page:

http://pypi.python.org/pypi/setuptools
http://peak.telecommunity.com/DevCenter/EasyInstall

Right now those are essentially the ultimate contact points for users.
A general Google seach for easy_install gets 357,000 hits; one
restricted to pypi.python.org gets 2,500. I don't know how many of the
latter are active packages on PyPi but certainly there must be hundreds
that currently use setuptools and document the use of easy_install for
their users. Those packages' documentation often directs users to
download the ez_install script to bootstrap setuptools; I believe there
are some packages that download it automatically, if necessary.

So, without some other intervention, users will continue to follow the
instructions with those hundreds of packages (or on those thousands of
websites) and manually or automatically try to install setuptools.
Which may or may not work - as witnessed with the release of 2.6.3.

Brett and others suggested including some sort of warnings in the python
release documentation. Is there consensus that should be done?
Starting with 2.6.4? Seems like that would be up to Barry and Guido to
pronounce.

Ultimately, to successfully migrate to Distribute is going to require a
good understanding of the use cases for easy_install (primarily those of
end users) and for the setuptools API (those of package developers).
Those use cases should be documented somehow, either in the roadmap or a
reference elsewhere. This is especially true if your intentions are to
deprecate the easy_install command in Distribute 0.7 and if APIs are
going to change for package developers. As Antoine points out, pip is
not equivalent to easy_install. BTW, keep in mind that today people
also use easy_install to manage the installation of packages that do not
use setuptools.

It will also be important to understand and address the major
distribution channels for Python installations, many of which already
include setuptools in their distributions. There are the multi-platform
distributors, like python.org (which does not currently distribute
setuptools), ActiveState, and Enthought. You've already mentioned some
of the Unix-y O/S ones: Debian, Fedora, et al. On the OS X side,
there's Apple and the major third-party providers of open-source
packages, MacPorts and Fink - they all distribute setuptools. I don't
have personal experience with the Windows world to know what all the
channels are there. Are there other operating systems who package
Python and possibly setuptools? They, one way or another, need to know
and have some sort of plan going forward for their distributions
regarding setuptools and Distribute.

And, of course, the stickiest issues in all this are not technical, but
rather social. For most of this, the most important part will likely be
giving a clear direction to all of the major groups involved: end users,
package developers, distributors. How to decide on and then effectively
communicate that direction is not at all trivial, I think.

Thanks again for your hard work on this, Tarek!

--
Ned Deily,
nad [at] acm

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Oct 8, 2009, 2:41 PM

Post #17 of 44 (1349 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Ned Deily <nad <at> acm.org> writes:
>
> How to decide on and then effectively
> communicate that direction is not at all trivial, I think.

I think it's quite trivial actually. Since everybody agrees (except perhaps PJE)
that Distribute should replace setuptools, the word will spread and trickle
quite quickly in the various layers of the community.

If you look at the XFree86 -> XOrg transition, I don't think there have been a
lot of problems to bring people to understand the change.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

nad at acm

Oct 8, 2009, 3:01 PM

Post #18 of 44 (1350 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

In article <loom.20091008T233831-780 [at] post>,
Antoine Pitrou <solipsis [at] pitrou> wrote:
> Ned Deily <nad <at> acm.org> writes:
> > How to decide on and then effectively
> > communicate that direction is not at all trivial, I think.
>
> I think it's quite trivial actually. Since everybody agrees (except perhaps
> PJE)
> that Distribute should replace setuptools, the word will spread and trickle
> quite quickly in the various layers of the community.
>
> If you look at the XFree86 -> XOrg transition, I don't think there have been
> a
> lot of problems to bring people to understand the change.

I'm not sure that's a such a good example: some of the impacts of that
transition are still affecting users.

The unspoken assumption here is that it is a good idea to make this
transition as painless as possible within some unspecified limits of
practicality. Sure, at one end of the spectrum, you can just throw
things out in the world and hope for the best. I think the Python
community has a well-deserved reputation for doing a lot better than
that.

--
Ned Deily,
nad [at] acm

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ziade.tarek at gmail

Oct 8, 2009, 3:55 PM

Post #19 of 44 (1346 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Thu, Oct 8, 2009 at 11:28 PM, Ned Deily <nad [at] acm> wrote:
[..]
> So, without some other intervention, users will continue to follow the
> instructions with those hundreds of packages (or on those thousands of
> websites) and manually or automatically try to install setuptools.

I think this will be in the hands of the projects developers : if they start to
use Distribute, they will start to document the way to install and use it,
and the word will spread eventually.

[..]
> Ultimately, to successfully migrate to Distribute is going to require a
> good understanding of the use cases for easy_install (primarily those of
> end users) and for the setuptools API (those of package developers).
> Those use cases should be documented somehow, either in the roadmap or a
> reference elsewhere. This is especially true if your intentions are to
> deprecate the easy_install command in Distribute 0.7 and if APIs are
> going to change for package developers. As Antoine points out, pip is
> not equivalent to easy_install. BTW, keep in mind that today people
> also use easy_install to manage the installation of packages that do not
> use setuptools.

The choice to deprecate easy_install in 0.7 is done because the Pip project
is not far to meet all uses cases easy_install users have, and we're betting
on the fact that Pip is active and will be much more advanced that what
we could do with a 'new' refactored easy_install by the time 0.7 is ready.

meaning : why doing this work twice ?

> Thanks again for your hard work on this, Tarek!

Thanks for the advices

Tarek
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kiorky at cryptelium

Oct 8, 2009, 4:11 PM

Post #20 of 44 (1356 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Tarek Ziad� a �crit :

> The choice to deprecate easy_install in 0.7 is done because the Pip project
> is not far to meet all uses cases easy_install users have, and we're betting
> on the fact that Pip is active and will be much more advanced that what
> we could do with a 'new' refactored easy_install by the time 0.7 is ready.

But how about retro-compatibility?
Like with all those buildout recipes which relies on setuptool APIs
(Requirement, PackageIndex and so on), it's a risk to break same at some point.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net

--
--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF

Attachments:

signature.asc (0.25 KB)

ziade.tarek at gmail

Oct 8, 2009, 4:21 PM

Post #21 of 44 (1337 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Fri, Oct 9, 2009 at 1:11 AM, kiorky <kiorky [at] cryptelium> wrote:
>
>
> Tarek Ziad� a �crit :
>
>> The choice to deprecate easy_install in 0.7 is done because the Pip project
>> is not far to meet all uses cases easy_install users have, and we're betting
>> on the fact that Pip is active and will be much more advanced that what
>> we could do with a 'new' refactored easy_install by the time 0.7 is ready.
>
> But how about retro-compatibility?
> Like with all those buildout recipes which relies on setuptool APIs
> (Requirement, PackageIndex and so on), it's a risk to break same at some point.

Keep in mind that Distribute 0.6.x will still be maintained and will
still provide easy_install and al,
and that you will be able to rely on it for your projects.

Distribute 0.7.x is to be considered as a new project in some ways :
all namespaces
will be different and the code will not interfer with a 0.6 installation.

So a buildout recipe will have the choice to use the old packages you
are mentioning and
at some point evolve and use the new tools 0.7 will provide.

Tarek
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

david.lyon at preisshare

Oct 8, 2009, 4:50 PM

Post #22 of 44 (1338 views)
Permalink

Re: Distutils and Distribute roadmap [In reply to]

On Thu, 8 Oct 2009 21:41:21 +0000 (UTC), Antoine Pitrou
<solipsis [at] pitrou> wrote:
> I think it's quite trivial actually. Since everybody agrees (except
perhaps
> PJE)
> that Distribute should replace setuptools, the word will spread and
trickle
> quite quickly in the various layers of the community.

I don't think that it's that black and white.

PJE put a lot of work (and creativity) into setuptools. There's some degree
of tragedy if his project gets forked off simply because of failures in
the testing and quality assurance department.

Personally, I find the setuptools code somewhat hard to follow. The style
goes back to early programming by 'masters' where the only person that
could follow it was the master himself.

Sadly, that is not what we need in open source. The newer trend is in
highly readily readable and easy to understand code.

Tarek is excellent at producing high quality code of the type that
is required. But his programming world (on the mac) isn't full of the
challenges that we poor other unfortunates face on other platforms face.

Coming back to the point, setuptools is a pretty poor package
to have to fork. It's very difficult to follow and the structure
and implementation is cryptic. Perphaps it's a thankless job
that Tarek has taken on in that regard - but he seems to be handling
the distribute fork quite well.

Being new to python, it's taken me a little while to figure out
the process. But I've decided to write a static metadata pep which
is underway now as an alternate proposal.

Being primarily a hacker, writing nice design documents doesn't come
easy.

If there is any distutils roadmap, I'd like to see improvements
that work on Windows. Yes.. python does work on windows and we
can't all get our bosses to change to os-x just yet.

There's many things in distutils that I could best describe as
being unfinished or partially done.

Let me make a few rambling direction suggestions...

- command line package installation for distutils

We need a simple script, not based on setuptools to
allow installation, listing and deinstallation of packages.

Currently *nothing* exists in distutils to do this.

It's a **major** failing.

- Improvements to operation on platform Windows

Package/Application installation on windows sucks...

For example, there's no documented way to get your
programs installed to "\Program Files\myApplication"

For non windows users, that's where programs are
supposed to go. Any other place is "evil" - like
where they are going now.

If it is only a documentation issue, well can we
fix that.

All I'm asking for is a matching "elif sys.platform = 'win32'"
for every "if sys.platform = 'darwin'".... or less. It's
not unreasonable...

It's probably a sensible idea for Tarek to keep going with
Distribute. My user experience with setuptools was that it
wasted a lot of my time unnecessarily.

I'm planning to put Distribute support in the Python Package
Manager project (pythonpkgmgr) for it's next release.

David

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ianb at colorstudy

Oct 8, 2009, 6:22 PM

Post #23 of 44 (1337 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

I'm coming in late and breaking threading, but wanted to reply to
Tarek's original email:

> - easy_install is going to be deprecated ! use Pip !

Cool! I wouldn't have written pip if I didn't think it would improve
substantially on easy_install.

Incidentally (because I know people get really enthused about this)
Carl Meyer just contributed a feature to pip to do atomic
uninstallation.

Someone mentioned that easy_install provided some things pip didn't;
outside of multi-versioned installs (which I'm not very enthusiastic
about) I'm not sure what this is?

> - distribute.resources: that's the old pkg_resources, but reorganized in clean, pep-8 modules. This package will
> only contain the query APIs and will focus on being PEP 376 compatible. We will promote its usage and see if Pip wants
> to use it as a basis. And maybe PyPM once it's open source ? (<hint> <hint>).
> It will probably shrink a lot though, once the stdlib provides PEP 376 support.

This seems easy enough to use in pip.

> - distribute.index: that's package_index and a few other things. everything required to interact with PyPI. We will promote
> its usage and see if Pip wants to use it as a basis.

This is a little tricky. Primarily because there's a fair amount of
logic involved in the indexing (going around to different URLs,
parsing links, finding stuff). So long as there is logic, something
can go wrong -- often not in the package itself, but simple user error
(e.g., it doesn't look where the user thinks it should, or a link is
malformed, etc). Because of this, and as a general design goal of
pip, I want to show as much as I can about what it is doing and why.
This is primarily tied into pip's logging system (which is oriented
towards command-line output, and isn't the standard logging system).
Also it tracks *why* it got to a certain links. These are the two
things I can think of where the index code in pip is tied to pip, and
why it would be hard to use an external system.

> = Virtualenv and the multiple version support in Distribute =
>
> (I am not saying "We" here because this part was not discussed yet
> with everyone)
>
> Virtualenv allows you to create an isolated environment to install some distribution without polluting the
> main site-packages, a bit like a user site-packages.
>
> My opinion is that this tool exists only because Python doesn't
> support the installation of multiple versions for the same
> distributions.
> But if PEP 376 and PEP 386 support are added in Python, we're not far
> from being able to provide multiple version support with
> the help of importlib.

Before making workingenv (virtualenv's predecessor) I actively tried
to use Setuptools' multi-version support. I found it very
unsuccessful. I don't think it was due to any problems with
Setuptools -- maybe a slight problem was the conflict between "active"
eggs and "multiversion" eggs (where active eggs would be more likely
to cause conflicts, while multiversion eggs aren't available until you
specifically require them). But that was just awkward, I don't think
it was the real problem.

The real problem is that a set of packages that makes up a working
application is something separate from any library. And you can only
know that an application works with an exact set of libraries. Every
update can break a working application (and with fairly high
probability). You can't know what updates are safe. And it's really
a bit tricky to even be sure you know what libraries a package really
requires -- lots of libraries might be incidentally available but no
longer formally required. (Someone mentioned a coworker that only
installed packages with easy_install -m, because he said it kept him
honest -- only packages that are explicitly required would be
available. But most people don't do this, and it really only solves
the one problem of undeclared dependencies)

The way both virtualenv and buildout handle this is that libraries
will have a single, static version until you explicitly do something
to update that version. Both are somewhat focused on a functional
unit -- like one virtualenv environment for one task, or one buildout
config for one application. Buildout allows for a globally shared set
of versioned eggs, but that's really just a little optimization (for
disk space or installation speed) -- each egg is brought in only
explicitly, at build time, and not as an option during the program's
runtime.

This is verifiable, stable, and to varying degrees concrete
(virtualenv being more concrete than buildout, which tends more
towards the declarative).

What virtualenv does could certainly be in the Python interpreter (and
much more compact as a result, I am sure). PYTHONHOME does it to a
degree, though binding a script to a environment through the
interpreter listed in #! is more stable than the implicit environment
of PYTHONHOME. workingenv used an environmental variable (PYTHONPATH,
before PYTHONHOME existed) and it caused problems. Also virtualenv
offers more system isolation.

If I had my way, buildout would use virtualenv and throw away its
funny script generation. If virtualenv had existed before buildout
began development, probably things would have gone this way. I think
it would make the environment more pleasant for buildout users. Also
I wish it used pip instead of its own installation procedure (based on
easy_install). I don't think the philosophical differences are that
great, and that it's more a matter of history -- because the code is
written, there's not much incentive for buildout to remove that code
and rely on other libraries (virtualenv and pip).

--
Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kiorky at cryptelium

Oct 9, 2009, 1:54 AM

Post #24 of 44 (1329 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

Ian Bicking a écrit :
> I'm coming in late and breaking threading, but wanted to reply to
> Tarek's original email:
>

> This is verifiable, stable, and to varying degrees concrete
> (virtualenv being more concrete than buildout, which tends more
> towards the declarative).

Is that a friday troll ? declarative over control ?

>
> What virtualenv does could certainly be in the Python interpreter (and
> much more compact as a result, I am sure). PYTHONHOME does it to a
> degree, though binding a script to a environment through the
> interpreter listed in #! is more stable than the implicit environment
> of PYTHONHOME. workingenv used an environmental variable (PYTHONPATH,
> before PYTHONHOME existed) and it caused problems. Also virtualenv
> offers more system isolation.
>
> If I had my way, buildout would use virtualenv and throw away its
> funny script generation. If virtualenv had existed before buildout

Which one, the one provided to generate scripts from entry points with the *.egg
recipes or the bin/buildout auto regeneration?

> began development, probably things would have gone this way. I think
> it would make the environment more pleasant for buildout users. Also

* I don't think so, buildout is the only tool atm that permit to have really
reproducible and isolated environments. Even, if you use the pip freezing
machinery, it is not equivalent to buildout, Control!
* Buildout can have single part to construct required eggs, at a specific
version and let you control that. Pip will just search for this version, see
that it's not available and fail. You have even recipes (like
minitage.recipe.egg that permit to construct eggs with special version when you
apply patches onto, thus, you can have the same egg in different flavors in the
same eggs cache available for different projects. Those projects will just have
to pin the right version to use, Control!.
* Another thing is the "funny script generation", you have not one global
site-packages for your project, but one global cache. But from this global
cache, your scripts will only have available the eggs you declared, see Control!
* Moreover buildout is not only a python packages manager, it's some of its
recipes that permit to use it as. Buildout is just a great deployment tool that
allow to script and manage your project in a "funny" and "flexible" way, Control!

> I wish it used pip instead of its own installation procedure (based on
> easy_install). I don't think the philosophical differences are that
> great, and that it's more a matter of history -- because the code is

And retro-compatibility.

> written, there's not much incentive for buildout to remove that code
> and rely on other libraries (virtualenv and pip).

* Virtualenv is not really needed as buildout uses "eggs caches" and do the
isolation from there. Scripts generate the adequate sys.path relying on the
application requirements (which, once more, you are in total control). I prefer
throught to use a virtualenv for each project (or a bunch of projects), and to
bootstrap my buildouts from there. For me, they are complementary to achieve
your deployment and isolation needs.

>
> --
> Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/kiorky%40cryptelium.net

--
--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF

Attachments:

signature.asc (0.25 KB)

ziade.tarek at gmail

Oct 9, 2009, 3:03 AM

Post #25 of 44 (1341 views)
Permalink

Re: Distutils and Distribute roadmap (and some words on Virtualenv, Pip) [In reply to]

On Fri, Oct 9, 2009 at 3:22 AM, Ian Bicking <ianb [at] colorstudy> wrote:
> I'm coming in late and breaking threading, but wanted to reply to
> Tarek's original email:
>
>> - easy_install is going to be deprecated ! use Pip !
>
> Cool! I wouldn't have written pip if I didn't think it would improve
> substantially on easy_install.
>
> Incidentally (because I know people get really enthused about this)
> Carl Meyer just contributed a feature to pip to do atomic
> uninstallation.

Yes I saw that, it's great. And he is now involved in PEP 376 work so
Pip could eventually become the first PEP 376 compliant installer/uninstaller.

>
> Someone mentioned that easy_install provided some things pip didn't;
> outside of multi-versioned installs (which I'm not very enthusiastic
> about) I'm not sure what this is?

Basically what you've listed on Pip front page I think, like 'not
tested under windows'
But I don't see any blocking point besides some testing, to move from
easy_install to pip,
and the deprecation of multi-versioned feature seem to go in the
direction of the community.

>> - distribute.index: that's package_index and a few other things. everything required to interact with PyPI. We will promote
>> its usage and see if Pip wants to use it as a basis.
>
> This is a little tricky. Primarily because there's a fair amount of
> logic involved in the indexing (going around to different URLs,
> parsing links, finding stuff). So long as there is logic, something
> can go wrong -- often not in the package itself, but simple user error
> (e.g., it doesn't look where the user thinks it should, or a link is
> malformed, etc). Because of this, and as a general design goal of
> pip, I want to show as much as I can about what it is doing and why.
> This is primarily tied into pip's logging system (which is oriented
> towards command-line output, and isn't the standard logging system).
> Also it tracks *why* it got to a certain links. These are the two
> things I can think of where the index code in pip is tied to pip, and
> why it would be hard to use an external system.

OK. Maybe this particular package could be used by another tool
that needs to work with PyPI. It will also include a set of APIs that
corresponds to PyPI XMLPRC services.

Regards
Tarek
--
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads