Mailing List Archive: Python: Dev

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

Index | Next | Previous | View Threaded

ncoghlan at gmail

Jul 3, 2009, 5:47 PM

Post #1 of 5 (537 views)
Permalink

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata

Tarek Ziadé wrote:
> 2009/7/3 Paul Moore <p.f.moore [at] gmail>:
>> Does this sound sensible? Tarek, would you be OK with me attempting to
>> modify your prototype to support this protocol? Are there any tests
>> for PEP 376, so that I can confirm I haven't completely broken
>> something? If I can, I'll knock up some simple prototype importers for
>> non-standard examples, and see how they work with all this.
>
> Yes that's exactly what I was thinking about after the discussion we
> had in the other
> thread. This change would allow much more flexibility.

It's also very similar to the ideas that started to tick over in the
back of my head after the previous discussion (only far more fleshed out
that anything I got to - fortunately I read Paul's message before
spending too much time on it!)

Specifically regarding this comment from Paul:

> Finder.uninstall(distribution_name, filter=callable, installer=name)
> Uninstall the given distribution. It's likely that many finders will
> be read-only. In that case, this function should return None.
> Otherwise, return a list of the "files" removed. (This may need some
> clarification, as many finders won't have a concept of a "file name").

PEP 302 finders and loaders actually *are* expected to have a concept of
a "file name". For Python modules they they are expected to have it so
they can populate __file__ correctly (runpy is also a lot happier with
them when they expose get_filename(), a relatively recent addition to
the PEP 302 API). For other resources they're expected to have it so the
get_data() API can work correctly.

One other thing to consider: it may be desirable to have hooks for
meta_path and path_hooks in distutils that are checked *before* the full
import hooks in the sys module. The reason I say this is that it would
allow someone to override just the metadata retrieval (e.g. to pick up
the extra dependency information saved by setuptools and friends)
without having to worry about breaking the actual imports from those
locations.

Cheers,
Nick.

P.S. +lots on using 'metadata' in the PEP 376 method names rather than
the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
gratuitous in this case.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

brett at python

Jul 4, 2009, 12:54 PM

Post #2 of 5 (505 views)
Permalink

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata [In reply to]

On Fri, Jul 3, 2009 at 17:47, Nick Coghlan <ncoghlan [at] gmail> wrote:

> Tarek ZiadĂŠ wrote:
> > 2009/7/3 Paul Moore <p.f.moore [at] gmail>:
> >> Does this sound sensible? Tarek, would you be OK with me attempting to
> >> modify your prototype to support this protocol? Are there any tests
> >> for PEP 376, so that I can confirm I haven't completely broken
> >> something? If I can, I'll knock up some simple prototype importers for
> >> non-standard examples, and see how they work with all this.
> >
> > Yes that's exactly what I was thinking about after the discussion we
> > had in the other
> > thread. This change would allow much more flexibility.
>
> It's also very similar to the ideas that started to tick over in the
> back of my head after the previous discussion (only far more fleshed out
> that anything I got to - fortunately I read Paul's message before
> spending too much time on it!)
>
> Specifically regarding this comment from Paul:
>
> > Finder.uninstall(distribution_name, filter=callable, installer=name)
> > Uninstall the given distribution. It's likely that many finders will
> > be read-only. In that case, this function should return None.
> > Otherwise, return a list of the "files" removed. (This may need some
> > clarification, as many finders won't have a concept of a "file name").
>
> PEP 302 finders and loaders actually *are* expected to have a concept of
> a "file name". For Python modules they they are expected to have it so
> they can populate __file__ correctly

Not only that, lots of code out there expects values from __path__ to
contain os.path.sep on top of __file__, so the idea of file paths is already
hard-coded into how import works, whether we like it or not.

> (runpy is also a lot happier with
> them when they expose get_filename(), a relatively recent addition to
> the PEP 302 API).

An addition I was not even aware of. Looks like importlib needs a little
updating.

> For other resources they're expected to have it so the
> get_data() API can work correctly.
>
> One other thing to consider: it may be desirable to have hooks for
> meta_path and path_hooks in distutils that are checked *before* the full
> import hooks in the sys module. The reason I say this is that it would
> allow someone to override just the metadata retrieval (e.g. to pick up
> the extra dependency information saved by setuptools and friends)
> without having to worry about breaking the actual imports from those
> locations.

So as the guy who plans to try to clean up import, I just want to say that
implicit finders will eventually go away. Do not start off making them
implicit, but you can have them installed on sys.meta_path or sys.path_hooks
by default.

>
>
> Cheers,
> Nick.
>
> P.S. +lots on using 'metadata' in the PEP 376 method names rather than
> the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly
> gratuitous in this case.

Ditto from here. Plus I have an aversion to terminology that goes down the
reptile route instead of the Monty Python route.

-Brett

mal at egenix

Jul 7, 2009, 12:50 AM

Post #3 of 5 (474 views)
Permalink

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata [In reply to]

Paul Moore wrote:
> 2009/7/7 Ben Finney <ben+python [at] benfinney>:
>> Paul Moore <p.f.moore [at] gmail> writes:
>>
>>> In fact, the above strongly suggests to me that PEP 376 must NOT
>>> follow setuptools conventions - otherwise, it will end up clashing
>>> with the files setuptools is putting on my system.
>> I think the argument for a separate â.pydistâ metadata directory,
>> normative in PEP 376, is a good one.
>
> I've been convinced otherwise. There's no long-term benefit (the
> directory layout and format is the same, so it's a purely cosmetic
> change) and we're doing no favours to anyone just adding a new format
> for the sake of it.

The PEP should take the chance to define not only the
directory, but also the complete set of files in there and
their format.

A typical setuptools dir looks like this:

dependency_links.txt
namespace_packages.txt
not-zip-safe
PKG-INFO
requires.txt
SOURCES.txt
top_level.txt

Ie. a complete mess of upper-case/lower-case names, names with
underscores or hyphens as word separator, files with extensions,
files without them.

This needs some serious cleanup.

A new dir name will allow to do this, so +1 on a new dir name.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Jul 07 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Jul 7, 2009, 2:04 AM

Post #4 of 5 (473 views)
Permalink

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata [In reply to]

2009/7/7 M.-A. Lemburg <mal [at] egenix>:
> The PEP should take the chance to define not only the
> directory, but also the complete set of files in there and
> their format.
>
> A typical setuptools dir looks like this:
>
> dependency_links.txt
> namespace_packages.txt
> not-zip-safe
> PKG-INFO
> requires.txt
> SOURCES.txt
> top_level.txt
>
> Ie. a complete mess of upper-case/lower-case names, names with
> underscores or hyphens as word separator, files with extensions,
> files without them.
>
> This needs some serious cleanup.
>
> A new dir name will allow to do this, so +1 on a new dir name.

I agree the current situation is a mess. I believe that everything you
mention is related to setuptools - so essentially, we have the
situation:

- the existing setuptools format is a mess (at least in the opinion of
MAL and me :-))
- PEP 376 is an opportunity to consider cleanup
- there are some strong supporters of keeping things setuptools-compatible
- nobody has come forward with any other real-world use case than setuptools
- consequently, those of us arguing for cleanup are talking theoretically :-(
- Do you (MAL) have any real use for this data? Any non-setuptools use
case, no matter how small, would really help here.

I suspect practicality will beat purity here. Which would be fine, if
the "practical" cases weren't just setuptools.

I understood that there was a lot of interest in a "distutils cleanup"
from the packaging community. And yet so far in this discussion, the
main participants have seemed to be (apologies to anyone if I've
misunderstood their position):

- Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed
- Me, advocating Windows issues and PEP 302 integration (to cover eggs
and general flexibility)
- Phillip (and occasional others), representing setuptools POV
- Interested python-dev participants

Nobody representing any other 3rd party packaging solution than setuptools.

Much as I'd like to, I can't really argue the case for breaking
setuptools compatibility without practical reasons. And if we're not
going to do that, PEP 376 reduces to a further stage of the
incremental move of setuptools into distutils core (by removing the
partial solution of a .egg-info file, and replacing it with a full
setuptools .egg-info directory, plus a few introspection APIs as a
sweetener).

2009/7/7 Ronald Oussoren <ronaldoussoren [at] mac>:
> But why break existing code without having any other benifits? If I read
> the discussion correctly the name would be changed without any changes to
> the contents of the metadata directory. I would be more inclined to be in
> favour if the name change had a sound technical reason, such as a change of
> the contents of the directory which would make setuptools "egg-info"
> directories incompatible with the PEP376 ones.

See MAL's comments above, and my response. If (and only if!) his
proposal is accepted, then a name change starts to be worth discussing
further. As things stand at the moment, the structure appears to be
setuptools-compatibile (other than the new RECORD file, which Phillip
has committed to adding) so a name change isn't worth it.

I can't comment myself on setuptools compatibility, so I'm assuming
here that Phillip will speak up if the proposed format is *not*
compatible...

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Jul 7, 2009, 2:02 PM

Post #5 of 5 (477 views)
Permalink

Re: PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata [In reply to]

Paul Moore wrote:
> 2009/7/7 Ben Finney <ben+python [at] benfinney>:
>>> - PEP 376 is an opportunity to consider cleanup
>>> - there are some strong supporters of keeping things
>>> setuptools-compatible
>> Your implication seems to be that these two are incompatible. It should
>> be noted that it's possible to clean up, by having a separate (e.g.
>> .pydist) directory as the new standard, with cleaned up standard
>> content, and continue backward compatibility in the actual tools while
>> deprecating the old .egg-info format for some time until finally
>> removing that support.
>
> Phillip argued strongly against this. I'm not going to repeat his
> arguments - they seemed to make sense to me at the time, but I haven't
> internalised them well enough that I'd be willing to try to repeat
> them here.
>
> The crux of all this seems to be setuptools' backwards compatibility
> issues, As someone who has a very strong dislike of setuptools (the
> implementation, not the ideas), I'm trying very, very hard to leave it
> to someone else to say "stuff setuptools, this is a core distutils
> proposal, setuptools can keep up or die" :-)

Actually, the approach Ben suggests is the same as the one I suggested
for dealing with the backwards compatibility problem (i.e. distutils
would install both the existing .egg-info for detection by existing
packaging tools, while putting the new stuff in a .pydist directory).

The argument against *that* approach to backwards compatibility is the
accumulation of cruft in site-packages (and other module installation
directories) since even a simple installation of a single module or
small package would now be generating at least *three* entries in the
relevant directory (.egg-info, .pydist and the module or package itself).

Retaining the .egg-info naming provides detection of installed
distributions by old packaging systems with a minimum of cruft on the
end user's system. The only possible advantage changing the extension
name might give us is making it fractionally easier to clean up the
contents of the .egg-info directory, but I don't think it will actually
make that much difference (i.e. it really shouldn't be that hard to keep
the new PEP 376 based filenames all UPPERCASE and distinct from any
filenames currently stored in the directory by setuptools or other
packaging utilities)

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads