Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Edits to Metadata 1.2 to add extras (optional dependencies)

 

 

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded


dholth at gmail

Aug 15, 2012, 7:49 AM

Post #1 of 50 (506 views)
Permalink
Edits to Metadata 1.2 to add extras (optional dependencies)

I've drafted some edits to Metadata 1.2 with valuable feedback from
distutils-sig (special thanks to Erik Bray), which seems to have no
more comments on the issue after about 6 weeks. Let me know if you
have an opinion, or if you will have one during some bounded time in
the future.

Metadata 1.2 (PEP 345), a non-final PEP that has been adopted by
approximately 10 of the latest sdists from pypy, cannot represent the
setuptools "extras" (optional dependencies) feature. This is a problem
because about 1600+ or 10% of the packages hosted on pypy define
"extras" as measured in May of this year.

The edit implements the extras feature by adding a new condition
"extra == 'name'" to the Metadata 1.2 environment markers.
Requirements with this marker are only installed when the named
optional feature is requested. Valid extras for a package must be
declared with Provides-Extra: name.

It also adds Setup-Requires-Dist as a way to specify requirements
needed during an install as opposed to during runtime.


Abbreviated highlights:

Setup-Requires-Dist (multiple use)

Like Requires-Dist, but names dependencies needed while the
distributions's distutils / packaging `setup.py` / `setup.cfg` is run.


Provides-Extra (multiple use)

A string containing the name of an optional feature.

Examples:

Requires-Dist: reportlab; extra == 'pdf'

Requires-Dist: nose; extra == 'test'

Requires-Dist: sphinx; extra == 'doc'


(full changeset on
https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068)

Thanks,

Daniel Holth
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ericsnowcurrently at gmail

Aug 15, 2012, 8:05 AM

Post #2 of 50 (502 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Wed, Aug 15, 2012 at 8:49 AM, Daniel Holth <dholth [at] gmail> wrote:
> I've drafted some edits to Metadata 1.2 with valuable feedback from
> distutils-sig (special thanks to Erik Bray), which seems to have no
> more comments on the issue after about 6 weeks. Let me know if you
> have an opinion, or if you will have one during some bounded time in
> the future.
>
> Metadata 1.2 (PEP 345), a non-final PEP that has been adopted by
> approximately 10 of the latest sdists from pypy, cannot represent the
> setuptools "extras" (optional dependencies) feature. This is a problem
> because about 1600+ or 10% of the packages hosted on pypy define
> "extras" as measured in May of this year.
>
> The edit implements the extras feature by adding a new condition
> "extra == 'name'" to the Metadata 1.2 environment markers.
> Requirements with this marker are only installed when the named
> optional feature is requested. Valid extras for a package must be
> declared with Provides-Extra: name.
>
> It also adds Setup-Requires-Dist as a way to specify requirements
> needed during an install as opposed to during runtime.
>
>
> Abbreviated highlights:
>
> Setup-Requires-Dist (multiple use)
>
> Like Requires-Dist, but names dependencies needed while the
> distributions's distutils / packaging `setup.py` / `setup.cfg` is run.
>
>
> Provides-Extra (multiple use)
>
> A string containing the name of an optional feature.
>
> Examples:
>
> Requires-Dist: reportlab; extra == 'pdf'
>
> Requires-Dist: nose; extra == 'test'
>
> Requires-Dist: sphinx; extra == 'doc'
>
>
> (full changeset on
> https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068)

s/pypy/PyPI/

-eric
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


dholth at gmail

Aug 27, 2012, 7:56 AM

Post #3 of 50 (457 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Wed, Aug 15, 2012 at 10:49 AM, Daniel Holth <dholth [at] gmail> wrote:
> I've drafted some edits to Metadata 1.2 with valuable feedback from
...
> (full changeset on https://bitbucket.org/dholth/python-peps/changeset/537e83bd4068)

Metadata 1.2 is nearly 8 years old and it's Accepted but not Final. Is
it better to continue editing it, or create a new PEP for Metadata
1.3?
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


petri at digip

Aug 27, 2012, 9:15 PM

Post #4 of 50 (456 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

Daniel Holth wrote:
> I don't know of a tool that doesn't reliably ignore extra fields, but
> I will put you down as being in favor of an X- fields paragraph:
>
> Extensions (X- Fields)
> ::::::::::::::::::::::
>
> Metadata files can contain fields that are not part of
> the specification, called *extensions*. These fields start with
> with `X-`.

See RFC 6648 for why such X-fields may not be a good idea:

http://tools.ietf.org/html/rfc6648
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


stephen at xemacs

Aug 27, 2012, 9:57 PM

Post #5 of 50 (457 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

Petri Lehtinen writes:
> Daniel Holth wrote:
> > I don't know of a tool that doesn't reliably ignore extra fields, but
> > I will put you down as being in favor of an X- fields paragraph:
> >
> > Extensions (X- Fields)
> > ::::::::::::::::::::::
> >
> > Metadata files can contain fields that are not part of
> > the specification, called *extensions*. These fields start with
> > with `X-`.
>
> See RFC 6648 for why such X-fields may not be a good idea:
>
> http://tools.ietf.org/html/rfc6648

But note that the RFC also says that the preferred solution to the
problem that X-fields are intended to solve is an easily accessible
name registry and a simple registration procedure. If Martin's "be
prepared for a ten-year period to acceptance" is serious, what should
be done about such a registry?
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


dholth at gmail

Aug 28, 2012, 4:04 AM

Post #6 of 50 (456 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Aug 28, 2012, at 1:20 AM, Nick Coghlan <ncoghlan [at] gmail> wrote:

> On Tue, Aug 28, 2012 at 6:29 AM, "Martin v. Löwis" <martin [at] v> wrote:
>> Am 27.08.12 16:56, schrieb Daniel Holth:
>>> Metadata 1.2 is nearly 8 years old and it's Accepted but not Final. Is
>>> it better to continue editing it, or create a new PEP for Metadata
>>> 1.3?
>>
>>
>> You can't add new fields to the format after the fact, unless the format had
>> provided for such additions (which it does not - there is
>> no mention of custom fields anywhere, and no elaboration on how
>> "unknown" fields should be processed).
>>
>> So if you want to add new fields, you need to create a new version
>> of the metadata.
>
> I agree with this point - the main reason the metadata PEP is still
> lingering at Accepted rather than Final is the tangled relationship
> between distutils and other projects that led to the complete
> distutils feature freeze. Until distutils2 makes it into the standard
> library as the packaging module, the standard library is going to be
> stuck at v1.1 of the metadata format.
>
>> Prepare for a ten-year period of acceptance - so it
>> would be good to be sure that no further additions are desired within
>> the next ten years before seeking approval for the PEP.
>
> However, this point I really don't agree with. The packaging ecosystem
> is currently evolving outside the standard library, but the
> standardisation process for the data interchange formats still falls
> under the authority of python-dev and the PEP process.
>
> If there are things missing from v1.2 of the metadata spec, then
> define v1.3 to address those known problems. Don't overengineer it in
> an attempt to anticipate every possible need that might come in the
> next decade. Tools outside the standard library are then free to adopt
> the new standard, even while the stdlib itself continues to lag
> behind.
>
> When the packaging module is finally added (hopefully 3.4, even if
> that means we have to temporarily cull the entire compiler
> subpackage), it will handle the most recent accepted version of the
> metadata format (as well as any previous versions). If more holes
> reveal themselves in the next 18 months, then it's OK if v1.4 is
> created when it becomes clear that it's necessary.
>
> At the very least, something v1.3 should make explicit is that custom
> metadata should NOT be put into the .dist-info/METADATA (PEP 376
> location, PKG-INFO, in distutils terms) file. Instead, that data
> should be placed in a *separate* file in the .dist-info directory.
> Something that *may* be appropriate is a new field in METADATA that
> explicitly calls out such custom metadata files by naming the PyPI
> distribution that is the authority for the relevant format (e.g.
> "Custom-Metadata: wheel" to indicate that 'wheel' defined metadata is
> present)

Setuptools just uses path.exists() when it needs a particular file and will not bother parsing pkg-info at all if it can help it. The metadata edits for 1.2 fold some of those files into metadata.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 4:45 AM

Post #7 of 50 (450 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 9:04 PM, Daniel Holth <dholth [at] gmail> wrote:
> Setuptools just uses path.exists() when it needs a particular file and will not bother parsing pkg-info at all if it can help it. The metadata edits for 1.2 fold some of those files into metadata.

You can't use path.exists() on metadata published by a webservice (or
still inside a zipfile), but you can download or read the main
metadata file.

Still, I don't really care whether or not such a field indicating the
presence of custom metadata is added, I'm mainly registering a strong
-1 on allowing extension fields (in the form of X- headers or CSS
style prefixed headers) in the metadata file itself.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


donald.stufft at gmail

Aug 28, 2012, 5:07 AM

Post #8 of 50 (450 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

I personally think that at a minimum we should have X-Fields that
get moved into the normal METADATA file, and personally I would
prefer to just drop the X- prefix completely.

I think any spec which doesn't include first class support for
extending it with new metadata is going to essentially kick
the can down the road and solve the problems of today without
leaving room to solve the problems of tomorrow.

I know that distutils2 have requires-dist, but for the sake of
argument pretend they don't. If there is first class support for
extending the metadata with new fields, a project could come
along, and add a requires-dist (or x-requires-dist) concept to
metadata. Tools that understand it would see that data and
be able to act on it, tools that don't understand it would simply
write it to the METADATA file incase in the future a tool that
does understand it needs to act on it.

Essentially first class support for extending the metadata outside
of a PEP process means that outside of the stdlib people can
experiment and try new things, existing tools will continue to work
and just ignore that extra data (but leave it intact), new tools will be
able to utilize it to do something useful. Ideally as a new concept is
tested externally and begins to gain acceptance a new metadata
version could be created that standardizes that field as part of the
spec instead of an extension.


On Tuesday, August 28, 2012 at 7:45 AM, Nick Coghlan wrote:

> On Tue, Aug 28, 2012 at 9:04 PM, Daniel Holth <dholth [at] gmail (mailto:dholth [at] gmail)> wrote:
> > Setuptools just uses path.exists() when it needs a particular file and will not bother parsing pkg-info at all if it can help it. The metadata edits for 1.2 fold some of those files into metadata.
>
>
> You can't use path.exists() on metadata published by a webservice (or
> still inside a zipfile), but you can download or read the main
> metadata file.
>
> Still, I don't really care whether or not such a field indicating the
> presence of custom metadata is added, I'm mainly registering a strong
> -1 on allowing extension fields (in the form of X- headers or CSS
> style prefixed headers) in the metadata file itself.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncoghlan [at] gmail (mailto:ncoghlan [at] gmail) | Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python (mailto:Python-Dev [at] python)
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com
>
>


dholth at gmail

Aug 28, 2012, 5:28 AM

Post #9 of 50 (455 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 8:07 AM, Donald Stufft <donald.stufft [at] gmail> wrote:
> I personally think that at a minimum we should have X-Fields that
> get moved into the normal METADATA file, and personally I would
> prefer to just drop the X- prefix completely.

That is my preference as well. The standard library basically ignores
every metadata field or metadata file inside or outside of metadata
currently, so where is the harm changing the official document to read
"you may add new metadata fields to metadata" with an updated standard
library that only ignores some of the metadata in metadata instead of
all of it. The community is small enough to handle it.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 5:28 AM

Post #10 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 10:07 PM, Donald Stufft <donald.stufft [at] gmail> wrote:
> I personally think that at a minimum we should have X-Fields that
> get moved into the normal METADATA file, and personally I would
> prefer to just drop the X- prefix completely.

Hell no. We've been down this road with setuptools and it *sucks*.
Everybody gets confused, because you can't tell just by looking at a
metadata file what's part of the standard and what's been added just
because a tool developer thought it was a good idea without being able
to obtain broad consensus (perhaps because others couldn't see the
point until the extension had been field tested for an extended
period).

Almost *nobody* reads metadata specs other than the people that helped
write them. Everyone else copies a file that works, and tweaks it to
suit, or they use a tool that generates the metadata for them based on
some other interface.

The least-awful widespread extension approach I'm aware of is CSS
vendor prefixes. X- headers suck because they only give you two
namespaces - the "standard" namespace and the "extension" namespace.
That means everyone is quickly forced back into seeking agreement and
consensus to avoid naming conflicts for extension fields.

However, I'm open to the idea of a properly namespaced extension
mechanism, which is exactly why I suggested separate files flagged in
the main metadata with the PyPI project that defines the format of
those extensions. I'm also open to the idea of extensions appearing in
[PyPI distribution] prefixed sections after the standard metadata so,
for example, there could be a [wheel] section in METADATA rather than
a separate WHEEL file.

We already have a namespace registry in the form of PyPI, so there's
no reason to invent a new one, and allowing *any* PyPI distribution to
add custom metadata fields without name conflicts would allow easy
experimentation while still making it clear which fields are defined
in PEPs and which are defined by particular projects.


> I know that distutils2 have requires-dist, but for the sake of
> argument pretend they don't. If there is first class support for
> extending the metadata with new fields, a project could come
> along, and add a requires-dist (or x-requires-dist) concept to
> metadata. Tools that understand it would see that data and
> be able to act on it, tools that don't understand it would simply
> write it to the METADATA file incase in the future a tool that
> does understand it needs to act on it.
>
> Essentially first class support for extending the metadata outside
> of a PEP process means that outside of the stdlib people can
> experiment and try new things, existing tools will continue to work
> and just ignore that extra data (but leave it intact), new tools will be
> able to utilize it to do something useful. Ideally as a new concept is
> tested externally and begins to gain acceptance a new metadata
> version could be created that standardizes that field as part of the
> spec instead of an extension.

Agreed, and this is the kind of thing a v1.3 metadata PEP could
define. It just needs to be properly namespaced, and the obvious
namespacing mechanism is PyPI project names.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 5:31 AM

Post #11 of 50 (450 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 10:28 PM, Daniel Holth <dholth [at] gmail> wrote:
> That is my preference as well. The standard library basically ignores
> every metadata field or metadata file inside or outside of metadata
> currently, so where is the harm changing the official document to read
> "you may add new metadata fields to metadata" with an updated standard
> library that only ignores some of the metadata in metadata instead of
> all of it. The community is small enough to handle it.

I will campaign ardently against any such proposal. Any extension
field must be clearly traceable to an authority that gets to define
what it means to avoid a repeat of the setuptools debacle.

Namespaces are a honkin' great idea, let's do more of those :P

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


dholth at gmail

Aug 28, 2012, 5:33 AM

Post #12 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 8:28 AM, Nick Coghlan <ncoghlan [at] gmail> wrote:
> On Tue, Aug 28, 2012 at 10:07 PM, Donald Stufft <donald.stufft [at] gmail> wrote:
>> I personally think that at a minimum we should have X-Fields that
>> get moved into the normal METADATA file, and personally I would
>> prefer to just drop the X- prefix completely.
>
> Hell no. We've been down this road with setuptools and it *sucks*.
> Everybody gets confused, because you can't tell just by looking at a
> metadata file what's part of the standard and what's been added just
> because a tool developer thought it was a good idea without being able
> to obtain broad consensus (perhaps because others couldn't see the
> point until the extension had been field tested for an extended
> period).
>
> Almost *nobody* reads metadata specs other than the people that helped
> write them. Everyone else copies a file that works, and tweaks it to
> suit, or they use a tool that generates the metadata for them based on
> some other interface.
>
> The least-awful widespread extension approach I'm aware of is CSS
> vendor prefixes. X- headers suck because they only give you two
> namespaces - the "standard" namespace and the "extension" namespace.
> That means everyone is quickly forced back into seeking agreement and
> consensus to avoid naming conflicts for extension fields.
>
> However, I'm open to the idea of a properly namespaced extension
> mechanism, which is exactly why I suggested separate files flagged in
> the main metadata with the PyPI project that defines the format of
> those extensions. I'm also open to the idea of extensions appearing in
> [PyPI distribution] prefixed sections after the standard metadata so,
> for example, there could be a [wheel] section in METADATA rather than
> a separate WHEEL file.
>
> We already have a namespace registry in the form of PyPI, so there's
> no reason to invent a new one, and allowing *any* PyPI distribution to
> add custom metadata fields without name conflicts would allow easy
> experimentation while still making it clear which fields are defined
> in PEPs and which are defined by particular projects.
>
>
>> I know that distutils2 have requires-dist, but for the sake of
>> argument pretend they don't. If there is first class support for
>> extending the metadata with new fields, a project could come
>> along, and add a requires-dist (or x-requires-dist) concept to
>> metadata. Tools that understand it would see that data and
>> be able to act on it, tools that don't understand it would simply
>> write it to the METADATA file incase in the future a tool that
>> does understand it needs to act on it.
>>
>> Essentially first class support for extending the metadata outside
>> of a PEP process means that outside of the stdlib people can
>> experiment and try new things, existing tools will continue to work
>> and just ignore that extra data (but leave it intact), new tools will be
>> able to utilize it to do something useful. Ideally as a new concept is
>> tested externally and begins to gain acceptance a new metadata
>> version could be created that standardizes that field as part of the
>> spec instead of an extension.
>
> Agreed, and this is the kind of thing a v1.3 metadata PEP could
> define. It just needs to be properly namespaced, and the obvious
> namespacing mechanism is PyPI project names.
>
> Cheers,
> Nick.

Wheel deals with this somewhat by including a

Packager: bdist_wheel

line in WHEEL so that you can deal with packager-specific bugs.

Bento uses indentation so you can have sections:

Key: value
Indented Key: value
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


dholth at gmail

Aug 28, 2012, 5:57 AM

Post #13 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

How about

Extensions are fields that start with a pypi-registered name followed
by a hyphen. A file that contains extension fields declares them with
Extension: name :

Extension: pypiname
pypiname-Field: value
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 5:59 AM

Post #14 of 50 (452 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 10:33 PM, Daniel Holth <dholth [at] gmail> wrote:
> Wheel deals with this somewhat by including a
>
> Packager: bdist_wheel
>
> line in WHEEL so that you can deal with packager-specific bugs.

Right, but the problem with that is it's defining a couple of *new*
namespaces to manage:
- the filenames within dist_info (although uppercasing a PyPI project
name is pretty safe)
- the "Packager" field (bdist_wheel is a distutils command rather than
a PyPI project)

By using PyPI distribution names to indicate custom sections in the
main metadata file, we would get to exploit an existing registry that
enforces uniqueness without imposing significant overhead.

> Bento uses indentation so you can have sections:
>
> Key: value
> Indented Key: value

Yes, the main metadata file could definitely go that way. The three
main ways I can see an extensible metadata format working are:

1. The way wheel currently works (separate WHEEL file, naming
conflicts resolved largely by first-in-first-served with no official
registry, no obvious indication which project defines the format)

2. PyPI as extension registry, with an ini-file inspired section
syntax inside dist-info/METADATA

<standard metadata must appear first>

[wheel]
Version: 0.9
Packager: bdist_wheel-0.1
Root-Is-Purelib: true

3. PyPI as extension registry, with an indented section syntax inside
dist-info/METADATA

<custom metadata sections may appear anywhere in the file>


Extended-Metadata: wheel
Version: 0.9
Packager: bdist_wheel-0.1
Root-Is-Purelib: true

My preference is currently for the ini-style variant, but I could
definitely live with the indented approach.Either way, any project
registered on PyPI would be free to add their own extensions without
fear of naming conflicts or any doubts about the relevant authority
for the meaning of the fields. Standard tools could just treat those
sections as opaque blocks of text to be preserved verbatim, or else
they could be constrained so that the standard tools could pick out
the individual key:value pairs.

Namespacing an extension mechanism based on PyPI distributions names
should be pretty straightforward and it will mean that a lot of
problems that can otherwise arise with extensible metadata systems
should simply never come up.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 6:09 AM

Post #15 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 10:57 PM, Daniel Holth <dholth [at] gmail> wrote:
> How about
>
> Extensions are fields that start with a pypi-registered name followed
> by a hyphen. A file that contains extension fields declares them with
> Extension: name :
>
> Extension: pypiname
> pypiname-Field: value

The repetition seems rather annoying. Compare the two section based
variants I just posted to:

Extension: wheel
wheel-Version: 0.9
wheel-Packager: bdist_wheel-0.1
wheel-Root-Is-Purelib: true

It does have the advantage that tools for manipulating the format can
remain dumber, but that doesn't seem like *that* much of an advantage,
especially since any such benefit could be eliminated completely by
just switching to a completely standard ConfigParser format by putting
the PEP defined settings into a [python] section.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


donald.stufft at gmail

Aug 28, 2012, 6:19 AM

Post #16 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tuesday, August 28, 2012 at 8:28 AM, Nick Coghlan wrote:
>
> Agreed, and this is the kind of thing a v1.3 metadata PEP could
> define. It just needs to be properly namespaced, and the obvious
> namespacing mechanism is PyPI project names.

The biggest reason I have against namespacing them is it makes
moving from experimental to standard easier, but I'm ok with some
form of a namespace. The biggest reason I see against using PyPI
names as the namespace is it needlessly ties a piece of data to
the original creator. Similar to how right now you could write a
less hacky setuptools, but in order to do so you need to continue
to use the setuptools package name (see distribute). Using PyPI
names means that in the requires-dist example it would be something
like setuptools-requires-dist, and even if I make my own tool that
supports the same concept as setuptools's requires-dist I would
need to use setuptools-requires-dist.

The concept of metadata I think should be divorced from specific
implementations. Obviously there are going to be some implementation
specific issues but I think it's much cleaner to have a x-requires-dist
that any implementation can use than to have whoever-invented-it-first-requires-dist
or a twenty-different-forms-of-requires-dist.


donald.stufft at gmail

Aug 28, 2012, 6:20 AM

Post #17 of 50 (453 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tuesday, August 28, 2012 at 9:09 AM, Nick Coghlan wrote:
> On Tue, Aug 28, 2012 at 10:57 PM, Daniel Holth <dholth [at] gmail (mailto:dholth [at] gmail)> wrote:
> > How about
> >
> > Extensions are fields that start with a pypi-registered name followed
> > by a hyphen. A file that contains extension fields declares them with
> > Extension: name :
> >
> > Extension: pypiname
> > pypiname-Field: value
> >
>
>
> The repetition seems rather annoying. Compare the two section based
> variants I just posted to:
>
> Extension: wheel
> wheel-Version: 0.9
> wheel-Packager: bdist_wheel-0.1
> wheel-Root-Is-Purelib: true
>
> It does have the advantage that tools for manipulating the format can
> remain dumber, but that doesn't seem like *that* much of an advantage,
> especially since any such benefit could be eliminated completely by
> just switching to a completely standard ConfigParser format by putting
> the PEP defined settings into a [python] section.
>
METADATA files are not ini files.


dholth at gmail

Aug 28, 2012, 6:20 AM

Post #18 of 50 (450 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 9:09 AM, Nick Coghlan <ncoghlan [at] gmail> wrote:
> On Tue, Aug 28, 2012 at 10:57 PM, Daniel Holth <dholth [at] gmail> wrote:
>> How about
>>
>> Extensions are fields that start with a pypi-registered name followed
>> by a hyphen. A file that contains extension fields declares them with
>> Extension: name :
>>
>> Extension: pypiname
>> pypiname-Field: value
>
> The repetition seems rather annoying. Compare the two section based
> variants I just posted to:
>
> Extension: wheel
> wheel-Version: 0.9
> wheel-Packager: bdist_wheel-0.1
> wheel-Root-Is-Purelib: true
>
> It does have the advantage that tools for manipulating the format can
> remain dumber, but that doesn't seem like *that* much of an advantage,
> especially since any such benefit could be eliminated completely by
> just switching to a completely standard ConfigParser format by putting
> the PEP defined settings into a [python] section.

Wheel is a little different because once it's installed it is no
longer a wheel, but it makes a decent example. That's not even
repetition, it's just longer tag names. Repetition is having one
Classifier: line for every trove classifier.

It would be quite inconvenient to change the parser for PKG-INFO. It's
a win to keep the file flat.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


donald.stufft at gmail

Aug 28, 2012, 6:23 AM

Post #19 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tuesday, August 28, 2012 at 9:09 AM, Nick Coghlan wrote:
>
> It does have the advantage that tools for manipulating the format can
> remain dumber, but that doesn't seem like *that* much of an advantage,
> especially since any such benefit could be eliminated completely by
> just switching to a completely standard ConfigParser format by putting
> the PEP defined settings into a [python] section.
>
To be more specific, there is setup.cfg (which I dislike for other reasons), and
then there is METADATA. setup.cfg is an ini file but METADATA is a simple
key: value file with a flat namespace so any namespacing you want to do in
METADATA needs to be done at the key level. You could translate:

[setuptools]
requires-dist=foo

in a setup.cfg into

setuptools-requires-dist: foo

in METADATA, but I'm not sure if that would be beneficial or not.


ncoghlan at gmail

Aug 28, 2012, 6:25 AM

Post #20 of 50 (451 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 11:20 PM, Daniel Holth <dholth [at] gmail> wrote:
> Wheel is a little different because once it's installed it is no
> longer a wheel, but it makes a decent example. That's not even
> repetition, it's just longer tag names. Repetition is having one
> Classifier: line for every trove classifier.
>
> It would be quite inconvenient to change the parser for PKG-INFO. It's
> a win to keep the file flat.

Cool, it's the namespace I care about. Every piece of extended
metadata must have an authority who gets to define what it means. If
that means people register a "virtual" PyPI project just to reserve an
extension namespace, I'm fine with that.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 28, 2012, 6:41 AM

Post #21 of 50 (449 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 11:23 PM, Donald Stufft <donald.stufft [at] gmail> wrote:
> To be more specific, there is setup.cfg (which I dislike for other reasons),
> and
> then there is METADATA. setup.cfg is an ini file but METADATA is a simple
> key: value file with a flat namespace so any namespacing you want to do in
> METADATA needs to be done at the key level.

We're talking about the format for v1.3 of the metadata. That format
is not defined yet, so it's not obligatory for it to remain a flat key
value store. However, there are advantages to keeping it as such, so
I'm fine with Daniel's suggested approach.

The only thing I really care about is the namespacing, for the same
reasons the IETF wrote RFC 6648, as Petri linked earlier [1].
Establishing proper name registration rules can categorically
eliminate a bunch of problems further down the line (such as the past
confusion between which metadata entries were defined by PEPs and
which were setuptools-specific extensions that other tools might not
understand).

With PyPI based namespacing we get clear orthogonal naming with clear
lines of authority:

1. PEPs continue to define the core metadata used by PyPI, the
standard library (once we get updated packaging support in place) and
most other tools
2. Any members of the community with a specific interest can register
a PyPI project to define additional metadata without risking naming
conflicts. This need may arise in the context of a specific project,
and thus use that project's name, or else it may be a project
registered for the express purpose of being a metadata namespace, and
not actually correspond to any installable module. The main point is
to take advantage of an existing automated Python-specific name and
resource registry to avoid naming conflicts without Java-style reverse
DNS based clutter, and without python-dev having to explicitly approve
each and every metadata extension.

Cheers,
Nick.

[1] https://tools.ietf.org/html/rfc6648#section-4

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


donald.stufft at gmail

Aug 28, 2012, 6:48 AM

Post #22 of 50 (449 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tuesday, August 28, 2012 at 9:41 AM, Nick Coghlan wrote:
>
> The only thing I really care about is the namespacing, for the same
> reasons the IETF wrote RFC 6648, as Petri linked earlier [1].
> Establishing proper name registration rules can categorically
> eliminate a bunch of problems further down the line (such as the past
> confusion between which metadata entries were defined by PEPs and
> which were setuptools-specific extensions that other tools might not
> understand).
>
>
I'm happy with any form of a namespace to be quite honest. I have a bit of
a preference for no or flat namespace but i'm perfectly fine with a PyPI based
namespace. The important part is a defined way to extend the data that
even when tools don't understand the extended data they can losslessly
move it around from setup.cfg/setup.py/whatever to METADATA and
any other format, even if they themselves don't utilize it, leaving it intact
for tools that _do_ utilize it.


ncoghlan at gmail

Aug 28, 2012, 7:15 AM

Post #23 of 50 (449 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

On Tue, Aug 28, 2012 at 11:48 PM, Donald Stufft <donald.stufft [at] gmail> wrote:
> On Tuesday, August 28, 2012 at 9:41 AM, Nick Coghlan wrote:
>
> The only thing I really care about is the namespacing, for the same
> reasons the IETF wrote RFC 6648, as Petri linked earlier [1].
> Establishing proper name registration rules can categorically
> eliminate a bunch of problems further down the line (such as the past
> confusion between which metadata entries were defined by PEPs and
> which were setuptools-specific extensions that other tools might not
> understand).
>
>
> I'm happy with any form of a namespace to be quite honest. I have a bit of
> a preference for no or flat namespace but i'm perfectly fine with a PyPI
> based
> namespace. The important part is a defined way to extend the data that
> even when tools don't understand the extended data they can losslessly
> move it around from setup.cfg/setup.py/whatever to METADATA and
> any other format, even if they themselves don't utilize it, leaving it
> intact
> for tools that _do_ utilize it.

Oh, yes, I care about that part, too, as without that there's no
reason to define a metadata extension format at all :)

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Aug 28, 2012, 7:24 AM

Post #24 of 50 (449 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

Am 28.08.12 14:28, schrieb Daniel Holth:
> On Tue, Aug 28, 2012 at 8:07 AM, Donald Stufft<donald.stufft [at] gmail> wrote:
>> I personally think that at a minimum we should have X-Fields that
>> get moved into the normal METADATA file, and personally I would
>> prefer to just drop the X- prefix completely.
>
> That is my preference as well. The standard library basically ignores
> every metadata field or metadata file inside or outside of metadata
> currently, so where is the harm changing the official document to read
> "you may add new metadata fields to metadata" with an updated standard
> library that only ignores some of the metadata in metadata instead of
> all of it. The community is small enough to handle it.

The problem with that (and the reason to introduce the X- prefix in RFC
822) is that allowing arbitrary additions will make evolution difficult:
if you want to standardize a certain field at some point, you either
need to pick a name that is unused in all implementations (which you
never can be really certain about), or you break some existing tool by
making the addition (unless the addition happens to have the exact
same syntax and semantics as the prior use).

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Aug 28, 2012, 7:43 AM

Post #25 of 50 (449 views)
Permalink
Re: Edits to Metadata 1.2 to add extras (optional dependencies) [In reply to]

> But note that the RFC also says that the preferred solution to the
> problem that X-fields are intended to solve is an easily accessible
> name registry and a simple registration procedure. If Martin's "be
> prepared for a ten-year period to acceptance" is serious, what should
> be done about such a registry?

I'm happy for PyPI to host such a registry. A specificaion for the
registry should be part of the PEP for the 1.3 format, but I would
propose this structure (without having researched in detail what
other registries feature, but with a rough idea what IANA registries
typically include):

- name of metadata field
- name of registrant (individual or PyPI package)
- contact email address (published)
- expiration date; by default, extensions expire 1 month after
their registration, unless renewed; maximum expiration time is
5 years
- English description of the field
- regular expression to validate the field

Deleting undesired extensions would not be possible, instead, one
would have to create another extension if the syntax or semantics
changes

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.