Mailing List Archive: Python: Dev

Packaging documentation and packaging.pypi API

Index | Next | Previous | View Threaded

p.f.moore at gmail

Jun 20, 2012, 8:29 AM

Post #1 of 7 (176 views)
Permalink

Packaging documentation and packaging.pypi API

On 20 June 2012 14:47, Paul Moore <p.f.moore [at] gmail> wrote:
> On 20 June 2012 14:16, Alexis Métaireau <alexis [at] notmyidea> wrote:
>> packaging.pypi is functionally working but IMO the API can (and probably
>> should) be improved (we really lack feedback to know that).
>
> I wasn't aware of this - I've had a look and my first thought is that
> the documentation needs completing. At the moment, there's a lot that
> isn't documented, and we should avoid getting into the same situation
> as with distutils where people have to use undocumented APIs to get
> anything done. There are a lot of examples, but not so much formal API
> documentation.

As a specific example, one thing I would like to do is to be able to
set up a packaging.pypi client object that lets me query and download
distributions. However, rather than just querying PyPI (the default)
I'd like to be able to set up a list of locations (PyPI, a local
server, and maybe some distribution files stored on my PC) and combine
results from all of them. This differs from the mirror support in that
I want to combine the lists, not use one as a fallback if the other
doesn't exist. From the documentation, I can't tell if this is
possible, or a feature request, or unsupported... (Actually, there's
not even any documentation saying how the URL(s) in index_url should
behave, so how exactly do I set up a local repository anyway...?)

On a similar note, at some point, crawler.get_releases('pywin32')
needs to work. I believe the issue here is technically with pywin32,
which uses non-standard version numbers (214) and is hosted externally
(Sourceforge) but I'd expect that packaging.pypi should be able to
access anything that's on PyPI, even if other APIs like
packaging.version can't deal with them.

Ideally, these would be simply things I'd raise as issues on
bugs.python.org. But as things stand, such issues aren't getting
fixed, and we don't move forward. And without the documentation to
back up a debate, it's hard to argue "X is a bug, Y is a feature
request, Z is behaving correctly".

Paul.

PS As I write this, it suggests to me that maybe even the "pick out
the stable APIs" approach isn't as simple as we'd like to think it is.
You can either read this email as a set of specific documentation
points to fix, or as evidence that we should drop packaging.pypi as
well for now, even though it's a pretty cool feature...
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

carl at oddbird

Jun 20, 2012, 9:07 AM

Post #2 of 7 (173 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

Hi Paul,

On 06/20/2012 09:29 AM, Paul Moore wrote:
> As a specific example, one thing I would like to do is to be able to
> set up a packaging.pypi client object that lets me query and download
> distributions. However, rather than just querying PyPI (the default)
> I'd like to be able to set up a list of locations (PyPI, a local
> server, and maybe some distribution files stored on my PC) and combine
> results from all of them. This differs from the mirror support in that
> I want to combine the lists, not use one as a fallback if the other
> doesn't exist. From the documentation, I can't tell if this is
> possible, or a feature request, or unsupported... (Actually, there's
> not even any documentation saying how the URL(s) in index_url should
> behave, so how exactly do I set up a local repository anyway...?)

This is perhaps a tangent, as your point here is to point out what the
API of packaging.pypi ought to allow - but pip's PackageFinder class can
easily do exactly this for you. Feel free to follow up with me for
details if this is actually still a problem you need to solve.

Carl
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Jun 20, 2012, 9:45 AM

Post #3 of 7 (165 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

On 20 June 2012 17:07, Carl Meyer <carl [at] oddbird> wrote:
> Hi Paul,
>
> On 06/20/2012 09:29 AM, Paul Moore wrote:
>> As a specific example, one thing I would like to do is to be able to
>> set up a packaging.pypi client object that lets me query and download
>> distributions. However, rather than just querying PyPI (the default)
>> I'd like to be able to set up a list of locations (PyPI, a local
>> server, and maybe some distribution files stored on my PC) and combine
>> results from all of them. This differs from the mirror support in that
>> I want to combine the lists, not use one as a fallback if the other
>> doesn't exist. From the documentation, I can't tell if this is
>> possible, or a feature request, or unsupported... (Actually, there's
>> not even any documentation saying how the URL(s) in index_url should
>> behave, so how exactly do I set up a local repository anyway...?)
>
> This is perhaps a tangent, as your point here is to point out what the
> API of packaging.pypi ought to allow - but pip's PackageFinder class can
> easily do exactly this for you. Feel free to follow up with me for
> details if this is actually still a problem you need to solve.

Thanks - as you say, it's not so much the actual problem as the
principle of what the packaging API offers that matters here. Although
it does make a good point - to what extent do the packaging APIs draw
on existing experience like that of pip? Given that tools like pip are
used widely to address real requirements, it would seem foolish to
*not* draw on that experience in designing a stdlib API.

(As regards my actual use, it's very much a "back burner" project of
mine - I keep dabbling with writing utilities to grab bdist_wininst
installers and unpack/install them in virtualenvs. The PackageFinder
class might well be useful there. I'll keep it in mind for the next
time I go back to that problem).

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

alexis at notmyidea

Jun 20, 2012, 9:57 AM

Post #4 of 7 (163 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

Le mer. 20 juin 2012 18:45:23 CEST, Paul Moore a Ã©crit :
> Thanks - as you say, it's not so much the actual problem as the
> principle of what the packaging API offers that matters here. Although
> it does make a good point - to what extent do the packaging APIs draw
> on existing experience like that of pip? Given that tools like pip are
> used widely to address real requirements, it would seem foolish to
> *not* draw on that experience in designing a stdlib API.

IIRC, pip relies nly onthe XML/RPC API to get information about the
distributions from the cheeseshop. the code that's in packaging.pypi
was built with the implementation in setuptools in mind, so we keep
compatibility with setuptools "easy_install".

But this is for the internal implementation. You're right and I will
have a deep look at what the API in pip looks like when it comes to
download packages from PyPI.

That is, this leverages one question more on my side: was/is pip
intended to be used as a library rather than as a tool / are there some
people that are actually building tools on top of pip this way?
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

alexis at notmyidea

Jun 20, 2012, 10:05 AM

Post #5 of 7 (166 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

On 20/06/2012 17:29, Paul Moore wrote:
>>
>> I wasn't aware of this - I've had a look and my first thought is that
>> the documentation needs completing. At the moment, there's a lot that
>> isn't documented, and we should avoid getting into the same situation
>> as with distutils where people have to use undocumented APIs to get
>> anything done. There are a lot of examples, but not so much formal API
>> documentation.

So that's something we definitely want to fix. The code is heavily annotated, and this had been made to generate the documentation automatically with sphinx in the first time, so… that would make no sense to not make it.
This is for the format API documentation, which seems to be easy to hook to sphinx.

I'll also review all the documentation there to make sure that it perfectly makes sense.

> As a specific example, one thing I would like to do is to be able to
> set up a packaging.pypi client object that lets me query and download
> distributions. However, rather than just querying PyPI (the default)
> I'd like to be able to set up a list of locations (PyPI, a local
> server, and maybe some distribution files stored on my PC) and combine
> results from all of them. This differs from the mirror support in that
> I want to combine the lists, not use one as a fallback if the other
> doesn't exist. From the documentation, I can't tell if this is
> possible, or a feature request, or unsupported... (Actually, there's
> not even any documentation saying how the URL(s) in index_url should
> behave, so how exactly do I set up a local repository anyway...?)
that's not something possible out of the box using the crawler the way
they are defined (iow, that's not one supported use case), *but* it's
possible to make a class on top of the existing ones which could provide
this kind of fallback feature. I'm not sure that we want or don't want
that to be a part of packaging.pypi, but that's definitely something
that this API makes possible without too much trouble.
> On a similar note, at some point, crawler.get_releases('pywin32')
> needs to work. I believe the issue here is technically with pywin32,
> which uses non-standard version numbers (214) and is hosted externally
> (Sourceforge) but I'd expect that packaging.pypi should be able to
> access anything that's on PyPI, even if other APIs like
> packaging.version can't deal with them.
If this is not working / following the links that are present in the
cheeseshp then this should be considered a bug.
> Ideally, these would be simply things I'd raise as issues on
> bugs.python.org. But as things stand, such issues aren't getting
> fixed, and we don't move forward. And without the documentation to
> back up a debate, it's hard to argue "X is a bug, Y is a feature
> request, Z is behaving correctly".
Alright, so this is a true documentation issue. I will make it clearer
what packaging.pypi makes and doesn't make possible.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

carl at oddbird

Jun 20, 2012, 10:09 AM

Post #6 of 7 (171 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

Hi Alexis,

On 06/20/2012 10:57 AM, Alexis MÃ©taireau wrote:
> Le mer. 20 juin 2012 18:45:23 CEST, Paul Moore a Ã©crit :
>> Thanks - as you say, it's not so much the actual problem as the
>> principle of what the packaging API offers that matters here. Although
>> it does make a good point - to what extent do the packaging APIs draw
>> on existing experience like that of pip? Given that tools like pip are
>> used widely to address real requirements, it would seem foolish to
>> *not* draw on that experience in designing a stdlib API.
>
> IIRC, pip relies nly onthe XML/RPC API to get information about the
> distributions from the cheeseshop. the code that's in packaging.pypi was
> built with the implementation in setuptools in mind, so we keep
> compatibility with setuptools "easy_install".

No, this is not accurate. Pip's PackageFinder uses setuptools-compatible
link-scraping, not the XMLRPC API, and it is the PackageFinder that is
used to actually find distributions to install. I think PackageFinder is
pretty much equivalent to what packaging.pypi is intended to do.

Pip does have a separate "search" command that uses the XMLRPC API -
this is the only part of pip that uses XMLRPC. I consider this a bug in
pip, because the results can be inconsistent with actual installation
using PackageFinder, and "search" can't be used with mirrors or private
indexes (unless they implement the XMLRPC API). The "search" command
should just use PackageFinder instead.

> That is, this leverages one question more on my side: was/is pip
> intended to be used as a library rather than as a tool / are there some
> people that are actually building tools on top of pip this way?

Pip's internal APIs are not documented, and they aren't the cleanest
APIs ever, but some of them (particularly PackageFinder and
InstallRequirement/RequirementSet) can be used without too much
difficulty, and some people are using them. Not a lot of people, I don't
think; I don't have hard numbers. I haven't seen much in the way of
public reusable tools built atop pip, but I've talked with a few people
building internal deployment tools that use pip as a library.

Carl
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

alexis at notmyidea

Jun 20, 2012, 10:44 AM

Post #7 of 7 (158 views)
Permalink

Re: Packaging documentation and packaging.pypi API [In reply to]

Hi Carl,

Thanks for clarifying this.
This means that indeed we have the same goals. I'll have a closer look
at the internal pip APIs, as they are probably really useful and already
used in production environment :)

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads