Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Install Hook [Was: Re: PEP 414 updated]

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


armin.ronacher at active-4

Mar 4, 2012, 6:43 AM

Post #1 of 7 (422 views)
Permalink
Install Hook [Was: Re: PEP 414 updated]

Hi,

Jut to reiterate what I wrote on IRC:

Please do not write or advocate for import hooks, especially not for
porting purposes. It would either mean that people start adding that
hook on their own to the code (and that awfully reminds me of the days
of 'require "rubygems"' in the Ruby world) or that the __init__.py has
to do that and that's a non trivial thing.

The hook on install time works perfectly fine and the only situation
where it might not work is when you're trying to use Python 3.2 for
development and also support down to 2.x by using the newly introduced
u-prefixes. In this case I would recommend using Python 3.3 for
development and running the testsuite periodically from Python 3.2 after
installating the library (into a virtualenv for instance).

The current work in progress install time hook can be found here:
https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook


Regards,
Armin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Mar 4, 2012, 8:44 AM

Post #2 of 7 (416 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

On Sun, Mar 4, 2012 at 6:43 AM, Armin Ronacher
<armin.ronacher [at] active-4> wrote:
> Please do not write or advocate for import hooks, especially not for
> porting purposes.  It would either mean that people start adding that
> hook on their own to the code (and that awfully reminds me of the days
> of 'require "rubygems"' in the Ruby world) or that the __init__.py has
> to do that and that's a non trivial thing.

I'd love a pointer to the rubygems debacle...

> The hook on install time works perfectly fine and the only situation
> where it might not work is when you're trying to use Python 3.2 for
> development and also support down to 2.x by using the newly introduced
> u-prefixes.  In this case I would recommend using Python 3.3 for
> development and running the testsuite periodically from Python 3.2 after
> installating the library (into a virtualenv for instance).

+1

> The current work in progress install time hook can be found here:
>  https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook

Yee!

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Mar 4, 2012, 1:00 PM

Post #3 of 7 (418 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

Armin Ronacher <armin.ronacher <at> active-4.com> writes:

> The current work in progress install time hook can be found here:
> https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook

I realise that the implementation is different, using tokenize rather than
lib2to3, but in terms of its effect on the transformed code, what are the
differences between this hook and running 2to3 with just the fix_unicode fixer?

Regards,

Vinay Sajip




_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


armin.ronacher at active-4

Mar 4, 2012, 2:31 PM

Post #4 of 7 (411 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

Hi,

On 3/4/12 4:44 PM, Guido van Rossum wrote:
> I'd love a pointer to the rubygems debacle...
Setuptools worked because Python had .pth files for a long, long time.
When the Ruby world started moving packages into nonstandard locations
(GameName/<the files in that gem>) something needed to activate that
import machinery hack. For a while all Ruby projects had the line
"require 'rubygems'" somewhere in the project. Some libraries even
shipped that line to bootstrap rubygems.

I think an article about that should be found here:
http://tomayko.com/writings/require-rubygems-antipattern

But since the page errors out currently I don't know if that is the one
I'm referring to.

Considering such an import hook has to run over all imports because it
would not know which to rewrite and which not I think it would be
equally problematic, especially if libraries would magically activate
that hook.


Regards,
Armin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


armin.ronacher at active-4

Mar 4, 2012, 2:44 PM

Post #5 of 7 (410 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

Hi,

On 3/4/12 9:00 PM, Vinay Sajip wrote:
> I realise that the implementation is different, using tokenize rather than
> lib2to3, but in terms of its effect on the transformed code, what are the
> differences between this hook and running 2to3 with just the fix_unicode fixer?
I would hope they both have the same effect. Namely stripping the 'u'
prefix in all variations.

Why did I go with the tokenize approach? Because I never even
considered a 2to3 solution. Part of the reason why I wrote this PEP was
that 2to3 is so awfully slow and I was assuming that this would be
largely based on the initial parsing step and not the fixers themselves.
Why did I not time it with just the unicode fixer? Because if you look
at how simple the tokenize version is you can see that this one did not
take me more than a good minute and maybe 10 more for the distutils hooking.


Regards,
Armin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Mar 4, 2012, 3:04 PM

Post #6 of 7 (410 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

Armin Ronacher <armin.ronacher <at> active-4.com> writes:

> Considering such an import hook has to run over all imports because it
> would not know which to rewrite and which not I think it would be
> equally problematic, especially if libraries would magically activate
> that hook.

You could be right, but it sounds a little alarmist to me - "problematic" -
"magical". For example, in the current implementation of uprefix, the hook does
nothing for files in the stdlib, could be refined to be more intelligent about
what to run on, etc. Plus, as Zbigniew pointed out in his post, ways could be
found (e.g. via a context manager) to give users control of when the hook runs.

I'm not sure your rubygems example is analogous - I would have thought the
equivalent for Python would be to stick "import setuptools" everywhere, which
is not an anti-pattern we lose sleep over, AFAIK.

It's early days, but it seems reasonable to document in the usage of the hook
that it is intended to be used in certain ways and not in others. IIRC Ryan's
post was doing just that - telling people how the requiring of rubygems should
work.

AFAIK the approach hasn't been tried before, and was suggested by Nick (so I
assume is not completely off the wall). My particular implementation might have
holes in it (feedback welcome on any such, and I'll try to fix them) but I
would think the approach could be given a chance in some realistic scenarios to
see what problems emerge in practice, rather than trying to shoot it down
before it's even got going.

Regards,

Vinay Sajip

"It is not enough merely to win; others must lose." - Gore Vidal

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Mar 4, 2012, 3:19 PM

Post #7 of 7 (411 views)
Permalink
Re: Install Hook [Was: Re: PEP 414 updated] [In reply to]

Armin Ronacher <armin.ronacher <at> active-4.com> writes:

> I would hope they both have the same effect. Namely stripping the 'u'
> prefix in all variations.

Okay, that's all I was curious about.

> Why did I go with the tokenize approach? Because I never even
> considered a 2to3 solution. Part of the reason why I wrote this PEP was
> that 2to3 is so awfully slow and I was assuming that this would be
> largely based on the initial parsing step and not the fixers themselves.
> Why did I not time it with just the unicode fixer? Because if you look
> at how simple the tokenize version is you can see that this one did not
> take me more than a good minute and maybe 10 more for the distutils hooking.

You don't need to justify your approach - to me, anyway ;-) I suppose tokenize
needed changing because of the grammar change, so it seems reasonable to put
the changed version to work.

I agree that 2to3 seems slow sometimes, but I can't say I've pinned it down as
to exactly where the time is spent. I assumed that it was just because it seems
to run over a lot of files each time, regardless of whether they've been
changed since the last run or not. (I believe there might be ways of optimising
that, but my understanding is that in the default/simple cases it runs over
everything.)

I factored out the transformation step in my hook into a method, so I should be
able to swap out the lib2to3 approach with a tokenize approach without too much
work, should that prove necessary or desirable.

Regards,

Vinay Sajip

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.