Mailing List Archive: Python: Dev

AST optimizer implemented in Python

1 2

View All

Index | Next | Previous | View Threaded

guido at python

Aug 13, 2012, 7:45 AM

Post #26 of 32 (214 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

Not so fast. If you make this a language feature you force all Python
implementations to support an identical AST API. That's a big step.

Not that AST manipulation isn't cool -- but I'd like to warn against
over-enthusiasm that might backfire on the language (or its community)
as a whole.

--Guido

On Mon, Aug 13, 2012 at 1:34 AM, Mark Shannon <mark [at] hotpy> wrote:
> Brett Cannon wrote:
>>
>>
>>
>> On Sat, Aug 11, 2012 at 8:16 PM, Eric Snow <ericsnowcurrently [at] gmail
>> <mailto:ericsnowcurrently [at] gmail>> wrote:
>>
>> On Sat, Aug 11, 2012 at 6:03 PM, Brett Cannon <brett [at] python
>> <mailto:brett [at] python>> wrote:
>> > It would also be very easy to expand importlib.abc.SourceLoader
>> to add a
>> > method which is called with source and returns the bytecode to be
>> written
>> > out
>>
>> Yes, please. Not having to hack around this would be nice.
>>
>>
>> http://bugs.python.org/issue15627
>
>
> AST transformation is a lot more general than just optimization.
>
> Adding an AST transformation is a relatively painless way to add to
> Python the last element of lisp-ness that it lacks:
> Namely being able to treat code as data and transform it at runtime,
> after parsing but before execution.
>
> Some examples:
> Profiling code be added by an AST transformation.
> IMO this would have been a more elegant way to implement CProfile
> and similar profilers than the current approach.
>
> AST transformations allow DSLs to be implemented in Python
> (I don't know if that is a + or - ).
>
> Access to the AST of a function at runtime would also be of use to
> method-based dynamic optimizers, or dynamic de-optimizers for static
> compilers.
>
> All for the price of adding a single method to SourceLoader.
> What a bargain :)
>
> Cheers,
> Mark.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

tjreedy at udel

Aug 13, 2012, 12:00 PM

Post #27 of 32 (212 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

On 8/13/2012 10:45 AM, Guido van Rossum wrote:
> Not so fast. If you make this a language feature you force all Python
> implementations to support an identical AST API. That's a big step.

I have been wondering about this. One could think from the manuals that
we are there already. From the beginning of the ast chapter:

"The ast module helps Python applications to process trees of *the*
Python abstract syntax grammar. ... An abstract syntax tree can be
generated by passing ast.PyCF_ONLY_AST as a flag to the compile()
built-in function" (emphasis on *the* added).

and the entry for compile(): "Compile the source into a code or AST object."

I see nothing about ast possibly being CPython only. Should there be?

--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Aug 13, 2012, 3:33 PM

Post #28 of 32 (214 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

Implementations are currently required to *have* an AST (or else declare
non compliance with that particular flag to compile). They're definitely
not required to have the *same* AST, thus all AST manipulation, like
bytecode manipulation, is necessarily implementation dependent.

We don't even guarantee AST compatibility between versions of CPython.

I believe the appropriate warnings are already present in the ast module
docs, but there may be misleading wording elsewhere that needs to be
cleaned up.

Regards,
Nick.

--
Sent from my phone, thus the relative brevity :)
On Aug 14, 2012 5:03 AM, "Terry Reedy" <tjreedy [at] udel> wrote:

> On 8/13/2012 10:45 AM, Guido van Rossum wrote:
>
>> Not so fast. If you make this a language feature you force all Python
>> implementations to support an identical AST API. That's a big step.
>>
>
> I have been wondering about this. One could think from the manuals that we
> are there already. From the beginning of the ast chapter:
>
> "The ast module helps Python applications to process trees of *the* Python
> abstract syntax grammar. ... An abstract syntax tree can be generated by
> passing ast.PyCF_ONLY_AST as a flag to the compile() built-in function"
> (emphasis on *the* added).
>
> and the entry for compile(): "Compile the source into a code or AST
> object."
>
> I see nothing about ast possibly being CPython only. Should there be?
>
> --
> Terry Jan Reedy
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> ncoghlan%40gmail.com<http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com>
>

kristjan at ccpgames

Aug 14, 2012, 3:25 AM

Post #29 of 32 (211 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

> -----Original Message-----
> I moved the script to a new dedicated project on Bitbucket:
> https://bitbucket.org/haypo/astoptimizer
>
> Join the project if you want to help me to build a better optimizer!
>
> It now works on Python 2.5-3.3.

I had the idea (perhaps not an original one) that peephole optimization would be much better
done in python than in C. The C code is clunky and unwieldly, wheras python would be much
better suited, being able to use nifty regexes and the like.

The problem is, there exists only bytecode disassembler, no corresponding assembler.

Then I stumbled upon this project:
http://code.google.com/p/byteplay/
Sounds like just the ticket, disassemble the code, do transformations on it, then reassemble.
Haven't gotten further than that though :)

K

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Aug 14, 2012, 6:32 AM

Post #30 of 32 (209 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

2012/8/14 Kristján Valur Jónsson <kristjan [at] ccpgames>:
>> I moved the script to a new dedicated project on Bitbucket:
>> https://bitbucket.org/haypo/astoptimizer
>>
>> Join the project if you want to help me to build a better optimizer!
>>
>> It now works on Python 2.5-3.3.
>
> I had the idea (perhaps not an original one) that peephole optimization would be much better
> done in python than in C. The C code is clunky and unwieldly, wheras python would be much
> better suited, being able to use nifty regexes and the like.
>
> The problem is, there exists only bytecode disassembler, no corresponding assembler.

Why would you like to work on bytecode instead of AST? The AST
contains much more information, you can implement better optimizations
in AST. AST is also more convinient than bytecode.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kristjan at ccpgames

Aug 14, 2012, 7:33 AM

Post #31 of 32 (209 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

> -----Original Message-----
> From: Victor Stinner [mailto:victor.stinner [at] gmail]
> Sent: 14. ágúst 2012 13:32
> To: Kristján Valur Jónsson
> Cc: Python Dev
> Subject: Re: [Python-Dev] AST optimizer implemented in Python
> > The problem is, there exists only bytecode disassembler, no corresponding
> assembler.
>
> Why would you like to work on bytecode instead of AST? The AST contains
> much more information, you can implement better optimizations in AST. AST
> is also more convinient than bytecode.
>

We already optimize bytecode. But it seems much more could be done there.
It also seems like a simpler goal. Also, AST will need to be changed to bytecode at some point, and that bytecode could still be optimized in ways not available to the AST, I imagine.
Also, I understand bytecode, more or less :)

K
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

hrvoje.niksic at avl

Aug 14, 2012, 8:09 AM

Post #32 of 32 (206 views)
Permalink

Re: AST optimizer implemented in Python [In reply to]

On 08/14/2012 03:32 PM, Victor Stinner wrote:
>> I had the idea (perhaps not an original one) that peephole optimization would be much better
>> done in python than in C. The C code is clunky and unwieldly, wheras python would be much
>> better suited, being able to use nifty regexes and the like.
>>
>> The problem is, there exists only bytecode disassembler, no corresponding assembler.
>
> Why would you like to work on bytecode instead of AST? The AST
> contains much more information, you can implement better optimizations

AST allows for better high-level optimizations, but a real peephole
optimization pass is actually designed to optimize generated code. This
allows eliminating some inefficiencies which would be fairly hard to
prevent at higher levels - wikipedia provides some examples.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads