Mailing List Archive: Python: Python

On-topic: alternate Python implementations

1 2

View All

Index | Next | Previous | View Threaded

no.email at nospam

Aug 4, 2012, 1:43 PM

Post #26 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Stefan Behnel <stefan_ml [at] behnel> writes:
>> Calling CPython hardly counts as compiling Python into C.
> CPython is written in C, though. So anything that CPython does can be
> done in C. It's not like the CPython project used a completely unusual
> way of writing C code.

CPython is a relatively simple interpreter, and executing code
by invoking such an interpreter IMHO doesn't count as "compiling" it in
any meaningful way.

> You will always need some kind of runtime infrastructure when you
> "compile Python into C", so you can just as well use CPython for that
> instead of reimplementing it completely from scratch.

Maybe there's parts of Cpython you can re-use, but having the CPython
interpreter be the execution engine for "compiled" Python generators
again fails the seriousness test of what it means to compile code. If
you mean something other than that, you might explain more clearly.

> Both Cython and Nuitka do exactly that,

I didn't know about Nuitka; it looks interesting but (at least after a
few minutes looking) I don't have much sense of how it works.

> No, you are going to compile only the generator function into a function
> that uses gotos, maybe with an additional in-out struct parameter that
> holds its state.

Yeah, ok, I guess that can work, given python generators are limited
to returning through just one stack level. You might want to avoid
copying locals by just putting everything into a struct, that has to
be retained across entries/exits.

> If you don't like that, you can experiment with anything from a dedicated
> GC to transactional memory.

OK, but then CPython is no longer managing the memory.

> Last I heard, PyPy had a couple of GCs to choose from,

PyPy doesn't compile to C, but I guess compiling to C doesn't preclude
precise GC, as long as the generated C code carefully tracks what C
objects can contain GC-able pointers, and follows some constraints about
when the GC can run. Some other compilers do this so it's not as big a
deal as it sounded like at first. OK.
>
>>> or provide none at all.
>> You're going to let the program just leak memory until it crashes??
> Well, it's not like CPython leaks memory until it crashes...

I was counting CPython's reference counting as a rudimentary form of GC,
though I guess that's terminology that not everyone agrees on.

> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
> compiler specifically for Lua code. How is that similar to the Jython
> runtime that runs *on top of* the JVM with its generic byte code based
> JIT compiler?

I thought LuaJIT compiles the existing Lua VM code, but I haven't
looked at it closely or used it.

>> It seems very hard to do reasonable optimizations in the presence of
>> standard Python techniques
>
> Sure. Even when targeting the CPython runtime with the generated C
> code (like Cython or Nuitka), you can still do a lot. And sure, static
> code analysis will never be able to infer everything that a JIT
> compiler can see.

I think even a JIT can't avoid a lot of pain and slowdown, without
complex whole-program analysis and requiring the application to follow
some special conventions, like never importing at "runtime".
--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 4, 2012, 2:24 PM

Post #27 of 43 (646 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Paul Rubin, 04.08.2012 22:43:
> Stefan Behnel writes:
>>> Calling CPython hardly counts as compiling Python into C.
>> CPython is written in C, though. So anything that CPython does can be
>> done in C. It's not like the CPython project used a completely unusual
>> way of writing C code.
>
> CPython is a relatively simple interpreter, and executing code
> by invoking such an interpreter IMHO doesn't count as "compiling" it in
> any meaningful way.

Oh, CPython is substantially more than an interpreter. The eval loop is
only *one* way to use the runtime environment. Remember that it has many
builtin types and functions as well as a huge standard library. Much of the
runtime environment is already written in C or can be compiled down to C.
If you compile Python code into C code that avoids the eval loop and only
uses the CPython runtime environment (which is what Cython does), I think
that qualifies as compiling Python code to C. It's definitely the most
practical and user friendly way to do it.

>> You will always need some kind of runtime infrastructure when you
>> "compile Python into C", so you can just as well use CPython for that
>> instead of reimplementing it completely from scratch.
>
> Maybe there's parts of Cpython you can re-use, but having the CPython
> interpreter be the execution engine for "compiled" Python generators
> again fails the seriousness test of what it means to compile code. If
> you mean something other than that, you might explain more clearly.

See above.

>> Both Cython and Nuitka do exactly that,
>
> I didn't know about Nuitka; it looks interesting but (at least after a
> few minutes looking) I don't have much sense of how it works.

It's mostly like Cython but without the type system, i.e. without all the
stuff that makes it useful in real life. Just a bare
Python-to-C++-in-CPython compiler, without much of a way to make it do what
you want.

>> Last I heard, PyPy had a couple of GCs to choose from,
>
> PyPy doesn't compile to C

RPython (usually) does, though, and my guess is that the memory management
part of the runtime is written in RPython.

> but I guess compiling to C doesn't preclude
> precise GC, as long as the generated C code carefully tracks what C
> objects can contain GC-able pointers, and follows some constraints about
> when the GC can run. Some other compilers do this so it's not as big a
> deal as it sounded like at first. OK.

Yep, C really becomes a lot nicer when you generate it.

>> Huh? LuaJIT is a reimplementation of Lua that uses an optimising JIT
>> compiler specifically for Lua code. How is that similar to the Jython
>> runtime that runs *on top of* the JVM with its generic byte code based
>> JIT compiler?
>
> I thought LuaJIT compiles the existing Lua VM code, but I haven't
> looked at it closely or used it.

Ok. It obviously reuses code, but the VM part of it is really different
from standard Lua.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

james.pye at gmail

Aug 4, 2012, 3:05 PM

Post #28 of 43 (645 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Friday, August 3, 2012 11:15:20 PM UTC-7, Steven D'Aprano wrote:
> WPython - another optimizing version of Python with wordcodes instead of
> bytecodes.
>
> http://code.google.com/p/wpython/

I remember reading about this a while ago. I thought this was eventually going to be committed to CPython... =\
--
http://mail.python.org/mailman/listinfo/python-list

jae+python at jaerhard

Aug 4, 2012, 4:25 PM

Post #29 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
> Steven D'Aprano, 04.08.2012 08:15:
> > Most people are aware, if only vaguely, of the big Four Python
> > implementations:
> >
>
> And not to forget Cython, which is the only static Python compiler that is
> widely used. Compiles and optimises Python to C code that uses the CPython
> runtime and allows for easy manual optimisations to get C-like performance
> out of it.

Cython is certainly *not* a Python *implementation*, since it always
uses the CPython runtime (and compiling Cython C files requires
Python.h).

None of the other implementations require Python for actually
compiling or running Python source.

Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
By embedding the Python runtime (dynamic linking with libpythonX.Y...
maybe static too? Didn't test, because it's irrelevant for making the
point).

Grits, J
--
http://mail.python.org/mailman/listinfo/python-list

steve+comp.lang.python at pearwood

Aug 4, 2012, 5:54 PM

Post #30 of 43 (645 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Sat, 04 Aug 2012 08:59:18 -0700, Paul Rubin wrote:

> C isn't so great for high-assurance stuff either, compared to (say) Ada.
> People do use it in critical apps, but that's just because it is (or
> anyway used to be) so ubiquitous.

And then they are shocked, SHOCKED I say!, when their app has enough
buffer overflow security vulnerabilities to sink a battleship.

[half a wink]

> Haskell doesn't sound all that great as a translation target for Python
> either, unfortunately, because its execution semantics are so different.

I have no opinion on that either way, except to say that if some
developer wants to experiment with Python-in-Haskell, good on him or her.
Trying something new is how progress is made.

[...]
> Finally, Python itself isn't all that well suited for compilation, given
> its high dynamicity. It will be interesting to see if the language
> evolves due to PyPy.

Python is a dynamic language, but most Python code is relatively static.
Runtime optimizations that target the common case, but fall back to
unoptimized code in the rare cases that the optimization doesn't apply,
offer the opportunity of big speedups for most code at the cost of
trivial slowdowns when you do something unusual.

--
Steven
--
http://mail.python.org/mailman/listinfo/python-list

no.email at nospam

Aug 4, 2012, 6:38 PM

Post #31 of 43 (646 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Steven D'Aprano <steve+comp.lang.python [at] pearwood> writes:
> Runtime optimizations that target the common case, but fall back to
> unoptimized code in the rare cases that the optimization doesn't apply,
> offer the opportunity of big speedups for most code at the cost of
> trivial slowdowns when you do something unusual.

The problem is you can't always tell if the unusual case is being
exercised without an expensive dynamic check, which in some cases must
be repeated in every iteration of a critical inner loop, even though it
turns out that the program never actually uses the unusual case.
--
http://mail.python.org/mailman/listinfo/python-list

steve+comp.lang.python at pearwood

Aug 4, 2012, 7:19 PM

Post #32 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:

> Steven D'Aprano <steve+comp.lang.python [at] pearwood> writes:
>> Runtime optimizations that target the common case, but fall back to
>> unoptimized code in the rare cases that the optimization doesn't apply,
>> offer the opportunity of big speedups for most code at the cost of
>> trivial slowdowns when you do something unusual.
>
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

I never said optimizing Python was easy :)

Obviously if the check is expensive enough, the optimization isn't going
to be worth doing. But often the check is not so expensive, or is just a
matter of tedious and careful book-keeping.

I don't wish to dispute that optimizing Python is hard, but it's not a
Hard Problem like factorizing huge integers, or solving the Palestine/
Israeli conflict. It's hard like cleaning your house after a gang of
drunken frat boys have partied all weekend.

--
Steven
--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 4, 2012, 10:37 PM

Post #33 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Paul Rubin, 05.08.2012 03:38:
> Steven D'Aprano writes:
>> Runtime optimizations that target the common case, but fall back to
>> unoptimized code in the rare cases that the optimization doesn't apply,
>> offer the opportunity of big speedups for most code at the cost of
>> trivial slowdowns when you do something unusual.
>
> The problem is you can't always tell if the unusual case is being
> exercised without an expensive dynamic check, which in some cases must
> be repeated in every iteration of a critical inner loop, even though it
> turns out that the program never actually uses the unusual case.

Cython does a lot of optimistic optimisations. That's where a large part of
that huge C file comes from that Cython generates from even simple Python code.

For example, in CPython, C function calls are so ridiculously faster than
Python function calls that it's worth some effort if it saves you from
packing an argument tuple to call into a Python function. In fact, we've
been thinking about ways to export C signatures from Python function
objects, so that code implemented in C (or a C compatible language) can be
called directly from other code implemented in C. That's very common in the
CPython ecosystem.

There are a lot of simple things that quickly add up into a much better
performance on average.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 4, 2012, 10:46 PM

Post #34 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Jürgen A. Erhard, 05.08.2012 01:25:
> On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
>> Steven D'Aprano, 04.08.2012 08:15:
>>> Most people are aware, if only vaguely, of the big Four Python
>>> implementations:
>>
>> And not to forget Cython, which is the only static Python compiler that is
>> widely used. Compiles and optimises Python to C code that uses the CPython
>> runtime and allows for easy manual optimisations to get C-like performance
>> out of it.
>
> Cython is certainly *not* a Python *implementation*, since it always
> uses the CPython runtime (and compiling Cython C files requires
> Python.h).

Yes, it avoids an unnecessary duplication of effort as well as a
substantial loss of compatibility that all non-CPython based
implementations suffer from.

You'd be surprised to see how much of Python we implement, though,
including some of the builtins. You might want to revise your opinion once
you start digging into it. It's always easy to disagree at the surface.

> None of the other implementations require Python for actually
> compiling or running Python source.

Nuitka was on the list as well.

> Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
> By embedding the Python runtime (dynamic linking with libpythonX.Y...
> maybe static too?

Sure, that works.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 5, 2012, 12:51 AM

Post #35 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Stefan Behnel, 05.08.2012 07:46:
> Jürgen A. Erhard, 05.08.2012 01:25:
>> None of the other implementations require Python for actually
>> compiling or running Python source.
>
> Nuitka was on the list as well.

Oh, and Stackless was also on Steven's list, as well as WPython. That means
that 50% of the "other implementations" that Steven presented are not
"implementations" according to your apparent definition.

BTW, what is you definition?

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

jae+python at jaerhard

Aug 5, 2012, 5:28 AM

Post #36 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote:
> J�rgen A. Erhard, 05.08.2012 01:25:
> > On Sat, Aug 04, 2012 at 08:40:16AM +0200, Stefan Behnel wrote:
> >> Steven D'Aprano, 04.08.2012 08:15:
> >>> Most people are aware, if only vaguely, of the big Four Python
> >>> implementations:
> >>
> >> And not to forget Cython, which is the only static Python compiler that is
> >> widely used. Compiles and optimises Python to C code that uses the CPython
> >> runtime and allows for easy manual optimisations to get C-like performance
> >> out of it.
> >
> > Cython is certainly *not* a Python *implementation*, since it always
> > uses the CPython runtime (and compiling Cython C files requires
> > Python.h).
>
> Yes, it avoids an unnecessary duplication of effort as well as a
> substantial loss of compatibility that all non-CPython based
> implementations suffer from.

But it's not an Python *implementation*, "just" an extension.

Mind you, this is not intended as a slight of Cython as such. I
really like it, though I haven't had need for it yet, but I sure
prefer it to writing extensions in pure C. *brrrr*

> > None of the other implementations require Python for actually
> > compiling or running Python source.
>
> Nuitka was on the list as well.

True, which I realized only after my missive. But doesn't change
much, only that the list is wrong.

> > Oh, yes, you can create a stand-alone... wait, a "stand-alone" app.
> > By embedding the Python runtime (dynamic linking with libpythonX.Y...
> > maybe static too?
>
> Sure, that works.

My definition, to also answer your following post, is "does not rely
on any executable part of the CPython source (which includes .c files
and executable code in header files if any, but of course can exclude
the stdlib)". Not sure that's precise enough, but... if it can't
run/work on a system that has no shred of CPython installed, it's not
an alternative *implementation*. The big three don't need CPython
(except PyPy for building, and even it can use a precompile PyPy I think).

Grits, J
--
http://mail.python.org/mailman/listinfo/python-list

ethan at stoneleaf

Aug 5, 2012, 7:27 AM

Post #37 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Mark Lawrence wrote:
> With arrogance like that German by any chance?

Comments like that are not appropriate on this list. Please don't make
them.

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

ben+python at benfinney

Aug 5, 2012, 8:21 AM

Post #38 of 43 (644 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Mark Lawrence <breamoreboy [at] yahoo> writes:

> With arrogance like that German by any chance?

Please keep derogatory national stereotypes off this forum and out of
our community. They are counter to our goals of diversity
<URL:http://www.python.org/community/diversity/>; you don't have to
subscribe to that, but if not then you thereby exclude yourself.

--
\ “You can stand tall without standing on someone. You can be a |
`\ victor without having victims.” —Harriet Woods, 1927–2007 |
_o__) |
Ben Finney
--
http://mail.python.org/mailman/listinfo/python-list

wuwei23 at gmail

Aug 5, 2012, 8:40 PM

Post #39 of 43 (645 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On Aug 4, 4:15�pm, Steven D'Aprano <steve
+comp.lang.pyt...@pearwood.info> wrote:
> But the Python ecosystem is a lot bigger than just those four. Here are
> just a few other implementations that you might be interested in:

There's also HotPy:

http://code.google.com/p/hotpy/
http://www.hotpy.org/

--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 5, 2012, 11:21 PM

Post #40 of 43 (645 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

alex23, 06.08.2012 05:40:
> On Aug 4, 4:15 pm, Steven D'Aprano <steve
> +comp.lang.pyt...@pearwood.info> wrote:
>> But the Python ecosystem is a lot bigger than just those four. Here are
>> just a few other implementations that you might be interested in:
>
> There's also HotPy:
>
> http://code.google.com/p/hotpy/
> http://www.hotpy.org/

And just in case anyone was wondering where the others are:

http://wiki.python.org/moin/PythonImplementations

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

stefan_ml at behnel

Aug 5, 2012, 11:46 PM

Post #41 of 43 (645 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Jürgen A. Erhard, 05.08.2012 14:28:
> On Sun, Aug 05, 2012 at 07:46:59AM +0200, Stefan Behnel wrote:
>> Jürgen A. Erhard, 05.08.2012 01:25:
>>> None of the other implementations require Python for actually
>>> compiling or running Python source.
>>
>> Nuitka was on the list as well.
>
> True, which I realized only after my missive. But doesn't change
> much, only that the list is wrong.

Agreed.

> My definition, to also answer your following post, is "does not rely
> on any executable part of the CPython source (which includes .c files
> and executable code in header files if any, but of course can exclude
> the stdlib)". Not sure that's precise enough, but... if it can't
> run/work on a system that has no shred of CPython installed, it's not
> an alternative *implementation*.

I can live with that definition. Cython is (by design) not an independent
reimplementation of Python.

Stefan

--
http://mail.python.org/mailman/listinfo/python-list

nagle at animats

Aug 6, 2012, 10:57 PM

Post #42 of 43 (647 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

On 8/4/2012 7:19 PM, Steven D'Aprano wrote:
> On Sat, 04 Aug 2012 18:38:33 -0700, Paul Rubin wrote:
>
>> Steven D'Aprano <steve+comp.lang.python [at] pearwood> writes:
>>> Runtime optimizations that target the common case, but fall back to
>>> unoptimized code in the rare cases that the optimization doesn't apply,
>>> offer the opportunity of big speedups for most code at the cost of
>>> trivial slowdowns when you do something unusual.
>>
>> The problem is you can't always tell if the unusual case is being
>> exercised without an expensive dynamic check, which in some cases must
>> be repeated in every iteration of a critical inner loop, even though it
>> turns out that the program never actually uses the unusual case.

There are other approaches. PyPy uses two interpreters and a JIT
compiler to handle the hard cases. When code does something unexpected
to other code, the backup interpreter is used to get control out of
the trouble spot so that the JIT compiler can then recompile the
code. (I think; I've read the paper but haven't looked at the
internals.)

This is hard to implement and hard to get right.

John Nagle
--
http://mail.python.org/mailman/listinfo/python-list

ulrich.eckhardt at dominolaser

Aug 8, 2012, 1:29 AM

Post #43 of 43 (621 views)
Permalink

Re: On-topic: alternate Python implementations [In reply to]

Am 04.08.2012 15:53, schrieb Stefan Behnel:
> So, if a C++ compiler takes a .c file and compiles it with C language
> semantics, it doesn't qualify as a C compiler? That implies a rather weird
> definition of a C compiler, I'd say.

I'd say that even a brainfuck compiler compiling a .py file with C
language semantics can shamelessly call itself a C compiler. :P

If a C++ compiler is given C code, it may or may not produce equivalent
executables. In most non-trivial cases it will just barf on the valid C
/ invalid C++ code and refuse to compile it. In few rare cases, it will
compile the code and produce different behaviour at runtime (e.g. for
"sizeof 'a'").

Uli
--
http://mail.python.org/mailman/listinfo/python-list

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads