On-topic: alternate Python implementations

  • Thread starter Steven D'Aprano
  • Start date
S

Steven D'Aprano

Most people are aware, if only vaguely, of the big Four Python
implementations:

CPython, or just Python, the reference implementation written in C.
IronPython, written in .NET.
Jython, written in Java.
PyPy, the optimizing implementation written in Python (actually, it's
written in a subset of Python, RPython).

But the Python ecosystem is a lot bigger than just those four. Here are
just a few other implementations that you might be interested in:


Stackless - the "forgetten Python", Stackless is, I believe, the oldest
implementation behind only CPython itself. It's a fork of CPython with
the calling stack removed and fast and lightweight microthreads, and is
used extensively in EVE Online.

http://www.stackless.com/


Nuitka - optimising Python compiler written in C++, supports Python 2.6
and 2.7, claims to be up to twice as fast as CPython.

http://nuitka.net/pages/overview.html


WPython - another optimizing version of Python with wordcodes instead of
bytecodes.

http://code.google.com/p/wpython/


CLPython, an implementation of Python written in Common Lisp.

http://common-lisp.net/project/clpython/


CapPython is an experimental restricted version of Python with
capabilities.

http://plash.beasts.org/wiki/CapPython
http://en.wikipedia.org/wiki/Object-capability_model


Berp - a compiler which works by translating Python to Haskell and
compiling that.

https://github.com/bjpop/berp/wiki



Give them some love!
 
C

Chris Angelico

CLPython, an implementation of Python written in Common Lisp.

Berp - a compiler which works by translating Python to Haskell and
compiling that.

Okay. WHY? CLPython gives some reason, but how often do you need to
bridge that particular pair of languages? And why compile Python via
Haskell, when C is available as a "high level assembly language"?

The mind boggles...

ChrisA
 
S

Stefan Behnel

Steven D'Aprano, 04.08.2012 08:15:
Most people are aware, if only vaguely, of the big Four Python
implementations:

CPython, or just Python, the reference implementation written in C.
IronPython, written in .NET.
Jython, written in Java.
PyPy, the optimizing implementation written in Python (actually, it's
written in a subset of Python, RPython).

But the Python ecosystem is a lot bigger than just those four. Here are
just a few other implementations that you might be interested in:


Stackless - the "forgetten Python", Stackless is, I believe, the oldest
implementation behind only CPython itself. It's a fork of CPython with
the calling stack removed and fast and lightweight microthreads, and is
used extensively in EVE Online.

http://www.stackless.com/


Nuitka - optimising Python compiler written in C++, supports Python 2.6
and 2.7, claims to be up to twice as fast as CPython.

http://nuitka.net/pages/overview.html


WPython - another optimizing version of Python with wordcodes instead of
bytecodes.

http://code.google.com/p/wpython/


CLPython, an implementation of Python written in Common Lisp.

http://common-lisp.net/project/clpython/


CapPython is an experimental restricted version of Python with
capabilities.

http://plash.beasts.org/wiki/CapPython
http://en.wikipedia.org/wiki/Object-capability_model


Berp - a compiler which works by translating Python to Haskell and
compiling that.

https://github.com/bjpop/berp/wiki

And not to forget Cython, which is the only static Python compiler that is
widely used. Compiles and optimises Python to C code that uses the CPython
runtime and allows for easy manual optimisations to get C-like performance
out of it.

http://cython.org/

Stefan
 
S

Steven D'Aprano

And not to forget Cython, which is the only static Python compiler that
is widely used. Compiles and optimises Python to C code that uses the
CPython runtime and allows for easy manual optimisations to get C-like
performance out of it.

http://cython.org/

Cython is great, but I question that it is a *Python* implementation.
That's not a criticism of Cython, but it is different from Python. Take
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

Cython is very Python-like, but there is no doubt in my mind that it is a
superset of Python and therefore a different language.
 
S

Stefan Behnel

Steven D'Aprano, 04.08.2012 09:49:
Cython is great, but I question that it is a *Python* implementation.
That's not a criticism of Cython, but it is different from Python. Take
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

We never met in person, so I can't comment on the last part.

However, the above is Cython code and, yes, that's a different language.
Note that it uses a different file extension: ".pyx". Try putting the above
code into a .py file and compiling that. Cython will reject it and tell you
that "cimport" is not valid Python syntax.

Cython is very Python-like, but there is no doubt in my mind that it is a
superset of Python and therefore a different language.

As long as you don't use any features of the Cython language, it's plain
Python. That makes it a Python compiler in my eyes. The fact that you can
easily use C features to optimise your code (also in Python syntax, BTW)
doesn't impact that.

You mentioned a couple of other implementations and I'm sure they have
features (and bugs) that CPython doesn't have. Interaction with Lisp code
in CLPython, for example. I don't think additional features or language
implementation bugs make a Python implementation non-Python per se.

Also note that most of the less widely known "alternative Python
implementations" do not even publish their results of running the CPython
test suite, so how do you actually know they can run Python code?

Stefan
 
M

Mark Lawrence

Cython is great, but I question that it is a *Python* implementation.
That's not a criticism of Cython, but it is different from Python. Take
this example code from the tutorial:

from libc.math cimport sin

cdef double f(double x):
return sin(x*x)

If that's Python code, then I'm Ethel the Aardvark.

Cython is very Python-like, but there is no doubt in my mind that it is a
superset of Python and therefore a different language.

I agree so it's off topic and can't be discussed here. Isn't that
right, Stefan?
 
S

Steven D'Aprano

Okay. WHY? CLPython gives some reason, but how often do you need to
bridge that particular pair of languages? And why compile Python via
Haskell, when C is available as a "high level assembly language"?

For much the same reason that PyPy uses RPython when C is available.
Because Haskell is available as a high level non-assembly language.

Berp is based on the Glasgow Haskell Compiler, which is a modern,
efficient, optimizing compiler capable of producing excellent quality
machine code on Windows, Mac, Linux and many Unixes. It gives you all the
advantages of a high-level language with high-level data structures, type
inference, and a compiler capable of generating optimized, fast, machine
code.

Who would want to deal with C's idiosyncrasies, low-powered explicit type
system, difficult syntax, and core-dumps, when you could use something
better? Apart from C programmers, of course.
 
S

Stefan Behnel

Mark Lawrence, 04.08.2012 12:05:
I agree so it's off topic and can't be discussed here. Isn't that right,
Stefan?

Hmm, in case you are referring to a recent friendly and diplomatic request
of mine regarding a couple of people who were burdening a public high
volume mailing list with a purely private back-and-forth chat about having
beer and getting drunk - then, no, I don't think the discussion in this
thread qualifies as yet another example for that so far.

Stefan
 
S

Stefan Krah

Steven D'Aprano said:
Who would want to deal with C's idiosyncrasies, low-powered explicit type
system, difficult syntax, and core-dumps, when you could use something
better?

In the free software world, apparently many people like C. C is also
quite popular in the zero-fault software world: Several verification
tools do exist and Leroy et al. are writing a certified compiler for
C to plug the hole between the verified source code and the generated
assembly.


Stefan Krah
 
S

Stefan Behnel

Steven D'Aprano, 04.08.2012 12:54:
Berp is based on the Glasgow Haskell Compiler, which is a modern,
efficient, optimizing compiler capable of producing excellent quality
machine code on Windows, Mac, Linux and many Unixes. It gives you all the
advantages of a high-level language with high-level data structures, type
inference, and a compiler capable of generating optimized, fast, machine
code.

Although all those optimisations don't mean that Python code would run fast
on top of it. Just because you translate Python to another language and
platform doesn't mean that there's any benefit from the underlying platform
optimisations. Both PyPy and Cython run Python code faster than CPython,
but not because they eventually translate it into machine code but because
they optimise and specialise it along the way, based on its high-level code
constructs. One big success of the Unladen Swallow project was to show that
bare JIT compilation is mostly worthless for high level languages.

Who would want to deal with C's idiosyncrasies, low-powered explicit type
system, difficult syntax, and core-dumps, when you could use something
better?

The core developers of both CPython and Cython aim for exactly that. They
write C so you don't have to. But keep in mind that C is still *the* lingua
franca of software development. A major reason why Python is (slowly)
catching up these days is that the main implementation is written in C and
makes it easy to interface with C code.

Stefan
 
T

Thomas Rachel

Am 04.08.2012 11:10 schrieb Stefan Behnel:
As long as you don't use any features of the Cython language, it's plain
Python. That makes it a Python compiler in my eyes.

Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless,
C and C++ are distinct languages and so are Python and Cython.


Thomas
 
S

Stefan Behnel

Thomas Rachel, 04.08.2012 14:51:
Am 04.08.2012 11:10 schrieb Stefan Behnel:

Tell that the C++ guys. C++ is mainly a superset of C. But nevertheless, C
and C++ are distinct languages and so are Python and Cython.

So, if a C++ compiler takes a .c file and compiles it with C language
semantics, it doesn't qualify as a C compiler? That implies a rather weird
definition of a C compiler, I'd say.

Stefan
 
S

Stefan Behnel

Stefan Behnel, 04.08.2012 15:53:
Thomas Rachel, 04.08.2012 14:51:

So, if a C++ compiler takes a .c file and compiles it with C language
semantics, it doesn't qualify as a C compiler? That implies a rather weird
definition of a C compiler, I'd say.

Ah, sorry. Got it. You were again talking about Cython the language. Sure,
Cython the language is different from Python the language. Cython the
compiler can compile both.

Stefan
 
P

Paul Rubin

Stefan Krah said:
In the free software world, apparently many people like C. C is also
quite popular in the zero-fault software world: Several verification
tools do exist and Leroy et al. are writing a certified compiler for
C to plug the hole between the verified source code and the generated
assembly.

C is pretty poor as a compiler target: how would you translate Python
generators into C, for example? How would you handle garbage
collection?

C isn't so great for high-assurance stuff either, compared to (say) Ada.
People do use it in critical apps, but that's just because it is (or
anyway used to be) so ubiquitous. I'm wondering what you mean about
verification tools, other than analyzers like Coverity that mainly check
for bugs that in a safer language would be caught by the compiler.
Compcert is not all that C-specific it has been adapted to compile a
Haskell-derived language (Habit).

Haskell doesn't sound all that great as a translation target for Python
either, unfortunately, because its execution semantics are so different.
GHC is a very powerful compiler but it was made to compile Haskell code
that people actually write, and may do less good of a job with compiler
output from an imperative language like Python. Compiling Python to
Scheme and then using a Scheme compiler might be a more natural fit.
But, compiling to Haskell was probably pretty convenient for that
particular project.

Finally, Python itself isn't all that well suited for compilation, given
its high dynamicity. It will be interesting to see if the language
evolves due to PyPy.
 
S

Stefan Behnel

Paul Rubin, 04.08.2012 17:59:
C is pretty poor as a compiler target: how would you translate Python
generators into C, for example?

Depends. If you have CPython available, that'd be a straight forward
extension type. Otherwise, I guess you'd either have a class for them in
C++ or a struct in C. Not exactly complex.

For the yielding, you can use labels and goto. Given that you generate the
code, that's pretty straight forward as well.

How would you handle garbage collection?

CPython does it automatically for us at least. Lacking that, you'd use one
of the available garbage collection implementations, or provide none at all.

Haskell doesn't sound all that great as a translation target for Python
either, unfortunately, because its execution semantics are so different.
GHC is a very powerful compiler but it was made to compile Haskell code
that people actually write, and may do less good of a job with compiler
output from an imperative language like Python. Compiling Python to
Scheme and then using a Scheme compiler might be a more natural fit.
But, compiling to Haskell was probably pretty convenient for that
particular project.

You'd have some kind of emulation layer that does most of the translation
at runtime. That's why I said that you shouldn't expect too much of a
performance gain from what the platform gives you for the underlying
implementation. It can optimise the emulator, but it won't see enough of
the Python code to make anything efficient out of it. Jython is an example
for that.

Finally, Python itself isn't all that well suited for compilation, given
its high dynamicity.

You can get pretty far with static code analysis, optimistic optimisations
and code specialisation. We've decided against whole program analysis in
Cython not only for compiler complexity reasons but also because it would
let the normal compilation time explode for gains that you can much easier
get by manual optimisation. Obviously, optimising JIT compilers can do much
more here (because they actually have to do less), although they won't
always be able to figure out the right thing to do either. That's where
manual optimisation wins again.

Stefan
 
P

Paul Rubin

Stefan Behnel said:
Depends. If you have CPython available, that'd be a straight forward
extension type.

Calling CPython hardly counts as compiling Python into C.
For the yielding, you can use labels and goto. Given that you generate
the code, that's pretty straight forward as well.

You're going to compile the whole Python program into a single C
function so that you can do gotos inside of it? What happens if the
program imports a generator?
CPython does it automatically for us at least.

You mean you're going to have all the same INCREF/DECREF stuff on every
operation in compiled data? Ugh.
Lacking that, you'd use one of the available garbage collection
implementations,

What implementations would those be? There's the Boehm GC which is
useful for some purposes but not really suitable at large scale, from
what I can tell. Is there something else?
or provide none at all.

You're going to let the program just leak memory until it crashes??
you shouldn't expect too much of a performance gain from what the
platform gives you for the underlying implementation. It can optimise
the emulator, but it won't see enough of the Python code to make
anything efficient out of it. Jython is an example for that.

Compare that to the performance gain of LuaJIT and it starts to look
like something is wrong with that approach, or maybe some issue inherent
in Python itself.
You can get pretty far with static code analysis, optimistic
optimisations and code specialisation.

It seems very hard to do reasonable optimizations in the presence of
standard Python techniques like dynamically poking class instance
attributes. I guess some optimizations are still possible, like storing
attributes named as literals in the program in fixed slots, saving some
dictionary lookups even though the slot contents would have to still be
mutable.
 
M

Mark Lawrence

Mark Lawrence, 04.08.2012 12:05:

Hmm, in case you are referring to a recent friendly and diplomatic request
of mine regarding a couple of people who were burdening a public high
volume mailing list with a purely private back-and-forth chat about having
beer and getting drunk - then, no, I don't think the discussion in this
thread qualifies as yet another example for that so far.

Stefan

With arrogance like that German by any chance?
 
T

Temia Eszteri

With arrogance like that German by any chance?

Hey now, cool it with the passive-aggression. We're here to discuss
code, right? If you want to fight it out, you can gladly do it by
e-mail.

~Temia
 
Z

Zero Piraeus

:

With arrogance like that German by any chance?

I didn't give a monkeys about the beer conversation personally, but
can we leave the national stereotypes out of it?

-[]z.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,900
Latest member
Nell636132

Latest Threads

Top