Does the world need another v0.1 python compiler?

[Grant Olson]

I'm feeling a little guilty here. I spent a lot of my free time last year
working on an x86 compiler for python. I made a reasonable amount of
progress, but my interests wandered off into other areas. I basically just
got bored and stopped working on the thing maybe 6 months ago. So today, I
went to the directory that contains all of my source checkouts, and saw this
v 0.1 compiler just sitting out there AGAIN.

It's not that I don't want to release the code, but it's just not in quite
good enough shape to just throw out there and hope that other people pick up
and get things working. I've seen more than a few half-assed compiler
implementations out there and part of me thinks I'd just be wasting my time
by just zipping up what I have and throwing it out there. Also, I don't
really expect any significant speed increases, which seems to be what people
want out of a compiler, but was primarily interested in creating standalone
..exes without the need for a 'runtime environment' (whatever that means).
There might be a little speed increase from unrolling the eval loop, but
nothing too serious.

The basic approach I took was compiling to bytecode, and then
transliterating python bytecode to x86 asm. And it is working a little bit.
There is a slightly modified version of the CPython interpreter that
introduces xfunction and xcode objects, that basically act the same as
function and code objects, but run native code. I've successfully generated
some very simple .pyds that can be imported into this interpreter. I also
have some tests that test each bytecode on a small scale and they all seem
to be working. I can just barely run pystones. The biggest problem right
now seems to be that reference counting isn't adding up, resulting in some
serious performance degradation.

On the downside, right now it's Windows only. There's nothing inherently
windows specific, but I don't have anything to build on other platforms. It
also assumes you've got the visual studio toolchain.

So like I said, I feel a little guilty about keeping this thing hostage on
my systems at home. On the other hand, if anyone else is interested,
they're going to need to be a VERY early adopter.

Does anyone have any serious interest in taking a look?

-Grant

 

[John Nagle]

The basic approach I took was compiling to bytecode, and then
transliterating python bytecode to x86 asm. And it is working a little bit.

An interesting option might be to generate the byte code used by
the SpiderMonkey engine in Mozilla. That's used to handle both
Javascript in Firefox and ActionScript in Flash. It has a just-in-time
compiler, so you get x86 machine code when you need it.
And the run time engine is tiny; there's a copy inside Flash, which is
only 2MB.

That could be a way to get a faster Python without bringing excess
baggage.

John Nagle

 

[Kay Schluehr]

An interesting option might be to generate the byte code used by
the SpiderMonkey engine in Mozilla. That's used to handle both
Javascript in Firefox and ActionScript in Flash. It has a just-in-time
compiler, so you get x86 machine code when you need it.
And the run time engine is tiny; there's a copy inside Flash, which is
only 2MB.

That could be a way to get a faster Python without bringing excess
baggage.

John Nagle

This code generation for an arbitrary backend sounds more like an
appropriate task for PyPy. I think Grant's or anyone elses compiler
could be a viable tool for augmenting the CPython interpreter in
particular in the presence of optional type annotations in Py3K.

 

[sturlamolden]

This code generation for an arbitrary backend sounds more like an
appropriate task for PyPy. I think Grant's or anyone elses compiler
could be a viable tool for augmenting the CPython interpreter in
particular in the presence of optional type annotations in Py3K.

IMHO, with the presence of static types in Py3K, we should have a
static compiler that can be invoked dynamically, just like Common
Lisp.

Something like

def foo(...):
bar = static_compile(foo, optimize=2)
bar(...)

JIT compilers are hyped, static compilers perform much better. This
way the programmer can decide what needs to be compiled. This is the
reason why CMUCL can compete with most C compilers.

 

[bearophileHUGS]

sturlamolden:

IMHO, with the presence of static types in Py3K, we should have a
static compiler that can be invoked dynamically, just like Common
Lisp.
Something like

def foo(...):
bar = static_compile(foo, optimize=2)
bar(...)

JIT compilers are hyped, static compilers perform much better. This
way the programmer can decide what needs to be compiled. This is the
reason why CMUCL can compete with most C compilers.

Lot of Python code uses Psyco, so maybe it may be better to extend
Psyco to that 'static compilation' functionality too:

def foo(...):
psyco.static_bind(foo)

At the moment I think this approach can't improve much the speed of
Python programs compared to what Psyco is already able to do.
PyPy's RPython and ShedSkin are also to be considered, recently
ShedSkin is going to support some of the usual Python forms of lazy
processing too.

Bye,
bearophile

 

[sturlamolden]

At the moment I think this approach can't improve much the speed of
Python programs compared to what Psyco is already able to do.

Pyrex generates code that competes with hand-written C. It is as close
to statically typed Python as it gets.

http://www.scipy.org/PerformancePython