Issues in compiling Python to machine code

[Johnathan Doe]

I've been thinking about what the issues would be in compiling
Python into native machine code, and since type information is
important in Python, it seems possible that Python code can
be compiled into native machine code (albeit with a lot of
extra effort).

For instance, type information is discovered when something is
assigned to a variable or an anonymous piece of data is used
in a program. Compiling Python bytecode into native machine
code could involve a pass over the bytecode to look at all the
uses of a variable or anonymous variable, call some Python
function to discover its variable, then record it in the symbol
table.

Another issue is what to do with objects. Finding the offset
to a pointer which is the object's constructor, for instance,
if that's the way it's done internally in Python.

And then freeing the memory used by an object when it's no
longer needed. Perhaps freeing memory for an object can be
done by inserting machine code to return memory back to the
heap after the first pass over the bytecode and recording the
place a variable is last used (hopefully then it won't be
used in some other way later of course... perhaps it can
record all cases where that variable is referred to by another
name and track them, too.)

Seems like compiled Python would need a lot of support code,
in any case, that gets run at certain times: before the program
starts, whenever a variable is accessed, before an object is
used (to allocate memory), whenever something happens to data
within an object (allocate/reallocate/destroy), and so on...

Are there any reasons why Python would be *impossible* to compile?

Thanks

 

[Harald Massa]

Johnathan

Are there any reasons why Python would be *impossible* to compile?

there is quite a FAQ to this question ))

Why is there no Python compiler ?
- because when it was time to do so, it was not possible to get a
diploma for creating a compiler, so sufficent motivation was not
available.


Actually there are some projects towards a "python compiler":

look at:

http://codespeak.net/pypy/

-> Python in Python

http://psyco.sourceforge.net/

-> Python special compiler. Creates x86 machine code out of Python "on
the fly"

http://web.mit.edu/msalib/www/urop/

Starkiller is some project to deal with types in Python

http://ironpython.com/

Jim Hugunin is doing a projekt for "Python 'compilation' to the .NET /
MONO platform"


Maybe it could be valuable to you to visit EuroPython
(http://www.europython.org/, the biggest Python Conference without US
travelling restrictions. Especially

http://www.europython.org/conferences/epc2004/info/talks/refereed/msalib0
1

http://www.europython.org/conferences/epc2004/info/talks/python_language/
arigo02

http://www.europython.org/conferences/epc2004/info/talks/python_language/
arigo01

may be of great interest for you. Registration is still open, you can
attend for 220€ normal fee.


Harald

 

[Ryan Paul]

I've been thinking about what the issues would be in compiling
Python into native machine code, and since type information is
important in Python, it seems possible that Python code can
be compiled into native machine code (albeit with a lot of
extra effort).

For instance, type information is discovered when something is
assigned to a variable or an anonymous piece of data is used
in a program. Compiling Python bytecode into native machine
code could involve a pass over the bytecode to look at all the
uses of a variable or anonymous variable, call some Python
function to discover its variable, then record it in the symbol
table.

Another issue is what to do with objects. Finding the offset
to a pointer which is the object's constructor, for instance,
if that's the way it's done internally in Python.

And then freeing the memory used by an object when it's no
longer needed. Perhaps freeing memory for an object can be
done by inserting machine code to return memory back to the
heap after the first pass over the bytecode and recording the
place a variable is last used (hopefully then it won't be
used in some other way later of course... perhaps it can
record all cases where that variable is referred to by another
name and track them, too.)

Seems like compiled Python would need a lot of support code,
in any case, that gets run at certain times: before the program
starts, whenever a variable is accessed, before an object is
used (to allocate memory), whenever something happens to data
within an object (allocate/reallocate/destroy), and so on...

Are there any reasons why Python would be *impossible* to compile?

Thanks

we have talked about this here before, I think. Native compilation wont
improve runtime speed of a dynamically typed language. Despite this, there
are actually a few projects out there endevouring to do it anyway. If you
want better speed from a python application, try a customized python
runtime engine like Psyco.

--SegPhault

 

[Johnathan Doe]

Thanks for all the links!

Obviously I've just shown myself to think out loud, and not to think
very much at all before doing the "out loud" part...

I'll take a look through all these.

Incidentally, this doesn't have anything to do with speeding up Python.
It's already fast enough for what I want. My main issue is writing
low level code in Python. It's possible to do great things like
autodetect hardware in Python for an OS, e.g. Anaconda/Kudzu, but why
not be able to write an OS completely in Python? I am sure it's
possible somehow. Just not sure yet

Possibly, GRUB can load the startup code, which contains an embedded
Python interpeter. Then the interpreter defines the kernel_main
function and takes over. I'd need some way of gluing up all the low
level stuff to the interpreted Python "platform" : interrupts and page
faults and whatnot.

Anyway, I should talk about this on alt.os.development

Johnathan

 

[Johnathan Doe]

we have talked about this here before, I think. Native compilation wont

improve runtime speed of a dynamically typed language. Despite this, there
are actually a few projects out there endevouring to do it anyway. If you
want better speed from a python application, try a customized python
runtime engine like Psyco.

That might be a good idea actually, to investigate Psyco and others.
I'm not interested in speeding up Python as such, I'm just interested in
being able to write system-level software in Python. I would like to be
able to write an OS in Python for instance.

Johnathan