Python compilers?

S

Svein Ove Aas

Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...
 
P

Peter Hansen

Svein said:
Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...

Lots of past threads on this, including this one:
http://groups.google.com/groups?&th=8f7b4867334c3d07

(Short answers: no, maybe, look at Psyco, PyPy, and others...)

-Peter
 
S

SeeBelow

Svein said:
Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...

I really wish there was a python-to-native compiler, a good one that
would produce fairly fast execution.

Mitchell Timin

--
"Many are stubborn in pursuit of the path they have chosen, few in
pursuit of the goal." - Friedrich Nietzsche

http://annevolve.sourceforge.net is what I'm into nowadays.
Humans may write to me at this address: zenguy at shaw dot ca
 
L

Leif K-Brooks

Svein said:
Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

What are you trying to achieve?

If it's faster code execution, the primary slowdown with a very
high-level language like Python is caused by high-level data structures
(introspection, everything being an object, etc.), not the code itself.
A native compiler would still have to use high-level data structures to
work with all Python code, so the speed increase wouldn't be very much.

If it's ease of distribution you're looking for, I think distutils can
make standalone programs on Windows, and most Linux distros have Python
installed by default.

If you just think a compiler would be cool (or would like to see how it
would be done), check out Psyco, Pyrex, and probably some other
projects. Pyrex even combats the speed issue by allowing native C types
to be used in addition to Python high-level types.
 
Y

Yermat

Svein said:
Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...

look at starkiller http://www.python.org/pycon/dc2004/papers/1/

The first step is to create a Type Inference system so there is star killer.
Michael Salib said he is working on a C++ translation for python...
 
S

Svein Ove Aas

Leif said:
What are you trying to achieve?

If it's faster code execution, the primary slowdown with a very
high-level language like Python is caused by high-level data structures
(introspection, everything being an object, etc.), not the code itself.
A native compiler would still have to use high-level data structures to
work with all Python code, so the speed increase wouldn't be very much.

I'd like to point out the usual suspects: Lisp compilers.
*They* somehow manage to get within 2x of C++ speed, so why can't Python?
If it's ease of distribution you're looking for, I think distutils can
make standalone programs on Windows, and most Linux distros have Python
installed by default.

Nope.
If you just think a compiler would be cool (or would like to see how it
would be done), check out Psyco, Pyrex, and probably some other
projects. Pyrex even combats the speed issue by allowing native C types
to be used in addition to Python high-level types.

Erk.
Seems to me that you want 'smarter', not 'worse'. I can't take a language
seriously if it says that 1/3 is 0.33333... .
 
B

Brian Quinlan

Svein said:
Erk.
Seems to me that you want 'smarter', not 'worse'. I can't take a language
seriously if it says that 1/3 is 0.33333... .

What is that? Are you arguing for an integer result, a fixed result or
a rational result?

Cheers,
Brian
 
M

Michael Hudson

Svein Ove Aas said:
I'd like to point out the usual suspects: Lisp compilers.
*They* somehow manage to get within 2x of C++ speed, so why can't Python?

Type declarations. Lots of effort by seriously smart cookies.

We're working on the latter :)

Cheers,
mwh

--
> say-hi-to-the-flying-pink-elephants-for-me-ly y'rs,
No way, the flying pink elephants are carrying MACHINE GUNS!
Aiiee!! Time for a kinder, gentler hallucinogen...
-- Barry Warsaw & Greg Ward, python-dev
 
J

Jacek Generowicz

Leif K-Brooks said:
What are you trying to achieve?

If it's faster code execution, the primary slowdown with a very
high-level language like Python is caused by high-level data
structures (introspection, everything being an object, etc.), not the
code itself. A native compiler would still have to use high-level data
structures to work with all Python code, so the speed increase
wouldn't be very much.

Oh please !

Native compilers for other languages just as dynamic as Python
exist. These compilers manage to achieve very significant speed
increases[*].

Psyco is a native compiler, of sorts, for Python, and it manages to
produce dramatic improvements (in the areas where it works).

While it's true that often speed doesn't matter, and many of the
criticisms levelled at Python for being too slow are completely
unfounded in real world situations, this is no reason for Pythonistas
to

a) be happy about the fact that Python is slow,

b) be convinced that Python _must_ be slow.

The sooner "we" stop believing that Python's flexibility comes at the
unavoidable cost of piss-poor runtime performance, and the sooner we
accept that it would be useful to have a Python which maintains its
flexibility but runs like a bat out of hell (ie, the sooner we stop
making excuses for Pythons lack of speed), the sooner we will get one.

Fortunately there is already a bunch of people who understand this,
and is trying to do something about it.


[*] A prime example is the Common Lisp implementation CMUCL. Ironically
enough, CMUCL's compiler is called ... Python.
 
C

Cameron Laird

.
.
.
I really wish there was a python-to-native compiler, a good one that
would produce fairly fast execution.
.
.
.
Maybe there is. There's enough ambiguity in "python-to-native
compiler" that it's possible what you (think you) want already
exists. See Leif K-Brooks's follow-up, especially.
 
H

Heiko Wundram

Am Dienstag, 18. Mai 2004 13:41 schrieb Jacek Generowicz:
Native compilers for other languages just as dynamic as Python
exist. These compilers manage to achieve very significant speed
increases[*].

You are again refering to LISP as an example of a dynamic language which when
compiled gives huge speed increases. This is true in some respect, in others
it isn't. LISP has the advantage that type-inference may be used throughout
the program to create one version of each function, which can then be
compiled. Of course it still has to call into runtime functions to do the
high-level work, but there is actually only one representation of each
finished LISP program, and only one set and one proper order of
runtime-functions to call.

In Python this isn't true. Python, instead of LISP, is "completely" dynamic,
meaning that it's pretty impossible to do type-inference for each function
that is called (even checking types isn't possible). E.g. how do you expect
type-inference to work with the pickle module? string -> something/Error
would be the best description what pickle does. For the function which calls
pickle, do you want to create versions for each possible output of Pickle?
Which outputs of Pickle are possible? (depends on the loaded modules, which
can be loaded at runtime) There is no (sane) way to create machine-code which
calls into the appropriate (low-level) Python-runtime functions (such as
Py_List*, Py_Dict*, etc.) for such a method, at least not at compile-time.

At runtime, this is possible. See what psyco does. There's a nice presentation
on the psyco-website which explains what it does, and I guess you'll
understand when you see that why writing a "compiler" for Python is pretty
impossible.

Heiko.
 
G

Grant Edwards

Seems to me that you want 'smarter', not 'worse'. I can't take a language
seriously if it says that 1/3 is 0.33333... .

Dude, didn't you take high-school math? 1/3 _is_ 0.33333...
 
S

Svein Ove Aas

Brian said:
What is that? Are you arguing for an integer result, a fixed result or
a rational result?

A rational result, of course. If I'm okay with losing precision, I'll
*tell* the language I'm okay with losing precision.
 
T

Tor Iver Wilhelmsen

Grant Edwards said:
Dude, didn't you take high-school math? 1/3 _is_ 0.33333...

No, because at some point you will stop writing 3's, either out of
boredom, exhaustion or because you need to pee. At that instant, you
introduce a rounding error, making 3 * 1/3 = 0.99999999999... instead
of 1.0
 
S

SeeBelow

Leif said:
What are you trying to achieve?

If it's faster code execution, the primary slowdown with a very
high-level language like Python is caused by high-level data structures
(introspection, everything being an object, etc.), not the code itself.
A native compiler would still have to use high-level data structures to
work with all Python code, so the speed increase wouldn't be very much.

Yes, fast execution. I have been using C. In my applications there is
a population of "chromosomes" which are arrays of floats, about 2 to 5 k
in length. Then there are subroutines which operate on a chromosome
using pointers. For example, the "crossover" routine uses two pointers
to swap portions of two chromosomes. My software sometimes runs for
hours, perform many millions of operations like these. Clearly, speed
of execution is of dramatic importance.

A related problem is that it seems to be a big deal to call C routines
from Python. I have not actually tried it, because when I read about
how its done I was not able to understand it.

Mitchell Timin

--
"Many are stubborn in pursuit of the path they have chosen, few in
pursuit of the goal." - Friedrich Nietzsche

http://annevolve.sourceforge.net is what I'm into nowadays.
Humans may write to me at this address: zenguy at shaw dot ca
 
S

Svein Ove Aas

Heiko said:
Am Dienstag, 18. Mai 2004 13:41 schrieb Jacek Generowicz:
Native compilers for other languages just as dynamic as Python
exist. These compilers manage to achieve very significant speed
increases[*].

You are again refering to LISP as an example of a dynamic language which
when compiled gives huge speed increases. This is true in some respect,
in others it isn't. LISP has the advantage that type-inference may be
used throughout the program to create one version of each function,
which can then be compiled. Of course it still has to call into runtime
functions to do the high-level work, but there is actually only one
representation of each finished LISP program, and only one set and one
proper order of runtime-functions to call.

In Python this isn't true. Python, instead of LISP, is "completely"
dynamic, meaning that it's pretty impossible to do type-inference for
each function that is called (even checking types isn't possible). E.g.
how do you expect type-inference to work with the pickle module? string
-> something/Error would be the best description what pickle does. For
the function which calls pickle, do you want to create versions for each
possible output of Pickle? Which outputs of Pickle are possible?
(depends on the loaded modules, which can be loaded at runtime) There is
no (sane) way to create machine-code which calls into the appropriate
(low-level) Python-runtime functions (such as Py_List*, Py_Dict*, etc.)
for such a method, at least not at compile-time.

I suppose that's true; pickle is an exception, and the compiler would pick
that up. (The Lisp (print) function is approximately equivalent, at least
when *print-readably* is true.)

What you're claiming, though, is that it's possible to write Python code
that can't easily be translated to equivalent Lisp code. Can you give an
example?
 
Y

Yermat

Heiko said:
[...]

In Python this isn't true. Python, instead of LISP, is "completely" dynamic,
meaning that it's pretty impossible to do type-inference for each function
that is called (even checking types isn't possible). E.g. how do you expect
type-inference to work with the pickle module? string -> something/Error
would be the best description what pickle does. For the function which calls
pickle, do you want to create versions for each possible output of Pickle?
Which outputs of Pickle are possible? (depends on the loaded modules, which
can be loaded at runtime) There is no (sane) way to create machine-code which
calls into the appropriate (low-level) Python-runtime functions (such as
Py_List*, Py_Dict*, etc.) for such a method, at least not at compile-time.

At runtime, this is possible. See what psyco does. There's a nice presentation
on the psyco-website which explains what it does, and I guess you'll
understand when you see that why writing a "compiler" for Python is pretty
impossible.

That is here where you are wrong !
It can be known at compile time if you know every modules that will be
imported. That is called "closed environment".

If it is the case, you would be able to compile a program even if you
would not be allowed to do incremental compilation... That just mean
that you will need to recompile everything each time you modified something.

The demonstration is quite easy:
In a "closed environment" there is a finite number of classes. So you
just have to create as much specialized functions as classes. Then where
you can infere type, call directly the good function. Elsewhere, just
call a runtime dispatcher. In fact, it is already used in some languages
like Eiffel with certain optimizations.

The problem is that many python programs are not "closed" at
compile-time, ie they import or eval stuff only known at run-time.

impossible n'est pas français ;-)
 
R

Roger Binns

A related problem is that it seems to be a big deal to call C routines
from Python. I have not actually tried it, because when I read about
how its done I was not able to understand it.

Use SWIG. If your C header file is well written, you may not need to
give any additional information to SWIG.

Roger
 
S

simo

If it's faster code execution, the primary slowdown with a very
high-level language like Python is caused by high-level data structures
(introspection, everything being an object, etc.), not the code itself.
A native compiler would still have to use high-level data structures to
work with all Python code, so the speed increase wouldn't be very much.

I think the main slowdown most people would see is the startup time -
the interpreter takes way too long to start (why the company where I
work won't replace Perl/PHP with Python).
If it's ease of distribution you're looking for, I think distutils can
make standalone programs on Windows, and most Linux distros have Python
installed by default.

Oh that one is funny - do most distro's have PyQT and wxPython (and
their supporting Qt/GKT libs) installed by default, oh and which ones
come with 2.3 by default?
If you just think a compiler would be cool (or would like to see how it
would be done), check out Psyco, Pyrex, and probably some other
projects. Pyrex even combats the speed issue by allowing native C types
to be used in addition to Python high-level types.

I thought Pyrex was a hybrid of C and Python (like Jython/Java) not
actually a Python-to-C convertor? And Pysco is just a different VM
isn't it?

I could really push Python where I work if there was a native
compiler, my company uses C/C++/Java/Qt and were looking at QSA as a
way to allow the user to script things, but as all of our products
integrate with our software protection system, we can't be
distributing source or easily decompiled bytecode!

We could replace the whole lot with PyQt and use an embedded Python
interpreter for the scripting! Ah the frustration :-(
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,756
Messages
2,569,535
Members
45,008
Latest member
obedient dusk

Latest Threads

Top