Python compilers?

[Carl Banks]

How could it possibly know? The reassignment of a.bar could happen
anytime, anywhere in the code. Maybe even in an eval.

And if it happens anytime, anywhere in the code, the compiler will see
it and create more general code. Or is that impossible?

As for eval, well it would be silly to even allow eval in a compiled
program; kind of defeats the purpose of compling. You might let it
run in its own namespace so it can only affect certain objects.

 

[asdf as]

Is anyone working on a python-to-native compiler?
I'd be interested in taking a look.

Come to think of it, is anyone working on a sexpr-enabled version of
Python, or anything similar? I really miss my macros whenever I try to
use it...

 

[Fuzzyman]

Various people on various projects... most still at the vapourware
stage.
In essence python is a dynamic interpreted language though - in order
to make it suitable for 'compilation' you have to remove some of the
advantages.

Funnily though, most people call Java a compiled language, but it only
compiles to Java bytecode which runs on the virtual machine. Python
precompiles to python bytecode which runs on the python virtual
machine. So arguably it is *as* compiled as Java.....

I think the main slowdown most people would see is the startup time -
the interpreter takes way too long to start (why the company where I
work won't replace Perl/PHP with Python).

It's not a problem I've been particularly aware of.
Lots of projects to increase execution speed. Psyco is *very* good and
very useable of x86 machines. I use it and see a real speed increase
of approx 10 times on a couple of applications I 've done - anagram
generator and binary interleaving... both of which are processor
intensive. It doesn't address startup speed though.........

Oh that one is funny - do most distro's have PyQT and wxPython (and
their supporting Qt/GKT libs) installed by default, oh and which ones
come with 2.3 by default?

No - and worse PyQT is commercial only on windoze.
You need py2exe (or similar) to make standalone programs that will run
without python. It's not difficult though.

I thought Pyrex was a hybrid of C and Python (like Jython/Java) not
actually a Python-to-C convertor? And Pysco is just a different VM
isn't it?

That's right - Pyrex is a new language that mixes python and C syntax.
You can also use it to easily build python extensions.

I could really push Python where I work if there was a native
compiler, my company uses C/C++/Java/Qt and were looking at QSA as a
way to allow the user to script things, but as all of our products
integrate with our software protection system, we can't be
distributing source or easily decompiled bytecode!

We could replace the whole lot with PyQt and use an embedded Python
interpreter for the scripting! Ah the frustration :-(

if you're lookibng at scripting then you'll be embedding an
interpreter *anyway*.
There *isn't* a native compiler... because python isn't a compiled
language.
*But* bytecode isn't *easily* decompiled (and decompyler is no longer
publicly available) and speed critical stuff/stuff that needs to be
compiled for security reasons can be compiled using Pyrex - which
handles all the interfacing of python to C and you get the best of
both worlds in terms of syntax.

No reason you shouldn't distribute the PyQt libraries *with* your
program - assuming you stick to the relevant licenses.

Regards,


Fuzzy

http://www.voidspace.org.uk/atlantibots/pythonutils.html

 

[Heather Coppersmith]

And if it happens anytime, anywhere in the code, the compiler will see
it and create more general code. Or is that impossible?

The compiler might not see it. Any function/method/module called while
a is in scope might change a.bar. It doesn't even take a perverted
introspection tool:

module foo:

import bar
class C:
pass
a = C( )
a.bar = 'bar'
bar.bar( a )
a.bar = a.bar + 'bar' # which 'add' instruction should this compile to?

module bar:

def bar( x ):
x.bar = 3

As for eval, well it would be silly to even allow eval in a compiled
program; kind of defeats the purpose of compling. You might let it
run in its own namespace so it can only affect certain objects.

I reiterate my comments regarding type declarations, add new comments
regarding backwards compatibility, and note that Lisp allows eval in
compiled code.

Regards,
Heather

 

[Peter Hansen]

Funnily though, most people call Java a compiled language, but it only
compiles to Java bytecode which runs on the virtual machine. Python
precompiles to python bytecode which runs on the python virtual
machine. So arguably it is *as* compiled as Java.....

Actually, there are compilers that produce native machine code from
Java for several CPUs available, and they are used at least in the
embedded world.

-Peter

 

[Svein Ove Aas]

Actually, there are compilers that produce native machine code from
Java for several CPUs available, and they are used at least in the
embedded world.

There is also GCJ as part of the GCC, which can compile both .class
and .java files. Its libraries aren't complete yet, but I'm sure it's
only a matter of time.

 

[Carl Banks]

The compiler might not see it. Any function/method/module called while
a is in scope might change a.bar. It doesn't even take a perverted
introspection tool:

module foo:

import bar
class C:
pass
a = C( )
a.bar = 'bar'
bar.bar( a )
a.bar = a.bar + 'bar' # which 'add' instruction should this compile to?

module bar:

def bar( x ):
x.bar = 3

Compiler builds a big call-tree. Compiler sees that the object "a" is
passed to a function that might rebind bar. Compiler thinks to
itself, "better mark C.bar as a dynamic attribute." Compiler sees
dynamic attirbute and addition. Compiler generates generic add code.

You could have all kinds of setattrs, evals, and unanalysable data
structures (a la pickle), that could turn the code into a nightmare
that a compiler couldn't do anything with. But surely, guarding
against all of that, a good enough static compiler can still reduce a
lot of good, maintainable Python code to it's static case.


And, frankly, I still don't see how this is different from Lisp.

(defun foo ()
(let ((x 1))
(setq x (bar x))
(setq x (+ x 1))))

In another package:

(defun bar (x)
#C(2.0 1.0))

If I recall, + can work on ints, floats, bignums, rationals, and
complex numbers, at least. What one instruction does + compile to
here?

I reiterate my comments regarding type declarations, add new comments
regarding backwards compatibility, and note that Lisp allows eval in
compiled code.

It's highly frowned upon cause it interferes with everything it
touches.

 

[Andrew MacIntyre]

There is also GCJ as part of the GCC, which can compile both .class
and .java files. Its libraries aren't complete yet, but I'm sure it's
only a matter of time.

Hmmm... anyone tried GCJ on Jython?

 

[Paul Rubin]

If I recall, + can work on ints, floats, bignums, rationals, and
complex numbers, at least. What one instruction does + compile to
here?

Lisp supports type declarations which advise the compiler in those
situations. A few such proposals have been made for Python, but none
have taken off so far.

 

[Carl Banks]

Lisp supports type declarations which advise the compiler in those
situations. A few such proposals have been made for Python, but none
have taken off so far.

Yes, that's been established. There's two questions remaining for me:

1. These claims that Lisp code can approach 50 percent the speed of C,
is that with or without the optional type declarations?

2. If you don't use the declarations, does compiling Lisp help? If it
does (and nothing I've read indicated that is doesn't), it
definitely casts some doubt on the claim that compiling Python
wouldn't help. That's kind of been my point all along.

I think (and I'm wrapping this up, cause I think I made my point)
compiling Python could help, even without type declarations, but
probably not as much as in Lisp. It could still make inferences or
educated guesses, like Lisp compilers do; just maybe not as often.

 

[Ville Vainio]

Andrew> Hmmm... anyone tried GCJ on Jython?

Native code will not help much if the created native code is of type:

arg=lookup(object1, "fooarg")
f = lookup(object2,"foomethod")
call(f,arg)

For the performance that is expected of native code we need direct
dispatching with the addresses of the functions known at the compile
time, or via direct indexing of linear virtual tables.

(I'm speaking of static compilation here - what I said may not apply
to psyco)

 

[Heather Coppersmith]

... There's two questions remaining for me:

1. These claims that Lisp code can approach 50 percent the speed
of C, is that with or without the optional type declarations?

That would be *with* type declarations.

2. If you don't use the declarations, does compiling Lisp help?
If it does (and nothing I've read indicated that is doesn't),
it definitely casts some doubt on the claim that compiling
Python wouldn't help. That's kind of been my point all
along.

In _Common Lisp The Language, Second Edition_, by Guy Steele
(CLTL2), on page 686, in section 25.1.3 ("Compilation
Environment") there is a lengthy list of assumptions the compiler
makes, including:

o ... within a named function, a recursive call to a function
of the same name refers to the same function [barring
notinline declarations] ...

o ... a call within the file being compiled to a named
function that is defined in that file refers to that
function [barring notinline declarations] ...

o ... the signature (or "interface contract") of all built-in
Common Lisp functions will not change ...

o ... the signature (or "interface contract") of functions
with ftype information will not change.

o ... any type definition made with defstruct or deftype in
the compile-type environment will retain the same definition
in the run-time environment. It may also assume that a
class defined in the compile-time environment will be
defined in the same run-time environment in such a way as to
have the same superclasses and metaclass ...

o ... if type declarations are present in the compile-time
environment, the corresponding variables and functions
present in the run-time environment will actually be of
those types ...

So compiled Lisp is a less dynamic than Python, and the built-in
functionality is large and guaranteed to match its textbook
definitions, which gives Lisp Compilers more than a fighting
chance. I'm pretty sure I've seen that summary on c.l.p before.

I think (and I'm wrapping this up, cause I think I made my
point) compiling Python could help, even without type
declarations, but probably not as much as in Lisp. It could
still make inferences or educated guesses, like Lisp compilers
do; just maybe not as often.

I agreem, with emphasis on "could" and some hesitation on how much
guessing I want my compiler to do.

Regards,
Heather

 

[Terry Reedy]

In _Common Lisp The Language, Second Edition_, by Guy Steele
(CLTL2), on page 686, in section 25.1.3 ("Compilation
Environment") there is a lengthy list of assumptions the compiler
makes, including:

o ... within a named function, a recursive call to a function
of the same name refers to the same function [barring
notinline declarations] ...

o ... a call within the file being compiled to a named
function that is defined in that file refers to that
function [barring notinline declarations] ...

o ... the signature (or "interface contract") of all built-in
Common Lisp functions will not change ...

Interesting. PyCode can be sped up by making the same assumptions.
Richard Hettinger's recent recipe implements these assumptions. (It was
rejected as part of the standard lib for being too implementation specific,
but remains available in the archives and Python Cookbook site.)

[snip less applicable to Python today stuff]

So compiled Lisp is a less dynamic than Python,

The balance between flexibity and speed continues to be debated by the
developers.

Terry J. Reedy

 

[Günter Jantzen]

The balance between flexibity and speed continues to be debated by the
developers.

And should be debated by the users, too.

Not all of Pythons dynamics is really necessary to write good programms.
Writing good software needs creativity and discipline. We like to work with
Python, because it allows us creativitity. Sometimes to enforce a grain of
discipline in Python would not be bad. There have been many discussions
about programming idioms and design patterns in the last years. They give
not so high level languages the ability to simulate high level features.
They could also help to give a very high level language more solid ground.
And they normalize the way to handle common classes of problems

"""There should be one-- and preferably only one --obvious way to do it.
"""
We could try to search patterns, idioms, antipatterns which could establish
a better way to use the Python language:
-- to avoid unnecessary dynamics
-- to avoid unnecessary features
-- to establish the 'obvious way to do it'
-- without loosing the power and expressiveness of python

"""In the face of ambiguity, refuse the temptation to guess.
"""
Possible sideeffect:.- it could be easier for mechanical tools (Pyrex,
Psycho, PyPy, Starkiller) to make Python faster

Not all people will agree about a useful set of patterns. But this is not
necessary now.
Patterns and idioms do not change a language immediately. They do not fall
from heaven, it is necessary to play around and to gain some experience.

I think patterns define a subset of a language and some constraints on
programms
Maybe this constrained subset (or subsets if there are controversal ideas)
can be formalized and be checked by 'pylintplus' or 'pythoncheckerX' (do not
google).
Or there would be a configurable universal 'Monsterchecker'. In the first
line of our sources we expose a checkpolicy and the checker will follow this
policy.
If patterns can not be formalized so easy, they can be established by
convention.

I do not know the result of this process ...
Now some ideas what could be done in the near future

1) Integration of Pyrex into Python.
[strong variant]
Syntactically this languages are close. Maybe they could be changed so, that
every valid Pyrex source is also a valid Python source with -hopefully- the
same semantics. Special Pyrex tokens would be allowed, but meaningless when
used in the Python context.
Benefit: more homogenous environment. Little changes in the program could be
made without a C-Compilation step.
Maybe the support of interfaces would be necessary in Python, but this is
not a bad idea.
Then rewrite some parts of the standard library in Pyrex

[weak variant]
Rewrite some parts of the standard library in Pyrex

2) Find a set of constraints which make it easier for Psyco (now) and
Starkiller (later) to do their job

3) Implement Monsterchecker modes for 1 and 2

Regards
Guenter

 

[Andrew MacIntyre]

Andrew> Hmmm... anyone tried GCJ on Jython?

Native code will not help much if the created native code is of type:

arg=lookup(object1, "fooarg")
f = lookup(object2,"foomethod")
call(f,arg)

For the performance that is expected of native code we need direct
dispatching with the addresses of the functions known at the compile
time, or via direct indexing of linear virtual tables.

(I'm speaking of static compilation here - what I said may not apply
to psyco)

I was asking more out of interest in the potential for creating
distributable binaries (which is usually one of the desires of people
looking for compilers), than performance specifically. Of course it
would be nice if a GCJ compiled Jython app could at least match the
performance of CPython for the same app.

 

[Konstantin Veretennicov]

Interesting. PyCode can be sped up by making the same assumptions.
Richard Hettinger's recent recipe implements these assumptions. (It was
rejected as part of the standard lib for being too implementation specific,
but remains available in the archives and Python Cookbook site.)

I believe that's /Raymond/ Hettinger.

Anyway, the recipe is a great stuff:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940


- kv

 

[Jacek Generowicz]

Am Dienstag, 18. Mai 2004 13:41 schrieb Jacek Generowicz:

Native compilers for other languages just as dynamic as Python
exist. These compilers manage to achieve very significant speed
increases[*].

In Python this isn't true. Python, instead of LISP, is "completely" dynamic,

In what way is Lisp less dynamic then Python? If what follows is the
basis of your argument, then think again?

meaning that it's pretty impossible to do type-inference for each function
that is called (even checking types isn't possible). E.g. how do you expect
type-inference to work with the pickle module? string -> something/Error
would be the best description what pickle does. For the function which calls
pickle, do you want to create versions for each possible output of Pickle?
Which outputs of Pickle are possible?

Which outputs of the standard Common Lisp function 'read'[*] are possible?

[*] http://www.lisp.org/HyperSpec/Body/chap-23.html

 

[Jacek Generowicz]

... def bar(self, x):
... return x*x
...

I'm afraid you'll have to try harder.


[1]> (defclass foo () ())
#<STANDARD-CLASS FOO>
[2]> (defmethod bar ((foo foo) x)
(* x x))
#<STANDARD-METHOD (#<STANDARD-CLASS FOO> #<BUILT-IN-CLASS T>)>
[3]> (defparameter a (make-instance 'foo))
A
[4]> (bar a 3)
9
[5]> (defmethod bar ((self (eql a)) x)
(* x x x))
#<STANDARD-METHOD ((EQL #<FOO #x203185F9>) #<BUILT-IN-CLASS T>)>
[6]> (bar a 3)
27

 

[Jacek Generowicz]

How could it possibly know? The reassignment of a.bar could happen
anytime, anywhere in the code. Maybe even in an eval.


Lisp compilers might have to do that sometimes, but Python compilers
would have to do it ALL the time. Psyco took one way out, basically
generating code at runtime and caching it for specific operand types,
but the result is considerable code expansion compared to precompilation.

I see how seasoned Lispers can get pretty tired of these sorts of
arguments. Please bear in mind that Lisp has been the playground for a
lot of very clever people whose aim was to solve difficult problems
efficiently ... and they've been at it for about 3 decades before
Python was even thought of. Instead of claiming that Python is
something revolutionarily new in the area of dynamicity (it isn't) and
that compiling it is impossible or futile (it isn't) have a look at
what the Lispers learned in their study of the subject over the four
decades that they have been at it. (You should only do this, of
course, if you are interested in how Python might benefit from being
compiled; if you don't care, then at least don't hurl unfounded
"opinions" about how it is impossible because Python is so amazingly
dynamic.)

Of course certain things are easier to compile to efficient machine
code than others. How does the existence of the problem spots take
away from the usefulness of compiling the easier parts? It's possible
to have significant gains. Bigger gains come with harder work. Bigger
gains can be had in some areas than in others. Certain areas are more
likely to be speed critical than others to a larger proportion of
users (number crunching, for instance). Different compiler
implementors make different choices regarding which areas of the
language they target with hard compiler optimizations.

Imagine that I would like to be able to write effecient numeric code
in pure Python, for example ... I really don't care that any instance
methods that I add will not be dispatched within a single cycle, do I?