[ann] first release of PyPy

[holger krekel]

Welcome to PyPy 0.6
--------------------

*The PyPy Development Team is happy to announce the first
public release of PyPy after two years of spare-time and
half a year of EU funded development. The 0.6 release
is eminently a preview release.*

What it is and where to start
-----------------------------

Getting started: http://codespeak.net/pypy/index.cgi?doc/getting_started.html

PyPy Documentation: http://codespeak.net/pypy/index.cgi?doc

PyPy Homepage: http://codespeak.net/pypy/

PyPy is a MIT-licensed reimplementation of Python written in
Python itself. The long term goals are an implementation that
is flexible and easy to experiment with and retarget to
different platforms (also non-C ones) and such that high
performance can be achieved through high-level implementations
of dynamic optimisation techniques.

The interpreter and object model implementations shipped with 0.6 can
be run on top of CPython and implement the core language features of
Python as of CPython 2.3. PyPy passes around 90% of the Python language
regression tests that do not depend deeply on C-extensions. Some of
that functionality is still made available by PyPy piggy-backing on
the host CPython interpreter. Double interpretation and abstractions
in the code-base make it so that PyPy running on CPython is quite slow
(around 2000x slower than CPython ), this is expected.

This release is intended for people that want to look and get a feel
into what we are doing, playing with interpreter and perusing the
codebase. Possibly to join in the fun and efforts.

Interesting bits and highlights
---------------------------------

The release is also a snap-shot of our ongoing efforts towards
low-level translation and experimenting with unique features.

* By default, PyPy is a Python version that works completely with
new-style-classes semantics. However, support for old-style classes
is still available. Implementations, mostly as user-level code, of
their metaclass and instance object are included and can be re-made
the default with the ``--oldstyle`` option.

* In PyPy, bytecode interpretation and object manipulations
are well separated between a bytecode interpreter and an
*object space* which implements operations on objects.
PyPy comes with experimental object spaces augmenting the
standard one through delegation:

* an experimental object space that does extensive tracing of
bytecode and object operations;

* the 'thunk' object space that implements lazy values and a 'become'
operation that can exchange object identities.

These spaces already give a glimpse in the flexibility potential of
PyPy. See demo/fibonacci.py and demo/sharedref.py for examples
about the 'thunk' object space.

* The 0.6 release also contains a snapshot of our translation-efforts
to lower level languages. For that we have developed an
annotator which is capable of infering type information
across our code base. The annotator right now is already
capable of successfully type annotating basically *all* of
PyPy code-base, and is included with 0.6.

* From type annotated code, low-level code needs to be generated.
Backends for various targets (C, LLVM,...) are included; they are
all somehow incomplete and have been and are quite in flux. What is
shipped with 0.6 is able to deal with more or less small/medium examples.


Ongoing work and near term goals
---------------------------------

Generating low-level code is the main area we are hammering on in the
next months; our plan is to produce a PyPy version in August/September
that does not need to be interpreted by CPython anymore and will
thus run considerably faster than the 0.6 preview release.

PyPy has been a community effort from the start and it would
not have got that far without the coding and feedback support
from numerous people. Please feel free to give feedback and
raise questions.

contact points: http://codespeak.net/pypy/index.cgi?contact

contributor list: http://codespeak.net/pypy/index.cgi?doc/contributor.html

have fun,

Armin Rigo, Samuele Pedroni,

Holger Krekel, Christian Tismer,

Carl Friedrich Bolz


PyPy development and activities happen as an open source project
and with the support of a consortium funded by a two year EU IST
research grant. Here is a list of partners of the EU project:

Heinrich-Heine University (Germany), AB Strakt (Sweden)

merlinux GmbH (Germany), tismerysoft GmbH(Germany)

Logilab Paris (France), DFKI GmbH (Germany)

ChangeMaker (Sweden)

 

[Kay Schluehr]

Welcome to PyPy 0.6
--------------------

*The PyPy Development Team is happy to announce the first
public release of PyPy after two years of spare-time and
half a year of EU funded development. The 0.6 release
is eminently a preview release.*

Congratulation to You and Your team!

PyPy is really awesome and if it succeeds in speed demands after the
translation phase I believe that the project will shift the power
within the Python community on the long run. There are moments I'm
almost shocked about it and think about the fate of other programming
programming languages like LISP. PyPy can be resolved to "Python in
Python" but also "Python multiplied/powered by itself" which is much
more triumphant. A short review of the 'thunks' objspace example gives
me the impression that the language development process as we know it
comes to an end and makes a kind of transition. This is both very
exciting and dangerous, like every philosophical event.

Regards,
Kay

 

[beliavsky]

Congratulation to You and Your team!

PyPy is really awesome and if it succeeds in speed demands after the
translation phase I believe that the project will shift the power
within the Python community on the long run.

Could you please explain this statement? Who will gain power, and who
will lose it? Are you suggesting that CPython and PyPy developers are
competing?

 

[Paul Rubin]

Could you please explain this statement? Who will gain power, and who
will lose it? Are you suggesting that CPython and PyPy developers are
competing?

I interpreted it to mean that it would bring power into the Python
community from outside, e.g., Python would become more competitive with
other languages that are currently beating it on performance.

 

[Christian Tismer]

Could you please explain this statement? Who will gain power, and who
will lose it?

The Python community will gain power, and nobody will loose some.
The big win is that we gain a new flexibility that did not
exist before, even if PyPy should completely miss its speed
promises. Having an extremely flexible implementation in a
very high-level language (which happens to be Python) enables
possibilities which have not been seen, before.

There is of course a chance for some community of C programmers
to loose interest, if PyPy really gets as efficient as we hope
for. But this is a) still a long, uncertain path and b) not
a real danger, but more likely an advantage for the involved people.

Are you suggesting that CPython and PyPy developers are
competing?

No idea how we could get onto this track. If there is a competition,
then only if PyPy gets into a position where it is comparable
with CPython. This is not the case, at least not in a well-ordered manner.
It is not really faster, but it is definately much more flexible.
Comparisons are not suitable at all, bcause there are too many
qualities to compare about.

And I see no point for any competition in any future. We all love
Python. It is a language, and languages are communities.
If a particular implementation gets more interest for some reasons,
then because it ibetter s more efficient or more interesting, whatever
reasoning gives it popularity. But we are all with Python!

Surely we are comparing our performance with CPython's. This is
not the real point. Note also, that many of the PyPy team members belong
to CPython core developers, as well. This is not a competition, but
a huge new branch, exploring what is doable and what not.

You might also give our website a try which is quite informative
and gives you an insight into what we are aiming for.

http://codespeak.net/pypy

PyPy is just a completely new approach to interpreted languages,
almost based upon known compiler technology, but applying this in a
consequent manner, that has no comparable prior example.

I wish to repeat the congratulations to the team for the first release!
-----------------------------------------------------------------------

ciao -- chris

--
Christian Tismer :^) <mailto:[email protected]>
tismerysoft GmbH : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

 

[Paul Rubin]

PyPy is just a completely new approach to interpreted languages,
almost based upon known compiler technology, but applying this in a
consequent manner, that has no comparable prior example.

Is there a web page describing what's new? Compile-and-go interactive
languages have been around for decades.

 

[Kay Schluehr]

The Python community will gain power, and nobody will loose some.
The big win is that we gain a new flexibility that did not
exist before, even if PyPy should completely miss its speed
promises. Having an extremely flexible implementation in a
very high-level language (which happens to be Python) enables
possibilities which have not been seen, before.

But not only flexibility IN the current language but also beyond it.
It's not anymore clear what the language as a set of well-defined
syntactical and semantical rules really is if You can change the
semantics in an arbitrary module representing an object-space. I
currently don't know how modular the parser is but adding syntax-rules
should not be that hard either. Once You get enough speed out of the
PyPy-runtime and the community shifts to it the PEP-process degenerates
in the view of a PyPythonista to discussions about aspects of the
std-objectspace and language design patterns. There will be some
CPython compliance - that's all.

Empowering the community means beheading the BDFL and that's currently
not only a person but a principle. Well, maybe that's o.k. but at least
inevitable and Guido finally finds the time to clip roses, writes his
memoirs, educates children and polishs his medals of honour.

Regards,
Kay

 

[Torsten Bronger]

Hallöchen!

[...]

[...] Once You get enough speed out of the PyPy-runtime and the
community shifts to it the PEP-process degenerates in the view of
a PyPythonista to discussions about aspects of the std-objectspace
and language design patterns. There will be some CPython
compliance - that's all.

Please could somebody explain to us non-CS people why PyPy could
have speed features CPython can't have?

Tschö,
Torsten.

 

[Paul Rubin]

Please could somebody explain to us non-CS people why PyPy could
have speed features CPython can't have?

Does the one-word answer "compiler" explain enough?

 

[Torsten Bronger]

Hallöchen!

Does the one-word answer "compiler" explain enough?

No, just more questions. ;-)

What's supposed to be compiled? Only PyPy itself or also the
programs it's "interpreting"?

<http://www.python.org/pycon/dc2004/papers/27/>: "In the next step
of the project, we will generate C code or machine code from the
source of Pypy, thereby reducing the speed penalty."

I've been told by so many books and on-line material that Python
cannot be compiled (unless you cheat). So how is this possible?

Tschö,
Torsten.

 

[Ville Vainio]

Torsten> What's supposed to be compiled? Only PyPy itself or also
Torsten> the programs it's "interpreting"?

PyPy is written in python, if it can be compiled then the programs can
be as well.

Torsten> I've been told by so many books and on-line material that
Torsten> Python cannot be compiled (unless you cheat). So how is
Torsten> this possible?

These guys are exploring a new territory. OTOH, Lisp is a dynamic
language like python and it can be compiled to native code. Pyrex
demonstrates the "trivial" way to compile python to native code, the
real problem is making the resulting code fast. Typically this
requires type inference (i.e. figuring out the type of an object from
the context because there are no type declarations) to avoid dict
lookups in method dispatch.

This is not about PyPy but it might help:

http://www.python.org/pycon/dc2004/papers/1/paper.pdf

 

[Ville Vainio]

Ville> This is not about PyPy but it might help:

Ville> http://www.python.org/pycon/dc2004/papers/1/paper.pdf

(It's about starkiller, sorry about the opaque url)

 

[Shane Hathaway]

Hallöchen!

[...]

[...] Once You get enough speed out of the PyPy-runtime and the
community shifts to it the PEP-process degenerates in the view of
a PyPythonista to discussions about aspects of the std-objectspace
and language design patterns. There will be some CPython
compliance - that's all.

Please could somebody explain to us non-CS people why PyPy could
have speed features CPython can't have?

The idea is to shift more of the responsibility to optimize code from
the human to the computer. Since C code is at a low level, the computer
can only infer low level intent and thus perform low level
optimizations. Humans optimize C by making thousands of speed-oriented
decisions directly in the code. Python code is at a much higher level,
which should enable the computer to discover more of the programmer's
intent and perform deep optimizations.

In the end, the computer's automated optimization could turn out better
than a human's manual optimization. Thus, by expressing the Python
interpreter in a high level manner, PyPy is a first step toward deep
optimizations that aren't possible to automate in C.

Even if things don't turn out that way, note that each generation of
programming languages builds on its predecessors, and PyPy could help
bootstrap the next generation. Assemblers first had to be written in
machine code; when it was possible to write assemblers in assembly,
people started writing complex grammars and came up with C. C compilers
first had to be written in assembly; when it was possible to write C
compilers in C, people started inventing high level languages. Now
people are experimenting with high level compilers written in high level
languages. Where will this pattern lead? Who knows.

Shane

 

[Kay Schluehr]

Hallöchen!



No, just more questions. ;-)

What's supposed to be compiled? Only PyPy itself or also the
programs it's "interpreting"?

There is no "PyPy itself". The distinction bewteen interpreter-level
code and application-level code is nothing but a set of coding
conventions which are usefull but not necessary to let the type
inferencer ( "annotator" in PyPy slang ) terminate definitely on a set
of machine-translateable types in case of interpreter-level code. At
least the interpreter should not interpret it's own code after an
inititalisation phase. Currently type annotated application level code
will still be compiled into bytecodes but it is not only possible to
JIT and to specialize it by means of Psyco, but it should be possible
to compile parts of it into native code like we do today with
C-extensions.

It's hard for me to recognize a fixedpoint in this process or a clear
boundary between interpreter and application level code. This is IMO a
pure heuristic not a categorial distinction but is also clear there
will ever be a remaining gap due to the dynanism of the language. I
think a lot of research understanding this distinction will follow in
the next years.

As a conclusion: with PyPy Python will still be interpreted, but a
large corpus of Python code may be compiled into native code of the
underlying machine.

Regards,
Kay

 

[Kay Schluehr]

Now people are experimenting with high level compilers written in high level
languages. Where will this pattern lead? Who knows.

Drift from old Europe ( greek Pythons ) to old India to "Nagas" and
other snake-beings and goddesses

http://www.khandro.net/mysterious_naga.htm#many-headed

Regards,
Kay

 

[Carl Friedrich Bolz]

Hi!

Torsten> What's supposed to be compiled? Only PyPy itself or also
Torsten> the programs it's "interpreting"?

PyPy is written in python, if it can be compiled then the programs can
be as well.

That's correct in the sense that if a program adherses to the same
staticness conditions as the PyPy code, it can be compiled. The core
parts of the PyPy interpreter are written in "Restricted Python"
(RPython), which imposes some limits to the features you are allowed to
use. This is done in such a way that the annotator can perform type
inference, e.g. you are not allowed to assign values with different
types to a variable (plus some more restrictions). See

http://codespeak.net/pypy/index.cgi?doc/coding-guide.html#restricted-python

for more details about RPython.

Torsten> I've been told by so many books and on-line material that
Torsten> Python cannot be compiled (unless you cheat). So how is
Torsten> this possible?

These guys are exploring a new territory. OTOH, Lisp is a dynamic
language like python and it can be compiled to native code. Pyrex
demonstrates the "trivial" way to compile python to native code, the
real problem is making the resulting code fast. Typically this
requires type inference (i.e. figuring out the type of an object from
the context because there are no type declarations) to avoid dict
lookups in method dispatch.

There is some preliminary documentation about the type infering (which
is called annotation here) and the translation process:

http://codespeak.net/pypy/index.cgi?doc/translation.html

Regards,

Carl Friedrich

 

[beliavsky]

Shane Hathaway wrote:

The idea is to shift more of the responsibility to optimize code from
the human to the computer. Since C code is at a low level, the computer
can only infer low level intent and thus perform low level
optimizations. Humans optimize C by making thousands of speed-oriented
decisions directly in the code. Python code is at a much higher level,
which should enable the computer to discover more of the programmer's
intent and perform deep optimizations.

In the end, the computer's automated optimization could turn out better
than a human's manual optimization. Thus, by expressing the Python
interpreter in a high level manner, PyPy is a first step toward deep
optimizations that aren't possible to automate in C.

I am less optimistic but hope I am wrong .

C++ is a higher level language than C, but it's not clear to me that
compilers are able to optimize C++ code using higher-level features
such as the Standard Library so that they run as fast as the equivalent
C code. OTOH, some C++ experts have advocated template metaprogramming
as a way of speeding up programs. I wonder how widely this technique is
used.

Fortran 95 is a considerably higher level language than Fortran 77, but
I get the impression from comp.lang.fortran that it is harder to
optimize. Fortran compilers compete in a performance-driven market, and
AFAIK they are written in C.

 

[John Roth]

Hallöchen!



No, just more questions. ;-)

What's supposed to be compiled? Only PyPy itself or also the
programs it's "interpreting"?

To be more specific, the (possible) speed increase will come
from JIT (Just In Time) compilation technology. JIT technology
is quite capable of handling dynamic languages. That has to
come after they get a compilable interpreter working, but I
believe it was in the original project vision statement.

A JIT compiler within the interpreter will put PyPy pretty
much on a par with Java as far as speed goes.

<http://www.python.org/pycon/dc2004/papers/27/>: "In the next step
of the project, we will generate C code or machine code from the
source of Pypy, thereby reducing the speed penalty."

I've been told by so many books and on-line material that Python
cannot be compiled (unless you cheat). So how is this possible?

JIT compilers cheat. Specifically, they compile for the observed
object environment of a statement, and then insert a test to make
sure that the actual environment matches the expected environment.
If it doesn't, it goes back to interpretation for that code segment.

John Roth

 

[Skip Montanaro]

beliavsky> C++ is a higher level language than C,

From the compiler's viewpoint C++ is not much higher level than C. It has

the same basic types, (structs, unions and C++ classes are really the same
thing data-wise, though C++ classes can be somewhat more complex
layout-wise) and supports pointers to those types as well as void pointers
(pointers to untyped memory). In addition, the operators are essentially
the same.

Python has a somewhat higher-level set of objects, doesn't have pointers,
nor does it allow untyped pointers to random chunks of memory. I would
think that a run-time specializing compiler like Psyco could potentially do
more with that than a C/C++ compiler can do with the data structures and
operations it has to work with.

Skip

 

[Mike Meyer]

Torsten Bronger wrote:
Even if things don't turn out that way, note that each generation of
programming languages builds on its predecessors, and PyPy could help
bootstrap the next generation. Assemblers first had to be written in
machine code; when it was possible to write assemblers in assembly,
people started writing complex grammars and came up with C. C compilers
first had to be written in assembly; when it was possible to write C
compilers in C, people started inventing high level languages. Now
people are experimenting with high level compilers written in high level
languages. Where will this pattern lead? Who knows.

Your history of programming languages skips so many steps that it's
misleading.

For instance, C didn't arrive ab initio. It was preceeded by B, which
was a derivative of BCPL. From the history at <URL:
http://www.cs.bell-labs.com/who/dmr/chist.html >, it seems that B
slowly evolved into C. B started life as an interpreted language, with
a compiler that generated pseudo-code. The first B compiler was
written in TMG, which was a high-level language designed for creating
compilers - well, sorta. Based in that history, it seems likely that
the first program that compiled a language called C was written in B.

I'm used to seeing the term "high level languages" used for languages
a lot like C, to distinguish them from assembler. See <URL:
http://www.computerhope.com/jargon/h/highll.htm > for one
definition. "Very high level languages" used to be popular, but I
haven't seen it used much. At least one person classifies Python as
such <URL: http://www.everything2.com/index.pl?node_id=735359 >. In
any case, powerful dynamic languages - of which python is an example -
date back to LISP. The first LISP compiler in LISP almost certainly
predates C.

Basically, there's a *lot* of history in programming languages. I'd
hate to see someone think that we went straight from assembler to C,
or that people didn't understand the value of dynamic languages very
early.

<mike