pyvm -- faster python

[Stelios Xanthakis]

Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/


Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes
in 55% of the time python takes
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers
new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to
make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios

 

[Kay Schluehr]

Hi.

pyvm is a program which can run python 2.4 bytecode (the .pyc files).
A demo pre-release is available at:
http://students.ceid.upatras.gr/~sxanth/pyvm/


Facts about pyvm:
- It's FAST. According to the "cooked-bench" benchmark suite it finishes
in 55% of the time python takes
- It's SMALL. Currently the source code is under 15k lines with the
builtin modules.
- It's new. Uses no code from CPython.
- It's incomplete. Not even near the stability and quality of python.
It needs A LOT of work before it can be compared to CPython.
Moreover, at the time it lacks many things like closures, long numbers
new style classes, etc.
- It's incompatible with CPython. Not all programs run.
- The C API is incompatible. You can't run C modules (a thin wrapper to
make pyvm appear as libpython *could* be possible but not a goal
AFAIC)
- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Hope it works!

Cheers,

Stelios

Hi Stelios,

could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. The last one I remember died shortly
after it was born:

http://www.python.org/pycon/papers/pymite/

This is sad because it is still challenging to offer a tiny interpreter
of a dynamic language for glue code/RAD on tiny hardware. A lot of
effort was spent to provide Java for microcontrollers especially for
SmartCards. I think a lot of people would show interest in Your project
if it gets somehow focussed and does not seem to be redundant.

Ciao,
Kay

 

[bearophileHUGS]

I've seen the benchmarks, they look quite interesting.

This project is probably a LOT of work; maybe people can tell us about
such efforts *before* doing so much work, so we can discuss it, and
avoid wasting time.

Maybe you can explain us why it is so fast, and/or maybe you can work
with the other developers to improve the speed of the normal CPython,
this can require equal or less work for you, and it can produce more
long-lasting results/consequences for your work.

Bye,
Bearophile

 

[Roger Binns]

could You tell us a bit more about Your motivation to create an

alternative C-Python interpreter?

I'd also be curious to know if the performance gains would remain
once it gets fleshed out with things like closures, long numbers,
new style classes and a C library compatibility shim.

Roger

 

[Peter Hansen]

could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. ..

If you check the URL more closely, you'll notice that this is a
university site.

It seem likely this being done at least partly as a school project,
which is certainly motivation enough for such a thing, *even* if it
doesn't survive (since it will pay for itself in learning).

-Peter

 

[Paul Rubin]

- The demo is an x86/linux binary only. You shouldn't trust binaries,
run it in a chrooted environment not as root!

Are you going to release the source? If not, it's a lot less interesting.

 

[djw]

Are you going to release the source? If not, it's a lot less interesting.

From the website:

"...the source will be released when it becomes more complete and stable."

 

[Dan Christensen]

From the website:

"...the source will be released when it becomes more complete and stable."

If he releases the source now, it'll probably become more complete and
stable sooner...

Remember, release early, release often!

Dan

 

[Stelios Xanthakis]

could You tell us a bit more about Your motivation to create an
alternative C-Python interpreter? There is AFAIK no such ambitious
project that has ever survived. The last one I remember died shortly
after it was born:

The motivation is that I just needed some bytecode engine. I deciced
to go with python bytecode, because well, IMO it's the best!
Eventually it turned out what it is.

I'm afraid this may end up dead before unborn too.
So it depends what people want. If nobody cares, I just learned python
the hard way and the project does to the abandonware cemetary.


Stelios

 

[Stelios Xanthakis]

This project is probably a LOT of work; maybe people can tell us about
such efforts *before* doing so much work, so we can discuss it, and
avoid wasting time.

It is a lot of work indeed.
Usually, when people announce "we shall create X", it doesn't happen.
And you don't know if you can do it at all *before*.

Maybe you can explain us why it is so fast, and/or maybe you can work
with the other developers to improve the speed of the normal CPython,
this can require equal or less work for you, and it can produce more
long-lasting results/consequences for your work.

The reason is that it's built from the scratch.
Guido would disagree with that, see py-dev thread:

http://www.mail-archive.com/[email protected]/msg01225.html


There *are* alternative ways to do some things in the vm and Joel
is simply wrong
Could this be .... Python 3000!!!?


Stelios

 

[Stelios Xanthakis]

I'd also be curious to know if the performance gains would remain
once it gets fleshed out with things like closures, long numbers,
new style classes and a C library compatibility shim.

I guess it will. There are several places with /*XXX:speed up*/
and a couple more ideas to gain exrta speed. On the other hand
pyvm does not check for Bad Internal Calls and other checks Cpython
does to be OK in all cases (for example if __cmp__ modifies a list
while it's being sorted, in pyvm you *can* get a crash).

There is no plan to implement closures; unless there is a way to
turn closures to default arguments with bytecode hacking. But yes,
closures are interesting, especially in generator expressions and
lambdas.

I am not very interested on C compatibility. Once the source is
open people can implement it if they want but I'd rather go with
an advanced ctypes module which will make loading dynamic libraries
a matter of python code.


Stelios

 

[=?iso-8859-1?Q?Fran=E7ois?= Pinard]

[Stelios Xanthakis]

I'm afraid this may end up dead before unborn too. So it depends what
people want. If nobody cares, [...]

People might not care so much about what could be done about your
project, unless you give them proper and complete means for evaluating
the state of affairs. Your project is very likely to die if you keep
your sources closed, and yourself loose interest in the project.

Opening your sources is no guarantee either that the community will
adopt your project. But this might give your project a better chance.

 

[Terry Reedy]

The reason is that it's built from the scratch.
Guido would disagree with that, see py-dev thread:

Guido, like me, believes in being correct before being fast.

http://www.mail-archive.com/[email protected]/msg01225.html


There *are* alternative ways to do some things in the vm and Joel
is simply wrong

The smiley doesn't negate your silliness. Joel explicitly talked about
complex, mature, debugged, in-use systems, with actual or potential
competitive replacements, not laboratory development toys and prototypes.
Three years between releases *is* a long time. And the Python codebase
*does* embody a lot of hard-won knowledge, hardly available elsewhere, that
should not be tossed. A particular example is how to get certain things to
work across numerous platforms in the face of ambuiguity and 'not defined'
behavior in the C standard and implementations.

Its possible that some future reference Python will be based on a new,
built-from-scratch core developed *in parallel* with current Python. But
that will only happen after it has been tested on a hundred different
platforms with a hundred different applications and packages (as happens,
more or less, with each new CPython release).

Terry J. Reedy

 

[Roger Binns]

I am not very interested on C compatibility.

That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon. What ultimately killed it was the problem of having
a decent library. You don't have to make the C library
compatibility totally high performance, but having any form of
it there will make adoption by others easier.

Roger

 

[Paul Rubin]

That will rule out all the gui frameworks, SSL, cryptography
and numerous other packages. Have a look at what happened to
Prothon.

I think it would be enough to retarget SWIG.

What ultimately killed it was the problem of having a decent
library.

Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?

 

[Roger Binns]

I think it would be enough to retarget SWIG.

That is also a lot of work, and a lot of toolkits that use Swig
for Python bindings are very strongly tied to the Swig-Python
internals. For example I doubt there are many where you could
tell Swig to output Java and it would just work. (I know none
of mine would work, nor would wxPython).

Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?

Err, you proved my point! Prothon was fine at the VM level. The
author couldn't figure out how to get a decent sized "standard"
library, and ultimately ended up abandoning his VM for .Net since
that gets you a large standard library. Jython also gets a large
standard library from the Java ones.

Roger

 

[Peter Hansen]

I think it would be enough to retarget SWIG.




Nah, I think people just lost interest. Otherwise, why isn't Jython
dead?

I thought Jython had full access to the entire library of standard Java,
plus at least the Python parts of the Python standard library...

-Peter

 

[Paul Rubin]

Err, you proved my point! Prothon was fine at the VM level. The
author couldn't figure out how to get a decent sized "standard"
library, and ultimately ended up abandoning his VM for .Net since
that gets you a large standard library. Jython also gets a large
standard library from the Java ones.

What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that shouldn't
be in C in the first place (the Python compiler and interpreter), etc.

I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.

 

[Roger Binns]

What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that shouldn't
be in C in the first place (the Python compiler and interpreter), etc.

Some examples are gui toolkits (eg wxPython), SSL (eg M2Crypto, pyopenssl)
and database (pysqlite, APSW). These aren't in the shipped with Python
library but are widely used.

I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.

You don't have to stay backwards compatible. It is best to provide
some sort of way of using the old extensions even if it is suboptimal
(eg some sort of mapping shim). The extensions can later be improved,
but if there is no simple way of recompiling and running then it will
take a long time for all the extensions to get ported.

I already get burnt out on the matrix of CPython versions and different
platforms. Adding another interpretter would make life even harder
for extension authors.

Roger

 

[Kay Schluehr]

What kind of stuff is in the existing Python C library that couldn't
be reimplemented or retargeted pretty easily? Most of it is either
wrappers for standard C functions (system calls), stuff that shouldn't
be in C in the first place (the Python compiler and interpreter),

etc.

Delete the "standard" and You still obtain huge librarys for .Net, Java
and Python. I also regret that Prothon starved in infancy but it might
be exeggerated to demand that each language designer or one of his
apostels should manage a huge community that enjoys doing redundant
stuff like writing Tk-bindings, regexp-engines and all that.

I hope that PyPy will replace CPython once it's solid enough. Trying
to stay backwards compatible with the legacy C API doesn't seem to me
to be that important a goal. Redoing the library may take more work
than the Prothon guy was willing to do for Prothon, but PyPy has more
community interest and maybe can attract more resources.

PyPy is a beast on it's own and provides interesting new ideas to
Python as well as runtime design in general. But I'm not quite shure
that it will stay in the boundarys of the current Python community
process and will subordinate to Guidos will. As a Marxist would say:
who controls the runtime also controls the language. Sooner or later
Python will go the LISP way of having a standardized "Common-Python" (
std_objectspace) and a number of dialects and DSLs running in their own
derived object spaces. Maybe Python 3000 is an illusion and will fade
away like a Fata Morgana the closer we seem come.

Kay