PyPy and RPython

[sarvi]

Is there a plan to adopt PyPy and RPython under the python foundation
in attempt to standardize both.

I have been watching PyPy and RPython evolve over the years.

PyPy seems to have momentum and is rapidly gaining followers and
performance.

PyPy JIT and performance would be a good thing for the Python
Community
And it seems to be well ahead of Unladen Swallow in performance and in
a position to improve quite a bit.


Secondly I have always fantasized of never having to write C code yet
get its compiled performance.
With RPython(a strict subset of Python), I can actually compile it to
C/Machine code


These 2 seem like spectacular advantages for Python to pickup on.
And all this by just showing the PyPy and the Python foundation's
support and direction to adopt them.


Yet I see this forum relatively quite on PyPy or Rpython ? Any
reasons???

Sarvi

 

[Benjamin Peterson]

Is there a plan to adopt PyPy and RPython under the python foundation
in attempt to standardize both.

There is not.

Secondly I have always fantasized of never having to write C code yet
get its compiled performance.
With RPython(a strict subset of Python), I can actually compile it to
C/Machine code

RPython is not supposed to be a general purpose language. As a PyPy developer
myself, I can testify that it is no fun.

Yet I see this forum relatively quite on PyPy or Rpython ? Any
reasons???

You should post to the PyPy list instead. (See pypy.org)

 

[alex23]

Yet I see this forum relatively quite on PyPy or Rpython ?  Any
reasons???

For me, it's two major ones:

1. PyPy only recently hit a stability/performance point that makes it
worth checking out,
2. Using non-pure-python modules wasn't straightforward (at least when
I last looked)

However, I've always felt the PyPy project was far more promising than
Unladen Swallow.

 

[sarvi]

RPython is not supposed to be a general purpose language. As a PyPy developer
myself, I can testify that it is no fun.

Can be worse than than writing C/C++
Compared to Java, having the interpreter during development is huge

I actually think yall at PyPy are hugely underestimating RPython.
http://olliwang.com/2009/12/20/aes-implementation-in-rpython/
http://alexgaynor.net/2010/may/15/pypy-future-python/

Look at all the alternatives we have. Cython? Shedskin?
I'll take PyPy anyday instead of them

We make performance tradeoffs all the the time. Look at Mercurial.
90% python and 5% C
Wouldn't you rather this be 90% Python and 5% RPython ???

Add to the possibility of writing Python extension module in RPython.
You could be winning a whole group of developer mindshare.

You should post to the PyPy list instead. (See pypy.org)

I tried. got bounced. Just subscribed.
Will try again.

Sarvi

 

[Stefan Behnel]

sarvi, 02.09.2010 07:06:

Look at all the alternatives we have. Cython? Shedskin?
I'll take PyPy anyday instead of them

Fell free to do so, but don't forget that the choice of a language always
depends on the specific requirements at hand. Cython has proven its
applicability in a couple of large projects, for example. And it has a lot
more third party libraries available than both PyPy and Shedskin together:
all Python libraries, pure Python and CPython binary extensions, as well as
tons of code written in Cython, C, C++, Fortran, and then some. And you
don't have to give up one bit of CPython compatibility to use all of that.
That alone counts as a pretty huge advantage to some people.

Stefan

 

[John Nagle]

Is there a plan to adopt PyPy and RPython under the python foundation
in attempt to standardize both.

I have been watching PyPy and RPython evolve over the years.

PyPy seems to have momentum and is rapidly gaining followers and
performance.

PyPy JIT and performance would be a good thing for the Python
Community
And it seems to be well ahead of Unladen Swallow in performance and in
a position to improve quite a bit.


Secondly I have always fantasized of never having to write C code yet
get its compiled performance.
With RPython(a strict subset of Python), I can actually compile it to
C/Machine code


These 2 seem like spectacular advantages for Python to pickup on.
And all this by just showing the PyPy and the Python foundation's
support and direction to adopt them.


Yet I see this forum relatively quiet on PyPy or Rpython ? Any
reasons???

Sarvi

The winner on performance, by a huge margin, is Shed Skin,
the optimizing type-inferring compiler for a restricted subset
of Python. PyPy and Unladen Swallow have run into the problem
that if you want to keep some of the less useful dynamic semantics
of Python, the heavy-duty optimizations become extremely difficult.

However, if we defined a High Performance Python language, with
some restrictions, the problem becomes much easier. The necessary
restrictions are roughly this:

-- Functions, once defined, cannot be redefined.
(Inlining and redefinition do not play well
together.)

-- Variables are implicitly typed for the base types:
integer, float, bool, and everything else. The
compiler figures this out automatically.
(Shed Skin does this now.)

-- Unless a class uses a "setattr" function or has
a __setattr__ method, its entire list of attributes is
known at compile time.
(In other words, you can't patch in new attributes
from outside the class unless the class indicates
it supports that. You can subclass, of course.)

-- Mutable objects (other than some form of synchronized
object) cannot be shared between threads. This is the
key step in getting rid of the Global Interpreter Lock.

-- "eval" must be restricted to the form that has a list of
the variables it can access.

-- Import after startup probably won't work.

Those are the essential restrictions. With those, Python
could go 20x to 60x faster than CPython. The failures
of PyPy and Unladen Swallow to get any significant
performance gains over CPython demonstrate the futility
of trying to make the current language go fast.

Reference counts aren't a huge issue. With some static
analysis, most reference count updates can be optimized out.
(As for how this is done, the key issue is to determine whether
each function "keeps" a reference to each parameter. For
any function which does not, that parameter doesn't have
to have reference count updates within the function.
Most math library functions have this property.
You do have to analyze the entire program globally, though.)

John Nagle

 

[sarvi]

When I think about it these restrictions below seem a very reasonable
tradeoff for performance.
And I can use this for just the modules/sections that are performance
critical.

Essentially, the PyPy interpreter can have a restricted mode that
enforces these restriction.
This will help write such RPython code that can then be compiled into
C/ASM using the PyPy Compiler which as I understand can do this today.

If Shedskin generated C++ code is faster than PyPy generate C++ code.
Isn't that just another reason why PyPy and Shedskin should be joining
forces.

Wouldn't we want PyPy generated C code to be just as fast as
Shedskin's

Afterall thats how the PyPy compiler is built, right? and we do want
that to be fast too?

Sarvi

 

[John Nagle]

When I think about it these restrictions below seem a very reasonable
tradeoff for performance.

Yes.

And I can use this for just the modules/sections that are performance
critical.

Not quite. Neither Shed Skin nor RPython let you call from
restricted code to unrestricted code. That tends to happen
implicitly as objects are passed around. It's the global
analysis that makes this work; when you call something, you
need to know more about it than how to call it.

John Nagle

 

[sarvi]

    Yes.


    Not quite.  Neither Shed Skin nor RPython let you call from
restricted code to unrestricted code.  That tends to happen
implicitly as objects are passed around.  It's the global
analysis that makes this work; when you call something, you
need to know more about it than how to call it.

It should technically be possible to allow Python to call a module
written in RPython?
It should also compile RPython to a python module.so right?

Sarvi

 

[Stefan Behnel]

sarvi, 03.09.2010 07:30:

It should technically be possible to allow Python to call a module
written in RPython?

What's "Python" here? CPython? Then likely yes. I don't see a benefit, though.

It should also compile RPython to a python module.so right?

Why (and how) would CPython do that?

If you want a binary extension module for CPython, you can try to push the
RPython module through Cython. However, in that case, you wouldn't be
restricted to RPython in the first place.

Stefan

 

[John Nagle]

It should technically be possible to allow Python to call a module
written in RPython?

The problem is that, in a language where everything is an object,
everything you call calls you back.

The basic performance problem with CPython comes from the fact
that it uses the worst-case code for almost everything. Avoiding
that requires global analysis to detect the places where the code
clearly isn't doing anything weird and simpler code can be used.
Again, look at Shed Skin, which represents considerable progress
made by one guy. With more resources, that could be a very
good system.

John Nagle

 

[sarvi]

Well then, wouldn't it make sense for PyPy to use Shedskin and its
definition of Restricted Python?

I have heard repeatedly that PyPy RPython is very difficult to use.
Then why isn't PyPy using Shedskin to compile its PyPy-Jit?

Sarvi