Is a "real" C-Python possible?

[Christian Heimes]

> Given that the Python core team has been mostly silent about JIT
compilation and Armin Rigos work in particular which started 5 years
ago ( Psyco will not be promoted towards Python 3.0 and there is no
indication that anyone but Armin would maintain Psyco ) I wonder about
this sudden insight. But maybe you can tell us more when you work on
such stuff for CPython yourself? Given your status in the core Python
team this would have a chance of not being just a loosely coupled
independent project as almost all the interesting stuff that happens
around Python these days.

I don't see an indication that anybody but the creator of Psyco does
understand the code base. *g*

Guido has stated his opinion about optimizations more than once. My own
opinion as core developer (which is quite similar to Guido's opinion) is:

We are happy and glad for every improvement regarding speed, memory
usage or features if and only if:

* The implementation must be clean, well designed, well documented well
written and platform independent / supported on all platforms. Python
runs on machines from mobile phones to large main frames.

* The improvement must NOT hinder or slow down future development at
all cost. If it's so complicated that it might slow down future
development than it's a no go. It's more important to us to have a clean
and understandable code base than to add hundreds of small improvements
which makes debugging a nightmare.

* You are willing to support and fix the improvement for X years where
X is between 4 and INF years.

* The modification must not slow down Python for common uses like a
single threaded, single CPU bound program or small script. This rules
out all existing attempts to remove the GIL from Python since they have
slowed down Python to 50% or less. However Guido said a few months ago
that he would endorse a SMP branch of Python aimed to multi core and
multi threaded apps.

* The code and all its dependencies must be compatible with Python license.

* The code must be written following the C89 standards.

By the way core development is open for everybody. Any patch is
appreciated, starting from fixing a typo in the docs over bug reports,
bug fixes to new features.

Read the PEPs! http://www.python.org/dev/peps/
Subscribe to the mailing lists (I suggest gmane.org)!
Get involved!

Christian

 

[Paul Rudin]

We are happy and glad for every improvement regarding speed, memory
usage or features if and only if: ...

... platform independent / supported on all platforms. Python runs
on machines from mobile phones to large main frames.

JOOI - there are things in the standard library that are not supported
on all platforms. Why would that be a basis for excluding some
psyco-like package?

 

[Christian Heimes]

class A(object):
foo = property:
def fget(self):
return self._foo
def fset(self, value):
self._foo = value

which was translated as follows:

class A(object):
def thunk():
def fget(self):
return self._foo
def fset(self, value):
self._foo = value
return vars()
foo = propery(**thunk())
del thunk

Python 2.6 and 3.0 have a more Pythonic way for the problem:

class A(object):
@property
def foo(self):
return self._foo

@foo.setter
def foo(self, value)
self._foo = value

@foo.deletter
def foo(self)
del self._foo

class B(A):
# one can even overwrite the getter in a subclass
@foo.getter
def foo(self):
return self._foo * 2

Christian

 

[Christian Heimes]

class A(object):
foo = property:
def fget(self):
return self._foo
def fset(self, value):
self._foo = value

which was translated as follows:

class A(object):
def thunk():
def fget(self):
return self._foo
def fset(self, value):
self._foo = value
return vars()
foo = propery(**thunk())
del thunk

Python 2.6 and 3.0 have a more Pythonic way for the problem:

class A(object):
@property
def foo(self):
return self._foo

@foo.setter
def foo(self, value)
self._foo = value

@foo.deletter
def foo(self)
del self._foo

class B(A):
# one can even overwrite the getter in a subclass
@foo.getter
def foo(self):
return self._foo * 2

Christian

 

[George Sakkis]

I vaguely remember a discussion a few years ago, where someone made
the quite reasonable suggestion of introducing some kind of
thunk_statement:

class A(object):
foo = property:
def fget(self):
return self._foo
def fset(self, value):
self._foo = value

That's almost identical to a recipe I had written once upon a time,
without requiring a syntax change: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410698

George

 

[George Sakkis]

Python 2.6 and 3.0 have a more Pythonic way for the problem:

class A(object):
@property
def foo(self):
return self._foo

@foo.setter
def foo(self, value)
self._foo = value

@foo.deletter
def foo(self)
del self._foo

class B(A):
# one can even overwrite the getter in a subclass
@foo.getter
def foo(self):
return self._foo * 2

Christian

This is by definition Pythonic since it was conceived by the BDFL.It
is also certainly an improvement over the current common practice of
polluting the class namespace with private getters and setters. Still
it's a kludge compared to languages with builtin support for
properties.

George

 

[Kay Schluehr]

I think it is one step closer to Lisp. I believe that it would be
worth considering adding defmacro statement. Any syntax, including if,
else, for, while, class, lambda, try, except, etc. would be
implemented with defmacros. We would only need a minimalistic syntax,
that would bootstrap a full Python syntax on startup. And as for
speed, we all know how Lisp compares to Python.

It would be great if Python could be speeded up to SBCL Lisp by just
transforming one parse tree into another one. But since Python is
compiled to bytecodes I dare to say that surface syntax is not the key
factor

Kay

 

[Chris Mellon]

This is by definition Pythonic since it was conceived by the BDFL.It
is also certainly an improvement over the current common practice of
polluting the class namespace with private getters and setters. Still
it's a kludge compared to languages with builtin support for
properties.

How exactly is this a kludge? This is almost identical syntax (but
with less indentation) to a C# property declaration. The only thing
that's simpler is auto-generation of trivial accessors via a
decoration, but those are useless in Python so only the case of
getters and setters that actually do something needs to be addressed.

If the only thing that's not a "kludge" is direct syntax support for a
feature, you've set a pretty high and uselessly arbitrary bar.

 

[George Sakkis]

How exactly is this a kludge?

In three (at least) ways:

1. The property name ('foo') is repeated *5* times for a single class.
Talk about DRY.
2. Total inconsistency: @property for the getter when it is defined
for the first time, @foo.setter/@foo.deletter for the setter/deletter,
@foo.getter when the getter is redefined. WTF ?!

This is almost identical syntax (but with less indentation) to a C# property declaration.

3. Less indentation is not an advantage here; typically ones wants all
two or three related functions that define the property to stand out
as a separate group, not be mixed with regular public methods.

Sorry, C# wins hands down on this.

George

 

[Aahz]

JOOI - there are things in the standard library that are not supported
on all platforms. Why would that be a basis for excluding some
psyco-like package?

As a stand-alone package (even shipping with Python), that's not a
problem; my understanding is that other issues have prevented including
Psyco. However, Christian was talking specifically about changes to the
CPython core for performance purposes.

 

[Kay Schluehr]

How exactly is this a kludge? This is almost identical syntax (but
with less indentation) to a C# property declaration.

C# properties are thunk statements:

private Object _foo = null;

public Object foo {
get { return this._foo; }
set { this._foo = value; }
}

In Python pseudo code this would translate to

foo:
def get(self): return self._foo
def set(self, value): self._foo = value

omitting the reference to a property constructor. This is the pure
essence: assign methods not to objects but object attributes, for
which certain protocols are defined. It could be generalized for GUI
applications using triggers or other dataflow related bindings. The
"pythonic" solution being mentioned is a rather obscure and convoluted
decorator hack. "Executable pseudocode" reads different and for the
latter assertion one doesn't need a BDFL stamped license. It
demonstrates the cleverness of the programmer more than it clarifies
the issue.

Kay

 

[Paul Boddie]

I don't see an indication that anybody but the creator of Psyco does
understand the code base. *g*

Then you haven't been reading the right IRC channel recently. ;-)

Guido has stated his opinion about optimizations more than once. My own
opinion as core developer (which is quite similar to Guido's opinion) is:

We are happy and glad for every improvement regarding speed, memory
usage or features if and only if:

* The implementation must be clean, well designed, well documented well
written and platform independent / supported on all platforms. Python
runs on machines from mobile phones to large main frames.

Indeed, but there's arguably a certain amount of deadlock around
making unpatched, released versions of Python available in all these
places, unless there's been some activity below the surface in the
python-dev community on things like cross-compilation and not running
distutils using the newly built Python executable (which, as I
remember, was but one of the problems). Your point stands, naturally,
but if there's potential for some real movement on some
uncontroversial issues, and yet we see no movement, one remains
skeptical about getting even slightly controversial improvements into
vanilla CPython.

* The improvement must NOT hinder or slow down future development at
all cost. If it's so complicated that it might slow down future
development than it's a no go. It's more important to us to have a clean
and understandable code base than to add hundreds of small improvements
which makes debugging a nightmare.

Perhaps, but what would people prefer: yet more language bolt-ons or
better performance?

* You are willing to support and fix the improvement for X years where
X is between 4 and INF years.

Can't argue with this one. ;-)

* The modification must not slow down Python for common uses like a
single threaded, single CPU bound program or small script. This rules
out all existing attempts to remove the GIL from Python since they have
slowed down Python to 50% or less. However Guido said a few months ago
that he would endorse a SMP branch of Python aimed to multi core and
multi threaded apps.

It will be interesting to see what happens with recent work on
improving threading within CPython. As for Psyco (which perhaps offers
concurrency benefits only through instruction-level parallelism, if we
briefly consider that topic), I can understand that just-in-time
compilation can bring certain penalties in terms of memory usage and
initialisation times (as Java virtual machines have demonstrated), but
there's a compelling argument for trying to make such technologies
available to CPython if they can be switched off and won't then incur
such penalties. But we presumably return to the point of people not
wanting to touch code that has anything to do with such features: a
combination of social factors and the priorities of the group.

Paul

 

[Christian Heimes]

Then you haven't been reading the right IRC channel recently. ;-)

What's the right channel? I'm on #python and #python-dev

Indeed, but there's arguably a certain amount of deadlock around
making unpatched, released versions of Python available in all these
places, unless there's been some activity below the surface in the
python-dev community on things like cross-compilation and not running
distutils using the newly built Python executable (which, as I
remember, was but one of the problems). Your point stands, naturally,
but if there's potential for some real movement on some
uncontroversial issues, and yet we see no movement, one remains
skeptical about getting even slightly controversial improvements into
vanilla CPython.

I don't get your point, especially when you talk about distutils. Please
elaborate.

(C)Python has a well known process to get new features or changes into
the language: Write a PEP, convince enough core developers and/or Guido,
implement the feature. I don't see a PEP about JIT in the list at
abouthttp://www.python.org/dev/peps/, do you? :]

Besides nobody is going to stop you from creating a fork. Christian
Tismer forked of stackless years ago. It's a successful branch with
useful additions to the language. It got never merged back because
Christian didn't feel right about it.

Perhaps, but what would people prefer: yet more language bolt-ons or
better performance?

I prefer a fast, stable and maintainable Python over a faster but unstable.

It will be interesting to see what happens with recent work on
improving threading within CPython. As for Psyco (which perhaps offers
concurrency benefits only through instruction-level parallelism, if we
briefly consider that topic), I can understand that just-in-time
compilation can bring certain penalties in terms of memory usage and
initialisation times (as Java virtual machines have demonstrated), but
there's a compelling argument for trying to make such technologies
available to CPython if they can be switched off and won't then incur
such penalties. But we presumably return to the point of people not
wanting to touch code that has anything to do with such features: a
combination of social factors and the priorities of the group.

Rhamph is working on a GIL-less Python version. It may become a compile
time option someday in the future. Others have worked hard to speed up
other parts of Python. We have multiple pending patches which speed up
small parts of Python. Some are related to peephole (byte code
optimizations), other patches speed up attribute access on classes or
globals. The global lookup patch makes globals as fast as locals.

I've done my share for the poor Windows souls when I created the VS 2008
PCbuild9 directory and enabled PGO builds. PGO builds are about 10%
faster than ordinary VS 2008 builds. VS 2008 should be slightly faster
than VS 2003 but I can bench mark it on my machine.

In my opinion an optional JIT as compile time or startup option has good
chances to become part of the CPython implementation. You "only" have to
replace ceval.c ... :]

Christian

 

[Christian Heimes]

Then you haven't been reading the right IRC channel recently. ;-)

What's the right channel? I'm on #python and #python-dev

Indeed, but there's arguably a certain amount of deadlock around
making unpatched, released versions of Python available in all these
places, unless there's been some activity below the surface in the
python-dev community on things like cross-compilation and not running
distutils using the newly built Python executable (which, as I
remember, was but one of the problems). Your point stands, naturally,
but if there's potential for some real movement on some
uncontroversial issues, and yet we see no movement, one remains
skeptical about getting even slightly controversial improvements into
vanilla CPython.

I don't get your point, especially when you talk about distutils. Please
elaborate.

(C)Python has a well known process to get new features or changes into
the language: Write a PEP, convince enough core developers and/or Guido,
implement the feature. I don't see a PEP about JIT in the list at
abouthttp://www.python.org/dev/peps/, do you? :]

Besides nobody is going to stop you from creating a fork. Christian
Tismer forked of stackless years ago. It's a successful branch with
useful additions to the language. It got never merged back because
Christian didn't feel right about it.

Perhaps, but what would people prefer: yet more language bolt-ons or
better performance?

I prefer a fast, stable and maintainable Python over a faster but unstable.

It will be interesting to see what happens with recent work on
improving threading within CPython. As for Psyco (which perhaps offers
concurrency benefits only through instruction-level parallelism, if we
briefly consider that topic), I can understand that just-in-time
compilation can bring certain penalties in terms of memory usage and
initialisation times (as Java virtual machines have demonstrated), but
there's a compelling argument for trying to make such technologies
available to CPython if they can be switched off and won't then incur
such penalties. But we presumably return to the point of people not
wanting to touch code that has anything to do with such features: a
combination of social factors and the priorities of the group.

Rhamph is working on a GIL-less Python version. It may become a compile
time option someday in the future. Others have worked hard to speed up
other parts of Python. We have multiple pending patches which speed up
small parts of Python. Some are related to peephole (byte code
optimizations), other patches speed up attribute access on classes or
globals. The global lookup patch makes globals as fast as locals.

I've done my share for the poor Windows souls when I created the VS 2008
PCbuild9 directory and enabled PGO builds. PGO builds are about 10%
faster than ordinary VS 2008 builds. VS 2008 should be slightly faster
than VS 2003 but I can bench mark it on my machine.

In my opinion an optional JIT as compile time or startup option has good
chances to become part of the CPython implementation. You "only" have to
replace ceval.c ... :]

Christian

 

[Bruno Desthuilliers]

sturlamolden a écrit :

I do not want to use Lisp. The syntax is awkward and strange, and does
not fit in my brain. I cannot read Lisp code and get a mental image of
what it does. Readability is what sets Python apart.

Part of this readability comes from opiniated choices wrt/ syntax.
Letting anyone invent it's own syntax could well ruin this.

 

[Paul Boddie]

What's the right channel? I'm on #python and #python-dev

But where are people who might know Psyco likely to hang out? ;-)
Anyway, it remains to be seen what happens, but by reading various
conversations I get the impression that something could be afoot. I
wouldn't want to preempt any announcements, however, so I'll say no
more on the matter.

[Cross-compilation]

I don't get your point, especially when you talk about distutils. Please
elaborate.

From memory, once the Python executable is built, there's some kind of
process where modules get built with the newly built Python (perhaps
the rule labelled "Build the shared modules" in the Makefile). This
doesn't go down well when cross-compiling Python.

(C)Python has a well known process to get new features or changes into
the language: Write a PEP, convince enough core developers and/or Guido,
implement the feature. I don't see a PEP about JIT in the list at
abouthttp://www.python.org/dev/peps/, do you? :]

PEPs are very much skewed towards language changes, which then
encourages everyone and their dog to submit language changes, of
course.

Besides nobody is going to stop you from creating a fork. Christian
Tismer forked of stackless years ago. It's a successful branch with
useful additions to the language. It got never merged back because
Christian didn't feel right about it.

I think we all appreciate the work done by the core developers to
improve Python's stability and performance; new language features
don't interest me quite as much: it was, after all, possible to write
working systems in Python 1.x, with the addition of Unicode probably
rounding out quite a decent subset of what the language offers today.
The demands for greater performance enhancements than those possible
by modifying the existing virtual machine conservatively may, however,
eventually lead people to consider other paths of development just as
Stackless emerged as a fork in order to offer things that CPython
could not.

I think the pressure to fork Python will only increase over time,
considering the above together with the not inconsiderable impact of
Python 3.0 and the dependencies on Python 2.x out there in lots of
places, typically out of sight (or at least, the immediate
consideration) of the core developers.

Paul

 

[Patrick Mullen]

1. The property name ('foo') is repeated *5* times for a single class.
Talk about DRY.
2. Total inconsistency: @property for the getter when it is defined
for the first time, @foo.setter/@foo.deletter for the setter/deletter,
@foo.getter when the getter is redefined. WTF ?!

Eww, I agree with George here, with respect to these two points. When
I looked at this my first wtf was the @property and then @foo.getter
business. I really don't mind the current way of doing things: attr =
property(get,set). Other mechanisms can be created with getattr
routines. I don't really like this new syntax at all. Too many @
marks, inconsistancies, and too many foos everywhere. Not to mention
how long it reads. For only getters, it's not bad though, and a
little better than property().

Decorators really don't feel pythonic to me at all, mostly due to the
@ symbol, but it looks really awful in this instance.

What about this, somewhat similar but not ugly syntax:

class A:
foo = property()
def foo.get():
return self._foo
def foo.delete():
del self._foo
def foo.set(val):
self._foo = val

Defining something with a dot is currently a syntax error. Ok, so
it's still too many foos. At least it's consistent. I'm not really
proposing this btw. I'd rather not introduce more specialized syntax.

How about abusing with:

class A:
with property("foo"):
def get
def set...

There's your thunk, and I really like with, but am saddened that it
has such limited use at the moment. Of course this isn't really what
with is for...

Can anyone tell me what's wrong about the current property() syntax,
besides namespace polution?

 

[Chris Mellon]

Eww, I agree with George here, with respect to these two points. When
I looked at this my first wtf was the @property and then @foo.getter
business. I really don't mind the current way of doing things: attr =
property(get,set). Other mechanisms can be created with getattr
routines. I don't really like this new syntax at all.

For the record, this is not new syntax. It's implemented this way
specifically to avoid the creation of new syntax for properties.

Too many @
marks, inconsistancies, and too many foos everywhere. Not to mention
how long it reads. For only getters, it's not bad though, and a
little better than property().

I don't feel that it's especially inconsistent, and I like decorators.
Having to write foo everywhere isn't that nice, but it's only mildly
worse than C# to me - I find the extra block levels really atrocious.

Decorators really don't feel pythonic to me at all, mostly due to the
@ symbol, but it looks really awful in this instance.

What about this, somewhat similar but not ugly syntax:

class A:
foo = property()
def foo.get():
return self._foo
def foo.delete():
del self._foo
def foo.set(val):
self._foo = val

Defining something with a dot is currently a syntax error. Ok, so
it's still too many foos. At least it's consistent. I'm not really
proposing this btw. I'd rather not introduce more specialized syntax.

How about abusing with:

class A:
with property("foo"):
def get
def set...

There's your thunk, and I really like with, but am saddened that it
has such limited use at the moment. Of course this isn't really what
with is for...

Can anyone tell me what's wrong about the current property() syntax,
besides namespace polution?

Nothing, except that people prefer decorators and they don't like the
namespace pollution. foo = property() isn't going away and if you
prefer it (I don't) you're free to use it. If you don't like
decorators in general it's fairly obvious that you won't be partial to
a decorator based feature.

It's not that big a deal anyway, of course, the use case for
properties in Python has a much smaller scope than in C#, and
getter-only properties (which you can create with just @property) are
the majority of those.

 

[Duncan Booth]

Python 2.6 and 3.0 have a more Pythonic way for the problem:

class A(object):
@property
def foo(self):
return self._foo

@foo.setter
def foo(self, value)
self._foo = value

@foo.deletter
def foo(self)
del self._foo

class B(A):
# one can even overwrite the getter in a subclass
@foo.getter
def foo(self):
return self._foo * 2

That would be great if it worked, but it doesn't.

Fixing your typos (missing colons, spelling of deleter, and in B the
decorator needs to refer to A.foo.getter):

Python 3.0a2 (r30a2:59405M, Dec 7 2007, 15:23:28) [MSC v.1500 32 bit
(Intel)] on win32
Type "copyright", "credits" or "license()" for more information.

****************************************************************
Personal firewall software may warn about the connection IDLE
makes to its subprocess using this computer's internal loopback
interface. This connection is not visible on any external
interface and no data is sent to or received from the Internet.
****************************************************************

IDLE 3.0a1 @property
def foo(self):
return self._foo

@foo.setter
def foo(self, value):
self._foo = value

@foo.deleter
def foo(self):
del self._foo

# one can even overwrite the getter in a subclass
@A.foo.getter
def foo(self):
return self._foo * 2

a = A()
a.foo = 5
a.foo 10
A.__dict__['foo']

B.__dict__['foo']

<property object at 0x01261F80>

Unfortunately as currently implemented, getter setter and deleter just
update the existing property, so the getter defined in B changes how the
property works in A as well. I think the intention may have been that they
should create a new property each time, but this isn't what has been
implemented.

 

[sturlamolden]

I don't feel that it's especially inconsistent, and I like decorators.
Having to write foo everywhere isn't that nice, but it's only mildly
worse than C# to me - I find the extra block levels really atrocious.

Personally I find properties atrocious and unsafe. One cannot
distinguish between a function call and binding an attribute in a
statement like:

foo.bar = 2 # Does this call a function or bind an attribute?
# Is this foo.setBar(2) or setattr(foo,'bar',2)?

Even worse: if we make a typo, the error will not be detected as the
syntax is still valid. Properties and dynamic binding do not mix.