subclassing extension type and assignment to __class__

[gregory lielens]

Hello,

I am currently writing python bindings to an existing C++ library, and I
just encountered a problem that I hope has been solved by more
experienced python programmers:

A C++ class (let's call it CClass) is binded using classical Python
extension API to _PClass, which is accesible through python without any
problem. The problem is that I want this class to be extended/extensible
in python, and expose the python-extended version (PClass) to library
users (_PClass should never be used directly nor be retruned by any
function).
The aim is to leave only performance critical methods in C++ so that the
binding work is minimal, and develop the other methods in python so that
they are easier to maintain/extend.

We thus have something like this

class PClass(_PClass):
def overide_method(self,...):
...
def new_method(self,...):
...

and I can define
a=PClass()
and use my new or overiden method
a.overide_method() a.new_method() as intended...

So far, so good, trouble begin when I have a method from another class
PClass2 derived from _PClass2 which bind the C++ class CClass2, that
should return objects of type PClass:

Let call this method troublesome_method:

b=_PClass2()
c=b.troublesome_method()
type(c) gives _PClass.

now I want to define a python class PClass2 for extending methods of
_PClass2 like I have done for _PClass, in particular I want that PClass2
troublesome_method return objects of type PClass instead of _PClass...

To this end I try something like this

class PClass2(_PClass2):
...
def troubelsome_method(self):
base=_PClass2.troublesome_method(self)
base.__class__=PClass

and I have python complaining about TypeError: __class__ assignment:
only for heap types...

It seems thus that my approach is not possible, but problem is that I
can not see another approach that would not involve far more efforts
that this one (I have the impression I have to forget about making my
PClass derived from _PClass, but instead embed a _PClass instance in
PClass, far from ideal cause I would have to reimplement all method, not
only those that I want to change )
This is particularly frustrating cause I also have the impression that
want I want to do was at one time possible in python, let say in
2002-2003, when __class__ was already assignable but before assignment
was forbidden for non-heap types

Any hint on this problem?

Thanks,

Greg.

 

[Lenard Lindstrom]

Hello,

I am currently writing python bindings to an existing C++ library, and
I just encountered a problem that I hope has been solved by more
experienced python programmers:

A C++ class (let's call it CClass) is binded using classical Python
extension API to _PClass, which is accesible through python without
any problem. The problem is that I want this class to be
extended/extensible in python, and expose the python-extended version
(PClass) to library users (_PClass should never be used directly nor
be retruned by any function).
The aim is to leave only performance critical methods in C++ so that
the binding work is minimal, and develop the other methods in python
so that they are easier to maintain/extend.

We thus have something like this

class PClass(_PClass):
def overide_method(self,...):
...
def new_method(self,...):
...

and I can define
a=PClass()
and use my new or overiden method
a.overide_method() a.new_method() as intended...

So far, so good, trouble begin when I have a method from another class
PClass2 derived from _PClass2 which bind the C++ class CClass2, that
should return objects of type PClass:

Let call this method troublesome_method:

b=_PClass2()
c=b.troublesome_method()
type(c) gives _PClass.

now I want to define a python class PClass2 for extending methods of
_PClass2 like I have done for _PClass, in particular I want that
PClass2 troublesome_method return objects of type PClass instead of
_PClass...

To this end I try something like this

class PClass2(_PClass2):
...
def troubelsome_method(self):
base=_PClass2.troublesome_method(self)
base.__class__=PClass

and I have python complaining about TypeError: __class__ assignment:
only for heap types...

It seems thus that my approach is not possible, but problem is that I
can not see another approach that would not involve far more efforts
that this one (I have the impression I have to forget about making my
PClass derived from _PClass, but instead embed a _PClass instance in
PClass, far from ideal cause I would have to reimplement all method,
not only those that I want to change )
This is particularly frustrating cause I also have the impression that
want I want to do was at one time possible in python, let say in
2002-2003, when __class__ was already assignable but before assignment
was forbidden for non-heap types

Any hint on this problem?

I have just run into the same thing with the builtin list type. The trick
was not to declare my subclass directly from list, but rather a subclass
of list. So try this:

class PClassBase(_PClass):
pass

class PClass(PClassBase):
...

class PClass2(PClassBase):
...

Why? I cannot say other than I have noted that Python new-style classes
and extension types are not quite the same thing. That is, a new-style
class is a particular kind of extension type.

Lenard Lindstrom
<[email protected]>

 

[gregory lielens]

Thanks for your answer, it means I am not the only one having this kind
of problem...

I have just run into the same thing with the builtin list type. The trick
was not to declare my subclass directly from list, but rather a subclass
of list. So try this:

class PClassBase(_PClass):
pass

class PClass(PClassBase):
...

class PClass2(PClassBase):
...

I think here you mean PClass2(PClassBase2), with a
PClassBase2(_PClass2): In my example the 2 class do not inherit fro the
same base class...

Why? I cannot say other than I have noted that Python new-style classes
and extension types are not quite the same thing. That is, a new-style
class is a particular kind of extension type.

I see how it can help me making PClassBase a PClass (or any type derived
from PClassBase), but what I fail to see is how this will allow me to
change a _PClass into one of it's derived types: this is necessary for
updating methods of _PClass2 that return object of type _PClass (and on
this I do not have much control, both _PClass and _PClass2 and all of
their methods are implemented in C++). To updates theses methods for the
pure python derived class PClass2, I have to make them return the python
class PClass instead of the original _PClass, which means I still have
to transform a _PClass object into its derived type PClassBase or
PClass, and at this time things go pear-shaped :-(....or is there
something I miss?



Greg.

 

[Lenard Lindstrom]

Thanks for your answer, it means I am not the only one having this
kind of problem...
I think here you mean PClass2(PClassBase2), with a
PClassBase2(_PClass2): In my example the 2 class do not inherit fro
the same base class...

I see how it can help me making PClassBase a PClass (or any type
derived from PClassBase), but what I fail to see is how this will
allow me to change a _PClass into one of it's derived types: this is
necessary for updating methods of _PClass2 that return object of type
_PClass (and on this I do not have much control, both _PClass and
_PClass2 and all of their methods are implemented in C++). To updates
theses methods for the pure python derived class PClass2, I have to
make them return the python class PClass instead of the original
_PClass, which means I still have to transform a _PClass object into
its derived type PClassBase or PClass, and at this time things go
pear-shaped :-(....or is there something I miss?

Sorry for the confusion. This looks like a situation best handled with
embedding rather than inheritance. The PClass and PClass2 wrapper classes
can share whatever relationship _PClass and _PClass2 share.

class PClass:
def __init__(inst=None):
if inst is None:
inst = _PClass()
self._inst = inst
def override_method(self, ...):
self._inst.override_method(...)
def new_method(self, ...):
...

class PClass2:
self._inst = _PClass2()
def troublesome_method(self):
base = PClass(self._inst.troublesome_method())
...
return base

Alternatively you could provide a way to pass PClass back to the extension module
for _PClass2.troublesome_method to use instead of _PClass when creating a return
object. But the embedding approach seems more appropriate when _PClass is not to
be used directly.

Lenard Lindstrom
<[email protected]>

 

[gregory lielens]

Sorry for the confusion. This looks like a situation best handled with
embedding rather than inheritance. The PClass and PClass2 wrapper classes
can share whatever relationship _PClass and _PClass2 share.

class PClass:
def __init__(inst=None):
if inst is None:
inst = _PClass()
self._inst = inst
def override_method(self, ...):
self._inst.override_method(...)
def new_method(self, ...):
...

class PClass2:
self._inst = _PClass2()
def troublesome_method(self):
base = PClass(self._inst.troublesome_method())
...
return base

Yes I was affraid this would be the conclusion: embedding instead of
inheritance...But this means that not only the method that need to be
modified need to be rewritten in python, all the other ones also just to
delegate to the embedded instance...
This is not too practical, even if such writing can be more or less
automatized...
For my purpose it seems that there was a big step backward between
python 2.2 and 2.3, and I am surprised this has been mostly
unnoticed...This problem should be common for many people trying to
write mixed C/C++ / python classes. I found some reference on this in
the list archive, a few people beeing anoyed by it, but the proposition
to make the test about assignment to class less strict was not retained
it seems...I plan to experiment a little with the python C
implementation to see what happen if I bypass this test...

Alternatively you could provide a way to pass PClass back to the extension module
for _PClass2.troublesome_method to use instead of _PClass when creating a return
object. But the embedding approach seems more appropriate when _PClass is not to
be used directly.

Yes but again this is not so practical, the purpose of the _PClass /
PClass implementation was to make the bindings as light, and as easy to
implement / extend as possible, and both the embedding and the creation
of PClass within _PClass2 methods defeat this purpose...

I think allowing assignment to class for native types should be done if
possible, for example if the new class is a derived type (possibly using
__slots__)... I saw a post on this by Guido, but the reason why it was
not possible is not clear to me and the thread died without further
explanations...Maybe my experiments will show me what happen, or someone
more knowledgeable than me can explain why this is not possible?

Greg.

 

[David Bolen]

(...)

Yes I was affraid this would be the conclusion: embedding instead of
inheritance...But this means that not only the method that need to be
modified need to be rewritten in python, all the other ones also just
to delegate to the embedded instance...
This is not too practical, even if such writing can be more or less
automatized...

Unless I'm misunderstanding, couldn't one of __getattr__ or
__getattribute__ make mapping other methods you don't override very
simple and practical, not to mention fully automated with 2-3 lines of
code? In particular, __getattr__ would seem good for your use since
it is only called for attributes that couldn't already be located.

I've had code that wrapped underlying objects in a number of cases, and
always found that to be a pretty robust mechanism.

-- David

 

[Gregory Lielens]

Unless I'm misunderstanding, couldn't one of __getattr__ or
__getattribute__ make mapping other methods you don't override very
simple and practical, not to mention fully automated with 2-3 lines of
code? In particular, __getattr__ would seem good for your use since
it is only called for attributes that couldn't already be located.

I've had code that wrapped underlying objects in a number of cases, and
always found that to be a pretty robust mechanism.

Thanks for mentioning this, after more research I came up with something
usable, using delegating technique from python cookbook:


#wrapper

def wrap_class(base):
class wrapper:
__base__=base
def __init__(self,*args,**kwargs):
if len(args)==1 and type(args[0])==self.__base__ and kwargs=={}:
self._base = args[0]
else:
self._base=self.__class__.__base__.__new__(self.__class__.__base__,*args,**kwargs)
def __getattr__(self,s):
return self._base.__getattribute__(s)
return wrapper



#wrap class
complex2=wrap_class(complex)
#extend wrapper class
def supaprint(self):
print "printed with supaprint(tm):"
print self
complex2.supaprint=supaprint

#define wrapper class from base class
c1=1+1j
c2=complex2(c1)
#test old functionalities
print c1==c2
print "c1=",c1
print "c2=",c1
#test extension
c2.supaprint()
c1.supaprint()

So this basically fit the bill, it even delegates special methods it
seems, although I am not sure why...It is like writting heritage ourself,
in a way

Remaning problem is that if we use the class generating
wrapper function (simple), we loose the classic class definition syntax
and rely on explicitely adding methods.
The alternative is to include the wrapper
machinery in every "home-derived" class, but you are right, it is not as bad as I
though

The biggest point I am not sure now is performance: Isn't a big penalty
associated to this embedding, compared to derivation? Python performance
is not so critical in our application, but I would be uncomfortable having
a factor 10 penalty in methd calling associated to this approach...For now, this will be used for
testing new implementations/extensions, that will be added to the C++
written binding afterwards.

I'd like to thanks the list for the hints, I will post the results of my
experimatations relaxing the assigment to __class__ test if they are
interesting.

Greg.