assigning to __class__ for an extension type: Is it still possible?

[gregory.lielens]

Hello,

We are currently writing python bindings to an existing C++ library,
and we encountered
a problem that some of you may have solved (or that has found
unsolvable ):

A C++ class (let's call it CClass) is binded using classical Python
extension API to _PClass, which is accesible through python without
any
problem. The problem is that I want this class to be extended/
extensible
in python, and expose the python-extended version (PClass) to library
users (_PClass should never be used directly nor be retruned by any
function).
The aim is to leave only performance critical methods in C++ so that
the
binding work is minimal, and develop the other methods in python so
that
they are easier to maintain/extend.

We thus have something like this

class PClass(_PClass):
def overide_method(self,...):
...
def new_method(self,...):
...

and I can define
a=PClass()
and use my new or overiden method
a.overide_method() a.new_method() as intended...

So far, so good, trouble begin when I have a method from another
class
PClass2 derived from _PClass2 which bind the C++ class CClass2, that
should return objects of type PClass:

Let call this method troublesome_method:

b=_PClass2()
c=b.troublesome_method()
type(c) gives _PClass.

Now I want to define a python class PClass2 for extending methods of
_PClass2 like I have done for _PClass, in particular I want that
PClass2
troublesome_method return objects of type PClass instead of _PClass...

To this end I try something like this

class PClass2(_PClass2):
...
def troubelsome_method(self):
base=_PClass2.troublesome_method(self)
base.__class__=PClass

and I have python complaining about TypeError: __class__ assignment:
only for heap types...

We have then added the Py_TPFLAGS_HEAPTYPE tp_flag, which turn _PClass
into a heap
class and should make this class assignment possible...or so I though:
When the _PClass is turned into a heaptype, assignent to
__class__trigger a test in
python's typeobject.c on tp_dealloc/tp_free witch raise an exception
TypeError: __class__ assignment: '_PClass' deallocator differs from
'PClass'
I have commented out this test, just to check what happen, and just
got an error later on
TypeError: __class__ assignment: '_PClass' object layout differs from
'PClass'

It seems thus that this approach is not possible, but problem is that
delegation ( embedding a _PClass instance in
PClass (lets say as _base attribute) and automatically forwarding all
methods calls/setattr/getattr to ._base) is far from ideal...
Indeed, we would have to change all other methods that accepted
_PClass, to give them PClass._base, this means wrapping a large
number of methods (from our c++ bindings) for argument-processing...

This is particularly frustrating cause I also have the impression
that
want we want to do was at one time possible in python, let say in
2002-2003, when __class__ was already assignable but before various
safety checks were implemented (assignmenent only for heaptypes, check
on tp_free/tp_dealloc, layout check,...)

Any hint on this problem?
In particular, a list of condition type A and B have to fullfull so
that a=A(); a.__class__=B is possible would be very nice, especially
when A is an extension type (I think the correct term is static type)
while B is defined by a class statement (dynamic type)...This is
something that is not easy to find in the docs, and seems to have
changed quite a lot between revisions 2.2/2.3/2.4/2.5...

Thanks,

Greg.

 

[gregory.lielens]

We have then added the Py_TPFLAGS_HEAPTYPE tp_flag, which turn _PClass
into a heap
class and should make this class assignment possible...

A precision: it seems that just addind Py_TPFLAGS_HEAPTYPE flag in
the
PyTypeObject tp_flags is not all you have to do to turn a static type
into a heap type: indeed, when doing such in the Noddy examples,
I have a segfault when just typing:
n=Noddy()
n

there is an access to the ht_name slot that is apparently non
initialized...

So Maybe the core of the problem is that I do not define the heap type
correctly....Do anybody have (or can tell me where to find) a small
example of an extension module defining a heap class? Similar to the
Noddy examples from the python doc?
I did not find any concrete example of Py_TPFLAGS_HEAPTYPE in the
current doc or on the net...

Best regards,

Greg.