subclassing extension type and assignment to __class__

Discussion in 'Python' started by gregory lielens, Nov 27, 2004.

  1. Hello,

    I am currently writing python bindings to an existing C++ library, and I
    just encountered a problem that I hope has been solved by more
    experienced python programmers:

    A C++ class (let's call it CClass) is binded using classical Python
    extension API to _PClass, which is accesible through python without any
    problem. The problem is that I want this class to be extended/extensible
    in python, and expose the python-extended version (PClass) to library
    users (_PClass should never be used directly nor be retruned by any
    function).
    The aim is to leave only performance critical methods in C++ so that the
    binding work is minimal, and develop the other methods in python so that
    they are easier to maintain/extend.

    We thus have something like this

    class PClass(_PClass):
    def overide_method(self,...):
    ...
    def new_method(self,...):
    ...

    and I can define
    a=PClass()
    and use my new or overiden method
    a.overide_method() a.new_method() as intended...

    So far, so good, trouble begin when I have a method from another class
    PClass2 derived from _PClass2 which bind the C++ class CClass2, that
    should return objects of type PClass:

    Let call this method troublesome_method:

    b=_PClass2()
    c=b.troublesome_method()
    type(c) gives _PClass.

    now I want to define a python class PClass2 for extending methods of
    _PClass2 like I have done for _PClass, in particular I want that PClass2
    troublesome_method return objects of type PClass instead of _PClass...

    To this end I try something like this

    class PClass2(_PClass2):
    ...
    def troubelsome_method(self):
    base=_PClass2.troublesome_method(self)
    base.__class__=PClass

    and I have python complaining about TypeError: __class__ assignment:
    only for heap types...

    It seems thus that my approach is not possible, but problem is that I
    can not see another approach that would not involve far more efforts
    that this one (I have the impression I have to forget about making my
    PClass derived from _PClass, but instead embed a _PClass instance in
    PClass, far from ideal cause I would have to reimplement all method, not
    only those that I want to change :( )
    This is particularly frustrating cause I also have the impression that
    want I want to do was at one time possible in python, let say in
    2002-2003, when __class__ was already assignable but before assignment
    was forbidden for non-heap types

    Any hint on this problem?

    Thanks,

    Greg.
    gregory lielens, Nov 27, 2004
    #1
    1. Advertising

  2. gregory lielens <> writes:

    > Hello,
    >
    > I am currently writing python bindings to an existing C++ library, and
    > I just encountered a problem that I hope has been solved by more
    > experienced python programmers:
    >
    > A C++ class (let's call it CClass) is binded using classical Python
    > extension API to _PClass, which is accesible through python without
    > any problem. The problem is that I want this class to be
    > extended/extensible in python, and expose the python-extended version
    > (PClass) to library users (_PClass should never be used directly nor
    > be retruned by any function).
    > The aim is to leave only performance critical methods in C++ so that
    > the binding work is minimal, and develop the other methods in python
    > so that they are easier to maintain/extend.
    >
    > We thus have something like this
    >
    > class PClass(_PClass):
    > def overide_method(self,...):
    > ...
    > def new_method(self,...):
    > ...
    >
    > and I can define
    > a=PClass()
    > and use my new or overiden method
    > a.overide_method() a.new_method() as intended...
    >
    > So far, so good, trouble begin when I have a method from another class
    > PClass2 derived from _PClass2 which bind the C++ class CClass2, that
    > should return objects of type PClass:
    >
    > Let call this method troublesome_method:
    >
    > b=_PClass2()
    > c=b.troublesome_method()
    > type(c) gives _PClass.
    >
    > now I want to define a python class PClass2 for extending methods of
    > _PClass2 like I have done for _PClass, in particular I want that
    > PClass2 troublesome_method return objects of type PClass instead of
    > _PClass...
    >
    > To this end I try something like this
    >
    > class PClass2(_PClass2):
    > ...
    > def troubelsome_method(self):
    > base=_PClass2.troublesome_method(self)
    > base.__class__=PClass
    >
    > and I have python complaining about TypeError: __class__ assignment:
    > only for heap types...
    >
    > It seems thus that my approach is not possible, but problem is that I
    > can not see another approach that would not involve far more efforts
    > that this one (I have the impression I have to forget about making my
    > PClass derived from _PClass, but instead embed a _PClass instance in
    > PClass, far from ideal cause I would have to reimplement all method,
    > not only those that I want to change :( )
    > This is particularly frustrating cause I also have the impression that
    > want I want to do was at one time possible in python, let say in
    > 2002-2003, when __class__ was already assignable but before assignment
    > was forbidden for non-heap types
    >
    > Any hint on this problem?
    >

    I have just run into the same thing with the builtin list type. The trick
    was not to declare my subclass directly from list, but rather a subclass
    of list. So try this:

    class PClassBase(_PClass):
    pass

    class PClass(PClassBase):
    ...

    class PClass2(PClassBase):
    ...

    Why? I cannot say other than I have noted that Python new-style classes
    and extension types are not quite the same thing. That is, a new-style
    class is a particular kind of extension type.

    Lenard Lindstrom
    <>
    Lenard Lindstrom, Nov 27, 2004
    #2
    1. Advertising

  3. Thanks for your answer, it means I am not the only one having this kind
    of problem...
    > I have just run into the same thing with the builtin list type. The trick
    > was not to declare my subclass directly from list, but rather a subclass
    > of list. So try this:
    >
    > class PClassBase(_PClass):
    > pass
    >
    > class PClass(PClassBase):
    > ...
    >
    > class PClass2(PClassBase):
    > ...
    >

    I think here you mean PClass2(PClassBase2), with a
    PClassBase2(_PClass2): In my example the 2 class do not inherit fro the
    same base class...
    > Why? I cannot say other than I have noted that Python new-style classes
    > and extension types are not quite the same thing. That is, a new-style
    > class is a particular kind of extension type.


    I see how it can help me making PClassBase a PClass (or any type derived
    from PClassBase), but what I fail to see is how this will allow me to
    change a _PClass into one of it's derived types: this is necessary for
    updating methods of _PClass2 that return object of type _PClass (and on
    this I do not have much control, both _PClass and _PClass2 and all of
    their methods are implemented in C++). To updates theses methods for the
    pure python derived class PClass2, I have to make them return the python
    class PClass instead of the original _PClass, which means I still have
    to transform a _PClass object into its derived type PClassBase or
    PClass, and at this time things go pear-shaped :-(....or is there
    something I miss?



    Greg.
    gregory lielens, Nov 27, 2004
    #3
  4. gregory lielens <> writes:

    > Thanks for your answer, it means I am not the only one having this
    > kind of problem...
    > > I have just run into the same thing with the builtin list type. The trick
    > > was not to declare my subclass directly from list, but rather a subclass
    > > of list. So try this:
    > > class PClassBase(_PClass):
    > > pass
    > > class PClass(PClassBase):
    > > ...
    > > class PClass2(PClassBase):
    > > ...
    > >

    > I think here you mean PClass2(PClassBase2), with a
    > PClassBase2(_PClass2): In my example the 2 class do not inherit fro
    > the same base class...
    > > Why? I cannot say other than I have noted that Python new-style classes
    > > and extension types are not quite the same thing. That is, a new-style
    > > class is a particular kind of extension type.

    >
    > I see how it can help me making PClassBase a PClass (or any type
    > derived from PClassBase), but what I fail to see is how this will
    > allow me to change a _PClass into one of it's derived types: this is
    > necessary for updating methods of _PClass2 that return object of type
    > _PClass (and on this I do not have much control, both _PClass and
    > _PClass2 and all of their methods are implemented in C++). To updates
    > theses methods for the pure python derived class PClass2, I have to
    > make them return the python class PClass instead of the original
    > _PClass, which means I still have to transform a _PClass object into
    > its derived type PClassBase or PClass, and at this time things go
    > pear-shaped :-(....or is there something I miss?
    >

    Sorry for the confusion. This looks like a situation best handled with
    embedding rather than inheritance. The PClass and PClass2 wrapper classes
    can share whatever relationship _PClass and _PClass2 share.

    class PClass:
    def __init__(inst=None):
    if inst is None:
    inst = _PClass()
    self._inst = inst
    def override_method(self, ...):
    self._inst.override_method(...)
    def new_method(self, ...):
    ...

    class PClass2:
    self._inst = _PClass2()
    def troublesome_method(self):
    base = PClass(self._inst.troublesome_method())
    ...
    return base

    Alternatively you could provide a way to pass PClass back to the extension module
    for _PClass2.troublesome_method to use instead of _PClass when creating a return
    object. But the embedding approach seems more appropriate when _PClass is not to
    be used directly.

    Lenard Lindstrom
    <>
    Lenard Lindstrom, Nov 28, 2004
    #4

  5. > Sorry for the confusion. This looks like a situation best handled with
    > embedding rather than inheritance. The PClass and PClass2 wrapper classes
    > can share whatever relationship _PClass and _PClass2 share.
    >
    > class PClass:
    > def __init__(inst=None):
    > if inst is None:
    > inst = _PClass()
    > self._inst = inst
    > def override_method(self, ...):
    > self._inst.override_method(...)
    > def new_method(self, ...):
    > ...
    >
    > class PClass2:
    > self._inst = _PClass2()
    > def troublesome_method(self):
    > base = PClass(self._inst.troublesome_method())
    > ...
    > return base


    Yes I was affraid this would be the conclusion: embedding instead of
    inheritance...But this means that not only the method that need to be
    modified need to be rewritten in python, all the other ones also just to
    delegate to the embedded instance...
    This is not too practical, even if such writing can be more or less
    automatized...
    For my purpose it seems that there was a big step backward between
    python 2.2 and 2.3, and I am surprised this has been mostly
    unnoticed...This problem should be common for many people trying to
    write mixed C/C++ / python classes. I found some reference on this in
    the list archive, a few people beeing anoyed by it, but the proposition
    to make the test about assignment to class less strict was not retained
    it seems...I plan to experiment a little with the python C
    implementation to see what happen if I bypass this test...

    > Alternatively you could provide a way to pass PClass back to the extension module
    > for _PClass2.troublesome_method to use instead of _PClass when creating a return
    > object. But the embedding approach seems more appropriate when _PClass is not to
    > be used directly.


    Yes but again this is not so practical, the purpose of the _PClass /
    PClass implementation was to make the bindings as light, and as easy to
    implement / extend as possible, and both the embedding and the creation
    of PClass within _PClass2 methods defeat this purpose...

    I think allowing assignment to class for native types should be done if
    possible, for example if the new class is a derived type (possibly using
    __slots__)... I saw a post on this by Guido, but the reason why it was
    not possible is not clear to me and the thread died without further
    explanations...Maybe my experiments will show me what happen, or someone
    more knowledgeable than me can explain why this is not possible?

    Greg.
    gregory lielens, Nov 28, 2004
    #5
  6. gregory lielens

    David Bolen Guest

    gregory lielens <> writes:

    (...)
    > Yes I was affraid this would be the conclusion: embedding instead of
    > inheritance...But this means that not only the method that need to be
    > modified need to be rewritten in python, all the other ones also just
    > to delegate to the embedded instance...
    > This is not too practical, even if such writing can be more or less
    > automatized...


    Unless I'm misunderstanding, couldn't one of __getattr__ or
    __getattribute__ make mapping other methods you don't override very
    simple and practical, not to mention fully automated with 2-3 lines of
    code? In particular, __getattr__ would seem good for your use since
    it is only called for attributes that couldn't already be located.

    I've had code that wrapped underlying objects in a number of cases, and
    always found that to be a pretty robust mechanism.

    -- David
    David Bolen, Nov 30, 2004
    #6

  7. > Unless I'm misunderstanding, couldn't one of __getattr__ or
    > __getattribute__ make mapping other methods you don't override very
    > simple and practical, not to mention fully automated with 2-3 lines of
    > code? In particular, __getattr__ would seem good for your use since
    > it is only called for attributes that couldn't already be located.
    >
    > I've had code that wrapped underlying objects in a number of cases, and
    > always found that to be a pretty robust mechanism.


    Thanks for mentioning this, after more research I came up with something
    usable, using delegating technique from python cookbook:


    #wrapper

    def wrap_class(base):
    class wrapper:
    __base__=base
    def __init__(self,*args,**kwargs):
    if len(args)==1 and type(args[0])==self.__base__ and kwargs=={}:
    self._base = args[0]
    else:
    self._base=self.__class__.__base__.__new__(self.__class__.__base__,*args,**kwargs)
    def __getattr__(self,s):
    return self._base.__getattribute__(s)
    return wrapper



    #wrap class
    complex2=wrap_class(complex)
    #extend wrapper class
    def supaprint(self):
    print "printed with supaprint(tm):"
    print self
    complex2.supaprint=supaprint

    #define wrapper class from base class
    c1=1+1j
    c2=complex2(c1)
    #test old functionalities
    print c1==c2
    print "c1=",c1
    print "c2=",c1
    #test extension
    c2.supaprint()
    c1.supaprint()

    So this basically fit the bill, it even delegates special methods it
    seems, although I am not sure why...It is like writting heritage ourself,
    in a way :)

    Remaning problem is that if we use the class generating
    wrapper function (simple), we loose the classic class definition syntax
    and rely on explicitely adding methods.
    The alternative is to include the wrapper
    machinery in every "home-derived" class, but you are right, it is not as bad as I
    though :)

    The biggest point I am not sure now is performance: Isn't a big penalty
    associated to this embedding, compared to derivation? Python performance
    is not so critical in our application, but I would be uncomfortable having
    a factor 10 penalty in methd calling associated to this approach...For now, this will be used for
    testing new implementations/extensions, that will be added to the C++
    written binding afterwards.

    I'd like to thanks the list for the hints, I will post the results of my
    experimatations relaxing the assigment to __class__ test if they are
    interesting.

    Greg.
    Gregory Lielens, Dec 2, 2004
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Jp Calderone
    Replies:
    1
    Views:
    512
    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=
    Jul 6, 2003
  2. Barry Kelly
    Replies:
    5
    Views:
    570
    Barry Kelly
    Jun 21, 2006
  3. Replies:
    1
    Views:
    400
  4. Ivan Yurchenko
    Replies:
    1
    Views:
    322
    Steven D'Aprano
    Feb 10, 2013
  5. Steven D'Aprano
    Replies:
    10
    Views:
    125
    Gregory Ewing
    Dec 18, 2013
Loading...

Share This Page