Decorator metaclass

Discussion in 'Python' started by thomas.karolski@googlemail.com, May 23, 2008.

  1. Guest

    Hi,
    I would like to create a Decorator metaclass, which automatically
    turns a class which inherits from the "Decorator" type into a
    decorator.
    A decorator in this case, is simply a class which has all of its
    decorator implementation inside a decorator() method. Every other
    attribute access is being proxied to decorator().getParent().

    Here's my attempt:
    -------------------------------------------------------
    from new import classobj

    class DecoratorType(type):
    def __new__(cls, name, bases, dct):
    dct2 = {}
    for key, val in dct.iteritems():
    dct2[key] = val

    # create a new class which will store all of the implementation
    impl = classobj('%sImpl'%name,(),dct2)

    # update the old class to implement this implementation
    def __init__(self, *args, **dargs):
    object.__setattr__(self, '__impl', impl(*args, **dargs))
    def decorator(self):
    return object.__getattribute__(self,'__impl')
    def __getattribute__(self, attr):
    if attr=="decorator":
    return object.__getattribute__(self,'decorator')
    return getattr(object.__getattribute__(self, 'decorator')
    ().getParent(), attr)
    dct = {}
    dct['__init__'] = __init__
    dct['decorator'] = decorator
    dct['__getattribute__'] = __getattribute__

    return type.__new__(cls, name, bases, dct)

    class Decorator(object):
    __metaclass__ = DecoratorType

    class HBar(Decorator):
    def __init__(self, number):
    Decorator.__init__(self)
    self._number = number
    def inc(self):
    self._number += 1
    def p(self):
    print self._number

    hbar = HBar(10)
    for each in dir(hbar.decorator()):
    print each

    hbar.decorator().p()
    hbar.decorator().inc()
    hbar.decorator().p()
    -------------------------------------------------------
    Unfortunately this does not work. The newly defined __init__ method
    inside __new__, does a call to impl(*args, **dargs). However, since
    the HBar.__init__ calls the Decorator.__init__ method, but the
    HBar.__init__ method no longer resides inside HBar, but rather inside
    HBarImpl (which is no longer a subtype of Decorator), the compiler
    complains that Decorator.__init__ is not being called with a Decorator
    instance as its first argument (which is true).
    I tried changing the definition of impl inside __new__ to have
    Decorator as one of its bases, but then for some reason impl(*args,
    **dargs) asks for 4 arguments (just like __new__) and I have no clue
    as to why that happens.

    Any help on this?

    Regards,
    Thomas K.
    , May 23, 2008
    #1
    1. Advertising

  2. Le Friday 23 May 2008 04:28:22 , vous avez
    écrit :
    > Hi,
    > I would like to create a Decorator metaclass, which automatically
    > turns a class which inherits from the "Decorator" type into a
    > decorator.
    > A decorator in this case, is simply a class which has all of its
    > decorator implementation inside a decorator() method. Every other
    > attribute access is being proxied to decorator().getParent().
    >
    > ...
    >
    > -------------------------------------------------------
    > Unfortunately this does not work. The newly defined __init__ method
    > inside __new__, does a call to impl(*args, **dargs). However, since
    > the HBar.__init__ calls the Decorator.__init__ method, but the
    > HBar.__init__ method no longer resides inside HBar, but rather inside
    > HBarImpl (which is no longer a subtype of Decorator), the compiler
    > complains that Decorator.__init__ is not being called with a Decorator
    > instance as its first argument (which is true).
    > I tried changing the definition of impl inside __new__ to have
    > Decorator as one of its bases, but then for some reason impl(*args,
    > **dargs) asks for 4 arguments (just like __new__) and I have no clue
    > as to why that happens.
    >
    > Any help on this?
    >


    The problem with kind of design is that you must break the rules of class
    inheritance, and it seems like a strange idea to implement decorators by
    inheritance.

    Of course you could do all sort of magic with python, but what is your goal ?
    In your example, should the implementation types inherit from each other ?
    In that case, do you want to preserve the same semantic for __init__ as in
    standard python class (this could be a hard job) ?

    This quick fix seems to work with your example, but add extra magic to
    automatically call the super __init__ of the parent implementation, this
    could be a bad idea, use with caution ! (I still think it's a bad design,
    using composition and proxy classes is much more simple and clear)

    class DecoratorType(type):
    def __new__(cls, name, bases, dct):

    # create a new class which will store all of the
    implementation
    parent_impl_type = bases[0] is object and object \
    or bases[0]._impl_type
    impl = type('%sImpl'%name,(parent_impl_type,),dict(dct))
    dectype = type.__new__(cls, name, bases, {'_impl_type' :
    impl })

    # update the old class to implement this implementation
    def __init__(self, *args, **dargs):
    print args, dargs
    new_impl = impl(*args, **dargs)
    super(dectype._impl_type, new_impl).__init__(*args,
    **dargs)
    object.__setattr__(self, '_impl', new_impl)
    def decorator(self):
    return object.__getattribute__(self,'_impl')
    def __getattribute__(self, attr):
    if attr=="decorator":
    return
    object.__getattribute__(self,'decorator')
    return getattr(object.__getattribute__(
    self, 'decorator')(), attr)
    dectype.__init__ = __init__
    dectype.decorator = decorator
    dectype.__getattribute__ = __getattribute__

    return dectype

    class Decorator(object):

    __metaclass__ = DecoratorType

    class HBar(Decorator):
    def __init__(self, number):
    print 'hb:', number
    self._number = number
    def inc(self):
    self._number += 1
    def p(self):
    print self._number

    class HBar2(HBar) :
    def __init__(self, number):
    print 'hb2:', number
    self._hb2 = number
    def inc2(self):
    self._hb2 += 1
    def p2(self):
    print self._hb2


    hbar = HBar(10)
    for each in dir(hbar.decorator()):
    print each

    hbar.decorator().p()
    hbar.decorator().inc()
    hbar.decorator().p()

    hb2 = HBar2(5)
    hb2.p()
    hb2.p2()
    hb2.inc()
    hb2.p()
    hb2.p2()
    hb2.inc2()
    hb2.p()
    hb2.p2()



    --
    _____________

    Maric Michaud
    _____________
    Maric Michaud, May 23, 2008
    #2
    1. Advertising

  3. Carl Banks Guest

    On May 22, 10:28 pm, wrote:
    > Hi,
    > I would like to create a Decorator metaclass, which automatically
    > turns a class which inherits from the "Decorator" type into a
    > decorator.
    > A decorator in this case, is simply a class which has all of its
    > decorator implementation inside a decorator() method. Every other
    > attribute access is being proxied to decorator().getParent().
    >
    > Here's my attempt:


    You got deep stuff going on there, chief, and some of it's wrong.
    I'll try to point it out.

    > -------------------------------------------------------
    > from new import classobj
    >
    > class DecoratorType(type):
    > def __new__(cls, name, bases, dct):
    > dct2 = {}
    > for key, val in dct.iteritems():
    > dct2[key] = val


    First of all, you can just do dct2 = dct.copy().
    Second, since you never use dct again, even copying it is unnecessary.


    > # create a new class which will store all of the implementation
    > impl = classobj('%sImpl'%name,(),dct2)


    classobj creates an old-style class, and I'm pretty sure you don't
    want to do that. To create a new-style class, use type:

    impl = type('%sImpl'%name,(),dct)


    > # update the old class to implement this implementation
    > def __init__(self, *args, **dargs):
    > object.__setattr__(self, '__impl', impl(*args, **dargs))


    As your code stands now, object.__setattr__ isn't necessary here; just
    using

    self.__impl = impl(*args,**dargs)

    should work fine. I'm guessing you intend to override __setattr__
    later?

    If you do use object.__setattr__, I suggest that you might want to
    call the superclass's __setattr__ instead of object's. I imagine in
    this case the superclass will rarely want to override __setattr__
    itself, but in general it's a good idea. In this particular
    circumstance, we don't yet have the class object (it doesn't come till
    after calling type.__new__) but we do have the parent class. So you
    might consider changing the definition of __init__ to this:

    basecls = bases[0] if bases else object
    def __init__(self, *args, **dargs):
    basecls.__setattr__(self, '__impl', impl(*args, **dargs))

    > def decorator(self):
    > return object.__getattribute__(self,'__impl')


    Again, consider changing it to

    def decorator(self):
    return basecls.__getattribute(self,'__impl')

    > def __getattribute__(self, attr):
    > if attr=="decorator":
    > return object.__getattribute__(self,'decorator')
    > return getattr(object.__getattribute__(self, 'decorator')
    > ().getParent(), attr)
    > dct = {}
    > dct['__init__'] = __init__
    > dct['decorator'] = decorator
    > dct['__getattribute__'] = __getattribute__
    >
    > return type.__new__(cls, name, bases, dct)
    >
    > class Decorator(object):
    > __metaclass__ = DecoratorType


    Parenthetical: I don't normally recommend this style, since it
    obscures the fact that you're using a custom metaclass to the user.
    That is something the user probably would benefit from knowing, if for
    no other reason than so they can make a mental note about where to
    look first if something goes wrong. I prefer to make the user use the
    __metaclass__ attribute.

    However, I could see it being desirable for some cases where you're
    trying to be as transparent as possible, and indeed it looks as if
    that's your goal here.


    > class HBar(Decorator):
    > def __init__(self, number):
    > Decorator.__init__(self)


    Ok, at this point you have to ask yourself what you want to do,
    because the solution you choose will involve trade-offs.

    You will note that Decorator does not define __init__. In fact,
    object.__init__ will be called, which does nothing. If you think that
    all classes with DecoratorType as their metaclass will be a direct
    subclass of Decorator, you can get away with not calling
    Decorator.__init__ at all.

    However, this can cause problems if a user wants to define their own
    base class with an __init__ that does something (either by using the
    __metaclass__ attribute, or by subclassing a Decorator subclass). In
    that case, you will have to make arrangements to pass the decorator
    object to the superclass instead of the decorated. This can be pretty
    hairy, and it beyond the scope of this reply.

    To do it completely transparently, your decorated class will probably
    have to maintain a reference to its decorator, and will also have to
    derive from a base class that delegates any method calls to the
    superclass of the decorator. (Phew.)

    That won't be as easy as it sounds.


    > self._number = number
    > def inc(self):
    > self._number += 1
    > def p(self):
    > print self._number
    >
    > hbar = HBar(10)
    > for each in dir(hbar.decorator()):
    > print each
    >
    > hbar.decorator().p()
    > hbar.decorator().inc()
    > hbar.decorator().p()
    > -------------------------------------------------------
    > Unfortunately this does not work. The newly defined __init__ method
    > inside __new__, does a call to impl(*args, **dargs). However, since
    > the HBar.__init__ calls the Decorator.__init__ method, but the
    > HBar.__init__ method no longer resides inside HBar, but rather inside
    > HBarImpl (which is no longer a subtype of Decorator), the compiler
    > complains that Decorator.__init__ is not being called with a Decorator
    > instance as its first argument (which is true).
    > I tried changing the definition of impl inside __new__ to have
    > Decorator as one of its bases, but then for some reason impl(*args,
    > **dargs) asks for 4 arguments (just like __new__) and I have no clue
    > as to why that happens.


    I believe it's happening because you mixed old-style and new-style
    classes. But it's not the right solution anyway.

    > Any help on this?


    Probably the best piece of advice is "Don't try to use Decorator
    pattern". :)

    Seriously, you might want to see what other people have done in
    similar cases. This stuff is tricky to get right, so maybe you should
    shamelessly ride the coattails of someone who already ran into all the
    issues. One example I can think of is the ZODB Persistent class (it's
    a proxy class, so some of the same issues are involved). Perhaps
    searching Python cookbook for some proxy or decorator class recipes
    will give you ideas.


    Carl Banks
    Carl Banks, May 23, 2008
    #3
  4. Thanks for pointing out all those mistakes. I think I'm already starting
    to grasp all of the python magic going on in there.

    > Parenthetical: I don't normally recommend this style, since it
    > obscures the fact that you're using a custom metaclass to the user.
    > That is something the user probably would benefit from knowing, if for
    > no other reason than so they can make a mental note about where to
    > look first if something goes wrong. I prefer to make the user use the
    > __metaclass__ attribute.


    Really just personal preference I think. I'm not really a friend of
    declaring variables if there is a more "intuitive" way.

    >> class HBar(Decorator):
    >> def __init__(self, number):
    >> Decorator.__init__(self)

    >
    > Ok, at this point you have to ask yourself what you want to do,
    > because the solution you choose will involve trade-offs.


    Yes, it was probably a bad example. I decided not to call the Decorator
    's __init__ method in my new version (which I have posted as a reply to
    the reply of Maric Michaud).

    > You will note that Decorator does not define __init__. In fact,
    > object.__init__ will be called, which does nothing. If you think that
    > all classes with DecoratorType as their metaclass will be a direct
    > subclass of Decorator, you can get away with not calling
    > Decorator.__init__ at all.


    Now, inside my new version, I have a class which inherits from both
    Decorator and Window, out of which the __init__ for Decorator is not
    called. Does this prove to be a problem?

    > Probably the best piece of advice is "Don't try to use Decorator
    > pattern". :)


    Well, I decided on the decorator pattern, because I want to be able to
    change the behavior of classes during run-time. I did not really find
    any other pattern which would let me do that.

    Regards,
    Thomas K.
    Thomas Karolski, May 23, 2008
    #4
  5. Carl Banks Guest

    On May 23, 11:42 am, Thomas Karolski <>
    wrote:

    > > You will note that Decorator does not define __init__. In fact,
    > > object.__init__ will be called, which does nothing. If you think that
    > > all classes with DecoratorType as their metaclass will be a direct
    > > subclass of Decorator, you can get away with not calling
    > > Decorator.__init__ at all.

    >
    > Now, inside my new version, I have a class which inherits from both
    > Decorator and Window, out of which the __init__ for Decorator is not
    > called. Does this prove to be a problem?


    It sounds like it could create infinite recursion. If your decorated
    class inherits from Decorator, it should also get the DecorateType
    metaclass, which should recursively try to create its own decorated
    class, ad infinitum. Unless I misunderstand what you meant.


    > > Probably the best piece of advice is "Don't try to use Decorator
    > > pattern". :)

    >
    > Well, I decided on the decorator pattern, because I want to be able to
    > change the behavior of classes during run-time. I did not really find
    > any other pattern which would let me do that.



    Ah.

    Well it is possible to do that in Python. though there's probably not
    an official design pattern for it (but then design patterns grew up
    around less flexible languages, partially as a way to cope with their
    lack of flexibility). Here are a couple things to think about:

    If you'd like to change the behavior of individual instances of a
    class, you can assign functions to individual instances which act just
    like methods. (Warning: this does not work with operators.)

    Here is an example:

    class A(object):
    def say_hello(self):
    print "hello, world"

    a = A()

    def custom_hello():
    print "hello, my name is Inigo Montoya"

    a.say_hello = custom_hello


    If you'd like to change the behavior of all instances of the class,
    then you can assign a new method directly to the class after it was
    created:

    def new_default_hello(self):
    print "hello, children"

    A.say_hello = new_default_hello


    Notice that you need to use self when assigning it to the class
    object, and not to use self when assigning it to an instance of the
    class.


    Carl Banks
    Carl Banks, May 23, 2008
    #5
  6. Turns out the first msg I sent did not reach the list, so I'll just post
    what I've achieved by now:
    ------------------------------------------
    class DecoratorDummy(object): pass

    class InheritedDecoratorType(type):
    def __new__(cls, name, bases, dct):
    # return if its a class which inherited from Decorator
    if Decorator in bases:
    return type.__new__(cls, name, bases, {})

    # if its a class which did not directly inherit from Decorator,
    # then it inherited from a class which has been manipulated using the
    # Decorator class.
    # in that case we change the bases of the inheriting class.
    # We'll split the manipulated class into Decorator and its implementation
    for b in bases:
    if type(b) is InheritedDecoratorType:
    break
    newbases = [x for x in bases]
    # remove the manipulated base class
    newbases.remove(b)
    # and add the impl of the manipulated base class
    newbases.append(b._impl_type)
    # and add the Decorator class
    newbases.append(Decorator)
    # now we'll have to make sure the dict of the new class shows the
    original base class
    # (which has been removed) as the implementation class
    dct[b.__name__] = b._impl_type
    # since we have added a Decorator class, we ought to get rid of it
    # through the DecoratorType metaclass
    r = DecoratorType.__new__(cls, name, tuple(newbases), dct)

    return r

    class DecoratorType(type):
    def __new__(cls, name, bases, dct):

    # if the first class is DecoratorDummy, then we're handling the
    # Decorator class, which is not supposed to be modified
    if bases[0] is DecoratorDummy:
    return type.__new__(cls, name, bases, {})

    # if one of the bases is the Decorator class
    b = [x for x in bases]
    if Decorator in b:
    # create a new impl type which inherits from every but the decorator
    class
    b.remove(Decorator)

    impl_type = type('%sImpl'%name, tuple(b), dict(dct))
    # make the returned type no longer a DecoratorType, but rather a
    normal type
    # Types which inherit from a class which inherited from Decorator,
    will thus
    # *not* be put through this metaclass.
    #dectype = type.__new__(type, name, tuple(b), {'_impl_type' : impl_type })
    dectype = type.__new__(InheritedDecoratorType, name, tuple(b),
    {'_impl_type' : impl_type })

    # update the old class to implement this implementation
    def __init__(self, *args, **dargs):
    new_impl = impl_type(*args, **dargs)
    super(dectype._impl_type, new_impl).__init__(*args,
    **dargs)
    object.__setattr__(self, '_impl', new_impl)
    def decorator(self):
    return object.__getattribute__(self, '_impl')
    def __getattribute__(self, attr):
    if attr=="decorator":
    return object.__getattribute__(self, 'decorator')

    # if we have a specified method inside the decorator().__decorate__ var,
    # then call decorator().attr(), otherwise proxy the call
    d = object.__getattribute__(self, 'decorator')()
    if attr in d.__decorate__:
    return getattr(d, attr)
    return getattr(d.getParent(), attr)
    dectype.__init__ = __init__
    dectype.decorator = decorator
    dectype.__getattribute__ = __getattribute__

    return dectype

    class Decorator(DecoratorDummy):
    __metaclass__ = DecoratorType

    class Window(object):
    def __init__(self, parent):
    print "Window::__init__(%s)"%self
    self._parent = parent

    def setParent(self, parent):
    self._parent = parent

    def getParent(self):
    return self._parent

    class Textarea(Window):
    def draw(self):
    print "Textarea::draw()"

    class HBar(Decorator, Window):
    __decorate__ = ["draw"]
    def __init__(self, parent):
    print "HBar::__init__(%s)"%self
    Window.__init__(self, parent=parent)

    self._progress = 0.0

    def setProgress(self, p):
    print "setting progress to %s"%p
    self._progress= p

    def getProgress(self):
    return self._progress

    def draw(self):
    self.getParent().draw()
    print "HBar::draw()"

    class HBarTrue(HBar):
    # HBar's bases: Window (Decorator removed by metaclass)
    # HBar's methods: __init__, decorator, __getattribute__
    # everything else is inside decorator()
    # we thus need to make calls to self within HBarTrue,
    # calls to self.decorator() - this is not possible
    # Otherwise we could also let HBarTrue inherit from HBarImpl,
    # however then a call to HBar.__init__ would be invalid inside
    # HBarTrue.__init__ unless we specify HBar internally as being HBarImpl
    # - this however is not possible
    # if we inherit from HBarImpl, then we no longer have the decorator
    # functionality. We'd thus have to include Decorator in the list of bases

    # Inherit normally from HBar and not Decorator.
    # Move all methods from HBarTrue to HBar.decorator()
    # create a custom __init__ method which calls HBar.decorator().__init__

    def __init__(self, parent):
    print "HBarTrue::__init__(%s)"%self
    for each in dir(self):
    print each
    HBar.__init__(self, parent)

    self.setProgress(0.0)

    myTextarea = Textarea("main.parent")
    myTextarea.draw()
    myTextarea = HBarTrue(myTextarea)
    myTextarea.draw()
    myTextarea.decorator().setProgress(100.0)
    ------------------------------------------
    The code above works only if I don't inherit from HBar like I did.
    As it is now, the HBar class, which inherits from Decorator and Window,
    is going to be manipulated by the metaclass. During this process the
    Decorator base-class is removed, the implementation of HBar moved into
    _impl_type and the methods __init__, decorator and __getattribute__ are
    being assigned to the new class.
    Everything works so far if I don't subclass HBar.
    If I do subclass HBar, then the subclass goes through the second
    metaclass (InheritedDecoratorType). In there I replace the base class
    HBar with HBarImpl and Decorator. This way the subclass inherits the
    implementation of HBar and through the Decorator class then the subclass
    goes through the same metaclass magic as HBar in the first step. Of
    course however, since HBarTrue is no longer a subclass of HBar (since
    that baseclass has been removed), the HBar.__init__ method won't accept
    the non HBar self parameter.

    Now the reason why I'm using decorators, is because I want to be ably to
    add the functionality dynamicly - without the need of construction
    classes for the different possibilities. Composite pattern does not help
    in this case, since I lose the ability to choose in what order I call
    the decorator's and the decorated's methods.

    I'll just keep trying. Any input greatly appreciated.

    Regards,
    Thomas K.
    Thomas Karolski, May 23, 2008
    #6
  7. En Fri, 23 May 2008 16:25:19 -0300, Thomas Karolski <> escribió:

    > Turns out the first msg I sent did not reach the list, so I'll just post
    > what I've achieved by now:


    [snip a couple of long metaclasses]

    > Now the reason why I'm using decorators, is because I want to be ably to
    > add the functionality dynamicly - without the need of construction
    > classes for the different possibilities. Composite pattern does not help
    > in this case, since I lose the ability to choose in what order I call
    > the decorator's and the decorated's methods.


    I haven't looked in detail to your code, but if that is what you want, you don't need a Decorator metaclass at all. Just add/replace the desired methods inside the class itself (Python is a dynamic language, remember).

    py> class A(object):
    .... def foo(self): print "In foo, self=", self
    ....
    py> def bar(self): print "In bar, self=", self
    ....
    py> a = A()
    py> a.foo()
    In foo, self= <__main__.A object at 0x00A3CD50>
    py>
    py> A.bar = bar
    py> a.bar
    <bound method A.bar of <__main__.A object at 0x00A3CD50>>
    py> a.bar()
    In bar, self= <__main__.A object at 0x00A3CD50>

    You can even add methods to individual instances:

    py> def baz(self): print "In baz, self=", self
    ....
    py> import new
    py> b = A()
    py> b.baz = new.instancemethod(baz, b, type(b))
    py> a.baz()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    AttributeError: 'A' object has no attribute 'baz'
    py> b.baz
    <bound method A.baz of <__main__.A object at 0x00A3CDD0>>
    py> b.baz()
    In baz, self= <__main__.A object at 0x00A3CDD0>

    --
    Gabriel Genellina
    Gabriel Genellina, May 25, 2008
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gustavo Niemeyer

    Metaclass discussons

    Gustavo Niemeyer, Aug 22, 2003, in forum: Python
    Replies:
    5
    Views:
    350
    Gustavo Niemeyer
    Aug 25, 2003
  2. Replies:
    9
    Views:
    356
    David Mertz
    Sep 10, 2003
  3. ironfroggy
    Replies:
    16
    Views:
    428
    Michele Simionato
    Jun 3, 2005
  4. glomde
    Replies:
    5
    Views:
    515
    glomde
    Mar 29, 2007
  5. Steven D'Aprano

    Metaclass of a metaclass

    Steven D'Aprano, Jun 5, 2012, in forum: Python
    Replies:
    1
    Views:
    298
Loading...

Share This Page