Using metaclasses to play with decorators.

Discussion in 'Python' started by Michael Sparks, Jun 15, 2004.

  1. [ I'm not really sure of the etiquette of the python-dev list, so I think
    I'll play "safe" and post this thought here... I know some of the
    developers do look here, and Guido's comment (before europython) on the
    dev list saying he's not interested in new syntaxes makes me think this
    is a better place for it... ]

    Anyway...

    At Europython Guido discussed with everyone the outstanding issue with
    decorators and there was a clear majority in favour of having them, which
    was good. From where I was sitting it looked like about 20:20 split on the
    following syntaxes:
    1 def func(arg1, arg2, arg3) [dec,dec,dec]:
    function...
    2 [dec,dec,dec] def func(arg1, arg2, arg3):
    function...

    When it came to the decision on whether 2.4 includin one of these two
    options or a waiting for a third unspecified, indeterminate syntax in a
    later version, it looked like around a 60:40 vote in favour of waiting.
    (ie 60:20:20 for waiting/syntax 1/syntax 2)

    Also, suggested use cases for decorators were:
    * Changing function type from method to staticmethod, classmethod
    * Adding metadata:
    * Author
    * Version
    * Deprecation
    * Processing rules - eg grammar rules
    * Framework annotations
    * Support for external bindings

    Some of these are clearly transformations of the methods, some are
    annotations - ie addition of attributes after creation.

    Transformations of methods can already be done using metaclasses.

    Attributes can already appended to methods, but these need to be done
    before transformations, and have to be done after the function has been
    created:

    class foo(object):
    def bar(hello):
    "This is"
    print hello
    bar.__RULE__ = "expr := expr"
    bar.__doc__ += " a test"
    bar.__author__ = "John"
    bar = staticmethod(bar)

    Suppose for a moment we wanted to do the same thing for the class foo - ie
    we had some attributes we wanted to add to it, and after creating it we
    wanted to transform it - what might we write?

    class foo(object):
    __RULE__ = "expr := expr"
    __doc__ += " a test"
    __author__ = "John"
    def bar(hello):
    "This is"
    print hello

    foo = someclasstransform(foo)

    That's a lot nicer. However if we treat foo as a class description, then
    someclasstransform tends to look similar to metaclass shenanigans.
    Does that mean we can use metaclasses to similate a syntax for things
    like staticmethods ?

    My opinion here is yes - it's relatively trivial to implement if you use a
    naming
    scheme for methods:
    class decorated_class_one(type):
    def __new__(cls, name, bases, dct):
    for key in dct.keys():
    if key[:12] == "classmethod_":
    dct[ key[12:] ] = classmethod(dct[key])
    del dct[key]
    if key[:7] == "static_":
    dct[ key[7:] ] = staticmethod(dct[key])
    del dct[key]
    return type.__new__(cls, name, bases, dct)

    class myclass(object):
    __metaclass__ = decorated_class_one
    def static_foo(arg1, arg2, arg3):
    print "Hello",arg1, arg2, arg3
    def classmethod_bar(cls,arg1, arg2, arg3):
    print "World",cls, arg1, arg2, arg3
    def baz(self,arg1, arg2, arg3):
    print "There",self, arg1, arg2, arg3

    The question then becomes what's the cleanest way of adding attributes
    using this approach?

    Since we've got a syntax that's similar to classes, you might argue one
    approach might be:
    class myclass(object):
    __metaclass__ = decorated_class
    def static_foo(arg1, arg2, arg3):
    __doc__ = "This is a static method"
    __author__ = "Tom"
    __deprecated__ = True
    print "Hello",arg1, arg2, arg3
    def classmethod_bar(cls,arg1, arg2, arg3):
    __doc__ = "This is a class method"
    __author__ = "Dick"
    __deprecated__ = False
    print "World",cls, arg1, arg2, arg3
    def baz(self,arg1, arg2, arg3):
    __doc__ = "This is a normal method"
    __author__ = "Harry"
    __deprecated__ = False
    print "There",self, arg1, arg2, arg3

    Whilst we can get at the variable names that these declare, we can't get
    at the values. What else can we do?

    We could choose a similar "meta" keyword - say "decorators_", and pass a
    dictionary instead?

    class myclass(object):
    __metaclass__ = decorated_class_two
    decorators_static_foo = {
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    def static_foo(arg1, arg2, arg3):
    print "Hello",arg1, arg2, arg3
    decorators_classmethod_bar = {
    '__doc__' : "This is a class method",
    '__author__' : "Dick",
    '__deprecated__' : False
    }
    def classmethod_bar(cls,arg1, arg2, arg3):
    print "World",cls, arg1, arg2, arg3
    decorators_baz = {
    '__doc__' : "This is a normal method",
    '__author__' : "Harry",
    '__deprecated__' : False
    }
    def baz(self,arg1, arg2, arg3):
    print "There",self, arg1, arg2, arg3

    Whilst it's not perfect, it's something we can actually use. Bear in mind
    we have to add attributes/decorators before we do transforms:

    class decorated_class_two(type):
    def __new__(cls, name, bases, dct):
    for key in dct.keys():
    if key[:11] == "decorators_":
    for attr_key in dct[key].keys():
    exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
    del dct[key]
    for key in dct.keys():
    if key[:12] == "classmethod_":
    dct[ key[12:] ] = classmethod(dct[key])
    del dct[key]
    if key[:7] == "static_":
    dct[ key[7:] ] = staticmethod(dct[key])
    del dct[key]
    return type.__new__(cls, name, bases, dct)

    Can we go one better? Can we make the following work?
    class myclass(object):
    __metaclass__ = decorated_class_three
    decorators_foo = {
    'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    def foo(arg1, arg2, arg3):
    print "Hello",arg1, arg2, arg3
    decorators_bar = {
    'transforms' : [classmethod],
    '__doc__' : "This is a class method",
    '__author__' : "Dick",
    '__deprecated__' : False
    }
    def bar(cls,arg1, arg2, arg3):
    print "World",cls, arg1, arg2, arg3
    decorators_baz = {
    '__doc__' : "This is a normal method",
    '__author__' : "Harry",
    '__deprecated__' : False
    }
    def baz(self,arg1, arg2, arg3):
    print "There",self, arg1, arg2, arg3

    Well, let's try:
    class decorated_class_three(type):
    def __new__(cls, name, bases, dct):
    for key in dct.keys():
    if key[:11] == "decorators_":
    transforms = []
    for attr_key in dct[key].keys():
    if attr_key == 'transforms':
    transforms = dct[key][attr_key]
    continue
    exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
    for transform in transforms:
    dct[key[11:]] = transform(dct[key[11:]])
    del dct[key]
    return type.__new__(cls, name, bases, dct)

    And hey presto! It works!

    The upshot is is that using a very simple metaclass, we can already have
    the functionality that decorators will give us, but the syntax is less
    than ideal:

    class myclass(object):
    __metaclass__ = decorated_class_three
    decorators_foo = {
    'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    def foo(arg1, arg2, arg3):
    print "Hello",arg1, arg2, arg3

    Now let's simplify this to what people currently commonly do:
    class myclass(object):
    def foo(arg1, arg2, arg3):
    "This is a static method"
    print "Hello",arg1, arg2, arg3
    foo = staticmethod(foo)

    In this situation python *does* do something special with the first value
    it finds as the first value inside the method - it uses it as a __doc__
    decorator. To my mind this treating the first value of the code as special
    strikes me as the ideal hook. You could, for example, have the following
    syntax:

    class myclass(object):
    def foo(arg1, arg2, arg3):
    { 'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    print "Hello",arg1, arg2, arg3

    But here's the really cool trick - we can actually use this. Now. Let's
    put our metaclass back, and make a small modification to make this work:
    class myclass(object):
    __metaclass__ = decorated_class_four
    def foo(arg1, arg2, arg3):
    """{ 'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }"""
    print "Hello",arg1, arg2, arg3

    We can then modify our metaclass to make this work:
    class decorated_class_four(type):
    def __new__(cls, name, bases, dct):
    for key in dct.keys():
    doc = dct[key].__doc__
    if doc:
    try:
    transforms = []
    decorators = eval(doc)
    for attr_key in decorators:
    if attr_key == 'transforms':
    transforms = decorators[attr_key]
    continue
    exec 'dct[key].'+attr_key+' = decorators[attr_key]'
    for transform in transforms:
    dct[key] = transform(dct[key])
    except SyntaxError:
    pass # no decorators
    return type.__new__(cls, name, bases, dct)

    So recasting our example using this final syntax, we gain:
    class myclass(object):
    __metaclass__ = decorated_class_four

    def foo(arg1, arg2, arg3):
    """{ 'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }"""
    print "Hello",arg1, arg2, arg3

    def bar(cls,arg1, arg2, arg3):
    """{ 'transforms' : [classmethod],
    '__doc__' : "This is a class method",
    '__author__' : "Dick",
    '__deprecated__' : False
    }"""
    print "World",cls, arg1, arg2, arg3

    def baz(self,arg1, arg2, arg3):
    """{ '__doc__' : "This is a normal method",
    '__author__' : "Harry",
    '__deprecated__' : False
    }"""
    print "There",self, arg1, arg2, arg3

    Each of the transformations chosen takes us between a variety of syntaxes,
    with various advantages/disadvantages. Personally I think the best variety
    here without changing python's syntax or semantics is this one:

    class myclass(object):
    __metaclass__ = decorated_class_three
    decorators_foo = {
    'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    def foo(arg1, arg2, arg3):
    print "Hello",arg1, arg2, arg3
    ....

    Partly the reason for this is because it's very clear what's going on
    here, and also you can use doc strings as normal:
    class myclass(object):
    __metaclass__ = decorated_class_three
    decorators_foo = {
    'transforms' : [staticmethod],
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    def foo(arg1, arg2, arg3):
    "This is a static method"
    print "Hello",arg1, arg2, arg3
    ....

    Whereas with a semantic change to the initial var, I think this form is
    nicer:
    class myclass(object):
    # no metaclass, requires change in semantics
    def foo(arg1, arg2, arg3):
    { 'transforms' : [staticmethod],
    '__doc__' : "This is a static method",
    '__author__' : "Tom",
    '__deprecated__' : True
    }
    print "Hello",arg1, arg2, arg3

    def bar(cls,arg1, arg2, arg3):
    { 'transforms' : [classmethod],
    '__doc__' : "This is a class method",
    '__author__' : "Dick",
    '__deprecated__' : False
    }
    print "World",cls, arg1, arg2, arg3
    ....


    Syntactically this compiles/runs fine right now, it just doesn't have the
    decorator semantics we need for this to work.

    And what would the semantic change be? Currently we have foo.__doc__ .
    This could be foo.__decorator_dict__ . Quite what anyone chooses to do
    with these would be entirely up to them.

    Classes that inherit from object could be defined to do something special
    - such as act in a similar way to the presented metaclass. Whereas for
    objects that aren't derived from object, nothing would change, except a
    small amount of extra information is made available, and *potentially*
    ignored.

    Anyway, for any of the python-dev team who do hang on out python-list I
    hope this has been food for thought, and whilst I've done a cursory check
    of the archives I'm not subbed to python-dev, so apologies if I'm going
    over old ground and raking up old ideas needlessly!

    One thing that does strike me regarding this is this:

    This relies on being able to pull out the first value at the start of the
    function/method definition. This _does_ currently happen anyway, and
    people are using the value there. Making it consistent in that you are
    able to pull out the value, no matter what the type strikes me as a very
    useful thing, and decorators can naturally fall out of it as well.

    As I said, hopefully useful food for thought, and hopefully not hacked
    anyone off at the length of this post!

    Cheers,


    Michael.
    Michael Sparks, Jun 15, 2004
    #1
    1. Advertising

  2. Michael Sparks <> wrote in message news:<>...
    > [ I'm not really sure of the etiquette of the python-dev list, so I think
    > I'll play "safe" and post this thought here... I know some of the
    > developers do look here, and Guido's comment (before europython) on the
    > dev list saying he's not interested in new syntaxes makes me think this
    > is a better place for it... ]
    >
    > Anyway...
    >
    > At Europython Guido discussed with everyone the outstanding issue with
    > decorators and there was a clear majority in favour of having them, which
    > was good. From where I was sitting it looked like about 20:20 split on the
    > following syntaxes:
    > 1 def func(arg1, arg2, arg3) [dec,dec,dec]:
    > function...
    > 2 [dec,dec,dec] def func(arg1, arg2, arg3):
    > function...
    >
    > When it came to the decision on whether 2.4 includin one of these two
    > options or a waiting for a third unspecified, indeterminate syntax in a
    > later version, it looked like around a 60:40 vote in favour of waiting.
    > (ie 60:20:20 for waiting/syntax 1/syntax 2)
    > <snip>


    The problem of metaclasses-based solutions is that they are hacks.
    We don't want everybody to implement his/her version of decorators
    (yes me too I have implemented mine). Decorators MUST be in the core
    language. Waiting will only favor the abuse of metaclasses.
    I do not particularly like each of two syntaxes (even if the second
    one looks better) but also I do not particularly dislike them.
    My preferred solution would be

    def func(...) is dec:
    ....

    *without* the ability to specify multiple decorators (i.e. let compose
    the multiple generators explicitely by hand, with a functional.compose
    function or something like that).

    But I prefer to have a syntax I don't like 100% then waiting for another
    year or so. Why don't put a given syntax in 2.4a, see how it goes, and
    change it in 2.4b if it is the case?

    Michele Simionato
    Michele Simionato, Jun 15, 2004
    #2
    1. Advertising

  3. Michael Sparks

    j_mckitrick Guest

    You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\

    Could someone give me an idea of what metaclasses and decorators _are_?

    I think metaclasses are classes with their base methods overridden, correct?

    And decorators are a form of delegation, right?

    If so, where am I wrong? And how do these techniques make for better code/coding?

    jonathon
    j_mckitrick, Jun 18, 2004
    #3
  4. On 17 Jun 2004 17:42:40 -0700, (j_mckitrick)
    wrote:

    >You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\
    >
    >Could someone give me an idea of what metaclasses and decorators _are_?


    The best I have found on metaclasses is Alex Martelli's explanation
    and example in Python in a Nutshell. There is also extensive
    documentation available at http://python.org/doc/newstyle.html I have
    also written a one-page explanation for engineering students in my
    Python OOP chapter at http://ece.arizona.edu/~edatools/Python

    Decorators are just modifiers you can add to a function definition,
    ususally to modify the behavior, as in

    def func(x,y) [staticmethod]:

    These are not yet in Python. See PEP 318 for more explanationa and
    examples.

    -- Dave


    >I think metaclasses are classes with their base methods overridden, correct?
    >
    >And decorators are a form of delegation, right?
    >
    >If so, where am I wrong? And how do these techniques make for better code/coding?
    >
    >jonathon
    David MacQuigg, Jun 18, 2004
    #4
  5. (j_mckitrick) wrote in message news:<>...
    > You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\
    >
    > Could someone give me an idea of what metaclasses and decorators _are_?
    >
    > I think metaclasses are classes with their base methods overridden, correct?
    >
    > And decorators are a form of delegation, right?
    >
    > If so, where am I wrong? And how do these techniques make for better code/coding?
    >
    > jonathon


    http://www.python.org/cgi-bin/moinmoin/MetaClasses

    for decorators, look at PEP318.
    Michele Simionato, Jun 18, 2004
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Sparks
    Replies:
    6
    Views:
    347
    David MacQuigg
    Jun 18, 2004
  2. Robert Brewer
    Replies:
    9
    Views:
    302
    David MacQuigg
    Jun 26, 2004
  3. Paul Morrow
    Replies:
    2
    Views:
    240
    Larry Bates
    Aug 24, 2004
  4. Anthony Baxter
    Replies:
    3
    Views:
    264
    Jess Austin
    Aug 26, 2004
  5. Paul Morrow
    Replies:
    83
    Views:
    1,208
    Hallvard B Furuseth
    Sep 6, 2004
Loading...

Share This Page