Using metaclasses to play with decorators.

M

Michael Sparks

[ I'm not really sure of the etiquette of the python-dev list, so I think
I'll play "safe" and post this thought here... I know some of the
developers do look here, and Guido's comment (before europython) on the
dev list saying he's not interested in new syntaxes makes me think this
is a better place for it... ]

Anyway...

At Europython Guido discussed with everyone the outstanding issue with
decorators and there was a clear majority in favour of having them, which
was good. From where I was sitting it looked like about 20:20 split on the
following syntaxes:
1 def func(arg1, arg2, arg3) [dec,dec,dec]:
function...
2 [dec,dec,dec] def func(arg1, arg2, arg3):
function...

When it came to the decision on whether 2.4 includin one of these two
options or a waiting for a third unspecified, indeterminate syntax in a
later version, it looked like around a 60:40 vote in favour of waiting.
(ie 60:20:20 for waiting/syntax 1/syntax 2)

Also, suggested use cases for decorators were:
* Changing function type from method to staticmethod, classmethod
* Adding metadata:
* Author
* Version
* Deprecation
* Processing rules - eg grammar rules
* Framework annotations
* Support for external bindings

Some of these are clearly transformations of the methods, some are
annotations - ie addition of attributes after creation.

Transformations of methods can already be done using metaclasses.

Attributes can already appended to methods, but these need to be done
before transformations, and have to be done after the function has been
created:

class foo(object):
def bar(hello):
"This is"
print hello
bar.__RULE__ = "expr := expr"
bar.__doc__ += " a test"
bar.__author__ = "John"
bar = staticmethod(bar)

Suppose for a moment we wanted to do the same thing for the class foo - ie
we had some attributes we wanted to add to it, and after creating it we
wanted to transform it - what might we write?

class foo(object):
__RULE__ = "expr := expr"
__doc__ += " a test"
__author__ = "John"
def bar(hello):
"This is"
print hello

foo = someclasstransform(foo)

That's a lot nicer. However if we treat foo as a class description, then
someclasstransform tends to look similar to metaclass shenanigans.
Does that mean we can use metaclasses to similate a syntax for things
like staticmethods ?

My opinion here is yes - it's relatively trivial to implement if you use a
naming
scheme for methods:
class decorated_class_one(type):
def __new__(cls, name, bases, dct):
for key in dct.keys():
if key[:12] == "classmethod_":
dct[ key[12:] ] = classmethod(dct[key])
del dct[key]
if key[:7] == "static_":
dct[ key[7:] ] = staticmethod(dct[key])
del dct[key]
return type.__new__(cls, name, bases, dct)

class myclass(object):
__metaclass__ = decorated_class_one
def static_foo(arg1, arg2, arg3):
print "Hello",arg1, arg2, arg3
def classmethod_bar(cls,arg1, arg2, arg3):
print "World",cls, arg1, arg2, arg3
def baz(self,arg1, arg2, arg3):
print "There",self, arg1, arg2, arg3

The question then becomes what's the cleanest way of adding attributes
using this approach?

Since we've got a syntax that's similar to classes, you might argue one
approach might be:
class myclass(object):
__metaclass__ = decorated_class
def static_foo(arg1, arg2, arg3):
__doc__ = "This is a static method"
__author__ = "Tom"
__deprecated__ = True
print "Hello",arg1, arg2, arg3
def classmethod_bar(cls,arg1, arg2, arg3):
__doc__ = "This is a class method"
__author__ = "Dick"
__deprecated__ = False
print "World",cls, arg1, arg2, arg3
def baz(self,arg1, arg2, arg3):
__doc__ = "This is a normal method"
__author__ = "Harry"
__deprecated__ = False
print "There",self, arg1, arg2, arg3

Whilst we can get at the variable names that these declare, we can't get
at the values. What else can we do?

We could choose a similar "meta" keyword - say "decorators_", and pass a
dictionary instead?

class myclass(object):
__metaclass__ = decorated_class_two
decorators_static_foo = {
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
def static_foo(arg1, arg2, arg3):
print "Hello",arg1, arg2, arg3
decorators_classmethod_bar = {
'__doc__' : "This is a class method",
'__author__' : "Dick",
'__deprecated__' : False
}
def classmethod_bar(cls,arg1, arg2, arg3):
print "World",cls, arg1, arg2, arg3
decorators_baz = {
'__doc__' : "This is a normal method",
'__author__' : "Harry",
'__deprecated__' : False
}
def baz(self,arg1, arg2, arg3):
print "There",self, arg1, arg2, arg3

Whilst it's not perfect, it's something we can actually use. Bear in mind
we have to add attributes/decorators before we do transforms:

class decorated_class_two(type):
def __new__(cls, name, bases, dct):
for key in dct.keys():
if key[:11] == "decorators_":
for attr_key in dct[key].keys():
exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
del dct[key]
for key in dct.keys():
if key[:12] == "classmethod_":
dct[ key[12:] ] = classmethod(dct[key])
del dct[key]
if key[:7] == "static_":
dct[ key[7:] ] = staticmethod(dct[key])
del dct[key]
return type.__new__(cls, name, bases, dct)

Can we go one better? Can we make the following work?
class myclass(object):
__metaclass__ = decorated_class_three
decorators_foo = {
'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
def foo(arg1, arg2, arg3):
print "Hello",arg1, arg2, arg3
decorators_bar = {
'transforms' : [classmethod],
'__doc__' : "This is a class method",
'__author__' : "Dick",
'__deprecated__' : False
}
def bar(cls,arg1, arg2, arg3):
print "World",cls, arg1, arg2, arg3
decorators_baz = {
'__doc__' : "This is a normal method",
'__author__' : "Harry",
'__deprecated__' : False
}
def baz(self,arg1, arg2, arg3):
print "There",self, arg1, arg2, arg3

Well, let's try:
class decorated_class_three(type):
def __new__(cls, name, bases, dct):
for key in dct.keys():
if key[:11] == "decorators_":
transforms = []
for attr_key in dct[key].keys():
if attr_key == 'transforms':
transforms = dct[key][attr_key]
continue
exec 'dct[key[11:]].'+attr_key+' = dct[key][attr_key]'
for transform in transforms:
dct[key[11:]] = transform(dct[key[11:]])
del dct[key]
return type.__new__(cls, name, bases, dct)

And hey presto! It works!

The upshot is is that using a very simple metaclass, we can already have
the functionality that decorators will give us, but the syntax is less
than ideal:

class myclass(object):
__metaclass__ = decorated_class_three
decorators_foo = {
'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
def foo(arg1, arg2, arg3):
print "Hello",arg1, arg2, arg3

Now let's simplify this to what people currently commonly do:
class myclass(object):
def foo(arg1, arg2, arg3):
"This is a static method"
print "Hello",arg1, arg2, arg3
foo = staticmethod(foo)

In this situation python *does* do something special with the first value
it finds as the first value inside the method - it uses it as a __doc__
decorator. To my mind this treating the first value of the code as special
strikes me as the ideal hook. You could, for example, have the following
syntax:

class myclass(object):
def foo(arg1, arg2, arg3):
{ 'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
print "Hello",arg1, arg2, arg3

But here's the really cool trick - we can actually use this. Now. Let's
put our metaclass back, and make a small modification to make this work:
class myclass(object):
__metaclass__ = decorated_class_four
def foo(arg1, arg2, arg3):
"""{ 'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}"""
print "Hello",arg1, arg2, arg3

We can then modify our metaclass to make this work:
class decorated_class_four(type):
def __new__(cls, name, bases, dct):
for key in dct.keys():
doc = dct[key].__doc__
if doc:
try:
transforms = []
decorators = eval(doc)
for attr_key in decorators:
if attr_key == 'transforms':
transforms = decorators[attr_key]
continue
exec 'dct[key].'+attr_key+' = decorators[attr_key]'
for transform in transforms:
dct[key] = transform(dct[key])
except SyntaxError:
pass # no decorators
return type.__new__(cls, name, bases, dct)

So recasting our example using this final syntax, we gain:
class myclass(object):
__metaclass__ = decorated_class_four

def foo(arg1, arg2, arg3):
"""{ 'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}"""
print "Hello",arg1, arg2, arg3

def bar(cls,arg1, arg2, arg3):
"""{ 'transforms' : [classmethod],
'__doc__' : "This is a class method",
'__author__' : "Dick",
'__deprecated__' : False
}"""
print "World",cls, arg1, arg2, arg3

def baz(self,arg1, arg2, arg3):
"""{ '__doc__' : "This is a normal method",
'__author__' : "Harry",
'__deprecated__' : False
}"""
print "There",self, arg1, arg2, arg3

Each of the transformations chosen takes us between a variety of syntaxes,
with various advantages/disadvantages. Personally I think the best variety
here without changing python's syntax or semantics is this one:

class myclass(object):
__metaclass__ = decorated_class_three
decorators_foo = {
'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
def foo(arg1, arg2, arg3):
print "Hello",arg1, arg2, arg3
....

Partly the reason for this is because it's very clear what's going on
here, and also you can use doc strings as normal:
class myclass(object):
__metaclass__ = decorated_class_three
decorators_foo = {
'transforms' : [staticmethod],
'__author__' : "Tom",
'__deprecated__' : True
}
def foo(arg1, arg2, arg3):
"This is a static method"
print "Hello",arg1, arg2, arg3
....

Whereas with a semantic change to the initial var, I think this form is
nicer:
class myclass(object):
# no metaclass, requires change in semantics
def foo(arg1, arg2, arg3):
{ 'transforms' : [staticmethod],
'__doc__' : "This is a static method",
'__author__' : "Tom",
'__deprecated__' : True
}
print "Hello",arg1, arg2, arg3

def bar(cls,arg1, arg2, arg3):
{ 'transforms' : [classmethod],
'__doc__' : "This is a class method",
'__author__' : "Dick",
'__deprecated__' : False
}
print "World",cls, arg1, arg2, arg3
....


Syntactically this compiles/runs fine right now, it just doesn't have the
decorator semantics we need for this to work.

And what would the semantic change be? Currently we have foo.__doc__ .
This could be foo.__decorator_dict__ . Quite what anyone chooses to do
with these would be entirely up to them.

Classes that inherit from object could be defined to do something special
- such as act in a similar way to the presented metaclass. Whereas for
objects that aren't derived from object, nothing would change, except a
small amount of extra information is made available, and *potentially*
ignored.

Anyway, for any of the python-dev team who do hang on out python-list I
hope this has been food for thought, and whilst I've done a cursory check
of the archives I'm not subbed to python-dev, so apologies if I'm going
over old ground and raking up old ideas needlessly!

One thing that does strike me regarding this is this:

This relies on being able to pull out the first value at the start of the
function/method definition. This _does_ currently happen anyway, and
people are using the value there. Making it consistent in that you are
able to pull out the value, no matter what the type strikes me as a very
useful thing, and decorators can naturally fall out of it as well.

As I said, hopefully useful food for thought, and hopefully not hacked
anyone off at the length of this post!

Cheers,


Michael.
 
M

Michele Simionato

Michael Sparks said:
[ I'm not really sure of the etiquette of the python-dev list, so I think
I'll play "safe" and post this thought here... I know some of the
developers do look here, and Guido's comment (before europython) on the
dev list saying he's not interested in new syntaxes makes me think this
is a better place for it... ]

Anyway...

At Europython Guido discussed with everyone the outstanding issue with
decorators and there was a clear majority in favour of having them, which
was good. From where I was sitting it looked like about 20:20 split on the
following syntaxes:
1 def func(arg1, arg2, arg3) [dec,dec,dec]:
function...
2 [dec,dec,dec] def func(arg1, arg2, arg3):
function...

When it came to the decision on whether 2.4 includin one of these two
options or a waiting for a third unspecified, indeterminate syntax in a
later version, it looked like around a 60:40 vote in favour of waiting.
(ie 60:20:20 for waiting/syntax 1/syntax 2)
<snip>

The problem of metaclasses-based solutions is that they are hacks.
We don't want everybody to implement his/her version of decorators
(yes me too I have implemented mine). Decorators MUST be in the core
language. Waiting will only favor the abuse of metaclasses.
I do not particularly like each of two syntaxes (even if the second
one looks better) but also I do not particularly dislike them.
My preferred solution would be

def func(...) is dec:
....

*without* the ability to specify multiple decorators (i.e. let compose
the multiple generators explicitely by hand, with a functional.compose
function or something like that).

But I prefer to have a syntax I don't like 100% then waiting for another
year or so. Why don't put a given syntax in 2.4a, see how it goes, and
change it in 2.4b if it is the case?

Michele Simionato
 
J

j_mckitrick

You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\

Could someone give me an idea of what metaclasses and decorators _are_?

I think metaclasses are classes with their base methods overridden, correct?

And decorators are a form of delegation, right?

If so, where am I wrong? And how do these techniques make for better code/coding?

jonathon
 
D

David MacQuigg

You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\

Could someone give me an idea of what metaclasses and decorators _are_?

The best I have found on metaclasses is Alex Martelli's explanation
and example in Python in a Nutshell. There is also extensive
documentation available at http://python.org/doc/newstyle.html I have
also written a one-page explanation for engineering students in my
Python OOP chapter at http://ece.arizona.edu/~edatools/Python

Decorators are just modifiers you can add to a function definition,
ususally to modify the behavior, as in

def func(x,y) [staticmethod]:

These are not yet in Python. See PEP 318 for more explanationa and
examples.

-- Dave
 
M

Michele Simionato

You guys are WAY over my head, and I'm supposed to be a C.S. junior! :-\

Could someone give me an idea of what metaclasses and decorators _are_?

I think metaclasses are classes with their base methods overridden, correct?

And decorators are a form of delegation, right?

If so, where am I wrong? And how do these techniques make for better code/coding?

jonathon

http://www.python.org/cgi-bin/moinmoin/MetaClasses

for decorators, look at PEP318.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top