Slight metaclass confusion

[ben]

I am slightly confused about the way metaclasses work, having read
"Metaclass Programming In Python, Parts 1 and 2"

I get the fact that the instance of a metaclass is a class, but in
this case I fail to see why the following does'nt work:
.... def __str__(cls):
.... print "I am " + repr(cls)
.... Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Whereas this does:
.... def __str__(self):
.... return "I am " + repr(self)
.... I am <__main__.Simple object at 0x402f676c>

The tutorial does mention this (slightly), but doesn't make it clear
why this is happening. I am probably missing something obvious though!

Thanks for the help,

Ben
---

 

[Stephan Diehl]

I am slightly confused about the way metaclasses work, having read
"Metaclass Programming In Python, Parts 1 and 2"

I get the fact that the instance of a metaclass is a class, but in
this case I fail to see why the following does'nt work:

... def __str__(cls):
... print "I am " + repr(cls)
...
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Whereas this does:

... def __str__(self):
... return "I am " + repr(self)
...
I am <__main__.Simple object at 0x402f676c>

The tutorial does mention this (slightly), but doesn't make it clear
why this is happening. I am probably missing something obvious though!

The problem is perhaps the terminology (Unfortunatelly,I don't know a better
one).

The metaclass initializes a class. Just imagine, that the textual class
definition comes fresh from the parser, so you have information about
possible base classes, methods, attributes. These information can be
changed within the metaclass __new__ method.
Think of the metaclass as a factory that turns a class definition into a
class instance.
Other than that, they don't interact.

What a metaclass can do, is to change that information, before the class is
made into a class object.

Basicly, the metaclass concept is orthogonal to the inheritence concept of
OOP.

Thanks for the help,

Ben
---

Hope that helps

Stephan

 

[Ben Young]

I am slightly confused about the way metaclasses work, having read
"Metaclass Programming In Python, Parts 1 and 2"

[snip]

The problem is perhaps the terminology (Unfortunatelly,I don't know a better
one).

The metaclass initializes a class. Just imagine, that the textual class
definition comes fresh from the parser, so you have information about
possible base classes, methods, attributes. These information can be
changed within the metaclass __new__ method.
Think of the metaclass as a factory that turns a class definition into a
class instance.
Other than that, they don't interact.

What a metaclass can do, is to change that information, before the class is
made into a class object.

Basicly, the metaclass concept is orthogonal to the inheritence concept of
OOP.

So the way it is implemented in python a class is NOT an instance of a
metaclass the same way that an instance is an instance of a class. This is
slightly confusing as this is not the way it is described in all the
places I can find that talk about python metaclasses.

In this case, what is the point of using a class when a simple function
can do the job?

Hope that helps

Stephan

Thanks for the reply

Ben
---

 

[Stephan Diehl]

Ben Young wrote:

[snip]

The problem is perhaps the terminology (Unfortunatelly,I don't know a
better one).

The metaclass initializes a class. Just imagine, that the textual class
definition comes fresh from the parser, so you have information about
possible base classes, methods, attributes. These information can be
changed within the metaclass __new__ method.
Think of the metaclass as a factory that turns a class definition into a
class instance.
Other than that, they don't interact.

What a metaclass can do, is to change that information, before the class
is made into a class object.

Basicly, the metaclass concept is orthogonal to the inheritence concept
of OOP.

So the way it is implemented in python a class is NOT an instance of a
metaclass the same way that an instance is an instance of a class. This is
slightly confusing as this is not the way it is described in all the
places I can find that talk about python metaclasses.

For all I know, a class is the instance of its metaclass, but this doesn't
mean that an object a of class A can access methods of A' (the metaclass of
A). (Please note that this is probably totally wrong. Other people might
shed light on this more theoretical question)

In this case, what is the point of using a class when a simple function
can do the job?

Well, actually, there is probably nothing you can do with metaclasses that
you couldn't do in other ways.

The point (if you are not a language lawyer) is that the resulting code is
(at least in some cases) better readable when using metaclasses.

The Python cookbook (http://aspn.activestate.com/ASPN/Cookbook/Python) has a
couple of recipies about metaclasses. Please have a look there if you are
interested in practical things you can do with metaclasses.

 

[Hans Nowak]

I am slightly confused about the way metaclasses work, having read
"Metaclass Programming In Python, Parts 1 and 2"

I get the fact that the instance of a metaclass is a class, but in
this case I fail to see why the following does'nt work:

I'm not 100% sure, but:

... def __str__(cls):
... print "I am " + repr(cls)
...

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

....here, you're calling __str__ on a *class*...

... def __str__(self):
... return "I am " + repr(self)
...


I am <__main__.Simple object at 0x402f676c>

....and here, you're calling it on an *instance*. The reason it doesn't work
probably has nothing to do with your metaclass, as demonstrated by this:
Traceback (most recent call last):
'<__main__.Foo object at 0x0162C750>'

I'm not sure how it all works internally... using str(Foo) works, but
Foo.__str__() doesn't. Notice that Foo.__str__(Foo) does work.

I don't think there's anything wrong with your metaclass (though see below ;-),
just don't call __str__() directly on the class.

On a side note, it's a bad idea to use 'print' in __str__ or __repr__, even for
informal test code. I tried the code above in PyCrust, and it broke the
automatic attribute lookup. This is not really important, but it was a
surprising side effect.

HTH,

 

[David Mertz]

(e-mail address removed) ([email protected]) wrote previously:
|"Metaclass Programming In Python, Parts 1 and 2"
|I get the fact that the instance of a metaclass is a class, but in
|this case I fail to see why the following does'nt work:

Read part 2 again. It takes a while to sink in.

Your problem is that you aren't considering the MRO of Class:
... def __str__(cls): # Use 'return', not 'print'
... return "I am " + repr(cls)
... def foo(cls):
... return "I am " + repr(cls)
... "I am <class '__main__.Fish'>"

Class is an instance of Meta, just like you expect. Calling the .foo()
method shows this. But when you call a method, the method name is
checked in the -method resolution order- (before the metaclass is
checked):
[<class '__main__.Fish'>, <type 'object'>]

Fish doesn't define a method .__str__(), so Python looks in Fish's
parent, object. Well, object -does- define, .__str__(), so your call
resolves to:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Of course, since object doesn't define a .foo(), resolution proceeds to
the metaclass. If you want to get Class' metaclass method, you can let
Python do the magic for you:
"I am <class '__main__.Fish'>"

Or if you want to be really explicit:
"I am <class '__main__.Fish'>"

Yours, David...

 

[Bengt Richter]

I am slightly confused about the way metaclasses work, having read
"Metaclass Programming In Python, Parts 1 and 2"

I get the fact that the instance of a metaclass is a class, but in
this case I fail to see why the following does'nt work:

... def __str__(cls):
... print "I am " + repr(cls)
...

UIAM, you subclassed type, but you didn't override __new__, so ...became essentially a pass-through call to type, i.e., effectively the same as
Class = type("Fish",(),{})
and since you passed it an empty dict to make its Class.__dict__ from,
went looking in Class.__dict__ for '__str__' and couldn't find it. So it
went looking for it according to Class.mro(), which led to finding it in
the object base class -- i.e., object.__dict__['__str__'] succeeded. But what it
got was an unbound method, because neither of two things (accessing an ordinary
method as an attribute of an instance object, or accessing a class method as an attribute
of either and instance or its class) was happening to bind to the method. So you got a complaint.
Note that it's complaining about __str__ of object, not __str__ of your class Meta.

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Note that your def __str__(cls) *does* exist in Meta.__dict__, but that wasn't the
dict used to construct Class.

Whereas this does:

... def __str__(self):
... return "I am " + repr(self)
...

In this case you are not overriding __new__ either, but the search for __new__ is finding
object.__new__, and that is making the object instance for you, of subtype Simple.
Actually, I think this amounts to Simple.__call__(), which must call cls.__new__() and go
down the mro chain to find object.__new__ and call it with cls (which would be Simple).
Looking for __str__ in this case is looking in type(obj).__dict__ first and finding it right there.
You are triggering it via attribute access, which dynamically creates the bound method that supplies
the self arg. Alternatively, type(obj).__str__ should be the unbound method, to which you could pass
obj, and type(obj).__dict__['__str__'] should be the plain function that becomes a method by the
dynamic magic. It can be called with obj as arg like the unbound method.

I am <__main__.Simple object at 0x402f676c>

The tutorial does mention this (slightly), but doesn't make it clear
why this is happening. I am probably missing something obvious though!

Thanks for the help,

To get the effect you originally wanted, one way could be:
... def __str__(cls): return 'I am '+ repr(cls)
... __str__ = classmethod(__str__)
... def __new__(cls, name): return type.__new__(cls, name, (),cls.__dict__.copy())
... "I am <class '__main__.Fish'>"

Interestingly,
"I am <class '__main__.Meta'>"

apparently the str builtin doing str(x) goes after type(x).__str__, e.g.,
"I am <class '__main__.Fish'>"

vs
"I am <class '__main__.Meta'>"


The above is my current view of how things work. I think/hope it's pretty close, but I am not
speaking from a code walk here, so there may be some details awry. Corrections welcome.

BTW, the __metaclass__ class variable and how that works is something that belongs in
a discussion of Python metaclass programming, but someone else can fill that in ;-)

HTH and does not mislead.

Regards,
Bengt Richter

 

[Bengt Richter]

UIAM, you subclassed type, but you didn't override __new__, so ...

ARGH. I was was partly mistaken, obviously.

became essentially a pass-through call to type, i.e., effectively the same as

bzzt. wrong ;-/ It would (I think) have called type.__new__(Meta, 'Fish', (), {})

Class = type("Fish",(),{})
and since you passed it an empty dict to make its Class.__dict__ from,

The empty dict did become the Class.__dict__ initial value though, I believe.

went looking in Class.__dict__ for '__str__' and couldn't find it. So it still true
went looking for it according to Class.mro(), which led to finding it in
the object base class -- i.e., object.__dict__['__str__'] succeeded. But what it
got was an unbound method, because neither of two things (accessing an ordinary
method as an attribute of an instance object, or accessing a class method as an attribute
of either and instance or its class) was happening to bind to the method. So you got a complaint.
Note that it's complaining about __str__ of object, not __str__ of your class Meta.

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Note that your def __str__(cls) *does* exist in Meta.__dict__, but that wasn't the
dict used to construct Class. True, but maybe misleading.

Whereas this does:

... def __str__(self):
... return "I am " + repr(self)
...

In this case you are not overriding __new__ either, but the search for __new__ is finding
object.__new__, and that is making the object instance for you, of subtype Simple.
Actually, I think this amounts to Simple.__call__(), which must call cls.__new__() and go
down the mro chain to find object.__new__ and call it with cls (which would be Simple).

I need to read code for a while, and not write stuff I have to qualify...

Looking for __str__ in this case is looking in type(obj).__dict__ first and finding it right there.
You are triggering it via attribute access, which dynamically creates the bound method that supplies
the self arg. Alternatively, type(obj).__str__ should be the unbound method, to which you could pass
obj, and type(obj).__dict__['__str__'] should be the plain function that becomes a method by the
dynamic magic. It can be called with obj as arg like the unbound method.

I am <__main__.Simple object at 0x402f676c>

The tutorial does mention this (slightly), but doesn't make it clear
why this is happening. I am probably missing something obvious though!

Thanks for the help,

To get the effect you originally wanted, one way could be:
... def __str__(cls): return 'I am '+ repr(cls)
... __str__ = classmethod(__str__)
... def __new__(cls, name): return type.__new__(cls, name, (),cls.__dict__.copy())
..."I am <class '__main__.Fish'>"

Interestingly,
"I am <class '__main__.Meta'>"

apparently the str builtin doing str(x) goes after type(x).__str__, e.g.,
"I am <class '__main__.Fish'>"

vs
"I am <class '__main__.Meta'>"


The above is my current view of how things work. I think/hope it's pretty close, but I am not
speaking from a code walk here, so there may be some details awry. Corrections welcome.

I see David Mertz's post now. I learned that going off the end of the mro search for foo
then goes on to the metaclass, which presumably means that __str__ would have been found
if object.__str__ hadn't snagged it first in the mro chain.

BTW, the __metaclass__ class variable and how that works is something that belongs in
a discussion of Python metaclass programming, but someone else can fill that in ;-)

HTH and does not mislead.

Sorry about the pass-through-to-type mislead. That was plain wrong, along with the unstated
implication that Meta.__str__ was plain unreachable from the resulting Class. At least I qualified
it with UIAM ;-/

Whoa, I'm late. gotta run...

Regards,
Bengt Richter

 

[Ben Young]

(e-mail address removed) ([email protected]) wrote previously:
|"Metaclass Programming In Python, Parts 1 and 2"
|I get the fact that the instance of a metaclass is a class, but in
|this case I fail to see why the following does'nt work:

Read part 2 again. It takes a while to sink in.

Your problem is that you aren't considering the MRO of Class:
... def __str__(cls): # Use 'return', not 'print'
... return "I am " + repr(cls)
... def foo(cls):
... return "I am " + repr(cls)
..."I am <class '__main__.Fish'>"

Class is an instance of Meta, just like you expect. Calling the .foo()
method shows this. But when you call a method, the method name is
checked in the -method resolution order- (before the metaclass is
checked):
[<class '__main__.Fish'>, <type 'object'>]

Fish doesn't define a method .__str__(), so Python looks in Fish's
parent, object. Well, object -does- define, .__str__(), so your call
resolves to:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: descriptor '__str__' of 'object' object needs an argument

Of course, since object doesn't define a .foo(), resolution proceeds to
the metaclass. If you want to get Class' metaclass method, you can let
Python do the magic for you:
"I am <class '__main__.Fish'>"

Or if you want to be really explicit:
"I am <class '__main__.Fish'>"

Yours, David...

Right I think i'm getting there now. Thanks to all those who replied!

I see now that *instances* if classes must always look in their class dist
because they _don't have an mro_, whereas classes do and this is always
looked at first, causing the unbound method error.

Does this mean function calls are done something like this (untested code)

mro = getattr(obj, "mro", None)
if mro:
for i in mro:
func = getattr(i, "function")
if func:
return func(*args, **kw)

#Fall through
return getattr(type(obj), "function")(obj, *args, **kw)

Ben
---

--
_/_/_/ THIS MESSAGE WAS BROUGHT TO YOU BY: Postmodern Enterprises _/_/_/
_/_/ ~~~~~~~~~~~~~~~~~~~~[[email protected]]~~~~~~~~~~~~~~~~~~~~~ _/_/
_/_/ The opinions expressed here must be those of my employer... _/_/
_/_/_/_/_/_/_/_/_/_/ Surely you don't think that *I* believe them! _/_/

 

[David Mertz]

|mro = getattr(obj, "mro", None)
|if mro:
| for i in mro:
| func = getattr(i, "function")
| if func:
| return func(*args, **kw)
|#Fall through
|return getattr(type(obj), "function")(obj, *args, **kw)

Pretty much, but it looks like you have a couple errors in your example.
I think this is better (also undertested):

def callmeth(obj, methname, *args, **kw):
meth = obj.__dict__.get(methname)
if meth is not None: # Might be in object's dict
return meth(*args, **kw)
if hasattr(obj, "mro"): # Might be in the ancestry
for klass in obj.mro():
meth = getattr(klass, methname)
if meth is not None:
return meth(*args, **kw)
# Or finally, look in class of obj
return getattr(type(obj), methname)(*args, **kw)

I think this does roughly the right thing if a non-callable "meth" is
encountered.

Yours, David...