Strange metaclass behaviour

[Christian Eder]

Hi,

I think I have discovered a problem in context of
metaclasses and multiple inheritance in python 2.4,
which I could finally reduce to a simple example:

Look at following code:

class M_A (type) :

def __new__ (meta, name, bases, dict) :
print "M.__new__", meta, name, bases
return super (M_A, meta).__new__ (meta, name, bases, dict)

class M_B (M_A) : pass

class A (object) : __metaclass__ = M_A

class B (object) : __metaclass__ = M_B

class D (A, B) : pass


One would expect that either
1) D inherits the metaclass M_A from A
2) or python raises the well known "Metatype conflict among bases"
error

Instead, if you run this code, you get the following output:

M.__new__ <class '__main__.M_A'> A (<type 'object'>,)
M.__new__ <class '__main__.M_B'> B (<type 'object'>,)
M.__new__ <class '__main__.M_A'> D (<class '__main__.A'>, <class
'__main__.B'>)
M.__new__ <class '__main__.M_B'> D (<class '__main__.A'>, <class
'__main__.B'>)

This means that when class D gets defined, the __new__ from M_A
get executed twice (first from M_A, then from M_B), which should not happen.
This suggests that either
1) cooperative supercalls do not work here
2) the metaclass of EACH of the bases of D does it's work independently
when D is defined.

Does anyone have a detailed explanation here ?
Is this problem already known ?

regards
chris

 

[Ziga Seilnacht]

Hi,

I think I have discovered a problem in context of
metaclasses and multiple inheritance in python 2.4,
which I could finally reduce to a simple example:

I don't know if this is a bug; but I will try to expain
what is happening; here is an example similar to yours:
.... def __new__(meta, name, bases, dict):
.... print 'metaclass:', meta.__name__, 'class:', name
.... return super(M_A, meta).__new__(meta, name, bases, dict)
........ pass
........ __metaclass__ = M_A
....
metaclass: M_A class: A.... __metaclass__ = M_B
....
metaclass: M_B class: B

So far everything is as expected.
.... __metaclass__ = M_B
....
metaclass: M_B class: C

If we explicitly declare that our derived class inherits
from the second base, which has a more derived metaclass,
everything is OK.
.... pass
....
metaclass: M_A class: D
metaclass: M_B class: D

Now this is where it gets interesting; what happens
is the following:
- Since D does not have a __metaclass__ attribute,
its type is determined from its bases.
- Since A is the first base, its type (M_A) is called;
unfortunately this is not the way metaclasses are
supposed to work; the most derived metaclass should
be selected.
- M_A's __new__ method calls the __new__ method of the
next class in MRO; that is, super(M_1, meta).__new__
is equal to type.__new__.
- In type.__new__, it is determined that M_A is not
the best type for D class; it should be actually M_B.
- Since type.__new__ was called with wrong metaclass
as the first argument, call the correct metaclass.
- This calls M_B.__new__, which again calls type.__new__,
but this time with M_B as the first argument, which
is correct.

As I said, I don't know if this is a bug or not,
but you can achieve what is expected if you do the
following in your __new__ method (warning, untested code):
.... """
.... Metaclass that follows type's behaviour in "metaclass
resolution".
....
.... Code is taken from Objects/typeobject.c and translated to
Python.
.... """
.... def __new__(meta, name, bases, dict):
.... winner = meta
.... for cls in bases:
.... candidate = type(cls)
.... if candidate is ClassType:
.... continue
.... if issubclass(winner, candidate):
.... continue
.... if issubclass(candidate, winner):
.... winner = candidate
.... continue
.... raise TypeError("metaclass conflict: ...")
.... if winner is not meta and winner.__new__ !=
AnyMeta.__new__:
.... return winner.__new__(winner, name, bases, dict)
.... # Do what you actually meant from here on
.... print 'metaclass:', winner.__name__, 'class:', name
.... return super(AnyMeta, winner).__new__(winner, name, bases,
dict)
........ pass
........ __metaclass__ = AnyMeta
....
metaclass: AnyMeta class: A.... __metaclass__ = OtherMeta
....
metaclass: OtherMeta class: B.... pass
....
metaclass: OtherMeta class: C

Does anyone have a detailed explanation here ?
Is this problem already known ?

regards
chris

I hope that above explanation helps.

Ziga

 

[Michele Simionato]

- Since D does not have a __metaclass__ attribute,
its type is determined from its bases.
- Since A is the first base, its type (M_A) is called;
unfortunately this is not the way metaclasses are
supposed to work; the most derived metaclass should
be selected.
- M_A's __new__ method calls the __new__ method of the
next class in MRO; that is, super(M_1, meta).__new__
is equal to type.__new__.
- In type.__new__, it is determined that M_A is not
the best type for D class; it should be actually M_B.
- Since type.__new__ was called with wrong metaclass
as the first argument, call the correct metaclass.
- This calls M_B.__new__, which again calls type.__new__,
but this time with M_B as the first argument, which
is correct.

This is a very good explanation and it should go somewhere in the
standard docs.
I remember I spent a significant amount of time and effort to reach the
same conclusion
a while ago, and now I have already started to forget eveything a again
:-(
Anyway, I will bookmark this post for future reference

Michele Simionato

 

[Michele Simionato]

Hi,

I think I have discovered a problem in context of
metaclasses and multiple inheritance in python 2.4,
which I could finally reduce to a simple example:

Look at following code:

class M_A (type) :

def __new__ (meta, name, bases, dict) :
print "M.__new__", meta, name, bases
return super (M_A, meta).__new__ (meta, name, bases, dict)

class M_B (M_A) : pass

class A (object) : __metaclass__ = M_A

class B (object) : __metaclass__ = M_B

class D (A, B) : pass


One would expect that either
1) D inherits the metaclass M_A from A
2) or python raises the well known "Metatype conflict among bases"
error

No, there is no conflict in this case: since M_B is a subclass of M_A,
the
metaclass of D is M_B. I don't think this is bug either: the fact is
that
type.__new__ works differently from object.__new__, so that it is
called twice in this case. Not sure if it could be avoided.

Speaking of the metatype conflict, I realized a while ago that it is
possible
to avoid it automatically even at the pure Python level, without
changing
the C code for type.__new__. However, the solution is too complicate.
According
to the Zen of Python "If the implementation is hard to explain, it's a
bad idea",
thus I have never used it.

Still, it is an interesting exercise if you are willing to risk the
melting of your brain,
so here is the code

# noconflict.py
"""Deep, **DEEP** magic to remove metaclass conflicts.

``noconflict`` provides the ``safetype`` metaclass, the mother of
conflict-free
metaclasses. I you do

from noconflict import safetype as type

on top of your module, all your metaclasses will be conflict safe.
If you override ``__new__`` when you derive from ``safetype``,
you should do it cooperatively."""

import inspect, types, __builtin__
try: set # python version >= 2.4
except NameError: # python version <= 2.3
from sets import Set as set

def skip_redundant(iterable, skipset=None):
"Redundant items are repeated items or items in the original
skipset."
if skipset is None: skipset = set()
for item in iterable:
if item not in skipset:
skipset.add(item)
yield item

memoized_metaclasses_map = {}

# utility function
def remove_redundant(metaclasses):
skipset = set([types.ClassType])
for meta in metaclasses: # determines the metaclasses to be skipped
skipset.update(inspect.getmro(meta)[1:])
return tuple(skip_redundant(metaclasses, skipset))

##################################################################
## now the core of the module: two mutually recursive functions ##
##################################################################

def get_noconflict_metaclass(bases, left_metas, right_metas):
"""Not intended to be used outside of this module, unless you know
what you are doing."""
# make tuple of needed metaclasses in specified priority order
metas = left_metas + tuple(map(type, bases)) + right_metas
needed_metas = remove_redundant(metas)

# return existing confict-solving meta, if any
if needed_metas in memoized_metaclasses_map:
return memoized_metaclasses_map[needed_metas]
# nope: compute, memoize and return needed conflict-solving meta
elif not needed_metas: # wee, a trivial case, happy us
meta = type
elif len(needed_metas) == 1: # another trivial case
meta = needed_metas[0]
# check for recursion, can happen i.e. for Zope ExtensionClasses
elif needed_metas == bases:
raise TypeError("Incompatible root metatypes", needed_metas)
else: # gotta work ...
metaname = '_' + ''.join([m.__name__ for m in needed_metas])
meta = classmaker()(metaname, needed_metas, {})
memoized_metaclasses_map[needed_metas] = meta
return meta

def classmaker(left_metas=(), right_metas=()):
def make_class(name, bases, adict):
metaclass = get_noconflict_metaclass(bases, left_metas,
right_metas)
return metaclass(name, bases, adict)
return make_class

#################################################################
## and now a conflict-safe replacement for 'type' ##
#################################################################

__type__=__builtin__.type # the aboriginal 'type'
# left available in case you decide to rebind __builtin__.type

class safetype(__type__):
"""Overrides the ``__new__`` method of the ``type`` metaclass,
making the
generation of classes conflict-proof."""
def __new__(mcl, *args):
nargs = len(args)
if nargs == 1: # works as __builtin__.type
return __type__(args[0])
elif nargs == 3: # creates the class using the appropriate
metaclass
n, b, d = args # name, bases and dictionary
meta = get_noconflict_metaclass(b, (mcl,), right_metas=())
if meta is mcl: # meta is trivial, dispatch to the default
__new__
return super(safetype, mcl).__new__(mcl, n, b, d)
else: # non-trivial metaclass, dispatch to the right
__new__
# (it will take a second round)
return super(mcl, meta).__new__(meta, n, b, d)
else:
raise TypeError('%s() takes 1 or 3 arguments' %
mcl.__name__)



Michele Simionato

 

[Christian Eder]

I hope that above explanation helps.

Thanks for your support.
I now understand what happens here,
but I'm not really happy with the situation.
Your solution is a nice workaround, but in a quite
huge and complex class framework with a lot a custom
metaclasses you don't want this code in each __new__
function. And in fact each __new__ which does not contain this
fix-code (and which is not completely side-effect free) might
break if someone adds additional classes deeps down in the
inheritance hierarchy (which is exactly what happened
for me). And this is clearly not what one should expect in
context of multiple inheritance and cooperative supercalls.

Raising a "metatype conflict among bases" error might be a
perfectly acceptable behavior here (though it would be better if
python resolves the conflict as your code does), but
double-executing code is not in my humble opinion.

Is this worth a bug-report on sourceforge ?

regards
chris

 

[Christian Eder]

Still, it is an interesting exercise if you are willing to risk the
melting of your brain,
so here is the code

I tried your code and it fixes the double execution, but the __new__ is
executed in context of M_A instead of M_B which would be the more
specific type here.

Anyway, I think I found an acceptable solution for the problem.
The question is "What Do I need to fix metaclasses ?"
And the answer, of course, is "a meta-meta-class !"

The following code solves the problem.
It's basically Ziga's code except that I added one level of abstraction
(to avoid adding this kludge code to each custom metaclasses'es __new__):

class _fixed_type_ (type) :

def __call__ (meta, name, bases, dict) :
meta = meta._get_meta (bases, dict)
cls = meta.__new__ (meta, name, bases, dict)
meta.__init__ (cls, name, bases, dict)
return cls
# end def __call__

def _get_meta (meta, bases, dict) :
if "__metaclass__" in dict :
return dict ["__metaclass__"]
winner = meta
for b in bases :
cand = type (b)
if cand in (types.ClassType, type) :
pass
elif issubclass (cand, winner) :
winner = cand
elif issubclass (winner, cand) :
pass
else :
raise TypeError ("Metatype conflict among bases")
return winner
# end def _get_meta

# end class _fixed_type_

class my_type (type) :

__metaclass__ = _fixed_type_ ### to fix metaclasses, we need
### meta-meta classes

# end class my_type


Then I made "my_type" the root of my metaclass hierarchy
(instead of "type") which solves all my problems.

After 5 years of Python, I still find it impressive how much
vodoo and mojo one can do here

regards
chris

 

[Michele Simionato]

After 5 years of Python, I still find it impressive how much

vodoo and mojo one can do here

True

However, I should point out that I never use this stuff in production
code.
I have found out that for my typical usages metaclasses are too much:
a class decorator would be enough and much less fragile. At the
present, we
do not have class decorators, but we can nearly fake them with a very
neat
trick:


def thisclass(proc, *args, **kw):
""" Example: ... thisclass(register)
...
registered
"""
# basic idea stolen from zope.interface, which credits P.J. Eby
frame = sys._getframe(1)
assert '__module__' in frame.f_locals # inside a class statement
def makecls(name, bases, dic):
try:
cls = type(name, bases, dic)
except TypeError, e:
if "can't have only classic bases" in str(e):
cls = type(name, bases + (object,), dic)
else: # other strange errors, such as __slots__ conflicts, etc
raise
del cls.__metaclass__
proc(cls, *args, **kw)
return cls
frame.f_locals["__metaclass__"] = makecls

Figured you would like this one

Michele Simionato

 

[Ziga Seilnacht]

Michele Simionato wrote:

<snip>

There is a minor bug in your code:

def thisclass(proc, *args, **kw):
""" Example:... thisclass(register)
...
registered
"""
# basic idea stolen from zope.interface, which credits P.J. Eby
frame = sys._getframe(1)
assert '__module__' in frame.f_locals # <----------------------------------- here
def makecls(name, bases, dic):
try:
cls = type(name, bases, dic)
except TypeError, e:
if "can't have only classic bases" in str(e):
cls = type(name, bases + (object,), dic)
else: # other strange errors, such as __slots__ conflicts, etc
raise
del cls.__metaclass__
proc(cls, *args, **kw)
return cls
frame.f_locals["__metaclass__"] = makecls

Figured you would like this one

Michele Simionato

See this example:
.... frame = sys._getframe(1)
.... return '__module__' in frame.f_locals
........ frame = sys._getframe(1)
.... return '__module__' in frame.f_locals and not \
.... '__module__' in frame.f_code.co_varnames
........ print in_class_statement1()
.... print in_class_statement2()
....
True
True.... __module__ = 1
.... print in_class_statement1()
.... print in_class_statement2()
....True
False

 

[Michele Simionato]

Well, I would not call it a bug, I would call it to cheat
The assert is there I just wanted to prevent accidents, not to *really*
ensure
that 'thisclass' is called inside a class statement. Do you know of any
reliable method to enforce that restriction?

Michele Simionato

 

[Michele Simionato]

... frame = sys._getframe(1)
... return '__module__' in frame.f_locals and not \
... '__module__' in frame.f_code.co_varnames

On second thought, to break this check is less easy than I expected, so
maybe it is reliable enough. BTW, if you are interested, you can check
the original code in zope.interface.advice.