New+old-style multiple inheritance

[stephenpas]

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<quote>
class Foo:
pass

class Bar(object):
pass

class Baz(Foo,Bar):
pass

# Monkey-patch Foo to add a special method
def my_nonzero(self):
print "my_nonzero called"
return False
Foo.__nonzero__ = my_nonzero

b = Baz()

print "doing the test on Baz(Foo,Bar). Should return false"
if b:
print "true"
else:
print "false"
</quote>

Produces this output:

doing the test on Baz(Foo,Bar). Should return false
true

With some experimentation it is clear that this behaviour only occurs
when you combine new+old-style multiple inheritance, monkey-patching
and special methods. If Foo and Bar are either old or new-style it
works. calling b.__nonzero__() directly works. Defining __nonzero__
within Foo works.

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

Any ideas?

Cheers,
Stephen.

 

[Jonathan Gardner]

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

You've already discovered why--you're mixing old and new style
classes.

Monkey patching is definitely unpythonic. You must be a Ruby guy. Why
don't you try doing something else to get the behavior you want,
something more explicit?

 

[Gabriel Genellina]

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance. In so doing we have
uncovered some unexpected behaviour:

<quote>
class Foo:
pass

class Bar(object):
pass

class Baz(Foo,Bar):
pass

# Monkey-patch Foo to add a special method
def my_nonzero(self):
print "my_nonzero called"
return False
Foo.__nonzero__ = my_nonzero

b = Baz()

print "doing the test on Baz(Foo,Bar). Should return false"
if b:
print "true"
else:
print "false"
</quote>

Produces this output:

doing the test on Baz(Foo,Bar). Should return false
true

With some experimentation it is clear that this behaviour only occurs
when you combine new+old-style multiple inheritance, monkey-patching
and special methods. If Foo and Bar are either old or new-style it
works. calling b.__nonzero__() directly works. Defining __nonzero__
within Foo works.

I know this level of messing with python internals is a bit risky but
I'm wondering why the above code doesn't work.

I think I can barely explain what happens here (but I may be
absolutely wrong! Please someone with greater knowledge of Python
innards correct whatever is wrong on my description!)

type objects contain "slots" (function pointers) corresponding to the
"magic" methods like __nonzero__ (this one is stored into the
nb_nonzero slot). new-style classes are types; old-style classes are
not (they're all instances of classobj).
The slots are populated when a new type is created (e.g., when
creating a new-style class) and are updated when a magic attribute is
set onto the type. By example, setting the __nonzero__ attribute on a
new-style class updates the nb_nonzero slot. old-style classes just
store the magic attribute in its __dict__. Note that if you patch Foo
*before* defining Baz, it works fine, because Baz sees the magic
attribute and can populate its nb_nonzero slot when the new type is
created.

When you define Baz, neither Foo nor Bar have a __nonzero__ at this
time, so the nb_nonzero slot on Baz is empty.
Later, when you alter Foo, Baz cannot notice it (Foo has no way to
notify Baz that something changed).

If Foo were a new-style class, things are different: new-style classes
maintain a list of subclasses, and the subclasses can then be notified
of changes. In particular, setting a magic attribute on a base class
notifies all its subclasses, and the corresponding slots are updated.

The problem seems to be exactly that: old-style base classes can't
notify its derived new-style classes when magic methods are added, so
the corresponding slots aren't updated. Always asking the base class
whether it has a magic method or not would slow down all method
lookups.

To force a slot update:

py> Baz.__nonzero__ = "xxx"
py> del Baz.__nonzero__
py> bool(b)
my_nonzero called
False

(setting or deleting __nonzero__ triggers the slot update)

This is *why* it happens. How to avoid this... well, don't do that in
the first place Or try to patch the base class *before* the new-
style derived class is defined. Or replace the derived class with a
new version of itself (once the base was patched). Or, if you know all
the derived classes, force a slot update on them as above.

 

[Carl Banks]

We are trying to monkey-patch a third-party library that mixes new and
old-style classes with multiple inheritance.

New library? Geez, if people are dumb enough to do this, are you sure
you want your application to depend on their library?

Sometimes you have to work with code that's not up to your standards,
but come on.


Carl Banks

 

[Carl Banks]

You've already discovered why--you're mixing old and new style
classes.

Monkey patching is definitely unpythonic. You must be a Ruby guy. Why
don't you try doing something else to get the behavior you want,
something more explicit?

Time and place.

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.


Carl Banks

 

[stephenpas]

Time and place.

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.

Carl Banks

Indeed, I chuckled at the idea I was a Ruby programmer. Of course
it's a bit of a hack but monkey-patching has been common place in the
Zope world for years (maybe Zope3 has rid itself of the problem).

Agreed, mixing new and old-style classes in multiple inheritance is
asking for trouble (although the Guido's essay on the new-style
classes suggests you can: http://www.python.org/download/releases/2.2.3/descrintro/#subclassing).
I won't name and shame the library involved but unfortunately we can't
always pick and choose the tools we work with unless we want to write
(or rewrite) everything ourselves.

Cheers,
Stephen.

 

[stephenpas]

Thank Gabriel,

That sounds exactly right and your work-around provides confirmation.
This code is an expedient solution for us so I expect we'll go with
forcing a slot update. Hopefully the library concerned is going to
fix the problem in a future release.

Cheers,
Stephen.

 

[Terry Reedy]

| Thank Gabriel,
|
| That sounds exactly right and your work-around provides confirmation.
| This code is an expedient solution for us so I expect we'll go with
| forcing a slot update. Hopefully the library concerned is going to
| fix the problem in a future release.

To work with 3.0, they will have to, or rather it will be done for them,
since there will only be new style classes.

 

[Michele Simionato]

A well-considered, timely monkey-patch can sometimes save all kinds of
workarounds and other headaches.

Carl Banks

+1

Michele Simionato

 

[George Sakkis]

New library? Geez, if people are dumb enough to do this, are you sure
you want your application to depend on their library?

Sometimes you have to work with code that's not up to your standards,
but come on.

Doing the armchair code reviewer without context is easy but my guess
would be that the old style library classes were written long before
the new style ones (perhaps even before the latter were introduced)
and/or written independently by different developers/groups, without
any plan to mix the two in the future. Also remember that up to 2.4
even the standard exception classes were old-style so it's not safe to
assume that you only have new style classes to worry about when the
very standard library includes lots of legacy code.

George

 

[Carl Banks]

Doing the armchair code reviewer without context is easy but my guess
would be that the old style library classes were written long before
the new style ones (perhaps even before the latter were introduced)
and/or written independently by different developers/groups, without
any plan to mix the two in the future. Also remember that up to 2.4
even the standard exception classes were old-style so it's not safe to
assume that you only have new style classes to worry about when the
very standard library includes lots of legacy code.

It's not the use of old-style classes, or even old-style and new-style
in the same package. It's multiple inheritance combining old- and new-
style classes that's so wrong here. They should have converted the
old-style to new when they decided to derive from both types (it's not
like that's terribly difficult).


Carl Banks

 

[George Sakkis]

They should have converted the
old-style to new when they decided to derive from both types (it's not
like that's terribly difficult).

Unless perhaps the old-style class is part of an stdlib or 3rd party
(or rather 4th party since "they" are the 3rd party here ;-)) module.
Even if it's technically possible and the change doesn't break other
things, I'd rather not have to maintain a patched version of the
stdlib.

George

 

[Bruno Desthuilliers]

Jonathan Gardner a écrit :

On Dec 18, 7:08 am, "(e-mail address removed)" (snip)
Monkey patching is definitely unpythonic. You must be a Ruby guy.

Strange enough, I learned monkey-patching with Python, years before I
first heard of a language named Ruby.

Why
don't you try doing something else to get the behavior you want,
something more explicit?

From experience, monkey-patching is sometimes the very best solution.

 

[Robert Kern]

Jonathan Gardner a écrit :

Strange enough, I learned monkey-patching with Python, years before I
first heard of a language named Ruby.

Indeed. I believe the term started at Zope Corp.

Ah, Wikipedia supports my recollection:

http://en.wikipedia.org/wiki/Monkey_patch

(And I swear that it did so before I checked it.)

It looks like some Ruby folks have taken to calling the practice "duck
punching", which has a certain charm of its own.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco