Needed: Real-world examples for Python's Cooperative Multiple Inheritance

[Raymond Hettinger]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Google searches take me to old papers for C++ and Eiffel, but that
don't seem to be relevant to most Python programmers (i.e. a
WalkingMenu example where a submenu is both a Entry in a Menu and a
Menu itself). Another published example is in a graphic library where
some widgets inherit GraphicalFeature methods such as location, size
and NestingGroupingFeatures such as finding parents, siblings, and
children. I don't find either of those examples compelling because
there is no particular reason that they would have to have overlapping
method names.

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

If you guys know of good examples, I would appreciate a link or a
recap.

Thanks,


Raymond

 

[Antoine Pitrou]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Google searches take me to old papers for C++ and Eiffel, but that
don't seem to be relevant to most Python programmers (i.e. a
WalkingMenu example where a submenu is both a Entry in a Menu and a
Menu itself). Another published example is in a graphic library where
some widgets inherit GraphicalFeature methods such as location, size
and NestingGroupingFeatures such as finding parents, siblings, and
children. I don't find either of those examples compelling because
there is no particular reason that they would have to have overlapping
method names.

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

If you guys know of good examples, I would appreciate a link or a
recap.

I have never seen a good use of cooperative multiple inheritance in
Python. My own experience trying to use it suggests me that I would
have been better with independent "handler" classes (urllib2-style).

Regards

Antoine.

 

[Alice Bevan–McGregor]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

The SocketServer module
(http://docs.python.org/library/socketserver.html) uses cooperative
multiple inheritance to implement threading / async using a
ThreadingMixIn class and multi-processing using a ForkingMixIn class,
which may not be as complicated a use case as you are looking for.

One thing that caught me up was the attribute resolution order; it's a
FIFO, with the first superclass being examined preferentially over
later superclasses in the declaration. (Mixins go before the class
they extend.)

— Alice.

 

[Raymond Hettinger]

The SocketServer module
(http://docs.python.org/library/socketserver.html) uses cooperative
multiple inheritance to implement threading / async using a
ThreadingMixIn class and multi-processing using a ForkingMixIn class,
which may not be as complicated a use case as you are looking for.

It is a great example of a style of using mixins, but isn't
actually an example of Python's cooperative multiple inheritance.
The code doesn't use super() and doesn't have a diamond problem.

It's hard to write a best practices document for super() when
the doesn't appear to be any practice at all


Raymond

 

[Steve Holden]

It's hard to write a best practices document for super() when
the doesn't appear to be any practice at all

Sounds like the Python community have voted with their feet. I seem to
remember that Alex Martelli's "Python in a Nutshell" contains a very
thorough description of the new-style class system with an example
showing the resolution of the diamond pattern.

regards
Steve

 

[Aahz]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

My previous job used this rather heavily, but I can't provide you with
code. I suspect we weren't the only people, but I have no clue how to
locate samples.

Were you searching code.google.com or something else?

 

[Dennis Lee Bieber]

Sounds like the Python community have voted with their feet. I seem to
remember that Alex Martelli's "Python in a Nutshell" contains a very
thorough description of the new-style class system with an example
showing the resolution of the diamond pattern.

Or it may just be a case of real-world problems not really needing
it...

 

[Paul Rubin]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

I'd mention the SocketServer library, except I'm not sure what you
mean by "cooperative", so I don't know if that counts.

The few times I've tried to use multiple inheritance, I think it wasn't
the right idea for what I was doing. It turns into a big mess quite
easily.

The classic example though is a window system, where you have a "window"
class, and a "scroll bar" class, and a "drop-down menu" class, etc. and
if you want a window with a scroll bar and a drop-down menu, you inherit
from all three of those classes. They each have to support (e.g.) a
"redraw" operation, so automatic method combination figures out what to
call.

I''ve mostly seen this stuff in connection with Lisp. If you look at a
CLOS manual you can probably find examples. There are also some
examples in the old CADR manual, using the MIT Flavors system which
predated CLOS. That manual is online now:

http://common-lisp.net/project/bknr/static/lmman/frontpage.html

The object system is documented in this chapter:

http://common-lisp.net/project/bknr/static/lmman/flavor.xml

The paragraph about "defwrapper" gives a clear view into the machinery.

 

[John Nagle]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Google searches take me to old papers for C++ and Eiffel, but that
don't seem to be relevant to most Python programmers (i.e. a
WalkingMenu example where a submenu is both a Entry in a Menu and a
Menu itself). Another published example is in a graphic library where
some widgets inherit GraphicalFeature methods such as location, size
and NestingGroupingFeatures such as finding parents, siblings, and
children. I don't find either of those examples compelling because
there is no particular reason that they would have to have overlapping
method names.

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

If you guys know of good examples, I would appreciate a link or a
recap.

Multiple inheritance in Python is basically what fell out of
CPython's internals, not a design. It's one of those areas where
order of execution matters, and that wasn't well worked out.
Allowing classes to form a directed acyclic graph isn't very
useful. It just fell out of the semantics of a naive interpreter.

Read "C3 method resolution order":

http://www.python.org/download/releases/2.3/mro/

which is what you have to understand to use the awful cases properly.
Originally, the semantics were just wrong. Now the awful cases
have well-defined semantics, but aren't very useful.

Part of the problem is the notion that if a base class is duplicated
in the hierarchy, there's only one copy. So if you inherit from two
classes, both of which inherit from "dict", there will be only one
"dict" at the bottom. (I think.) This probably won't do what the
authors of any of the classes involved expected. The author
of the class which does the multiple inheritance might not even be
aware that there's a clash further up in the hierarchy. This is
one of those areas where all the code looks right locally, but it's
wrong globally.

Best practice for this is "don't do it." Some name clashes ought
to simply be detected as errors, rather than being given such
complex semantics.

John Nagle

 

[Mark Wooding]

Multiple inheritance in Python is basically what fell out of
CPython's internals, not a design. It's one of those areas where
order of execution matters, and that wasn't well worked out.

I'm not sure about the history, but this doesn't sound right to me.

Allowing classes to form a directed acyclic graph isn't very
useful.

This is simply wrong.

It just fell out of the semantics of a naive interpreter.

No: there's some essential work needed to make it happen.

Originally, the semantics were just wrong.
Agreed.

Now the awful cases have well-defined semantics, but aren't very
useful.

Disagree strongly. I think linearization is the only coherent approach
to multiple inheritance, and the C3 linearization seems to have almost
all of the necessary properties. I'm not quite sure what you mean by
`awful' here: the old Python linearization rules were wrong even for
very simple graphs (they failed to respect the superclass ordering
properly). The CLOS, Dylan and C3 linearizations agree on most commonly
occurring class graphs, including many graphs for which the old Python
orderings disagreed; the exceptions are where CLOS or Dylan failed to be
monotonic or to obey the extended precedence graph. The semantics are
extremely useful in the hands of a careful designer.

Part of the problem is the notion that if a base class is duplicated
in the hierarchy, there's only one copy.

This is a problem? No! Duplicating superclass state (as is done in C++
and, I believe, Eiffel) is incoherent.

So if you inherit from two classes, both of which inherit from "dict",
there will be only one "dict" at the bottom. (I think.)

Yes. You end up (probably) with an enhanced dictionary which supports
both protocols. This happens frequently, and is very useful.

This probably won't do what the authors of any of the classes involved
expected. The author of the class which does the multiple inheritance
might not even be aware that there's a clash further up in the
hierarchy. This is one of those areas where all the code looks right
locally, but it's wrong globally.

This is only likely if there's a misdesign -- probably using inheritance
where containership is required.

Best practice for this is "don't do it." Some name clashes ought
to simply be detected as errors, rather than being given such
complex semantics.

It sounds like you've been scarred by experiences with C++'s dementedly
inadequate object system, with its bizarre `repeated inheritance' rules
and hopelessly primitive manual (static!) method combination -- possibly
even to the extent of believing that they're in some way `right' or
`necessary'. I'm sorry. You have my pity.

Python's object system is different, and doesn't have those problems.
You probably need a viewpoint shift to be able to think about it
properly, but that won't happen if you continue to cling to the idea
that C++'s approach is anything like a model of how to do it right.

-- [mdw]

 

[Steven D'Aprano]

Part of the problem is the notion that if a base class is duplicated in
the hierarchy, there's only one copy.

Why is that a problem? I would expect it to be a problem if it showed up
twice.

So if you inherit from two
classes, both of which inherit from "dict", there will be only one
"dict" at the bottom. (I think.) This probably won't do what the
authors of any of the classes involved expected. The author of the
class which does the multiple inheritance might not even be aware that
there's a clash further up in the hierarchy. This is one of those areas
where all the code looks right locally, but it's wrong globally.

Why do you assume it is "wrong"? The whole point of the "complex
semantics" that you are complaining about is to ensure that multiple
inheritance does the right thing, rather than the wrong thing.

You simply can't safely inherit from arbitrary classes without
understanding them. This is equally true for single and multiple
inheritance, but in multiple inheritance there are more places for things
to go wrong. That is simply because the semantics of "inherit from
classes A and B" are more complex than the semantics of "inherit from
class A".

The problem is that people expect multiple inheritance to "just work"
mechanically, without giving any thought to the process. That it does
often just work is a credit to the "complex semantics" that you are
complaining about, but this lulls people into a false sense of security
and makes it more surprising when it doesn't work.

Best practice for this is "don't do it." Some name clashes ought
to simply be detected as errors, rather than being given such complex
semantics.

If that's the case, you would have to prohibit ALL multiple inheritance
from new-style classes, because all new-style classes inherit from object.

 

[Raymond Hettinger]

    Multiple inheritance in Python is basically what fell out of
CPython's internals, not a design.  

Sorry to disagree. That is historically inaccurate.
Guido designed super() on purpose. He took his cues from
"Putting Metaclasses to Work" by Ira Forman and Scott Danforth.

The method resolution order was based on yet another paper
which showed the defects in the two earlier designs for
multiple inheritance.

It may not be great, but it sure wasn't an accident.

    Best practice for this is "don't do it."  Some name clashes ought
to simply be detected as errors, rather than being given such
complex semantics.

That may well be true. If a coder has enough control over the classes
to be make sure they use super() in a way that supports cooperative
multiple inheritance, then they have enough control to just
rename the methods to prevent name diamond shaped name clashes.

OTOH, sometimes you don't have control over the names if they
are magic methods or standard names like close(), save(), flush(),
__init__(), etc.

Raymond

 

[Raymond Hettinger]

[Paul Rubin]

I'd mention the SocketServer library, except I'm not sure what you
mean by "cooperative", so I don't know if that counts.

Cooperative multiple inheritance is a specific problem when there
is a diamond diagram with the same method name needing to be called
on multiple paths and each name should only be called exactly once.

http://www.python.org/download/releases/2.2.3/descrintro/#cooperation


Raymond

 

[Michele Simionato]

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

I do not know of other use cases either.

Michele Simionato

 

[John Nagle]

That may well be true. If a coder has enough control over the classes
to be make sure they use super() in a way that supports cooperative
multiple inheritance, then they have enough control to just
rename the methods to prevent name diamond shaped name clashes.

OTOH, sometimes you don't have control over the names if they
are magic methods or standard names like close(), save(), flush(),
__init__(), etc.

I'd argue that a better implementation would require
that when there's a name clash, you have to specify the class
containing the name. In other words, if A is a subclass of B,
then B.foo() overrides A.foo(). But if C is a subclass of A and
B, and there's an A.foo() and a B.foo(), calling self.foo() in
C should be an error, because it's ambiguous. You should have
to specify the parent class, using "super".

The same issue applies in the other direction. If
A and B are subclasses of D, and A.foo() and B.foo() are
defined, and D calls self.foo(), that's an ambiguity and
should be reported. This catches the case where two
classed both inherit from, say "threading.thread", each
expecting to have a private thread. Someone then inherits
from both classes, getting totally unexpected results when
one of the "run" methods is chosen somewhat arbitrarily.

Detecting a clash requires machinery CPython's lookup
machinery currently lacks. That's why I wrote that the
current semantics fell out of the implementation.

Again, this is a form of global bug - each class can
appear internally correct, but the combination of them
won't work as expected. This is the sort of thing which
produces obscure bugs when the components come from different
sources. Python has relatively few dark corners like this,
but this is one of them. (Although, in practice, nobody
seems to use this stuff, so it may not matter.)

John Nagle

 

[Mark Wooding]

I'd argue that a better implementation would require that when there's
a name clash, you have to specify the class containing the name. In
other words, if A is a subclass of B, then B.foo() overrides
A.foo(). But if C is a subclass of A and B, and there's an A.foo() and
a B.foo(), calling self.foo() in C should be an error, because it's
ambiguous. You should have to specify the parent class, using "super".

How peculiar. `super' is /specifically/ for working out dynamically
which superclass to delegate behaviour to, because working it out
statically isn't possible in general.

The same issue applies in the other direction. If A and B are
subclasses of D, and A.foo() and B.foo() are defined, and D calls
self.foo(), that's an ambiguity and should be reported.

No it isn't. It's downward delegation, which is an essential feature of
object oriented design.

@This catches the case where two classed both inherit from, say
"threading.thread", each expecting to have a private thread.

Why on earth would anyone do such a bizarre thing? If you want a
private thread, then attach one as an attribute. Inheriting is simply
madness.

Someone then inherits from both classes, getting totally unexpected
results when one of the "run" methods is chosen somewhat arbitrarily.

But only because the original design was crazy. If you make a subclass
of `thread' it's because you wanted a more specific kind of thread. If
you subclass two other subclasses of `thread', it's because you wanted a
very specific kind of thread which included the behaviour of those other
two subclasses.

Detecting a clash requires machinery CPython's lookup machinery
currently lacks.

What you see as a `clash' is more likely deliberate factoring of
behaviour.

Again, you seem to think that C++ is, for some reason, the `one true
way'. It really isn't. Actually, it's a pretty poor way. But that's
not the point: the point is that there are other ways, and if you
stopped whining about how Python's object system might theoretically go
wrong if you try to use it like it was C++ and started thinking about
how to actually make effective use of the features it offers -- features
which long predate Python, and have been thought about over many years,
in languages such as Zetalisp, Common Lisp, Dylan, Scheme, and variants
of Smalltalk -- you might got on much better.

-- [mdw]

 

[John Nagle]

How peculiar. `super' is /specifically/ for working out dynamically
which superclass to delegate behaviour to, because working it out
statically isn't possible in general.


No it isn't. It's downward delegation, which is an essential feature of
object oriented design.


Why on earth would anyone do such a bizarre thing? If you want a
private thread, then attach one as an attribute. Inheriting is simply
madness.

This must be from someone who hasn't used threads in Python.

The usual way to write a thread in Python is to subclass
"threading.thread". The subclass provides a "run" function, which
will be called from the new thread.

If you
stopped whining about how Python's object system might theoretically go
wrong if you try to use it like it was C++ and started thinking about
how to actually make effective use of the features it offers -- features
which long predate Python, and have been thought about over many years,
in languages such as Zetalisp, Common Lisp, Dylan, Scheme, and variants
of Smalltalk -- you might got on much better.

Ah, fanboys.

Of those, I've written code in Common Lisp, Scheme, and Smalltalk.
Most of the LISP variants really did objects very well; objects were
an afterthought. Smalltalk went a bit too far in the other direction;
the "everything is an object" mindset was overdoing it.

Python is reasonably well balanced in the object area.
Everything isn't an object. There are explicit classes, unlike the
instance-copying model of Self and Javascript. However,
multiple inheritance is something of a mess, as the original
starter of this thread found when he tried to document how to use it.

If you have real trouble writing documentation for a feature, it's
usually because the feature is badly designed.

John Nagle

 

[Mark Wooding]

This must be from someone who hasn't used threads in Python.

Wrong again.

The usual way to write a thread in Python is to subclass
"threading.thread". The subclass provides a "run" function, which
will be called from the new thread.

Yes, it is. Does that conflict with what I wrote? No.

If you want you class to have a private thread, make /another/ class to
represent the behaviour of this private thread, and attach an instance
of this to the first class. Or you can pass a closure or a bound method
to the thread constructor. (That's probably cleaner, actually, but
doesn't fit culturally.)

Ah, fanboys.

Not exactly. I collect programming languages like some people collect
postage stamps; it gives one a useful perspective. I mentioned those
languages because they seem most historically relevant. Zetalisp's
`Flavors' system introduced multiple inheritance; Common Lisp and Dylan
fix the linearization properly (eventually culminating in the C3
linearization algorithm); Scheme has no standardized object system, but
there are a number of -- mainly CLOS-like -- object systems available;
and Smalltalk is both the classic dynamic object-oriented language and a
single-dispatch contrast to the CLOS/Dylan generic-functions approach.

Of those, I've written code in Common Lisp, Scheme, and Smalltalk.
Most of the LISP variants really did objects very well; objects were
an afterthought. Smalltalk went a bit too far in the other direction;
the "everything is an object" mindset was overdoing it.

CLOS actually does a remarkable integration job, bringing the existing
types into the object system; but, yes, the seams are still visible,
because you can't subclass some of the classes.

Python is reasonably well balanced in the object area. Everything
isn't an object. There are explicit classes, unlike the
instance-copying model of Self and Javascript.

Is that a major win? Self's prototype-based approach seems quite
capable of expressing anything you might want to express with classes,
and a few other things besides. (The implementation works by attempting
to deduce class structures dynamically, so there's an isomorphism here,
of a sort, but the dynamism would make inventing classes on the fly
rather inconvenient.) There's a significant difference between
Javascript and Self, by the way: a Self object can have multiple
`parent' slots, consequently with a form of multiple inheritance, while
Javascript is limited to single inheritance (unless you fake it up).

However, multiple inheritance is something of a mess, as the original
starter of this thread found when he tried to document how to use it.

Python's object system certainly isn't ideal.

The main problem that I see is that dynamic delegation with `super' is
hideously inconvenient to use, which means that programmers will tend
towards C++'s static delegation instead.

The Python object construction protocol (__new__ and __init__) is
somewhat simplistic; constructing instances of multiply inherited
classes in general requires a somewhat complicated dance with *args and
**kw arguments -- and you lose the ability to reject unknown arguments
-- which again leads to programmers taking shortcuts.

Most of the rest of the object system seems pretty sound to me.

If you have real trouble writing documentation for a feature, it's
usually because the feature is badly designed.

It might be that it was misunderstood.

There seem to be two obvious ways of learning a programming language.
One is to try and interpret its concepts in terms of concepts that you
already understand. This works, but you end up having to `translate'
between the new language; if the translation is imperfect then you'll be
confused or frustrated. There's a constant temptation to force one's
existing conceptual framework onto the new language -- to use it as if
it worked just like something else one is more familiar with that
doesn't quite work `right'. The new language is `broken Blub with funny
syntax'.

The other is to try to understand it on its own terms. This is the
harder road that leads to mastery.

-- [mdw]

 

[Steven D'Aprano]

On 11/26/2010 4:21 PM, Mark Wooding wrote: [...]

Why on earth would anyone do such a bizarre thing? If you want a
private thread, then attach one as an attribute. Inheriting is simply
madness.

This must be from someone who hasn't used threads in Python.

The usual way to write a thread in Python is to subclass
"threading.thread". The subclass provides a "run" function, which will
be called from the new thread.

Wanting a private thread, and writing a thread, are not the same. Of
course you have to subclass threading.thread to write a thread. But if
you want a private thread, there's no need to subclass thread just to get
one. You would add a thread as an attribute, exactly as you would add a
list or a dict or an int as an attribute if you wanted a private list or
dict or int. Trying to juggle access from *multiple* classes to a
*private* thread doesn't seem sensible, which was Mark's point.

Ah, fanboys.

Of those, I've written code in Common Lisp, Scheme, and Smalltalk.
Most of the LISP variants really did objects very well; objects were an
afterthought.

Describing something as an afterthought is normally meant as a
pejorative. To say that Lisp "really did objects very well" *and* that
they were afterthoughts seems like a contradiction.

Smalltalk went a bit too far in the other direction; the
"everything is an object" mindset was overdoing it.

Python is reasonably well balanced in the object area.
Everything isn't an object.

I don't understand what you mean by this. Can you give an example of a
thing in Python that is not an object?

There are explicit classes, unlike the
instance-copying model of Self and Javascript. However, multiple
inheritance is something of a mess, as the original starter of this
thread found when he tried to document how to use it.

If you have real trouble writing documentation for a feature, it's
usually because the feature is badly designed.

The problem isn't writing documentation for the feature, but coming up
with real-world use-cases. The documentation for super and the MRO is
extensive and detailed. It's also complicated, because multiple
inheritance is complicated. But it seems that multiple inheritance might
not be that useful outside of a small number of cases.

 

[Steve Holden]

On 11/27/2010 4:34 PM, Steven D'Aprano wrote:
[...]

The problem isn't writing documentation for the feature, but coming up
with real-world use-cases. The documentation for super and the MRO is
extensive and detailed. It's also complicated, because multiple
inheritance is complicated. But it seems that multiple inheritance might
not be that useful outside of a small number of cases.

It isn't. Even inheritance itself isn't as useful as it at first
appears, and composition turns out in practice to be much more useful.
That goes double for multiple inheritance.

regards
Steve