Needed: Real-world examples for Python's Cooperative Multiple Inheritance

[coldpizza]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Google searches take me to old papers for C++ and Eiffel, but that
don't seem to be relevant to most Python programmers (i.e. a
WalkingMenu example where a submenu is both a Entry in a Menu and a
Menu itself).  Another published example is in a graphic library where
some widgets inherit GraphicalFeature methods such as location, size
and NestingGroupingFeatures such as finding parents, siblings, and
children.  I don't find either of those examples compelling because
there is no particular reason that they would have to have overlapping
method names.

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

If you guys know of good examples, I would appreciate a link or a
recap.

Thanks,

Raymond

Did you try google code search? It is *not* the same as google code
hosting.
The site is http://www.google.com/codesearch and you can select Python
in the 'language' dropdown.

 

[Mark Wooding]

It isn't. Even inheritance itself isn't as useful as it at first
appears, and composition turns out in practice to be much more useful.
That goes double for multiple inheritance.

Composition /with a convenient notation for delegation/ works fairly
well. Indeed, this can be seen as the basis of Self. But downwards
delegation -- where a superclass leaves part of its behaviour
unspecified and requires (concrete) subclasses to fill in the resulting
blanks -- is hard to express like this without some kind of means of
identifying the original recipient of the delegated message. Once
you've done that, there isn't much of a difference between a superclass
and a component with implicit delegation.

-- [mdw]

 

[Kirill Simonov]

Hi Raymond,

We've been using cooperative inheritance to implement stacked utilities
such as WSGI middleware or connecting to a database.

An example of a WSGI middleware stack:

# Declare the interface and provide the default implementation.
class WSGI(Utility):
def __call__(self, environ, start_response):
# The main WSGI application is implemented here.
start_response("200 OK", [('Content-Type', 'text/plain')])
return ["Hello World!"]

# GZip middleware (may be defined in a different module or a plugin)
class GZIP(WSGI):
# To indicate relative position in the middleware stack
weights(100)
def __call__(self, environ, start_response):
# Call the next middleware in the stack to generate data.
# Also, need to wrap start_response here...
generator = super(GZIP, self).__call__(environ, start_response)
# Pack the output...

# Error handling middleware (defined in a different module or a plugin)
class LogErrors(WSGI):
weights(1000)
def __call__(self, environ, start_response):
# Call the next middleware in the stack, catch any errors.
try:
generator = super(LogErrors, self).__call__(environ,
start_response)
except:
# Log errors...

# Now glue them all together
def wsgi(environ, start_response):
wsgi = WSGI() # !!!
return wsgi(environ, start_response)

The trick here is that the constructor of `WSGI` (actually,
`Utility.__new__()`) is overridden. Instead of producing a new instance
of `WSGI` , it does the following:
- use `__subclasses__()` to find all components of the utility;
- order the components by their weights;
- create a new class: `type(name, list_of_components, {})`;
- return an instance of the class.

Here is another example, database connection.

# The interface, no default implementation.
class Connect(Utility):
def __init__(self, host, port, user, password, database):
self.host = host
self.port = port
self.user = user
self.password = password
self.database = database
def __call__(self):
raise NotImplementedError()

# Public API
def connect(host, port, user, password, database):
# Same trick here.
connect = Connect(host, port, user, password, database)
return connect()

# PostgreSQL implementation (defined in a plugin)
import psycopg2
class PGSQLConnect(Connect):
def __call__(self):
return psycopg2.connect(...)

# Connection pooling (defined in a plugin)
class Pooling(Connect):
weights(100)
def __call__(self):
# Check if we could reuse an existing connection
# ...
# If no free connections available
connection = super(Pooling, self).__call__()
# Save it somewhere and return it...

Note that utility instances are short-lived so any persistent state must
be kept elsewhere.


We also use the same pattern to implement Cecil/Diesel-style
multimethods and general predicate dispatch, but that's probably outside
the scope of your question.

A public version of the code lives here:
https://bitbucket.org/prometheus/htsql
Unfortunately it doesn't exactly match my examples above: connection
pooling and most of the wsgi middleware are still to be ported,
`weights()` is missing, etc.


Hope it helps.

Thanks,
Kirill

 

[André Malo]

Even inheritance itself isn't as useful as it at first
appears, and composition turns out in practice to be much more useful..
That goes double for multiple inheritance.

Amen.

nd

 

[Raymond Hettinger]

Did you try google code search? It is *not* the same as google code
hosting.
The site ishttp://www.google.com/codesearchand you can select Python
in the 'language' dropdown.

Yes, I use Google's code search frequently and did try it for super().
However, you still need to drill into many of the hits manually,
because it is difficult to disambiguate a single inheritance use
of super() (which is very common) from a design with cooperative
multiple inheritance. You have to read a lot of code and can still
come
up empty handed.


Raymond

 

[Michele Simionato]

It isn't. Even inheritance itself isn't as useful as it at first
appears, and composition turns out in practice to be much more useful.
That goes double for multiple inheritance.

Composition /with a convenient notation for delegation/ works fairly
well.  Indeed, this can be seen as the basis of Self.  But downwards
delegation -- where a superclass leaves part of its behaviour
unspecified and requires (concrete) subclasses to fill in the resulting
blanks -- is hard to express like this without some kind of means of
identifying the original recipient of the delegated message.  Once
you've done that, there isn't much of a difference between a superclass
and a component with implicit delegation.

-- [mdw]

For a long time I had the feeling that in a language with pattern
matching inheritance (both single and double) is basically useless.
You can easily define objects as functions responding to messages and
classes becomes useless. However I have never implemented a large
project with such techniques, so I
am not sure how much my gut feeling is sound. Apparently here at work
we are going to use Erlang in the near future and I hope to get my
hand dirty and see in practice how well one can work with a language
without inheritance. BTW, is there anybody here with experience on
such languages and caring to share his learned lessons?

Michele Simionato

 

[Kirill Simonov]

Hi Raymond,

Another example: extensions in Mercurial. Mercurial is a VCS with a
typical command line syntax:

$ hg <command> <command arguments and options>

Mercurial has an extension mechanism for adding new and modifying
existing commands. A big chunk of Mercurial functionality is
implemented in `ui` and `repo` classes and extensions often patch those
to override the default behavior. For instance, you could check the
`color` extension, which patches `ui` to override `write*` methods:

http://selenic.com/hg/file/3790452d499b/hgext/color.py#l152


Thanks,
Kirill

 

[Giampaolo Rodolà]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Google searches take me to old papers for C++ and Eiffel, but that
don't seem to be relevant to most Python programmers (i.e. a
WalkingMenu example where a submenu is both a Entry in a Menu and a
Menu itself).  Another published example is in a graphic library where
some widgets inherit GraphicalFeature methods such as location, size
and NestingGroupingFeatures such as finding parents, siblings, and
children.  I don't find either of those examples compelling because
there is no particular reason that they would have to have overlapping
method names.

So far, the only situation I can find where method names necessarily
overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

If you guys know of good examples, I would appreciate a link or a
recap.

Thanks,


Raymond

In pyftpdlib I used multiple inheritance to implement FTP over SSL:
http://code.google.com/p/pyftpdlib/source/browse/tags/release-0.5.2/demo/tls_ftpd.py#125

I can't say it's been natural/intuitive though, and it took me a while
to make it work properly.
super() is one of the few things in Python I really don't understand properly.


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/

 

[Gregory Ewing]

The classic example though is a window system, where you have a "window"
class, and a "scroll bar" class, and a "drop-down menu" class, etc. and
if you want a window with a scroll bar and a drop-down menu, you inherit
from all three of those classes.

Not in any GUI library I've ever seen. Normally there would
be three objects involved in such an arrangement, a Window,
a ScrollBar and a DropDownMenu, connected to each other in
some way.

 

[Daniel Urban]

So far, the only situation I can find where method names necessarily

overlap is for the basics like __init__(), close(), flush(), and
save() where multiple parents need to have their own initialization
and finalization.

One other possibility is subclasses of the JSONEncoder class. For
example the writer of class X provides a class, XJSONEncoder, which is
able to serialize X instances (by extending the default method).
Similarly there is a YJSONEncoder class, which can serialize Y
instances. Those classes implement the default method like this:

def default(self, o):
if isinstance(o, X):
... # serialize the X instance
else:
return super().default(o) # let the next in the MRO try to handle it

If YJSONEncoder encodes Y instances similarly, one can create an
encoder class, which can encode both X and Y instances:

class XYJSONEncoder(XJSONEncoder, YJSONEncoder): pass

It is usable this way:
json.dumps([X(), Y()], cls=XYJSONEncoder)


Regards,
Daniel

 

[Ethan Furman]

I'm writing-up more guidance on how to use super() and would like to
point at some real-world Python examples of cooperative multiple
inheritance.

Don't know if you are still looking for examples, but I recently came
across a thread in Python-Dev which had an example using unittest:

Ricardo Kirkner wrote [much snippage]:

I'll give you the example I came upon:

I have a TestCase class, which inherits from both Django's TestCase
and from some custom TestCases that act as mixin classes. So I have
something like

class MyTestCase(TestCase, Mixin1, Mixin2):
...

Since I explicitely base off 3 classes, I expected all 3
classes to be initialized, and I expect the setUp method to be called
on all of them.

As written this example failed because TestCase (from django) was based
on unittest2.TestCase, which was not calling super. I understand,
however, that if the mixins were listed before TestCase that it would work.

Hope this helps.

~Ethan~

 

[John Nagle]

Multiple inheritance in Python is so badly designed that it
probably should not be used in production code.

Generalizing multiple inheritance from a tree to a directed
acyclic graph is usually a mistake.

John Nagle