Python's super() considered super!

[Raymond Hettinger]

I just posted a tutorial and how-to guide for making effective use of
super().

One of the reviewers, David Beazley, said, "Wow, that's really
great! I see this becoming the definitive post on the subject"

The direct link is:

http://rhettinger.wordpress.com/2011/05/26/super-considered-super/

It would also be great if some of you would upvote it on HackerNews.


Raymond Hettinger

 

[Raymond Hettinger]

It would also be great if some of you would upvote it on HackerNews.


Here's a link to the super() how-to-guide and commentary: bit.ly/
iFm8g3

Raymod

 

[Dotan Cohen]

Here's a link to the super() how-to-guide and commentary:  bit.ly/
iFm8g3

Is that the same link as in the OP? I don't click on blind links, so
if it isn't then please post a direct link. Thanks.

 

[Ian Kelly]

Is that the same link as in the OP? I don't click on blind links, so
if it isn't then please post a direct link. Thanks.

It's a link to ycombinator:

http://news.ycombinator.com/item?id=2588262

 

[Dotan Cohen]

It's a link to ycombinator:

http://news.ycombinator.com/item?id=2588262

Thanks.

 

[Terry Reedy]

Is that the same link as in the OP? I don't click on blind links, so
if it isn't then please post a direct link. Thanks.

It is a link to HackerNews
http://news.ycombinator.com/item?id=2588262

 

[Raymond Hettinger]

We also, though, need *real* URLs. Blind URLs through obfuscation
services have their uses, but surely not in a forum like this. The real
URL is <URL:http://news.ycombinator.com/item?id=2588262>.

Fair enough. I had copied the link from Jesse's tweet (where shorter
is better).

Hope you enjoyed the post.


Raymond

 

[Steven D'Aprano]

I certainly did.

But I'm not better enlightened on why ‘super’ is a good thing.

Perhaps Raymond assumed that by now everybody knows that multiple
inheritance in Python that doesn't use super is buggy. super() was
introduced in version 2.2 in order to overcome bugs in MI, making it
about 8 years old now.

(Technically, it's only MI with diamond-shaped inheritance, but that
applies to *all* new-style classes. If you're writing multiple
inheritance in Python 3 without using super, your code is a land-mine
waiting to go off. If you're writing single inheritance, it's *still* a
land-mine, just waiting for some poor caller to use it in a MI context.)

But I do agree with you in that I expected to see at least some
discussion of why super should be actively preferred over calling the
parent class directly.

The
exquisite care that you describe programmers needing to maintain is IMO
just as much a deterrent as the super-is-harmful essay.

I don't see that Raymond describes anything needing "exquisite care".
It's more common sense really: ensure that your method signature and that
of your parents' match, plus good work-arounds for when they don't.
Anyone using inheritance is almost certainly 98% of the way there, unless
they're writing classes like this and wondering why they don't work

class MyBrokenList(list):
def __len__(self):
n = list.__len__(self, extra=42)
return n + 1
def __getitem__(self, i):
return list.__getitem__(self) + 1
def last_item(self):
return list.last_item(self) + 1


I was thrilled to learn a new trick, popping keyword arguments before
calling super, and wondered why I hadn't thought of that myself. How on
earth did I fail to realise that a kwarg dict was mutable and therefore
you can remove keyword args, or inject new ones in?

Given the plethora of articles that take a dim, if not outright negative,
view of super, it is good to see one that takes a positive view. Thanks
Raymond!

 

[sturlamolden]

I just posted a tutorial and how-to guide for making effective use of
super().

One of the reviewers, David Beazley, said, "Wow,  that's really
great!    I see this becoming the definitive post on the subject"

The direct link is:

 http://rhettinger.wordpress.com/2011/05/26/super-considered-super/

I really don't like the Python 2 syntax of super, as it violates
the DRY principle: Why do I need to write super(type(self),self)
when super() will do? Assuming that 'self' will always be named
'self' in my code, I tend to patch __builtins__.super like this:

import sys
def super():
self = sys._getframe().f_back.f_locals['self']
return __builtins__.super(type(self),self)

This way the nice Python 3.x syntax can be used in Python 2.x.


Sturla

 

[Mel]

I really don't like the Python 2 syntax of super, as it violates
the DRY principle: Why do I need to write super(type(self),self)
when super() will do? Assuming that 'self' will always be named
'self' in my code, I tend to patch __builtins__.super like this:

import sys
def super():
self = sys._getframe().f_back.f_locals['self']
return __builtins__.super(type(self),self)

This way the nice Python 3.x syntax can be used in Python 2.x.

Python causes trouble by letting the users get at the internals, but things
like this make it worthwhile.

Mel.

 

[Steven D'Aprano]

I really don't like the Python 2 syntax of super, as it violates the
DRY principle: Why do I need to write super(type(self),self) when
super() will do? Assuming that 'self' will always be named 'self' in my
code, I tend to patch __builtins__.super like this:

import sys
def super():
self = sys._getframe().f_back.f_locals['self']
return __builtins__.super(type(self),self)

This way the nice Python 3.x syntax can be used in Python 2.x.

Python causes trouble by letting the users get at the internals, but
things like this make it worthwhile.

Only if by "worthwhile" you mean "buggy as hell".

.... self = sys._getframe().f_back.f_locals['self']
.... return __builtins__.super(type(self),self)
........ def __init__(self):
.... super().__init__()
........ def __init__(self):
.... super().__init__()
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __init__
[...]
File "<stdin>", line 3, in __init__
RuntimeError: maximum recursion depth exceeded


Do not use super(type(self), self), because it does not do what you think
it does:

b = B() calls B.__init__(self)
.... which calls super(type(self), self) = super(B, self)
.... which calls A.__init__(self)
.... which calls super(type(self), self) = super(B, self) *not* A
.... which loops forever

type(self) does not return B inside B methods and A inside A methods, it
returns the class of the instance.

 

[harrismh777]

Only if by "worthwhile" you mean "buggy as hell".

I *don't* believe this... the king of metaphors and bogus analogies
has come up with 'buggy as hell' !!?

No no, you might have buggy as grubs-under-a-damp-log, or buggy as
'moths-around-an-incandecent-lamp' , or you could have 'hot-as-hell,'
but 'buggy-as-hell' just doesn't say what needs say'in...


.... I'm just saying....

 

[sturlamolden]

Assuming that 'self' will always be named
'self' in my code, I tend to patch __builtins__.super like this:

import sys
def super():
    self = sys._getframe().f_back.f_locals['self']
    return __builtins__.super(type(self),self)

A monkey-patch to __builtins__.super would probably also work.

Assuming the first argument to the callee is 'self' or 'cls':

import sys
_super = __builtins__.super
def monkeypatch(*args, **kwargs):
if (args == ()) and (kwargs=={}):
try:
obj = sys._getframe().f_back.f_locals['self']
except KeyError:
obj = sys._getframe().f_back.f_locals['cls']
return _super(type(obj),obj)
else:
return _super(*args, **kwargs)

class patcher(object):
def __init__(self):
__builtins__.super = monkeypatch
def __del__(self):
__builtins__.super = _super

_patch = patcher()



Sturla

 

[sturlamolden]

... infinite recursion follows ...

That's true

 

[sturlamolden]

Oops. There's a reason why Python 2 requires you to be explicit about
the class; you simply cannot work it out automatically at run time.
Python 3 fixes this by working it out at compile time, but for Python 2
there is no way around it.

Then it should be a keyword, not a function.

Sturla

 

[Steven D'Aprano]

Then it should be a keyword, not a function.

Why? The fault is not that super is a function, or that you monkey-
patched it, or that you used a private function to do that monkey-
patching. The fault was that you made a common, but silly, mistake when
reasoning about type(self) inside a class.

I made the same mistake: assume that type(self) will always be the same
class as that the method is defined in. But of course it won't be. With
the luxury of hindsight, it is a silly mistake to make, but I promise you
that you're not the first, and won't be the last, to make it.

 

[Ethan Furman]

Probably because most of the time it is better to avoid mutating kwargs.
Instead of popping an argument you simply declare it as a named argument in
the method signature. Instead of injecting new ones you can pass them as
named arguments.


def foo(x=None, **kwargs):
bar(y=2, **kwargs)


def bar(**kwargs):
print(kwargs)

Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
foo(x=1, y=2, z=3)
File "<pyshell#4>", line 2, in foo
bar(y=2, **kwargs)
TypeError: bar() got multiple values for keyword argument 'y'

And the above error is exactly why you don't want to use named arguments
in MI -- because you don't know in what order the methods will be
called, you cannot know which named arguments to supply to the method
that super() will call next.

~Ethan~

 

[Ian Kelly]

I just posted a tutorial and how-to guide for making effective use of
super().

I posted this already on the HackerNews thread but it seems to have
largely gone unnoticed, so I'm reposting it here.

It seems to me that the example of combining built-in dictionary
classes is naively optimistic. For starters, OrderedDict, as it
happens, does not use super! It calls the dict super-class methods
directly. Since dict luckily happens to be the next class in the MRO,
this doesn't really matter for the purpose of this example, but I can
envision a scenario where some plucky programmer inherits from both
OrderedCounter and some other dict subclass, and the result doesn't
work because OrderedDict does the wrong thing.

And OrderedDict isn't the only one. Maybe for some reason I would like
to have an OrderedCounter where all the counts default to 42. So I do
this:

class DefaultOrderedCounter(defaultdict, OrderedCounter):
pass
doc = DefaultOrderedCounter(lambda: 42)
doc.update('abracadabra')

Which results in:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "c:\python32\lib\collections.py", line 507, in update
_count_elements(self, iterable)
File "c:\python32\lib\collections.py", line 63, in __setitem__
self.__map[key] = link = Link()
AttributeError: 'DefaultOrderedCounter' object has no attribute
'_OrderedDict__map'

Whoops! Apparently defaultdict doesn't use super either. Of course a
better way to do this would be to subclass OrderedCounter and just
override the __missing__ method by hand, but that's not the point.

The article goes into "How to Incorporate a Non-cooperative Class",
which basically says "wrap it up in a proxy class". But that's not
really going to work here, since the result would be two separate
dicts, with the defaultdictwrapper methods operating on one dict, and
the other methods operating on the other.

 

[Chris Angelico]

It seems to me that the example of combining built-in dictionary
classes is naively optimistic.

So... Can anyone offer a non-trivial example of multiple inheritance
that *doesn't* have pitfalls? From what I've seen, MI always seems to
require cooperation from the authors of all involved classes. It may
be a useful tool when you control everything, but whenever you use
someone else's code, there seems to be this massive barrier of risk
(if that makes sense). For the DefaultOrderedCounter, I would be
strongly inclined to inherit singly, and then manually implement the
other half (whichever half is easier); in this case that happens to be
trivial (override __missing__), but even were it not, it would be a
means of being certain that things won't break.

Chris Angelico

 

[John Nagle]

So... Can anyone offer a non-trivial example of multiple inheritance
that *doesn't* have pitfalls? From what I've seen, MI always seems > to require cooperation from

the authors of all involved classes.

Good point.

Multiple inheritance is messy enough when the structure is just
a tree. When the structure is allowed to be a directed acyclic
graph, the whole thing just gets too complicated.

It doesn't even necessarily do what you want. If, say, you
have two classes that need dictionaries, and were implemented
by inheriting from "dict", a class that imports both has one
"dict", not two - something that was not anticipated in the
design of the original classes. That ought to be an error,
not a single "dict" shared by two unconnected classes.

What causes this kind of mess is a determination not to
report anything as an error if it can be given some kind of
meaningful semantics, even if the semantics have marginal
value. That's the kind of thinking which leads to

[1,2] * 2

returning

[1,2,1,2]



John Nagle