Python's super() considered super!

Discussion in 'Python' started by Raymond Hettinger, May 26, 2011.

  1. I just posted a tutorial and how-to guide for making effective use of
    super().

    One of the reviewers, David Beazley, said, "Wow, that's really
    great! I see this becoming the definitive post on the subject"

    The direct link is:

    http://rhettinger.wordpress.com/2011/05/26/super-considered-super/

    It would also be great if some of you would upvote it on HackerNews.


    Raymond Hettinger
    -----------------------
    follow my python tips on twitter: @raymondh
    Raymond Hettinger, May 26, 2011
    #1
    1. Advertising

  2. > It would also be great if some of you would upvote it on HackerNews.


    Here's a link to the super() how-to-guide and commentary: bit.ly/
    iFm8g3

    Raymod
    Raymond Hettinger, May 26, 2011
    #2
    1. Advertising

  3. Raymond Hettinger

    Dotan Cohen Guest

    On Thu, May 26, 2011 at 19:39, Raymond Hettinger <> wrote:
    >> It would also be great if some of you would upvote it on HackerNews.

    >
    >
    > Here's a link to the super() how-to-guide and commentary:  bit.ly/
    > iFm8g3
    >


    Is that the same link as in the OP? I don't click on blind links, so
    if it isn't then please post a direct link. Thanks.

    --
    Dotan Cohen

    http://gibberish.co.il
    http://what-is-what.com
    Dotan Cohen, May 26, 2011
    #3
  4. Raymond Hettinger

    Ian Kelly Guest

    On Thu, May 26, 2011 at 12:13 PM, Dotan Cohen <> wrote:
    > On Thu, May 26, 2011 at 19:39, Raymond Hettinger <> wrote:
    >>> It would also be great if some of you would upvote it on HackerNews.

    >>
    >>
    >> Here's a link to the super() how-to-guide and commentary:  bit.ly/
    >> iFm8g3
    >>

    >
    > Is that the same link as in the OP? I don't click on blind links, so
    > if it isn't then please post a direct link. Thanks.


    It's a link to ycombinator:

    http://news.ycombinator.com/item?id=2588262
    Ian Kelly, May 26, 2011
    #4
  5. Raymond Hettinger

    Dotan Cohen Guest

    Dotan Cohen, May 26, 2011
    #5
  6. Raymond Hettinger

    Terry Reedy Guest

    On 5/26/2011 2:13 PM, Dotan Cohen wrote:
    > On Thu, May 26, 2011 at 19:39, Raymond Hettinger<> wrote:
    >>> It would also be great if some of you would upvote it on HackerNews.

    >>
    >>
    >> Here's a link to the super() how-to-guide and commentary: bit.ly/
    >> iFm8g3
    >>

    >
    > Is that the same link as in the OP? I don't click on blind links, so
    > if it isn't then please post a direct link. Thanks.


    It is a link to HackerNews
    http://news.ycombinator.com/item?id=2588262

    --
    Terry Jan Reedy
    Terry Reedy, May 26, 2011
    #6
  7. On May 26, 6:39 pm, Ben Finney <> wrote:
    > We also, though, need *real* URLs. Blind URLs through obfuscation
    > services have their uses, but surely not in a forum like this. The real
    > URL is <URL:http://news.ycombinator.com/item?id=2588262>.


    Fair enough. I had copied the link from Jesse's tweet (where shorter
    is better).

    Hope you enjoyed the post.


    Raymond
    Raymond Hettinger, May 27, 2011
    #7
  8. On Fri, 27 May 2011 18:49:52 +1000, Ben Finney wrote:

    > Raymond Hettinger <> writes:
    >
    >> Hope you enjoyed the post.

    >
    > I certainly did.
    >
    > But I'm not better enlightened on why ‘super’ is a good thing.


    Perhaps Raymond assumed that by now everybody knows that multiple
    inheritance in Python that doesn't use super is buggy. super() was
    introduced in version 2.2 in order to overcome bugs in MI, making it
    about 8 years old now.

    (Technically, it's only MI with diamond-shaped inheritance, but that
    applies to *all* new-style classes. If you're writing multiple
    inheritance in Python 3 without using super, your code is a land-mine
    waiting to go off. If you're writing single inheritance, it's *still* a
    land-mine, just waiting for some poor caller to use it in a MI context.)

    But I do agree with you in that I expected to see at least some
    discussion of why super should be actively preferred over calling the
    parent class directly.


    > The
    > exquisite care that you describe programmers needing to maintain is IMO
    > just as much a deterrent as the super-is-harmful essay.


    I don't see that Raymond describes anything needing "exquisite care".
    It's more common sense really: ensure that your method signature and that
    of your parents' match, plus good work-arounds for when they don't.
    Anyone using inheritance is almost certainly 98% of the way there, unless
    they're writing classes like this and wondering why they don't work :)

    class MyBrokenList(list):
    def __len__(self):
    n = list.__len__(self, extra=42)
    return n + 1
    def __getitem__(self, i):
    return list.__getitem__(self) + 1
    def last_item(self):
    return list.last_item(self) + 1


    I was thrilled to learn a new trick, popping keyword arguments before
    calling super, and wondered why I hadn't thought of that myself. How on
    earth did I fail to realise that a kwarg dict was mutable and therefore
    you can remove keyword args, or inject new ones in?

    Given the plethora of articles that take a dim, if not outright negative,
    view of super, it is good to see one that takes a positive view. Thanks
    Raymond!



    --
    Steven
    Steven D'Aprano, May 27, 2011
    #8
  9. Raymond Hettinger

    sturlamolden Guest

    On 26 Mai, 18:31, Raymond Hettinger <> wrote:
    > I just posted a tutorial and how-to guide for making effective use of
    > super().
    >
    > One of the reviewers, David Beazley, said, "Wow,  that's really
    > great!    I see this becoming the definitive post on the subject"
    >
    > The direct link is:
    >
    >  http://rhettinger.wordpress.com/2011/05/26/super-considered-super/


    I really don't like the Python 2 syntax of super, as it violates
    the DRY principle: Why do I need to write super(type(self),self)
    when super() will do? Assuming that 'self' will always be named
    'self' in my code, I tend to patch __builtins__.super like this:

    import sys
    def super():
    self = sys._getframe().f_back.f_locals['self']
    return __builtins__.super(type(self),self)

    This way the nice Python 3.x syntax can be used in Python 2.x.


    Sturla
    sturlamolden, May 27, 2011
    #9
  10. Raymond Hettinger

    Mel Guest

    sturlamolden wrote:

    > I really don't like the Python 2 syntax of super, as it violates
    > the DRY principle: Why do I need to write super(type(self),self)
    > when super() will do? Assuming that 'self' will always be named
    > 'self' in my code, I tend to patch __builtins__.super like this:
    >
    > import sys
    > def super():
    > self = sys._getframe().f_back.f_locals['self']
    > return __builtins__.super(type(self),self)
    >
    > This way the nice Python 3.x syntax can be used in Python 2.x.


    Python causes trouble by letting the users get at the internals, but things
    like this make it worthwhile.

    Mel.
    Mel, May 27, 2011
    #10
  11. On Fri, 27 May 2011 10:33:20 -0400, Mel wrote:

    > sturlamolden wrote:
    >
    >> I really don't like the Python 2 syntax of super, as it violates the
    >> DRY principle: Why do I need to write super(type(self),self) when
    >> super() will do? Assuming that 'self' will always be named 'self' in my
    >> code, I tend to patch __builtins__.super like this:
    >>
    >> import sys
    >> def super():
    >> self = sys._getframe().f_back.f_locals['self']
    >> return __builtins__.super(type(self),self)
    >>
    >> This way the nice Python 3.x syntax can be used in Python 2.x.

    >
    > Python causes trouble by letting the users get at the internals, but
    > things like this make it worthwhile.


    Only if by "worthwhile" you mean "buggy as hell".


    >>> import sys
    >>>
    >>> def super():

    .... self = sys._getframe().f_back.f_locals['self']
    .... return __builtins__.super(type(self),self)
    ....
    >>> class A(object):

    .... def __init__(self):
    .... super().__init__()
    ....
    >>> class B(A):

    .... def __init__(self):
    .... super().__init__()
    ....
    >>> a = A()
    >>> b = B()

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 3, in __init__
    [...]
    File "<stdin>", line 3, in __init__
    RuntimeError: maximum recursion depth exceeded


    Do not use super(type(self), self), because it does not do what you think
    it does:

    b = B() calls B.__init__(self)
    .... which calls super(type(self), self) = super(B, self)
    .... which calls A.__init__(self)
    .... which calls super(type(self), self) = super(B, self) *not* A
    .... which loops forever

    type(self) does not return B inside B methods and A inside A methods, it
    returns the class of the instance.


    --
    Steven
    Steven D'Aprano, May 27, 2011
    #11
  12. Raymond Hettinger

    harrismh777 Guest

    Steven D'Aprano wrote:
    >> Python causes trouble by letting the users get at the internals, but
    >> > things like this make it worthwhile.


    > Only if by "worthwhile" you mean "buggy as hell".


    I *don't* believe this... the king of metaphors and bogus analogies
    has come up with 'buggy as hell' !!?

    No no, you might have buggy as grubs-under-a-damp-log, or buggy as
    'moths-around-an-incandecent-lamp' , or you could have 'hot-as-hell,'
    but 'buggy-as-hell' just doesn't say what needs say'in...


    .... I'm just saying....




    :)
    harrismh777, May 27, 2011
    #12
  13. Raymond Hettinger

    sturlamolden Guest

    On 27 Mai, 16:27, sturlamolden <> wrote:

    > Assuming that 'self' will always be named
    > 'self' in my code, I tend to patch __builtins__.super like this:
    >
    > import sys
    > def super():
    >     self = sys._getframe().f_back.f_locals['self']
    >     return __builtins__.super(type(self),self)



    A monkey-patch to __builtins__.super would probably also work.

    Assuming the first argument to the callee is 'self' or 'cls':

    import sys
    _super = __builtins__.super
    def monkeypatch(*args, **kwargs):
    if (args == ()) and (kwargs=={}):
    try:
    obj = sys._getframe().f_back.f_locals['self']
    except KeyError:
    obj = sys._getframe().f_back.f_locals['cls']
    return _super(type(obj),obj)
    else:
    return _super(*args, **kwargs)

    class patcher(object):
    def __init__(self):
    __builtins__.super = monkeypatch
    def __del__(self):
    __builtins__.super = _super

    _patch = patcher()



    Sturla
    sturlamolden, May 27, 2011
    #13
  14. Raymond Hettinger

    sturlamolden Guest

    On 27 Mai, 17:05, Duncan Booth <> wrote:

    > >>> class C(B): pass
    > >>> C().foo()

    >
    > ... infinite recursion follows ...


    That's true :(
    sturlamolden, May 27, 2011
    #14
  15. Raymond Hettinger

    sturlamolden Guest

    On 27 Mai, 17:05, Duncan Booth <> wrote:

    > Oops. There's a reason why Python 2 requires you to be explicit about
    > the class; you simply cannot work it out automatically at run time.
    > Python 3 fixes this by working it out at compile time, but for Python 2
    > there is no way around it.


    Then it should be a keyword, not a function.

    Sturla
    sturlamolden, May 27, 2011
    #15
  16. On Fri, 27 May 2011 08:31:40 -0700, sturlamolden wrote:

    > On 27 Mai, 17:05, Duncan Booth <> wrote:
    >
    >> Oops. There's a reason why Python 2 requires you to be explicit about
    >> the class; you simply cannot work it out automatically at run time.
    >> Python 3 fixes this by working it out at compile time, but for Python 2
    >> there is no way around it.

    >
    > Then it should be a keyword, not a function.


    Why? The fault is not that super is a function, or that you monkey-
    patched it, or that you used a private function to do that monkey-
    patching. The fault was that you made a common, but silly, mistake when
    reasoning about type(self) inside a class.

    I made the same mistake: assume that type(self) will always be the same
    class as that the method is defined in. But of course it won't be. With
    the luxury of hindsight, it is a silly mistake to make, but I promise you
    that you're not the first, and won't be the last, to make it.



    --
    Steven
    Steven D'Aprano, May 27, 2011
    #16
  17. Raymond Hettinger

    Ethan Furman Guest

    Duncan Booth wrote:
    > Steven D'Aprano <> wrote:
    >
    >> I was thrilled to learn a new trick, popping keyword arguments before
    >> calling super, and wondered why I hadn't thought of that myself. How on
    >> earth did I fail to realise that a kwarg dict was mutable and therefore
    >> you can remove keyword args, or inject new ones in?
    >>

    > Probably because most of the time it is better to avoid mutating kwargs.
    > Instead of popping an argument you simply declare it as a named argument in
    > the method signature. Instead of injecting new ones you can pass them as
    > named arguments.
    >
    >
    > def foo(x=None, **kwargs):
    > bar(y=2, **kwargs)
    >
    >
    > def bar(**kwargs):
    > print(kwargs)
    >
    >>>> foo(x=1, z=3)

    > {'y': 2, 'z': 3}
    >>>> foo(x=1, y=2, z=3)

    > Traceback (most recent call last):
    > File "<pyshell#8>", line 1, in <module>
    > foo(x=1, y=2, z=3)
    > File "<pyshell#4>", line 2, in foo
    > bar(y=2, **kwargs)
    > TypeError: bar() got multiple values for keyword argument 'y'


    And the above error is exactly why you don't want to use named arguments
    in MI -- because you don't know in what order the methods will be
    called, you cannot know which named arguments to supply to the method
    that super() will call next.

    ~Ethan~
    Ethan Furman, May 27, 2011
    #17
  18. Raymond Hettinger

    Ian Kelly Guest

    On Thu, May 26, 2011 at 10:31 AM, Raymond Hettinger <> wrote:
    > I just posted a tutorial and how-to guide for making effective use of
    > super().


    I posted this already on the HackerNews thread but it seems to have
    largely gone unnoticed, so I'm reposting it here.

    It seems to me that the example of combining built-in dictionary
    classes is naively optimistic. For starters, OrderedDict, as it
    happens, does not use super! It calls the dict super-class methods
    directly. Since dict luckily happens to be the next class in the MRO,
    this doesn't really matter for the purpose of this example, but I can
    envision a scenario where some plucky programmer inherits from both
    OrderedCounter and some other dict subclass, and the result doesn't
    work because OrderedDict does the wrong thing.

    And OrderedDict isn't the only one. Maybe for some reason I would like
    to have an OrderedCounter where all the counts default to 42. So I do
    this:

    class DefaultOrderedCounter(defaultdict, OrderedCounter):
    pass
    doc = DefaultOrderedCounter(lambda: 42)
    doc.update('abracadabra')

    Which results in:

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "c:\python32\lib\collections.py", line 507, in update
    _count_elements(self, iterable)
    File "c:\python32\lib\collections.py", line 63, in __setitem__
    self.__map[key] = link = Link()
    AttributeError: 'DefaultOrderedCounter' object has no attribute
    '_OrderedDict__map'

    Whoops! Apparently defaultdict doesn't use super either. Of course a
    better way to do this would be to subclass OrderedCounter and just
    override the __missing__ method by hand, but that's not the point.

    The article goes into "How to Incorporate a Non-cooperative Class",
    which basically says "wrap it up in a proxy class". But that's not
    really going to work here, since the result would be two separate
    dicts, with the defaultdictwrapper methods operating on one dict, and
    the other methods operating on the other.
    Ian Kelly, May 27, 2011
    #18
  19. On Sat, May 28, 2011 at 4:31 AM, Ian Kelly <> wrote:
    > It seems to me that the example of combining built-in dictionary
    > classes is naively optimistic.


    So... Can anyone offer a non-trivial example of multiple inheritance
    that *doesn't* have pitfalls? From what I've seen, MI always seems to
    require cooperation from the authors of all involved classes. It may
    be a useful tool when you control everything, but whenever you use
    someone else's code, there seems to be this massive barrier of risk
    (if that makes sense). For the DefaultOrderedCounter, I would be
    strongly inclined to inherit singly, and then manually implement the
    other half (whichever half is easier); in this case that happens to be
    trivial (override __missing__), but even were it not, it would be a
    means of being certain that things won't break.

    Chris Angelico
    Chris Angelico, May 27, 2011
    #19
  20. Raymond Hettinger

    John Nagle Guest

    On 5/27/2011 11:46 AM, Chris Angelico wrote:
    > On Sat, May 28, 2011 at 4:31 AM, Ian Kelly<> wrote:
    >> It seems to me that the example of combining built-in dictionary
    >> classes is naively optimistic.

    >
    > So... Can anyone offer a non-trivial example of multiple inheritance
    > that *doesn't* have pitfalls? From what I've seen, MI always seems > to require cooperation from

    the authors of all involved classes.

    Good point.

    Multiple inheritance is messy enough when the structure is just
    a tree. When the structure is allowed to be a directed acyclic
    graph, the whole thing just gets too complicated.

    It doesn't even necessarily do what you want. If, say, you
    have two classes that need dictionaries, and were implemented
    by inheriting from "dict", a class that imports both has one
    "dict", not two - something that was not anticipated in the
    design of the original classes. That ought to be an error,
    not a single "dict" shared by two unconnected classes.

    What causes this kind of mess is a determination not to
    report anything as an error if it can be given some kind of
    meaningful semantics, even if the semantics have marginal
    value. That's the kind of thinking which leads to

    [1,2] * 2

    returning

    [1,2,1,2]



    John Nagle
    John Nagle, May 27, 2011
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Guest

    super.super.super how?

    Guest, Feb 19, 2005, in forum: Java
    Replies:
    24
    Views:
    10,758
    Darryl Pierce
    Feb 24, 2005
  2. Fernando Rodriguez

    Getting the super class via the super() function

    Fernando Rodriguez, Nov 21, 2003, in forum: Python
    Replies:
    2
    Views:
    720
    Bob Willan
    Nov 22, 2003
  3. Kerim Borchaev

    super. could there be a simpler super?

    Kerim Borchaev, Jan 15, 2004, in forum: Python
    Replies:
    4
    Views:
    476
    Michele Simionato
    Jan 15, 2004
  4. Michele Simionato

    Re: Python's super() considered super!

    Michele Simionato, May 27, 2011, in forum: Python
    Replies:
    0
    Views:
    386
    Michele Simionato
    May 27, 2011
  5. Michele Simionato

    Re: Python's super() considered super!

    Michele Simionato, May 27, 2011, in forum: Python
    Replies:
    0
    Views:
    418
    Michele Simionato
    May 27, 2011
Loading...

Share This Page