sum() requires number, not simply __add__

Discussion in 'Python' started by Buck Golemon, Feb 23, 2012.

  1. Buck Golemon

    Buck Golemon Guest

    I feel like the design of sum() is inconsistent with other language
    features of python. Often python doesn't require a specific type, only
    that the type implement certain methods.

    Given a class that implements __add__ why should sum() not be able to
    operate on that class?

    We can fix this in a backward-compatible way, I believe.

    Demonstration:
    I'd expect these two error messages to be identical, but they are
    not.

    >>> class C(object): pass
    >>> c = C()
    >>> sum((c,c))

    TypeError: unsupported operand type(s) for +: 'int' and 'C'
    >>> c + c

    TypeError: unsupported operand type(s) for +: 'C' and 'C'
    Buck Golemon, Feb 23, 2012
    #1
    1. Advertising

  2. Buck Golemon

    Buck Golemon Guest

    On Feb 23, 1:19 pm, Buck Golemon <> wrote:
    > I feel like the design of sum() is inconsistent with other language
    > features of python. Often python doesn't require a specific type, only
    > that the type implement certain methods.
    >
    > Given a class that implements __add__ why should sum() not be able to
    > operate on that class?
    >
    > We can fix this in a backward-compatible way, I believe.
    >
    > Demonstration:
    >     I'd expect these two error messages to be identical, but they are
    > not.
    >
    >      >>> class C(object): pass
    >      >>> c = C()
    >      >>> sum((c,c))
    >     TypeError: unsupported operand type(s) for +: 'int' and 'C'
    >     >>> c + c
    >     TypeError: unsupported operand type(s) for +: 'C' and 'C'


    Proposal:

    def sum(values,
    base=0):
    values =
    iter(values)

    try:
    result = values.next()
    except StopIteration:
    return base

    for value in values:
    result += value
    return result
    Buck Golemon, Feb 23, 2012
    #2
    1. Advertising

  3. On 23 February 2012 21:19, Buck Golemon <> wrote:
    > I feel like the design of sum() is inconsistent with other language
    > features of python. Often python doesn't require a specific type, only
    > that the type implement certain methods.
    >
    > Given a class that implements __add__ why should sum() not be able to
    > operate on that class?


    It can. You need to pass a second argument which will be the start
    value. Try help(sum) for details.

    --
    Arnaud
    Arnaud Delobelle, Feb 23, 2012
    #3
  4. Buck Golemon

    Chris Rebert Guest

    On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <> wrote:
    > I feel like the design of sum() is inconsistent with other language
    > features of python. Often python doesn't require a specific type, only
    > that the type implement certain methods.
    >
    > Given a class that implements __add__ why should sum() not be able to
    > operate on that class?


    The time machine strikes again! sum() already can. You just need to
    specify an appropriate initial value (the empty list in this example)
    for the accumulator :

    Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
    [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> sum([[1,2],[3,4]], [])

    [1, 2, 3, 4]

    Cheers,
    Chris
    --
    http://rebertia.com
    Chris Rebert, Feb 23, 2012
    #4
  5. Buck Golemon

    Buck Golemon Guest

    On Feb 23, 1:32 pm, Chris Rebert <> wrote:
    > On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <> wrote:
    > > I feel like the design of sum() is inconsistent with other language
    > > features of python. Often python doesn't require a specific type, only
    > > that the type implement certain methods.

    >
    > > Given a class that implements __add__ why should sum() not be able to
    > > operate on that class?

    >
    > The time machine strikes again! sum() already can. You just need to
    > specify an appropriate initial value (the empty list in this example)
    > for the accumulator :
    >
    > Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
    > [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
    > Type "help", "copyright", "credits" or "license" for more information.>>>sum([[1,2],[3,4]], [])
    >
    > [1, 2, 3, 4]
    >
    > Cheers,
    > Chris
    > --http://rebertia.com


    Thanks. I did not know that!

    My proposal is still *slightly* superior in two ways:

    1) It reduces the number of __add__ operations by one
    2) The second argument isn't strictly necessary, if you don't mind
    that the 'null sum' will produce zero.

    def sum(values, base=0):
    values = iter(values)

    try:
    result = values.next()
    except StopIteration:
    return base

    for value in values:
    result += value

    return result
    Buck Golemon, Feb 23, 2012
    #5
  6. On 23 February 2012 21:23, Buck Golemon <> wrote:
    > def sum(values,
    > base=0):
    >      values =
    > iter(values)
    >
    >      try:
    >          result = values.next()
    >      except StopIteration:
    >          return base
    >
    >      for value in values:
    >          result += value
    >      return result


    This is definitely not backward compatible. To get something that has
    a better chance of working with existing code, try this (untested):

    _sentinel = object()

    def sum(iterable, start=_sentinel):
    if start is _sentinel:
    iterable = iter(iterable)
    try:
    start = iterable.next()
    except StopIteration:
    return 0
    for x in iterable:
    start += x
    return start

    del _sentinel

    --
    Arnaud
    Arnaud Delobelle, Feb 23, 2012
    #6
  7. Chris Rebert, 23.02.2012 22:32:
    > On Thu, Feb 23, 2012 at 1:19 PM, Buck Golemon <> wrote:
    >> I feel like the design of sum() is inconsistent with other language
    >> features of python. Often python doesn't require a specific type, only
    >> that the type implement certain methods.
    >>
    >> Given a class that implements __add__ why should sum() not be able to
    >> operate on that class?

    >
    > The time machine strikes again! sum() already can. You just need to
    > specify an appropriate initial value (the empty list in this example)
    > for the accumulator :
    >
    > Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
    > [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
    > Type "help", "copyright", "credits" or "license" for more information.
    >>>> sum([[1,2],[3,4]], [])

    > [1, 2, 3, 4]


    I know that you just meant this as an example, but it's worth mentioning in
    this context that it's not exactly efficient to "sum up" lists this way
    because there is a lot of copying involved. Each adding of two lists
    creates a third one and copies all elements into it. So it eats a lot of
    time and space.

    Stefan
    Stefan Behnel, Feb 23, 2012
    #7
  8. On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <> wrote:
    > _sentinel = object()
    >
    > def sum(iterable, start=_sentinel):
    >    if start is _sentinel:
    >
    > del _sentinel


    Somewhat off-topic: Doesn't the if statement there do a lookup for a
    global, which would mean that 'del _sentinel' will cause it to fail?
    Or have I missed something here?

    ChrisA
    Chris Angelico, Feb 23, 2012
    #8
  9. Buck Golemon

    Ian Kelly Guest

    On Thu, Feb 23, 2012 at 2:38 PM, Buck Golemon <> wrote:
    > My proposal is still *slightly* superior in two ways:
    >
    > 1) It reduces the number of __add__ operations by one
    > 2) The second argument isn't strictly necessary, if you don't mind
    > that the 'null sum' will produce zero.


    It produces the wrong result, though:

    >>> sum([3,4], base=12)

    7

    If I'm starting with 12 and summing 3 and 4, I expect to get 19.

    Ideally the second argument should be ignored only if it isn't passed
    in at all, and I don't know off-hand why the built-in sum doesn't do
    this. We really don't need to replace it, though. If you want a
    different sum behavior, just write your own.

    def sum(iterable, *args):
    return reduce(operator.add, iterable, *args)

    >>> sum([3,4])

    7
    >>> sum([3,4], 12)

    19
    >>> sum(['hello', 'world'])

    'helloworld'

    Cheers,
    Ian
    Ian Kelly, Feb 23, 2012
    #9
  10. On 23 February 2012 21:53, Chris Angelico <> wrote:
    > On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <> wrote:
    >> _sentinel = object()
    >>
    >> def sum(iterable, start=_sentinel):
    >>    if start is _sentinel:
    >>
    >> del _sentinel

    >
    > Somewhat off-topic: Doesn't the if statement there do a lookup for a
    > global, which would mean that 'del _sentinel' will cause it to fail?
    > Or have I missed something here?


    Yes, you're right :) Change the signature to

    def sum(iterable, start=_sentinel, _sentinel=_sentinel):

    This is not pretty...

    --
    Arnaud
    Arnaud Delobelle, Feb 23, 2012
    #10
  11. Buck Golemon

    Ian Kelly Guest

    On Thu, Feb 23, 2012 at 2:53 PM, Chris Angelico <> wrote:
    > On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <> wrote:
    >> _sentinel = object()
    >>
    >> def sum(iterable, start=_sentinel):
    >>    if start is _sentinel:
    >>
    >> del _sentinel

    >
    > Somewhat off-topic: Doesn't the if statement there do a lookup for a
    > global, which would mean that 'del _sentinel' will cause it to fail?
    > Or have I missed something here?


    I believe you're correct. If you really want to delete the _sentinel
    reference though, you could do:

    def sum(iterable, start=object()):
    if start is sum.func_defaults[0]:
    ...

    Cheers,
    Ian
    Ian Kelly, Feb 23, 2012
    #11
  12. On Fri, Feb 24, 2012 at 8:59 AM, Arnaud Delobelle <> wrote:
    > def sum(iterable, start=_sentinel, _sentinel=_sentinel):


    Is this a reason for Python to introduce a new syntax, such as:

    def foo(blah, optional=del):
    if optional is del: print("No argument was provided")

    Basically, 'del' is treated like a unique non-providable object, only
    possible in an argument list and only if the argument was omitted. No
    more proliferation of individual sentinels... what do you think?

    (I picked "del" because it's an existing keyword. Fairly arbitrary
    choice though.)

    Chris Angelico
    Chris Angelico, Feb 23, 2012
    #12
  13. On Fri, 24 Feb 2012 08:53:49 +1100, Chris Angelico wrote:

    > On Fri, Feb 24, 2012 at 8:41 AM, Arnaud Delobelle <>
    > wrote:
    >> _sentinel = object()
    >>
    >> def sum(iterable, start=_sentinel):
    >>    if start is _sentinel:
    >>
    >> del _sentinel

    >
    > Somewhat off-topic: Doesn't the if statement there do a lookup for a
    > global, which would mean that 'del _sentinel' will cause it to fail? Or
    > have I missed something here?


    Yes, deleting _sentinel will cause the custom sum to fail, and yes, you
    have missed something.

    If the caller wants to mess with your library and break it, they have
    many, many ways to do so apart from deleting your private variables.


    del _sentinel
    _sentinel = "something else"
    sum.__defaults__ = (42,) # mess with the function defaults
    sum.__code__ = (lambda a, b=None: 100).__code__ # and with func internals
    sum = None # change your custom sum to something else
    del sum # or just delete it completely
    len = 42 # shadow a built-in
    import builtins; del builtins.range # really screw with you


    If your application stops working after you carelessly mess with
    components your application relies on, the right answer is usually:

    "Don't do that then."

    Python doesn't try to prevent people from shooting themselves in the foot.


    Monkey-patching-by-actual-monkeys-for-fun-and-profit-ly y'rs,


    --
    Steven
    Steven D'Aprano, Feb 23, 2012
    #13
  14. On Fri, Feb 24, 2012 at 10:33 AM, Steven D'Aprano
    <> wrote:
    > Yes, deleting _sentinel will cause the custom sum to fail, and yes, you
    > have missed something.
    >
    > If the caller wants to mess with your library and break it, they have
    > many, many ways to do so apart from deleting your private variables.


    I was looking at the module breaking itself, though, not even waiting
    for the caller to do it.

    ChrisA
    Chris Angelico, Feb 23, 2012
    #14
  15. Buck Golemon

    Peter Otten Guest

    Buck Golemon wrote:

    > I feel like the design of sum() is inconsistent with other language
    > features of python. Often python doesn't require a specific type, only
    > that the type implement certain methods.
    >
    > Given a class that implements __add__ why should sum() not be able to
    > operate on that class?
    >
    > We can fix this in a backward-compatible way, I believe.
    >
    > Demonstration:
    > I'd expect these two error messages to be identical, but they are
    > not.
    >
    > >>> class C(object): pass
    > >>> c = C()
    > >>> sum((c,c))

    > TypeError: unsupported operand type(s) for +: 'int' and 'C'
    > >>> c + c

    > TypeError: unsupported operand type(s) for +: 'C' and 'C'


    You could explicitly provide a null object:

    >>> class Null(object):

    .... def __add__(self, other):
    .... return other
    ....
    >>> null = Null()
    >>> class A(object):

    .... def __init__(self, v):
    .... self.v = v
    .... def __add__(self, other):
    .... return A("%s+%s" % (self, other))
    .... def __str__(self):
    .... return self.v
    .... def __repr__(self):
    .... return "A(%r)" % self. v
    ....
    >>> sum(map(A, "abc"))

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: unsupported operand type(s) for +: 'int' and 'A'
    >>> sum(map(A, "abc"), null)

    A('a+b+c')
    Peter Otten, Feb 24, 2012
    #15
  16. On 02/24/2012 12:33 AM, Steven D'Aprano wrote:
    > If your application stops working after you carelessly mess with
    > components your application relies on, the right answer is usually:
    >
    > "Don't do that then."
    >
    > Python doesn't try to prevent people from shooting themselves in the foot.
    >

    Yes it does! A simple example is None as a keyword to prevent
    assignments to it.

    --
    Antoon Pardon
    Antoon Pardon, Feb 24, 2012
    #16
  17. Buck Golemon

    Roy Smith Guest

    In article <>,
    Antoon Pardon <> wrote:

    > > Python doesn't try to prevent people from shooting themselves in the foot.
    > >

    > Yes it does! A simple example is None as a keyword to prevent
    > assignments to it.


    Hmmm. Just playing around with some bizarre things to do with None, and
    discovered this:

    >>> import sys as None


    doesn't give an error, but also doesn't assign the module to the symbol
    'None'. Weird.
    Roy Smith, Feb 24, 2012
    #17
  18. Buck Golemon

    Terry Reedy Guest

    On 2/24/2012 8:23 AM, Roy Smith wrote:
    > In article<>,
    > Antoon Pardon<> wrote:
    >
    >>> Python doesn't try to prevent people from shooting themselves in the foot.
    >>>

    >> Yes it does! A simple example is None as a keyword to prevent
    >> assignments to it.

    >
    > Hmmm. Just playing around with some bizarre things to do with None, and
    > discovered this:
    >
    >>>> import sys as None

    >
    > doesn't give an error, but also doesn't assign the module to the symbol
    > 'None'. Weird.


    In 3.2
    >>> import sys as None

    SyntaxError: invalid syntax

    --
    Terry Jan Reedy
    Terry Reedy, Feb 24, 2012
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Weston Weems
    Replies:
    2
    Views:
    432
    Weston Weems
    Sep 3, 2004
  2. Ronny Mandal

    Difference: __iadd__ and __add__

    Ronny Mandal, Feb 17, 2006, in forum: Python
    Replies:
    3
    Views:
    316
    Peter Hansen
    Feb 18, 2006
  3. Siwat Saibua
    Replies:
    1
    Views:
    838
    Siwat Saibua
    Sep 21, 2007
  4. Magnus Schuster

    proxy class and __add__ method

    Magnus Schuster, Jul 29, 2008, in forum: Python
    Replies:
    0
    Views:
    241
    Magnus Schuster
    Jul 29, 2008
  5. Ðндрей Симурзин

    Don't work __getattr__ with __add__

    Ðндрей Симурзин, Mar 4, 2010, in forum: Python
    Replies:
    2
    Views:
    346
    Andrey Simurzin
    Mar 4, 2010
Loading...

Share This Page