reduce to be removed?

Discussion in 'Python' started by Dustan, Nov 11, 2006.

  1. Dustan

    Dustan Guest

    According to the following page on Wikipedia:
    http://en.wikipedia.org/wiki/Python_(programming_language)#Future_development
    reduce is going to be removed in python 3.0. It talks of an
    accumulation loop; I have no idea what that's supposed to mean. So,

    ===============================
    >>> x =\

    [[1,2,3],
    [4,5,6],
    [7,8,9]]
    >>> reduce(lambda a,b:a+b, x, [])

    [1, 2, 3, 4, 5, 6, 7, 8, 9]
    ===============================

    What's an accumulation loop, and how would I convert this code so it's
    compatible with the future 3.0 (preferably in a short sweet expression
    that I can embed in a list comprehension)?
    Dustan, Nov 11, 2006
    #1
    1. Advertising

  2. Dustan wrote:

    > What's an accumulation loop, and how would I convert this code so it's
    > compatible with the future 3.0


    the release of Python 3.0 is far away, and nobody knows how it's going
    to look. trying to be future-compatible at this time is a major waste
    of time and (not quite as wasteful as reopening yet another old "let's
    make some pointless change to the language" thread, but almost).

    surely you must have something better to do with your time ?

    </F>
    Fredrik Lundh, Nov 11, 2006
    #2
    1. Advertising

  3. Dustan

    Dustan Guest

    Fredrik Lundh wrote:
    > Dustan wrote:
    >
    > > What's an accumulation loop, and how would I convert this code so it's
    > > compatible with the future 3.0

    >
    > the release of Python 3.0 is far away, and nobody knows how it's going
    > to look. trying to be future-compatible at this time is a major waste
    > of time and (not quite as wasteful as reopening yet another old "let's
    > make some pointless change to the language" thread, but almost).
    >
    > surely you must have something better to do with your time ?
    >
    > </F>


    It's always nice to know there are such good-natured people ready to
    help on this group. Anyway, I figured out a way to get the builtin
    function 'sum' to work as I need:
    sum([[1,2,3],[4,5,6],[7,8,9]], [])

    On an unrelated note, can anyone explain this unpredictable behavior on
    IDLE? I swear I never hit ctrl-c...
    ==============================
    >>> help(sum)

    Help on built-in function sum in module __builtin__:

    sum(...)
    sum(sequence, start=0) -> value

    Returns the sum of a sequence of numbers (NOT strings) plus the
    value
    of parameter 'start'. When the sequence is empty, returns start.
    Traceback (most recent call last):
    File "<pyshell#0>", line 1, in <module>
    help(sum)
    File "C:\Python25\lib\site.py", line 346, in __call__
    return pydoc.help(*args, **kwds)
    File "C:\Python25\lib\pydoc.py", line 1642, in __call__
    self.help(request)
    File "C:\Python25\lib\pydoc.py", line 1687, in help
    self.output.write('\n')
    File "C:\Python25\lib\idlelib\PyShell.py", line 1246, in write
    self.shell.write(s, self.tags)
    File "C:\Python25\lib\idlelib\PyShell.py", line 1235, in write
    raise KeyboardInterrupt
    KeyboardInterrupt
    ==============================
    Dustan, Nov 11, 2006
    #3
  4. Dustan wrote:
    > According to the following page on Wikipedia:
    > http://en.wikipedia.org/wiki/Python_(programming_language)#Future_development
    > reduce is going to be removed in python 3.0. It talks of an
    > accumulation loop; I have no idea what that's supposed to mean. So,
    >
    > ===============================
    > >>> x =\

    > [[1,2,3],
    > [4,5,6],
    > [7,8,9]]
    > >>> reduce(lambda a,b:a+b, x, [])

    > [1, 2, 3, 4, 5, 6, 7, 8, 9]
    > ===============================
    >
    > What's an accumulation loop, and how would I convert this code so it's
    > compatible with the future 3.0 (preferably in a short sweet expression
    > that I can embed in a list comprehension)?


    itertools.chain or sum(x,[])
    Virgil Dupras, Nov 11, 2006
    #4
  5. Dustan wrote:

    > It's always nice to know there are such good-natured people ready to
    > help on this group.


    any special reason why you keep pretending that some random wikipedia
    editor knows more about a future Python release than the people that
    develops Python ?

    > Anyway, I figured out a way to get the builtin
    > function 'sum' to work as I need:
    > sum([[1,2,3],[4,5,6],[7,8,9]], [])


    sum() is designed for adding numbers, not sequences. abusing it
    for sequences leads to inefficient code, and extremely bad worst-
    case behaviour, since you end up copying the same data over and
    over and over again -- the function even checks for strings for
    this very reason:

    >>> sum(["123", "456", "789"], "")

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    TypeError: sum() can't sum strings [use ''.join(seq) instead]

    (maybe it should check for other well-known containers as well?)

    if you care about writing robust code, why not just use a for-loop,
    and the list extend method?

    </F>
    Fredrik Lundh, Nov 11, 2006
    #5
  6. Dustan

    Dustan Guest

    Fredrik Lundh wrote:
    > Dustan wrote:
    >
    > > It's always nice to know there are such good-natured people ready to
    > > help on this group.

    >
    > any special reason why you keep pretending that some random wikipedia
    > editor knows more about a future Python release than the people that
    > develops Python ?


    Be careful how you word that - this is the first time I've ever
    referenced wikipedia in a question on this forum.

    > > Anyway, I figured out a way to get the builtin
    > > function 'sum' to work as I need:
    > > sum([[1,2,3],[4,5,6],[7,8,9]], [])

    >
    > sum() is designed for adding numbers, not sequences. abusing it
    > for sequences leads to inefficient code, and extremely bad worst-
    > case behaviour, since you end up copying the same data over and
    > over and over again


    Thanks for some quality feedback for a change. Why can't you do that
    more often?

    > -- the function even checks for strings for
    > this very reason:
    >
    > >>> sum(["123", "456", "789"], "")

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > TypeError: sum() can't sum strings [use ''.join(seq) instead]
    >
    > (maybe it should check for other well-known containers as well?)
    >
    > if you care about writing robust code, why not just use a for-loop,
    > and the list extend method?


    Because I'm embedding this expression in a list comprehension (as I
    stated in my original post), and last time I checked, it's not possible
    to treat a for-loop as an expression (which is what a list
    comprehension requires).

    > </F>
    Dustan, Nov 11, 2006
    #6
  7. Dustan wrote:

    > Because I'm embedding this expression in a list comprehension


    because?

    </F>
    Fredrik Lundh, Nov 11, 2006
    #7
  8. Dustan

    Dustan Guest

    Fredrik Lundh wrote:
    > Dustan wrote:
    >
    > > Because I'm embedding this expression in a list comprehension

    >
    > because?
    >
    > </F>


    Because I thought I would be able to get an answer without revealing
    the exact details of what I am doing. I didn't realize that wasn't an
    option. I'll try once more to give you an idea of what I'm trying to
    accomplish without letting on the details.

    >>> foo =\

    [[[1,2,3],[4,5,6],[7,8,9]],
    [[3,2,1],[6,5,4],[9,8,7]]]

    Here, foo appears to be a 3-dimensional list - except it's supposed to
    be 2-dimensional. The inner-list-of-lists is a result of how I'm
    producing the data, and now I want to do a mass-concatenation (or
    extending) of the inner-list-of-lists, and come up with this:

    >>> foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

    True

    What's the best way to accomplish this?

    It's not quite this simple, but let's just see what you can come up
    with the information at hand, and I'll see if I can adapt it to my
    needs.
    Dustan, Nov 12, 2006
    #8
  9. Dustan wrote:

    >> > Because I'm embedding this expression in a list comprehension

    >>
    >> because?

    >
    > Because I thought I would be able to get an answer without revealing
    > the exact details of what I am doing.


    alright, let's try again: why do you need a self-contained reduce
    replacement that can be embedded inside a list comprehension ?

    </F>
    Fredrik Lundh, Nov 12, 2006
    #9
  10. Dustan

    Dustan Guest

    Fredrik Lundh wrote:
    > Dustan wrote:
    >
    > >> > Because I'm embedding this expression in a list comprehension
    > >>
    > >> because?

    > >
    > > Because I thought I would be able to get an answer without revealing
    > > the exact details of what I am doing.

    >
    > alright, let's try again: why do you need a self-contained reduce
    > replacement that can be embedded inside a list comprehension ?
    >
    > </F>



    >>> foo =\

    [[[1,2,3],[4,5,6],[7,8,9]],
    [[3,2,1],[6,5,4],[9,8,7]]]

    Here, foo appears to be a 3-dimensional list - except it's supposed to
    be 2-dimensional. The inner-list-of-lists is a result of how I'm
    producing the data, and now I want to do a mass-concatenation (or
    extending) of the inner-list-of-lists, and come up with this result:

    >>> foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

    True

    What's the best way to accomplish this?

    It's not quite this simple, but let's just see what you can come up
    with the information at hand, and I'll see if I can adapt it to my
    needs.
    Dustan, Nov 12, 2006
    #10
  11. Dustan

    Robert Kern Guest

    Dustan wrote:
    > Fredrik Lundh wrote:


    >> if you care about writing robust code, why not just use a for-loop,
    >> and the list extend method?

    >
    > Because I'm embedding this expression in a list comprehension (as I
    > stated in my original post), and last time I checked, it's not possible
    > to treat a for-loop as an expression (which is what a list
    > comprehension requires).


    As with all such things, you stick the implementation in a well-named function
    and simply call the function everywhere. The implementation never needs to be a
    one-liner expression.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Nov 12, 2006
    #11
  12. Dustan wrote:

    >>>> foo =\

    > [[[1,2,3],[4,5,6],[7,8,9]],
    > [[3,2,1],[6,5,4],[9,8,7]]]
    >
    > Here, foo appears to be a 3-dimensional list - except it's supposed to
    > be 2-dimensional. The inner-list-of-lists is a result of how I'm
    > producing the data, and now I want to do a mass-concatenation (or
    > extending) of the inner-list-of-lists, and come up with this result:
    >
    >>>> foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

    > True


    that still doesn't explain your "the expression must be used in a list
    comprehension" requirement, though. assuming that the sizes are
    varying, and at least sometimes a lot larger than 3x3, I'd probably
    write the above as

    for index, item in enumerate(foo):
    this = []
    for i in item:
    this.extend(i)
    foo[index] = this

    which should be pretty efficient, since it avoids unnecessary function
    calls, and is amortized linear time instead of O(N**2).

    or, if I was in a hurry, and didn't really care if the inner sequences
    were lists or tuples:

    foo = map(Tkinter._flatten, foo)

    </F>
    Fredrik Lundh, Nov 12, 2006
    #12
  13. On Sat, 11 Nov 2006 17:42:32 -0800, Dustan wrote:

    >> alright, let's try again: why do you need a self-contained reduce
    >> replacement that can be embedded inside a list comprehension ?
    >>
    >> </F>

    >
    >
    >>>> foo =\

    > [[[1,2,3],[4,5,6],[7,8,9]],
    > [[3,2,1],[6,5,4],[9,8,7]]]
    >
    > Here, foo appears to be a 3-dimensional list - except it's supposed to
    > be 2-dimensional. The inner-list-of-lists is a result of how I'm
    > producing the data, and now I want to do a mass-concatenation (or
    > extending) of the inner-list-of-lists, and come up with this result:
    >
    >>>> foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

    > True
    >
    > What's the best way to accomplish this?


    I don't know if this is the best, but it didn't take long to come up with
    it:

    >>> foo = [[[1,2,3],[4,5,6],[7,8,9]],

    .... [[3,2,1],[6,5,4],[9,8,7]]]
    >>>
    >>> def unroll(list3d):

    .... newl = []
    .... for sublist in list3d:
    .... newl.append(sum(sublist, []))
    .... return newl
    ....
    >>> bar = unroll(foo)
    >>> bar

    [[1, 2, 3, 4, 5, 6, 7, 8, 9], [3, 2, 1, 6, 5, 4, 9, 8, 7]]


    Repeat after me:

    "Not everything has to be a one-liner."


    If sum() is too slow, because your sub-lists are huge, you can easily
    factor that out and replace it with something using extend.


    --
    Steven.
    Steven D'Aprano, Nov 12, 2006
    #13
  14. Dustan

    Dustan Guest

    Robert Kern wrote:
    > Dustan wrote:
    > > Fredrik Lundh wrote:

    >
    > >> if you care about writing robust code, why not just use a for-loop,
    > >> and the list extend method?

    > >
    > > Because I'm embedding this expression in a list comprehension (as I
    > > stated in my original post), and last time I checked, it's not possible
    > > to treat a for-loop as an expression (which is what a list
    > > comprehension requires).

    >
    > As with all such things, you stick the implementation in a well-named function
    > and simply call the function everywhere. The implementation never needs to be a
    > one-liner expression.


    It's already in a function, but in order to convert the reduce function
    into a for-loop that I can use as an expression, I would have to create
    another independent function, which would make that code more cluttered
    than it already is.

    Unless you're saying to perform the list comprehension manually. That
    would be two for-loops I use in the list comprehension, plus another
    one to take place of the reduce function. Sure, that spreads out each
    individual step a little more, but it also makes it more difficult to
    understand what the overall goal of the code is (I'm going for
    readability as well as easy maintenance here).

    > --
    > Robert Kern
    >
    > "I have come to believe that the whole world is an enigma, a harmless enigma
    > that is made terrible by our own mad attempt to interpret it as though it had
    > an underlying truth."
    > -- Umberto Eco
    Dustan, Nov 12, 2006
    #14
  15. Dustan

    Dustan Guest

    Fredrik Lundh wrote:
    > Dustan wrote:
    >
    > >>>> foo =\

    > > [[[1,2,3],[4,5,6],[7,8,9]],
    > > [[3,2,1],[6,5,4],[9,8,7]]]
    > >
    > > Here, foo appears to be a 3-dimensional list - except it's supposed to
    > > be 2-dimensional. The inner-list-of-lists is a result of how I'm
    > > producing the data, and now I want to do a mass-concatenation (or
    > > extending) of the inner-list-of-lists, and come up with this result:
    > >
    > >>>> foo == [[1,2,3,4,5,6,7,8,9],[3,2,1,6,5,4,9,8,7]]

    > > True

    >
    > that still doesn't explain your "the expression must be used in a list
    > comprehension" requirement, though.


    Oh, right; sorry about the confusion. The list isn't quite that simple,
    and in order to pull the right pieces together, I use a list
    comprehension.

    > assuming that the sizes are
    > varying, and at least sometimes a lot larger than 3x3, I'd probably
    > write the above as
    >
    > for index, item in enumerate(foo):
    > this = []
    > for i in item:
    > this.extend(i)
    > foo[index] = this


    I'll see if that works.

    > which should be pretty efficient, since it avoids unnecessary function
    > calls, and is amortized linear time instead of O(N**2).
    >
    > or, if I was in a hurry, and didn't really care if the inner sequences
    > were lists or tuples:
    >
    > foo = map(Tkinter._flatten, foo)
    >
    > </F>


    Steven D'Aprano wrote:
    > I don't know if this is the best, but it didn't take long to come up with
    > it:
    >
    > >>> foo = [[[1,2,3],[4,5,6],[7,8,9]],

    > ... [[3,2,1],[6,5,4],[9,8,7]]]
    > >>>
    > >>> def unroll(list3d):

    > ... newl = []
    > ... for sublist in list3d:
    > ... newl.append(sum(sublist, []))
    > ... return newl


    deja vu...

    > >>> bar = unroll(foo)
    > >>> bar

    > [[1, 2, 3, 4, 5, 6, 7, 8, 9], [3, 2, 1, 6, 5, 4, 9, 8, 7]]
    >
    >
    > Repeat after me:
    >
    > "Not everything has to be a one-liner."


    Not everything has to be a one-liner. But readability helps.

    > If sum() is too slow, because your sub-lists are huge, you can easily
    > factor that out and replace it with something using extend.
    >
    >
    > --
    > Steven.
    Dustan, Nov 12, 2006
    #15
  16. Dustan

    Robert Kern Guest

    Dustan wrote:
    > Robert Kern wrote:
    >> Dustan wrote:
    >>> Fredrik Lundh wrote:
    >>>> if you care about writing robust code, why not just use a for-loop,
    >>>> and the list extend method?
    >>> Because I'm embedding this expression in a list comprehension (as I
    >>> stated in my original post), and last time I checked, it's not possible
    >>> to treat a for-loop as an expression (which is what a list
    >>> comprehension requires).

    >> As with all such things, you stick the implementation in a well-named function
    >> and simply call the function everywhere. The implementation never needs to be a
    >> one-liner expression.

    >
    > It's already in a function, but in order to convert the reduce function
    > into a for-loop that I can use as an expression, I would have to create
    > another independent function, which would make that code more cluttered
    > than it already is.
    >
    > Unless you're saying to perform the list comprehension manually. That
    > would be two for-loops I use in the list comprehension, plus another
    > one to take place of the reduce function. Sure, that spreads out each
    > individual step a little more, but it also makes it more difficult to
    > understand what the overall goal of the code is (I'm going for
    > readability as well as easy maintenance here).


    Let's just say that we disagree entirely on what constitutes readability and
    maintainability.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Nov 12, 2006
    #16
  17. Dustan

    Paddy Guest

    Dustan wrote:

    > Anyway, I figured out a way to get the builtin
    > function 'sum' to work as I need:
    > sum([[1,2,3],[4,5,6],[7,8,9]], [])
    >


    Hah!
    No-one expects sum to be used on anything but numbers.

    Except lists as above.

    No-one expects sum to be used on anything but numbers, and maybe lists
    too.

    ;-)


    (My apologies for my poor grasp of the Spanish Inquisition sketch; and
    I was alluding to
    this thread:
    http://groups.google.com/group/comp.lang.python/msg/f34525384f8028b5 )
    Paddy, Nov 12, 2006
    #17
  18. Dustan

    Ben Finney Guest

    "Dustan" <> writes:

    > Steven D'Aprano wrote:
    > > Repeat after me:
    > >
    > > "Not everything has to be a one-liner."

    >
    > Not everything has to be a one-liner. But readability helps.


    Indeed. Complex one-liners are rarely as readable as a well-named
    function call, implemented with several explicit readable lines in
    one place, and then re-used as many times as necessary.

    --
    \ "Anyone who believes exponential growth can go on forever in a |
    `\ finite world is either a madman or an economist." -- Kenneth |
    _o__) Boulding |
    Ben Finney
    Ben Finney, Nov 12, 2006
    #18
  19. Dustan

    Dustan Guest

    Alright, I can see I'm a bit outvoted here. I tried your suggestions
    and it worked fine.

    I'll also try to consider in the future that part of the problem might
    be lack of information conveyed on my part.
    Dustan, Nov 12, 2006
    #19
  20. Dustan wrote:

    > Alright, I can see I'm a bit outvoted here. I tried your suggestions
    > and it worked fine.
    >
    > I'll also try to consider in the future that part of the problem might
    > be lack of information conveyed on my part.


    If you insist on one-liners, it can be done without sum(), though it
    probably doesn't buy you much in readability:

    from itertools import chain
    [list(chain(*row)) for row in foo]

    By the way, if this was not a toy example and you're doing serious work
    with n-dimensional arrays, make yourself a favor and install NumPy;
    it's usually both faster and more elegant for array manipulations than
    pure python.

    George
    George Sakkis, Nov 13, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Earl Teigrob
    Replies:
    3
    Views:
    342
    Alvin Bruney [MVP]
    Mar 4, 2004
  2. Sammy
    Replies:
    2
    Views:
    423
    Steven Cheng[MSFT]
    May 18, 2004
  3. Phil Winstanley [Microsoft MVP ASP.NET]
    Replies:
    1
    Views:
    428
    Sammy
    May 17, 2004
  4. Don
    Replies:
    0
    Views:
    333
  5. Sean McLeod
    Replies:
    0
    Views:
    396
    Sean McLeod
    Jun 10, 2005
Loading...

Share This Page