Re: groupby behaviour

Discussion in 'Python' started by andrea crotti, Feb 26, 2013.

  1. 2013/2/26 Ian Kelly <>:
    > On Tue, Feb 26, 2013 at 9:27 AM, andrea crotti
    > <> wrote:
    >> So I was trying to use groupby (which I used in the past), but I
    >> noticed a very strange thing if using list on
    >> the result:

    >
    > As stated in the docs:
    >
    > """
    > The returned group is itself an iterator that shares the underlying
    > iterable with groupby(). Because the source is shared, when the
    > groupby() object is advanced, the previous group is no longer visible.
    > So, if that data is needed later, it should be stored as a list:
    > """
    > --
    > http://mail.python.org/mailman/listinfo/python-list



    I should have read more carefully sorry, I was in the funny situation
    where it would have actually worked in the production code but it was
    failing in the unit tests (because I was using list only there).

    It's very weird though this sharing and still doesn't really look
    rightl, is it done just for performance reasons?
     
    andrea crotti, Feb 26, 2013
    #1
    1. Advertising

  2. andrea crotti

    Paul Rubin Guest

    andrea crotti <> writes:
    > It's very weird though this sharing and still doesn't really look
    > rightl, is it done just for performance reasons?


    It could consume unbounded amounts of memory otherwise. E.g. if there
    are millions of items in the group.

    If you're not worried about that situation it's simplest to just
    convert each group to a list for processing it:

    for k,g in groupby(...):
    gs = list(g)
    # do stuff with gs

    calling list(g) reads all the elements in the group, advancing groupby's
    internal iterator to the next group. Since you've copied out all the
    group elements, you don't have to worry about the sharing effect.
     
    Paul Rubin, Feb 26, 2013
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. G?nter Jantzen

    whatsnew 2.4 about itertools.groupby:

    G?nter Jantzen, Jun 9, 2004, in forum: Python
    Replies:
    0
    Views:
    292
    G?nter Jantzen
    Jun 9, 2004
  2. Bryan

    groupby

    Bryan, May 23, 2006, in forum: Python
    Replies:
    4
    Views:
    590
    Paul McGuire
    May 27, 2006
  3. Replies:
    3
    Views:
    336
    Fredrik Lundh
    May 25, 2006
  4. andrea crotti

    groupby behaviour

    andrea crotti, Feb 26, 2013, in forum: Python
    Replies:
    0
    Views:
    98
    andrea crotti
    Feb 26, 2013
  5. Ian Kelly

    Re: groupby behaviour

    Ian Kelly, Feb 26, 2013, in forum: Python
    Replies:
    0
    Views:
    110
    Ian Kelly
    Feb 26, 2013
Loading...

Share This Page