Generator expressions vs. comprehensions

Discussion in 'Python' started by Ian Kelly, May 24, 2010.

  1. Ian Kelly

    Ian Kelly Guest

    Hi all,

    I just ran into an interesting but rather subtle little wart today.
    The following expressions are not functionally equivalent, even in
    Python 3:

    tuple(iterator.next() for i in xrange(n))

    tuple([iterator.next() for i in xrange(n)])

    In the first case, if iterator.next() raises a StopIteration, the
    exception is swallowed by the generator expression. The expression
    evaluates to a truncated tuple, and the StopIteration is not
    propagated.

    In the second case, the StopIteration exception is propagated as
    expected by the list comprehension. Set and dict comprehensions also
    behave this way in Python 3.

    Is this distinction generally known? The generator expression
    behavior is understandable since a generator would do the same thing,
    but I'm disappointed that the inconsistency exists and wasn't fixed in
    Python 3 when we had the chance.

    Cheers,
    Ian
    Ian Kelly, May 24, 2010
    #1
    1. Advertising

  2. Ian Kelly

    Carl Banks Guest

    On May 24, 3:31 pm, Ian Kelly <> wrote:
    > Hi all,
    >
    > I just ran into an interesting but rather subtle little wart today.
    > The following expressions are not functionally equivalent, even in
    > Python 3:
    >
    > tuple(iterator.next() for i in xrange(n))
    >
    > tuple([iterator.next() for i in xrange(n)])
    >
    > In the first case, if iterator.next() raises a StopIteration, the
    > exception is swallowed by the generator expression.  The expression
    > evaluates to a truncated tuple, and the StopIteration is not
    > propagated.
    >
    > In the second case, the StopIteration exception is propagated as
    > expected by the list comprehension.  Set and dict comprehensions also
    > behave this way in Python 3.
    >
    > Is this distinction generally known?  The generator expression
    > behavior is understandable since a generator would do the same thing,
    > but I'm disappointed that the inconsistency exists and wasn't fixed in
    > Python 3 when we had the chance.



    As a general rule you shouldn't call the next() method/function
    without arranging to catch StopIteration, unless you know the iterator
    will never raise StopIteration (such as it if it itertools.count()).
    The problem is that if a StopIteration "leaks", as it does with the
    generator examples, another iterator further up the stack might catch
    it and exit.

    The situation here is known. It can't be corrected, even in Python 3,
    without modifying iterator protocol to tie StopIteration to a specific
    iterator. This is possible and might be worth it to avoid hard-to-
    diagnose bugs but it would complicate iterator protocol, which becomes
    less useful as it becomes more complex.


    Carl Banks
    Carl Banks, May 24, 2010
    #2
    1. Advertising

  3. On Mon, 24 May 2010 15:47:32 -0700 (PDT)
    Carl Banks <> wrote:
    > >
    > > Is this distinction generally known?


    Yes, it is.

    > > The generator expression
    > > behavior is understandable since a generator would do the same thing,
    > > but I'm disappointed that the inconsistency exists and wasn't fixed in
    > > Python 3 when we had the chance.


    Why don't you use itertools.islice()?
    Antoine Pitrou, May 25, 2010
    #3
  4. On May 25, 12:47 am, Carl Banks <> wrote:
    > The situation here is known.  It can't be corrected, even in Python 3,
    > without modifying iterator protocol to tie StopIteration to a specific
    > iterator.  This is possible and might be worth it to avoid hard-to-
    > diagnose bugs but it would complicate iterator protocol, which becomes
    > less useful as it becomes more complex.


    The situation here is a known and could be corrected by changing the
    meaning of list comprehension,
    for instance by having [x for x in iterable] to be an alias for list(x
    for x in iterable). In such a way the StopIteration exception would be
    always swallowed and there would be consistency with generator
    expressions (by construction). However, the list comprehension would
    become non-equivalent to the corresponding for-loop with an .append,
    so somebody would be un happy anyway :-/
    Michele Simionato, May 25, 2010
    #4
  5. Ian Kelly

    Peter Otten Guest

    Michele Simionato wrote:

    > On May 25, 12:47 am, Carl Banks <> wrote:
    >> The situation here is known. It can't be corrected, even in Python 3,
    >> without modifying iterator protocol to tie StopIteration to a specific
    >> iterator. This is possible and might be worth it to avoid hard-to-
    >> diagnose bugs but it would complicate iterator protocol, which becomes
    >> less useful as it becomes more complex.

    >
    > The situation here is a known and could be corrected by changing the
    > meaning of list comprehension,
    > for instance by having [x for x in iterable] to be an alias for list(x
    > for x in iterable). In such a way the StopIteration exception would be
    > always swallowed and there would be consistency with generator
    > expressions (by construction). However, the list comprehension would
    > become non-equivalent to the corresponding for-loop with an .append,
    > so somebody would be un happy anyway :-/


    But the list comprehension is already non-equivalent to the for loop as the
    loop variable isn't leaked anymore. We do have three similar constructs with
    subtle differences.

    I think not turning the list-comp into syntactic sugar for list(genexp) in
    py3 is a missed opportunity.

    Peter
    Peter Otten, May 25, 2010
    #5
  6. On May 25, 9:08 am, Peter Otten <> wrote:
    > But the list comprehension is already non-equivalent to the for loop as the
    > loop variable isn't leaked anymore. We do have three similar constructs with
    > subtle differences.
    >
    > I think not turning the list-comp into syntactic sugar for list(genexp) in
    > py3 is a missed opportunity.


    Actually I do agree with the feeling, and this is not the only missed
    opportunity in Python 3 :-/
    Michele Simionato, May 25, 2010
    #6
  7. Ian Kelly

    Terry Reedy Guest

    On 5/25/2010 3:08 AM, Peter Otten wrote:
    > Michele Simionato wrote:


    > I think not turning the list-comp into syntactic sugar for list(genexp) in
    > py3 is a missed opportunity.


    Implementing it that way was tried but was much slower than the current
    implementation. If one uses StopIteration as it is intended to be used
    (and is so documented), then, I believe, they are equivalent. There was
    a conscious decision to not slow comprehensions for the many to cater to
    the very few.

    Terry Jan Reedy
    Terry Reedy, May 25, 2010
    #7
  8. Ian Kelly

    Terry Reedy Guest

    On 5/25/2010 1:09 PM, Terry Reedy wrote:
    > On 5/25/2010 3:08 AM, Peter Otten wrote:
    >> Michele Simionato wrote:

    >
    >> I think not turning the list-comp into syntactic sugar for
    >> list(genexp) in
    >> py3 is a missed opportunity.

    >
    > Implementing it that way was tried but was much slower than the current
    > implementation. If one uses StopIteration as it is intended to be used
    > (and is so documented), then, I believe, they are equivalent. There was
    > a conscious decision to not slow comprehensions for the many to cater to
    > the very few.


    And those few can always write list(genexp) instead of [genexp] (or
    set...) when the minute difference actually matters.

    > Terry Jan Reedy
    >
    Terry Reedy, May 25, 2010
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. seguso
    Replies:
    9
    Views:
    386
    seguso
    Dec 22, 2004
  2. Jeff Epler
    Replies:
    2
    Views:
    313
    Jeff Epler
    Aug 27, 2003
  3. Mahesh Padmanabhan

    Generator expressions v/s list comprehensions

    Mahesh Padmanabhan, Aug 28, 2004, in forum: Python
    Replies:
    24
    Views:
    646
    Raymond Hettinger
    Sep 1, 2004
  4. Replies:
    10
    Views:
    505
    Caleb Hattingh
    Feb 8, 2005
  5. Mike Meyer
    Replies:
    23
    Views:
    667
    Bengt Richter
    Apr 27, 2005
Loading...

Share This Page