StopIteration in the if clause of a generator expression

Discussion in 'Python' started by Peter Otten, Apr 1, 2005.

  1. Peter Otten

    Peter Otten Guest

    To confuse a newbies and old hands alike, Bengt Richter wrote:

    > Need something more straightforward, e.g., a wrapped one-liner:
    >
    > >>> def guess(n=3): print ("You're right!", 'No more tries for
    > >>> you!!!')[n-1 in

    > ... (x for x in xrange(n) for t in [raw_input('Guess my name:
    > ')=='Ben']
    > ... if not t or iter([]).next())]
    > ...
    > >>> guess()


    To make it a bit clearer, a StopIteration raised in a generator expression
    silently terminates that generator:

    >>> def stop(): raise StopIteration

    ....
    >>> list(i for i in range(10) if i < 5 or stop())

    [0, 1, 2, 3, 4]

    In a list comprehension, on the other hand, it is propagated:

    >>> [i for i in range(10) if i < 5 or stop()]

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 1, in stop
    StopIteration

    Is that an intentional difference?

    Peter
     
    Peter Otten, Apr 1, 2005
    #1
    1. Advertising

  2. Peter Otten

    Carl Banks Guest

    Peter Otten wrote:
    > To confuse a newbies and old hands alike, Bengt Richter wrote:
    >
    > > Need something more straightforward, e.g., a wrapped one-liner:
    > >
    > > >>> def guess(n=3): print ("You're right!", 'No more tries for
    > > >>> you!!!')[n-1 in

    > > ... (x for x in xrange(n) for t in [raw_input('Guess my name:
    > > ')=='Ben']
    > > ... if not t or iter([]).next())]
    > > ...
    > > >>> guess()

    >
    > To make it a bit clearer, a StopIteration raised in a generator

    expression
    > silently terminates that generator:
    >
    > >>> def stop(): raise StopIteration

    > ...
    > >>> list(i for i in range(10) if i < 5 or stop())

    > [0, 1, 2, 3, 4]
    >
    > In a list comprehension, on the other hand, it is propagated:
    >
    > >>> [i for i in range(10) if i < 5 or stop()]

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > File "<stdin>", line 1, in stop
    > StopIteration
    >
    > Is that an intentional difference?



    Very interesting. I'm not sure if the designers even considered this
    particular subtlety. Why it happens is pretty plain. In the generator
    expression case, the generator expression does propogate the
    StopIteration, but list() traps it. List comprehensions are internally
    treated as a for-loop (kind of), which doesn't trap StopIteration.
    Maybe it should.

    The list comprehension [ x for x in y ] is currently treated as
    equivalent to the following, with byte-code optimizations:

    .. _ = []
    .. for x in y:
    .. _.append(x)


    Perhaps it ought to be equivalent to:

    .. _ = []
    .. try:
    .. for x in y:
    .. _.append(x)
    .. except StopIteration:
    .. pass


    However, I would guess the Python gods wouldn't approve of this use of
    StopIteration, and so would make no sacrifices to get it.
    Nevertheless, it seems likely to be how a list comprehension would
    behave in Python 3.0, so maybe we should do it.


    --
    CARL BANKS
     
    Carl Banks, Apr 1, 2005
    #2
    1. Advertising

  3. [Peter Otten]
    > a StopIteration raised in a generator expression
    > silently terminates that generator:
    >
    > >>> def stop(): raise StopIteration

    > ...
    > >>> list(i for i in range(10) if i < 5 or stop())

    > [0, 1, 2, 3, 4]
    >
    > In a list comprehension, on the other hand, it is propagated:
    >
    > >>> [i for i in range(10) if i < 5 or stop()]

    > Traceback (most recent call last):
    > File "<stdin>", line 1, in ?
    > File "<stdin>", line 1, in stop
    > StopIteration
    >
    > Is that an intentional difference?


    I would call it an unfortunate assymmetry -- one the never comes up unless
    you're up to no good ;-)

    In a way, both behave identically. They both raise StopIteration. In the case
    of the generator expression, that StopIteration is intercepted by the enclosing
    list() call. That becomes obvious if you write a pure python equivalent for
    list:

    def lyst(s):
    it = iter(s)
    result = []
    try:
    while 1:
    result.append(it.next())
    except StopIteration: # guess who trapped StopIter
    return result


    Raymond Hettinger
     
    Raymond Hettinger, Apr 1, 2005
    #3
  4. On Fri, 01 Apr 2005 16:34:32 GMT, "Raymond Hettinger" <> wrote:

    >[Peter Otten]
    >> a StopIteration raised in a generator expression
    >> silently terminates that generator:
    >>
    >> >>> def stop(): raise StopIteration

    >> ...
    >> >>> list(i for i in range(10) if i < 5 or stop())

    >> [0, 1, 2, 3, 4]
    >>
    >> In a list comprehension, on the other hand, it is propagated:
    >>
    >> >>> [i for i in range(10) if i < 5 or stop()]

    >> Traceback (most recent call last):
    >> File "<stdin>", line 1, in ?
    >> File "<stdin>", line 1, in stop
    >> StopIteration
    >>
    >> Is that an intentional difference?

    >
    >I would call it an unfortunate assymmetry -- one the never comes up unless
    >you're up to no good ;-)

    ;-)
    >
    >In a way, both behave identically. They both raise StopIteration. In the case
    >of the generator expression, that StopIteration is intercepted by the enclosing
    >list() call. That becomes obvious if you write a pure python equivalent for
    >list:
    >
    > def lyst(s):
    > it = iter(s)
    > result = []
    > try:
    > while 1:
    > result.append(it.next())
    > except StopIteration: # guess who trapped StopIter
    > return result
    >
    >

    I assumed that all standard sequence consumers (including list, of course) would intercept
    the StopIteration of a sequence given them in the form of a generator expression, so your
    lyst example would have an analogue for other sequence consumers as well, right?
    I.e., there's not a hidden list(genex) in those others I would hope ;-)

    E.g., "in" in my toy exposed more clearly, using Peter's stop:

    >>> def show(x): print x,; return x

    ...
    >>> def stop(): raise StopIteration

    ...
    >>> 2 in (x for x in xrange(5) if show(x)<4 or stop())

    0 1 2
    True
    >>> 7 in (x for x in xrange(5) if show(x)<4 or stop())

    0 1 2 3 4
    False

    BTW I notice that this also nicely shortcuts when the 2 is found.

    Regards,
    Bengt Richter
     
    Bengt Richter, Apr 1, 2005
    #4
  5. > I assumed that all standard sequence consumers (including list, of course)
    would intercept
    > the StopIteration of a sequence given them in the form of a generator

    expression, so your
    > lyst example would have an analogue for other sequence consumers as well,

    right?
    > I.e., there's not a hidden list(genex) in those others I would hope ;-)


    Right.



    > E.g., "in" in my toy exposed more clearly, using Peter's stop:
    >
    > >>> def show(x): print x,; return x

    > ...
    > >>> def stop(): raise StopIteration

    > ...
    > >>> 2 in (x for x in xrange(5) if show(x)<4 or stop())

    > 0 1 2
    > True
    > >>> 7 in (x for x in xrange(5) if show(x)<4 or stop())

    > 0 1 2 3 4
    > False
    >
    > BTW I notice that this also nicely shortcuts when the 2 is found.


    That's a fact.


    Raymond
     
    Raymond Hettinger, Apr 2, 2005
    #5
  6. Peter Otten

    jfj Guest

    Peter Otten wrote:
    > To confuse a newbies and old hands alike, Bengt Richter wrote:


    got me for one:)


    >
    > To make it a bit clearer, a StopIteration raised in a generator expression
    > silently terminates that generator:


    *any* exception raised from a generator, terminates the generator


    jfj
     
    jfj, Apr 2, 2005
    #6
  7. Peter Otten

    Peter Otten Guest

    Raymond Hettinger wrote:

    (quoting Bengt)
    >> I assumed that all standard sequence consumers (including list, of
    >> course) would intercept the StopIteration of a sequence given them in the
    >> form of a generator expression, so your lyst example would have an
    >> analogue for other sequence consumers as well, right?
    >> I.e., there's not a hidden list(genex) in those others I would hope ;-)

    >
    > Right.


    I see I followed the historical evolvement and saw generator expressions as
    a lazy listcomp rather than a cool new way to write a generator. That
    turned out to be the road to confusion.

    Thanks Carl, thanks Raymond for setting me straight.

    > I would call it an unfortunate assymmetry -- one the never comes up unless
    > you're up to no good ;-)


    Do you see any chance that list comprehensions will be redefined as an
    alternative spelling for list(<generator expression>)?

    Peter
     
    Peter Otten, Apr 3, 2005
    #7
  8. Peter Otten

    Peter Otten Guest

    jfj wrote:

    >> To make it a bit clearer, a StopIteration raised in a generator
    >> expression silently terminates that generator:

    >
    > *any* exception raised from a generator, terminates the generator


    Yeah, but StopIteration is the only expected exception and therefore the
    only one that client code (nearly) always knows to deal with:

    >>> def choke(): raise ValueError

    ....
    >>> list(i for i in range(10) if i < 3 or choke())

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 1, in <generator expression>
    File "<stdin>", line 1, in choke
    ValueError
    >>> [i for i in range(10) if i < 3 or choke()]

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "<stdin>", line 1, in choke
    ValueError

    Here you can *not* tell apart list(genexp) and listcomp.

    (Of course, as has since been pointed out, the StopIteration is actually
    caught in the list constructor, so nothing magic to the example in my
    initial post)

    Peter
     
    Peter Otten, Apr 3, 2005
    #8
  9. [Peter Otten]
    > Do you see any chance that list comprehensions will be redefined as an
    > alternative spelling for list(<generator expression>)?


    Not likely. It is possible that the latter spelling would make it possible for
    Py3.0. eliminate list comps entirely. However, they are very popular and
    practical, so my bet is that they will live on.

    The more likely change is that in Py3.0 list comps will no longer expose the
    loop variable outside the loop.


    Raymond Hettinger
     
    Raymond Hettinger, Apr 3, 2005
    #9
  10. Raymond Hettinger wrote:
    > [Peter Otten]
    >
    >>Do you see any chance that list comprehensions will be redefined as an
    >>alternative spelling for list(<generator expression>)?

    >
    > Not likely. It is possible that the latter spelling would make it possible for
    > Py3.0. eliminate list comps entirely. However, they are very popular and
    > practical, so my bet is that they will live on.


    I suspect you're right, but I certainly wouldn't complain if list comps
    disappeared. TOOWTDI and all, and I often find myself alternating
    between the two when I can't decide which one seems more Pythonic.
    (These days I generally write a listcomp, but I wouldn't put any money
    on my code being entirely consistent about this...)

    STeVe
     
    Steven Bethard, Apr 3, 2005
    #10
  11. Peter Otten

    Guest

    This is all just making everything far too complicated. What you really
    want to do is quite simple:

    import itertools
    def condition(x): return x < 5

    list(itertools.takewhile(condition, (i for i in range(10))))

    The 'Stop Iteration In Generator Expression' problem was solved in the
    language that List Comprehensions came from, Haskell. Haskell's basic
    library, prelude, had a series of functions that have found their way
    into the itertools toolbox. I highly recommend having a read of the
    itertools docs if you want to continue hacking around with generators.

    Regards,
    Stephen Thorne
     
    , Apr 3, 2005
    #11
  12. [Steven Bethard]
    > and I often find myself alternating
    > between the two when I can't decide which one seems more Pythonic.


    Both are pythonic.

    Use a genexp when you need a generator
    and use a listcomp when you need a list.


    Raymond Hettinger
     
    Raymond Hettinger, Apr 4, 2005
    #12
  13. Raymond Hettinger wrote:
    > [Steven Bethard]
    >
    >>and I often find myself alternating
    >>between the two when I can't decide which one seems more Pythonic.

    >
    > Both are pythonic.
    >
    > Use a genexp when you need a generator
    > and use a listcomp when you need a list.


    So do I read this right in preferring
    [<x> for <y> in <z>]
    over
    list(<x> for <y> in <z>)
    ?

    STeVe
     
    Steven Bethard, Apr 4, 2005
    #13
  14. > > Use a genexp when you need a generator
    > > and use a listcomp when you need a list.


    [Steven Bethard]
    > So do I read this right in preferring
    > [<x> for <y> in <z>]
    > over
    > list(<x> for <y> in <z>)


    Yes!


    Raymond
     
    Raymond Hettinger, Apr 5, 2005
    #14
  15. On Apr 5, 2005 2:04 AM, Raymond Hettinger <> wrote:
    > [Steven Bethard]
    > > So do I read this right in preferring
    > > [<x> for <y> in <z>]
    > > over
    > > list(<x> for <y> in <z>)

    >
    > Yes!


    Why? (Serious question. I'm sure that you have a good reason - I just
    can't figure out what it is.)

    The generator expression has the advantage of not leaking references
    into the enclosing namespace. What's advantage of the list comp?

    --
    Cheers,
    Simon B,
    ,
    http://www.brunningonline.net/simon/blog/
     
    Simon Brunning, Apr 5, 2005
    #15
  16. Peter Otten

    Duncan Booth Guest

    Simon Brunning wrote:

    > On Apr 5, 2005 2:04 AM, Raymond Hettinger <> wrote:
    >> [Steven Bethard]
    >> > So do I read this right in preferring
    >> > [<x> for <y> in <z>]
    >> > over
    >> > list(<x> for <y> in <z>)

    >>
    >> Yes!

    >
    > Why? (Serious question. I'm sure that you have a good reason - I just
    > can't figure out what it is.)
    >
    > The generator expression has the advantage of not leaking references
    > into the enclosing namespace. What's advantage of the list comp?
    >

    The list comprehension is about 15-20% faster according to timeit.py:

    C:\Python24\Lib>..\python.exe timeit.py -s "t = range(1000)" "[ x for x in t]"
    10000 loops, best of 3: 116 usec per loop

    C:\Python24\Lib>..\python.exe timeit.py -s "t = range(1000)" "list(x for x in t)"
    1000 loops, best of 3: 144 usec per loop

    C:\Python24\Lib>..\python.exe timeit.py -s "t = range(100000)" "[ x for x in t]"
    10 loops, best of 3: 13.9 msec per loop

    C:\Python24\Lib>..\python.exe timeit.py -s "t = range(100000)" "list(x for x in t)"
    10 loops, best of 3: 16.3 msec per loop

    Alternatively you could just regard the list comprehension as having
    less clutter on the screen so it may be clearer.
     
    Duncan Booth, Apr 5, 2005
    #16
  17. > > [Steven Bethard]
    > > > So do I read this right in preferring
    > > > [<x> for <y> in <z>]
    > > > over
    > > > list(<x> for <y> in <z>)


    [Raymond Hettinger]
    > > Yes!


    [Simon Brunning]
    > Why? (Serious question. I'm sure that you have a good reason - I just
    > can't figure out what it is.)
    >
    > The generator expression has the advantage of not leaking references
    > into the enclosing namespace. What's advantage of the list comp?


    One advantage relates to mental parsing and chunking.
    A list comp reads as a single step: "make a list".
    The genexp form reads as "make a generator and turn it into a list."

    The listcomp form has slightly more economy of expression (it is succinct) and
    the brackets are a nice visual cue that may save a neuron or two.

    Another advantage is that listcomps are older. They tend to be better
    understood already. And, they run on older pythons.

    The design rule, "use listcomps to make lists and genexps to make generators",
    encourages data centric thinking.With being distracting, it helps maintain an
    awareness of whether you're filling memory or
    generating elements one-at-a-time.

    Internally, there are performance differences favoring listcomps when the
    desired output is a list. List comprehensions are interpreted immediately
    through syntax rather than a global lookup of the list() builtin. The code for
    listcomps does not have to create, switch between, and destroy a separate
    stackframe. The compiler creates custom code for list comps that takes
    advantage of the new LIST_APPEND opcode.

    Partially balancing out all of the above are some small advantages for the
    list(somegen) form. It lets you forget about listcomps and it works with other
    contructors, deque(somegen) or set(somegen) for example.

    To my tastes, the net balance favors using listcomps whenever you need to create
    a list.


    Raymond Hettinger
     
    Raymond Hettinger, Apr 5, 2005
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Using StopIteration

    , May 8, 2006, in forum: Python
    Replies:
    5
    Views:
    486
  2. Kris Kowal
    Replies:
    5
    Views:
    336
  3. Replies:
    5
    Views:
    1,266
    Chris
    Jun 17, 2008
  4. grocery_stocker
    Replies:
    2
    Views:
    465
    Terry Reedy
    Apr 5, 2009
  5. Roald de Vries
    Replies:
    0
    Views:
    217
    Roald de Vries
    Aug 7, 2010
Loading...

Share This Page