Re: Dangerous behavior of list(generator)

Discussion in 'Python' started by exarkun@twistedmatrix.com, Dec 14, 2009.

  1. Guest

    On 02:58 pm, wrote:
    > wrote:
    >>On 08:45 am, wrote:
    >>>Tom Machinski wrote:
    >>>>In most cases, `list(generator)` works as expected. Thus,
    >>>>`list(<generator expression>)` is generally equivalent to
    >>>>`[<generator
    >>>>expression>]`.
    >>>>
    >>>>Here's a minimal case where this equivalence breaks, causing a
    >>>>serious
    >>>>and hard-to-detect bug in a program:
    >>>>
    >>>> >>> def sit(): raise StopIteration()
    >>>
    >>>StopIteration is intended to be used only within the .__next__ method
    >>>of iterators. The devs know that other 'off-label' use results in the
    >>>inconsistency you noted, but their and my view is 'don't do that'.

    >>
    >>Which is unfortunate, because it's not that hard to get StopIteration
    >>without explicitly raising it yourself and this behavior makes it
    >>difficult to debug such situations.
    >>
    >>What's with this view, exactly? Is it just that it's hard to
    >>implement
    >>the more desirable behavior?

    >
    >I'm not exactly sure what you're asking for.
    >
    >The StopIteration exception originated as part of the for-loop
    >protocol. Later on it was generalized to apply to generators
    >as well.
    >
    >The reason for using an exception is simple: raising and catching
    >exceptions is fast at C level and since the machinery for
    >communicating exceptions up the call stack was already there
    >(and doesn't interfere with the regular return values), this
    >was a convenient method to let the upper call levels know
    >that an iteration has ended (e.g. a for-loop 4 levels up the
    >stack).
    >
    >I'm not sure whether that answers your question, but it's the
    >reason for things being as they are :)


    I'm asking about why the behavior of a StopIteration exception being
    handled from the `expression` of a generator expression to mean "stop
    the loop" is accepted by "the devs" as acceptable. To continue your
    comparison to for loops, it's as if a loop like this:

    for a in b:
    c

    actually meant this:

    for a in b:
    try:
    c
    except StopIteration:
    break

    Note, I know *why* the implementation leads to this behavior. I'm
    asking why "the devs" *accept* this.

    Jean-Paul
    , Dec 14, 2009
    #1
    1. Advertising

  2. Mel Guest

    wrote:
    [ ... ]
    it's as if a loop like this:
    >
    > for a in b:
    > c
    >
    > actually meant this:
    >
    > for a in b:
    > try:
    > c
    > except StopIteration:
    > break
    >
    > Note, I know *why* the implementation leads to this behavior. I'm
    > asking why "the devs" *accept* this.


    It's part of the price Python pays for letting people get their hands on the
    controls. Consider also:

    Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
    [GCC 4.3.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> class dict2(dict):

    .... def __getitem__ (self, key):
    .... if key == 'fatal':
    .... raise KeyError
    ....
    >>> d = dict2()
    >>> d['fatal'] = 'Hello, world!'
    >>> print d['fatal']

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 4, in __getitem__
    KeyError
    >>>



    "KeyError when we just put the item into the dict?"
    "Yep."


    Mel.

    >
    > Jean-Paul
    Mel, Dec 14, 2009
    #2
    1. Advertising

  3. Carl Banks Guest

    On Dec 14, 7:21 am, wrote:
    > Note, I know *why* the implementation leads to this behavior.  I'm
    > asking why "the devs" *accept* this.


    As noted, the problem isn't with generators but with iteration
    protocol. The devs "allowed" this because it was a necessary evil for
    correct functionality. As the system is set up it can't discriminate
    between a legitimate and a spurrious StopIteration. (Which is why
    Steven and others called you out for suggesting that the compiler has
    to read your mind.)

    However, as far as I'm concerned there is no reasonable argument that
    this behavior is good. So how, hypothetically, would one go about
    fixing it, short of ripping out and replacing the existing machinery?

    The first argument is that StopIteration has no place within a
    generator expression. Therefore a generator expression (but not a
    generator function) could deliberately catch StopIteration and raise a
    different exception.

    I don't like it, though: who says that StopIteration has no place
    within a generator expression? Currently it's possible to do
    something like this to terminate a genexp early, and I won't the one
    saying you shouldn't do it.

    def stop(): raise StopIteration

    list(x or stop() for x in stream)

    (Though personally I'd bite the bullet and write it as a generator
    function).

    What else? The way I see it, when you throw StopIteration you are
    trying to stop a specific generator. Therefore StopIteration should
    (somehow) contain a reference to the generator that it's being applied
    to. Perhaps it can be obtained by crawling ths stack, which shouldn't
    be a significant penalty (it'd be only called once per iteration, and
    most of the time it'd be only one or two frames up). The looping
    logic within Python should check whether the reference matches the
    object it's iterating over; if not it raises a LeakyLoopException or
    something like that.

    I haven't thought this out though, I'm just kind of throwing this out
    there. Any issues?


    But to answer your question, I think "simple is better than complex"
    rules the day. Right now StopIteration stops an iteration, simple as
    that. Any fix would add complexity.


    Carl Banks
    Carl Banks, Dec 14, 2009
    #3
  4. On Dec 14, 11:05 pm, Carl Banks <> wrote:
    > But to answer your question, I think "simple is better than complex"
    > rules the day.  Right now StopIteration stops an iteration, simple as
    > that.  Any fix would add complexity.


    +1
    Michele Simionato, Dec 16, 2009
    #4
  5. > I'm asking about why the behavior of a StopIteration exception being
    > handled from the `expression` of a generator expression to mean "stop
    > the loop" is accepted by "the devs" as acceptable.


    I may be late to this discussion, but the answer is "most definitely
    yes". *Any* exception leads to termination of the iterator, and
    StopIteration is no different:

    py> def stop(e):
    .... def f():
    .... raise e
    .... return f
    ....
    py> g = (f() for f in (lambda:1,stop(StopIteration),lambda:2))
    py> g.next
    <method-wrapper 'next' of generator object at 0xb7960fac>
    py> g.next()
    1
    py> g.next()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in <genexpr>
    File "<stdin>", line 3, in f
    StopIteration
    py> g.next()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    StopIteration
    py> g = (f() for f in (lambda:1,stop(ValueError),lambda:2))
    py> g.next()
    1
    py> g.next()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 1, in <genexpr>
    File "<stdin>", line 3, in f
    ValueError
    py> g.next()
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    StopIteration

    Regards,
    Martin
    Martin v. Loewis, Jan 2, 2010
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Martin Maurer
    Replies:
    3
    Views:
    4,783
    Peter
    Apr 19, 2006
  2. Tom Machinski

    Dangerous behavior of list(generator)

    Tom Machinski, Dec 13, 2009, in forum: Python
    Replies:
    0
    Views:
    225
    Tom Machinski
    Dec 13, 2009
  3. Gabriel Genellina

    Re: Dangerous behavior of list(generator)

    Gabriel Genellina, Dec 13, 2009, in forum: Python
    Replies:
    14
    Views:
    377
    Martin v. Loewis
    Jan 2, 2010
  4. Replies:
    10
    Views:
    425
    Peter Otten
    Dec 14, 2009
  5. Replies:
    3
    Views:
    280
    Steven D'Aprano
    Dec 15, 2009
Loading...

Share This Page