Why are "broken iterators" broken?

Discussion in 'Python' started by Steven D'Aprano, Sep 21, 2008.

  1. According to the Python docs, once an iterator raises StopIteration, it
    should continue to raise StopIteration forever. Iterators that fail to
    behave in this fashion are deemed to be "broken":

    http://docs.python.org/lib/typeiter.html

    I don't understand the reasoning behind this. As I understand it, an
    iterator is something like a stream. There's no constraint that once a
    stream is empty it must remain empty forever.

    Can somebody explain why "broken iterators" are broken?


    --
    Steven
     
    Steven D'Aprano, Sep 21, 2008
    #1
    1. Advertising

  2. Steven D'Aprano wrote:

    > According to the Python docs, once an iterator raises StopIteration, it
    > should continue to raise StopIteration forever. Iterators that fail to
    > behave in this fashion are deemed to be "broken":
    >
    > http://docs.python.org/lib/typeiter.html
    >
    > I don't understand the reasoning behind this. As I understand it, an
    > iterator is something like a stream. There's no constraint that once a
    > stream is empty it must remain empty forever.


    it's a design guideline, not an absolute rule.

    but I disagree that an iterator is "something like a stream". it's
    rather "something like a pointer or an index", that is, an object that
    helps you iterate over all members in a collection.

    </F>
     
    Fredrik Lundh, Sep 21, 2008
    #2
    1. Advertising

  3. Steven D'Aprano

    Roy Smith Guest

    In article <>,
    Fredrik Lundh <> wrote:

    > Steven D'Aprano wrote:
    >
    > > According to the Python docs, once an iterator raises StopIteration, it
    > > should continue to raise StopIteration forever. Iterators that fail to
    > > behave in this fashion are deemed to be "broken":
    > >
    > > http://docs.python.org/lib/typeiter.html
    > >
    > > I don't understand the reasoning behind this. As I understand it, an
    > > iterator is something like a stream. There's no constraint that once a
    > > stream is empty it must remain empty forever.

    >
    > it's a design guideline, not an absolute rule.
    >
    > but I disagree that an iterator is "something like a stream". it's
    > rather "something like a pointer or an index", that is, an object that
    > helps you iterate over all members in a collection.
    >
    > </F>


    There are plausible examples of collections which grow while you're
    iterating over them. I'm thinking specifically of a queue in a
    multi-threaded application. One thread pushes work onto the back of the
    queue while another pops from the front. The queue could certainly go
    empty at times. But, maybe a Python iterator is just the wrong way to
    model such behavior.
     
    Roy Smith, Sep 21, 2008
    #3
  4. Roy Smith wrote:

    > There are plausible examples of collections which grow while you're
    > iterating over them. I'm thinking specifically of a queue in a
    > multi-threaded application. One thread pushes work onto the back of the
    > queue while another pops from the front. The queue could certainly go
    > empty at times. But, maybe a Python iterator is just the wrong way to
    > model such behavior.


    you probably want the consumer thread to block when it catches up with
    the producer, rather than exit.

    (that's the default behaviour of Python's Queue object, btw)

    </F>
     
    Fredrik Lundh, Sep 21, 2008
    #4
  5. Steven D'Aprano

    Terry Reedy Guest

    Steven D'Aprano wrote:
    > According to the Python docs, once an iterator raises StopIteration, it
    > should continue to raise StopIteration forever. Iterators that fail to
    > behave in this fashion are deemed to be "broken":
    >
    > http://docs.python.org/lib/typeiter.html
    >
    > I don't understand the reasoning behind this. As I understand it, an
    > iterator is something like a stream. There's no constraint that once a
    > stream is empty it must remain empty forever.


    It is quite possible that a stream reader will return '' on one call and
    then something non-empty the next. An iterator that reads a stream
    and yields chunks of whatever size should either block until it gets
    sufficient data or yield nulls as long as the stream is open and not
    raise StopIteration until the steam is closed and it has yielded the
    last chunk of data.

    > Can somebody explain why "broken iterators" are broken?


    There is an important different between a store that is closed until the
    next day and one that closed - out of business. Similarly, there is a
    difference between an item being out-of-stock until the next delivery
    and out-of-stock and discontinued permanently, or between a road closed
    for repairs versus removal for something else. Using the same sign or
    signal for temporary and permanent conditions is confusing and therefore
    'broken'.

    Terry Jan Reedy
     
    Terry Reedy, Sep 21, 2008
    #5
  6. Steven D'Aprano

    Miles Guest

    On Sun, Sep 21, 2008 at 11:13 AM, Steven D'Aprano
    <> wrote:
    > According to the Python docs, once an iterator raises StopIteration, it
    > should continue to raise StopIteration forever. Iterators that fail to
    > behave in this fashion are deemed to be "broken":
    >
    > http://docs.python.org/lib/typeiter.html
    >
    > I don't understand the reasoning behind this. As I understand it, an
    > iterator is something like a stream. There's no constraint that once a
    > stream is empty it must remain empty forever.
    >
    > Can somebody explain why "broken iterators" are broken?


    It's not a terribly onerous restriction. If you're iterating with a
    for-loop, you can make the iterable return a new iterator object when
    the old one is exhausted, and if the intent is for the next()-method
    to be called directly, you don't have to conform to the iterator
    protocol.

    Strictly speaking, file objects are broken iterators:

    Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:16)
    >>> f = open('foo')
    >>> it = iter(f)
    >>> it.next()

    'hi\n'
    >>> it.next()

    'bye\n'
    >>> it.next()

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    StopIteration
    >>> f.seek(0)
    >>> it.next()

    'hi\n'

    -Miles
     
    Miles, Sep 22, 2008
    #6
  7. Steven D'Aprano

    Craig Allen Guest

    I'm interested what others think of this because at first I couldn't
    get it... I have an object which can iterate over its parts... and at
    first I thought, what? I'm supposed to create a new object every time
    the user needs to iterate the contents?

    In the end I interpreted that statement as if "unless __iter__()" is
    called again, in which case it makes sense that an iterator should
    stay "finished" until it's told to start to iterate again.

    Then even the stream analogy holds, you don't expect a stream to say
    EOF then start giving you bytes (from beyond the end, I guess)...
    instead, such a stream would more likely block if it got to the end of
    available data.

    I'm probably being stupid in this... perhaps I'm the only one that at
    first interpreted the phrase as being regardless of a fresh call to
    __iter__()... and that the OP was worried about some other
    implication. But if I was wrong to think it's ok for one and the same
    iterator to reset when __iter__() is called again, then I seriously
    don't understand.

    what does "forever" mean in that dictum?
     
    Craig Allen, Sep 23, 2008
    #7
  8. Steven D'Aprano

    greg Guest

    Craig Allen wrote:

    > In the end I interpreted that statement as if "unless __iter__()" is
    > called again


    It seems you were confusing iterators and iterables. The
    iterator is the object that is returned by calling
    __iter__() on an iterable, and yes, you are expected to
    get a new one each time you want to start iterating
    again.

    --
    Greg
     
    greg, Sep 27, 2008
    #8
  9. Steven D'Aprano

    Lie Guest

    On Sep 21, 10:13 pm, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:
    > According to the Python docs, once an iterator raises StopIteration, it
    > should continue to raise StopIteration forever. Iterators that fail to
    > behave in this fashion are deemed to be "broken":
    >
    > http://docs.python.org/lib/typeiter.html
    >
    > I don't understand the reasoning behind this. As I understand it, an
    > iterator is something like a stream. There's no constraint that once a
    > stream is empty it must remain empty forever.


    I think empty != StopIteration. StopIteration (IMHO) shouldn't be
    raised when the stream is empty, instead a sentinel value specifying
    that "there is no data yet, but if you wait there might be" should be
    returned (possibly None or empty string). When you raise
    StopIteration, it is a signal that I don't have any more data and
    there is no use in waiting.

    > Can somebody explain why "broken iterators" are broken?
    >
    > --
    > Steven
     
    Lie, Sep 28, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?VGltOjouLg==?=

    Why, why, why???

    =?Utf-8?B?VGltOjouLg==?=, Jan 27, 2005, in forum: ASP .Net
    Replies:
    6
    Views:
    574
    Juan T. Llibre
    Jan 27, 2005
  2. Mr. SweatyFinger

    why why why why why

    Mr. SweatyFinger, Nov 28, 2006, in forum: ASP .Net
    Replies:
    4
    Views:
    913
    Mark Rae
    Dec 21, 2006
  3. Mr. SweatyFinger
    Replies:
    2
    Views:
    2,009
    Smokey Grindel
    Dec 2, 2006
  4. Cameron Simpson

    Re: Why are "broken iterators" broken?

    Cameron Simpson, Sep 22, 2008, in forum: Python
    Replies:
    0
    Views:
    592
    Cameron Simpson
    Sep 22, 2008
  5. Fredrik Lundh

    Re: Why are "broken iterators" broken?

    Fredrik Lundh, Sep 22, 2008, in forum: Python
    Replies:
    0
    Views:
    608
    Fredrik Lundh
    Sep 22, 2008
Loading...

Share This Page