What makes an iterator an iterator?

Discussion in 'Python' started by Steven D'Aprano, Apr 18, 2007.

  1. I thought that an iterator was any object that follows the iterator
    protocol, that is, it has a next() method and an __iter__() method.

    But I'm having problems writing a class that acts as an iterator. I have:

    class Parrot(object):
    def __iter__(self):
    return self
    def __init__(self):
    self.next = self._next()
    def _next(self):
    for word in "Norwegian Blue's have beautiful plumage!".split():
    yield word

    But this is what I get:

    >>> P = Parrot()
    >>> for word in P:

    .... print word
    ....
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: iter() returned non-iterator of type 'Parrot'

    Why is my instance not an iterator?

    But I can do this:

    >>> for word in P.next:

    .... print word
    ....
    Norwegian
    Blue's
    have
    beautiful
    plumage!

    I find myself perplexed as to this behaviour.


    --
    Steven D'Aprano
     
    Steven D'Aprano, Apr 18, 2007
    #1
    1. Advertising

  2. Steven D'Aprano

    I V Guest

    On Wed, 18 Apr 2007 15:39:22 +1000, Steven D'Aprano wrote:
    > I thought that an iterator was any object that follows the iterator
    > protocol, that is, it has a next() method and an __iter__() method.

    ....
    > class Parrot(object):

    ....
    > def __init__(self):
    > self.next = self._next()


    self.next isn't a method here, it's a generator. You could do:

    def __init__(self):
    self.next = self._next().next

    But, having tested, that doesn't appear to work either - I think
    this is because, again, self.next is not strictly a method, it's an
    attribute that happens to be callable.

    The python manual gives you a possible solution:

    ---QUOTE http://docs.python.org/lib/typeiter.html ---
    Python's generators provide a convenient way to implement the iterator
    protocol. If a container object's __iter__() method is implemented as a
    generator, it will automatically return an iterator object (technically, a
    generator object) supplying the __iter__() and next() methods.
    ---END QUOTE---

    i.e., just rename your _next function to __iter__ . Your class won't
    itself be an iterator, but it will be usable in for statements and so one,
    and convertable to an iterator with the iter builtin.
     
    I V, Apr 18, 2007
    #2
    1. Advertising

  3. On Wed, 18 Apr 2007 15:39:22 +1000, Steven D'Aprano
    <> declaimed the following in
    comp.lang.python:

    > I thought that an iterator was any object that follows the iterator
    > protocol, that is, it has a next() method and an __iter__() method.
    >
    > But I'm having problems writing a class that acts as an iterator. I have:
    >
    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):
    > self.next = self._next()
    > def _next(self):


    >>> class Parrot(object):

    .... def __iter__(self):
    .... return self.next
    .... def __init__(self):
    .... self.next = self._next()
    .... def _next(self):
    .... for word in "Norwegian Blue's have beautiful
    plumage!".split():
    .... yield word
    ....
    >>> p = Parrot()
    >>> for word in p:

    .... print word
    ....
    Norwegian
    Blue's
    have
    beautiful
    plumage!
    >>>

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Apr 18, 2007
    #3
  4. Steven D'Aprano

    Ben Finney Guest

    Steven D'Aprano <> writes:

    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):
    > self.next = self._next()
    > def _next(self):
    > for word in "Norwegian Blue's have beautiful plumage!".split():
    > yield word


    Clearly the problem is you've misused an apostrophe. Python doesn't
    like the plural getting an apostrophe.

    <URL:http://www.angryflower.com/bobsqu.gif>

    --
    \ "Speech is conveniently located midway between thought and |
    `\ action, where it often substitutes for both." -- John Andrew |
    _o__) Holmes, _Wisdom in Small Doses_ |
    Ben Finney
     
    Ben Finney, Apr 18, 2007
    #4
  5. Steven D'Aprano

    Stefan Rank Guest

    on 18.04.2007 07:39 Steven D'Aprano said the following:
    > I thought that an iterator was any object that follows the iterator


    replace object with "instance of a class", i.e. the relevant methods are
    looked up in the __class__ not in the instance (I think).
    I had the same troubles trying to dynamically reassign a __call__ method...

    > protocol, that is, it has a next() method and an __iter__() method.
    >
    > But I'm having problems writing a class that acts as an iterator. I have:
    >
    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):
    > self.next = self._next()
    > def _next(self):
    > for word in "Norwegian Blue's have beautiful plumage!".split():
    > yield word


    Try this::

    class Parrot(object):
    def __iter__(self):
    return self
    def __init__(self):
    self.__class__.next = self._next().next # see post by I V
    def _next(self):
    for word in "Norwegian Blue's have beautiful plumage!".split():
    yield word

    This works but practically forces the class to be used as a singleton...
    not very helpful :)

    Better:

    * use the '__iter__ returns/is a generator' way,
    * or if you need the object to be the iterator, implement the next
    method directly on the class::

    class Parrot(object):
    def _next(self):
    for word in "Norwegian Blue's have beautiful plumage!".split():
    yield word
    def __iter__(self):
    self.generator = self._next()
    return self
    def next(self):
    return self.generator.next()

    cheers,
    stefan
     
    Stefan Rank, Apr 18, 2007
    #5
  6. On Wed, 18 Apr 2007 16:58:23 +1000, Ben Finney wrote:

    > Steven D'Aprano <> writes:
    >
    >> class Parrot(object):
    >> def __iter__(self):
    >> return self
    >> def __init__(self):
    >> self.next = self._next()
    >> def _next(self):
    >> for word in "Norwegian Blue's have beautiful plumage!".split():
    >> yield word

    >
    > Clearly the problem is you've misused an apostrophe. Python doesn't
    > like the plural getting an apostrophe.
    >
    > <URL:http://www.angryflower.com/bobsqu.gif>


    I thought the rule wa's that any time you 'see an 'S, you put an
    apo'strophe before it. If that's wrong, 'shouldn't it rai'se an exception?




    --
    'Steven D'Aprano
     
    Steven D'Aprano, Apr 18, 2007
    #6
  7. On Wed, 18 Apr 2007 06:13:39 +0000, I V wrote:

    > On Wed, 18 Apr 2007 15:39:22 +1000, Steven D'Aprano wrote:
    >> I thought that an iterator was any object that follows the iterator
    >> protocol, that is, it has a next() method and an __iter__() method.


    [snip]

    > i.e., just rename your _next function to __iter__ . Your class won't
    > itself be an iterator, but it will be usable in for statements and so one,
    > and convertable to an iterator with the iter builtin.



    Thanks to all those who helped, this fixed my problem.

    For the record, this is what I actually wanted: a four-line self-sorting
    dictionary:

    class SortedDict(dict):
    def __iter__(self):
    for key in sorted(self.keys()):
    yield key

    Note that using sorted(self) does not work.

    Iterating over a SortedDictionary returns the keys in sorted order. This
    minimalist implementation doesn't sort the values, items or string
    representation of the dict, but they should be easy to implement.



    --
    Steven D'Aprano
     
    Steven D'Aprano, Apr 18, 2007
    #7
  8. Steven D'Aprano

    Paul McGuire Guest

    On Apr 18, 3:32 am, Steven D'Aprano <>
    wrote:
    > On Wed, 18 Apr 2007 06:13:39 +0000, I V wrote:
    > > On Wed, 18 Apr 2007 15:39:22 +1000, Steven D'Aprano wrote:
    > >> I thought that an iterator was any object that follows the iterator
    > >> protocol, that is, it has a next() method and an __iter__() method.

    >
    > [snip]
    >
    > > i.e., just rename your _next function to __iter__ . Your class won't
    > > itself be an iterator, but it will be usable in for statements and so one,
    > > and convertable to an iterator with the iter builtin.

    >
    > Thanks to all those who helped, this fixed my problem.
    >
    > For the record, this is what I actually wanted: a four-line self-sorting
    > dictionary:
    >
    > class SortedDict(dict):
    > def __iter__(self):
    > for key in sorted(self.keys()):
    > yield key
    >
    > Note that using sorted(self) does not work.
    >
    > Iterating over a SortedDictionary returns the keys in sorted order. This
    > minimalist implementation doesn't sort the values, items or string
    > representation of the dict, but they should be easy to implement.
    >
    > --
    > Steven D'Aprano


    Very neat. Why not this?

    class SortedDict(dict):
    def __iter__(self):
    return iter(sorted(self.keys()))

    -- Paul
     
    Paul McGuire, Apr 18, 2007
    #8
  9. Steven D'Aprano

    Peter Otten Guest

    Steven D'Aprano wrote:

    > class SortedDict(dict):
    >     def __iter__(self):
    >         for key in sorted(self.keys()):
    >             yield key
    >
    > Note that using sorted(self) does not work.


    That's because sorted() invokes __iter__() if present. To prevent the
    recursion you can explicitly invoke dict.__iter__():

    >>> class SortedDict(dict):

    .... def __iter__(self):
    .... return iter(sorted(super(SortedDict, self).__iter__()))
    ....
    >>> sd = SortedDict(a=1, b=2, c=3)
    >>> list(sd)

    ['a', 'b', 'c']

    Note that a list of keys is still built before the first key is yielded,
    and, unlike dict, you can modify your SortedDict while iterating over it:

    >>> for k in sd:

    .... if k == "b": sd["x"] = 42
    ....
    >>> sd

    {'a': 1, 'x': 42, 'c': 3, 'b': 2}

    whereas:

    >>> d = dict(a=1, b=2, c=3)
    >>> for k in d:

    .... if k == "b": d["x"] = 42
    ....
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    RuntimeError: dictionary changed size during iteration

    By the way, I think it would be worthwile to change super() to allow
    e. g. super(SomeClass, self)[...] as an alternate spelling for
    super(SomeClass, self).__getitem__(...) etc. With such an enhancement
    SortedDict would become

    class SortedDict(dict):
    def __iter__(self):
    # doesn't work in current Python
    iter(sorted(super(SortedDict, self)))


    Peter
     
    Peter Otten, Apr 18, 2007
    #9
  10. Steven D'Aprano

    Georg Brandl Guest

    Stefan Rank schrieb:
    > on 18.04.2007 07:39 Steven D'Aprano said the following:
    >> I thought that an iterator was any object that follows the iterator

    >
    > replace object with "instance of a class", i.e. the relevant methods are
    > looked up in the __class__ not in the instance (I think).
    > I had the same troubles trying to dynamically reassign a __call__ method...


    This is correct.

    It's not properly documented though, and not applied consistently, e.g.
    __enter__ and __exit__ are looked up in the instance itself.

    Georg
     
    Georg Brandl, Apr 18, 2007
    #10
  11. >
    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):



    Typo right here!!!!

    > self.next = self._next()


    write:
    self.next = self._next

    no parenthesis.

    > def _next(self):
    > for word in "Norwegian Blue's have beautiful plumage!".split():
    > yield word
    >



    See previous explanation.

    thanks,

    - Isaac.
     
    Isaac Rodriguez, Apr 18, 2007
    #11
  12. Sorry, my previous post was incomplete. I didn't realized that you
    implemented _next() as a generator funcition. Besides changing
    __init__() from

    self.next = self._next()

    to
    self.next = self._next

    you need to implement __iter__() as:

    return self.next()


    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):
    > self.next = self._next()
    > def _next(self):
    > for word in "Norwegian Blue's have beautiful plumage!".split():
    > yield word
    >
     
    Isaac Rodriguez, Apr 18, 2007
    #12
  13. Steven D'Aprano <> wrote:

    > I thought that an iterator was any object that follows the iterator
    > protocol, that is, it has a next() method and an __iter__() method.


    The special methods need to be on the type -- having attributes of those
    names on the instance doesn't help (applies to all special methods in
    the normal, aka newstyle, object model; legacy, aka classic, classes,
    work by slightly different and not entirely self-consistent semantics).


    Alex
     
    Alex Martelli, Apr 18, 2007
    #13
  14. Steven D'Aprano

    Guest


    > I find myself perplexed as to this behaviour.


    You can not iterate over a dead object!
     
    , Apr 18, 2007
    #14
  15. Steven D'Aprano

    Guest

    On Apr 18, 10:36 am, wrote:
    > > I find myself perplexed as to this behaviour.

    >
    > You can not iterate over a dead object!


    It's not dead, it's restin'. All shagged out over a long squak.
     
    , Apr 18, 2007
    #15
  16. Steven D'Aprano

    7stud Guest

    On Apr 18, 8:50 am, (Alex Martelli) wrote:
    > The special methods need to be on the type -- having attributes of those
    > names on the instance doesn't help (applies to all special methods in
    > the normal, aka newstyle, object model; legacy, aka classic, classes,
    > work by slightly different and not entirely self-consistent semantics).
    >
    > Alex


    Can you explain some of the details of why this code fails:

    ---
    class Parrot(object):
    def __iter__(self):
    return self
    def __init__(self):
    self.next = self.next().next
    def next(self):
    for word in "Norwegian Blue's have beautiful
    plumage!".split():
    yield word

    P = Parrot()
    for word in P:
    print word
    ------

    It causes an infinite loop that just prints out the iterator object
    returned when you call the generator function.

    If I try this:

    -----
    class Parrot(object):
    def __iter__(self):
    return self
    def __init__(self):
    print self.next()
    print self.next().next
    #self.next = self.next().next
    def next(self):
    for word in "Norwegian Blue's have beautiful
    plumage!".split():
    yield word

    P = Parrot()

    '''
    for word in P:
    print word
    '''
    ----------

    the output is:

    <generator object at 0x5b080>
    <method-wrapper object at 0x55d30>

    Based on that output, self.next() is the iterator object that wraps
    the generator function, and it has a next() method. If 'i' is the
    iterator object, then the statement:

    self.next = self.next().next

    is equivalent to:

    self.next = i.next

    and therefor calling P.next() is equivalent to i.next(), which appears
    to be exactly what you want. As I understand it, this is how the for
    loop works:

    1) The for loop causes the built in, global iter() function to be
    called with P as an argument: iter(P). That calls P's __iter__()
    method, which just returns P.

    2) The for loop repeatedly calls next() on whatever object is returned
    by 1), so that results in calls to P.next().

    3) P.next() is equivalent to i.next()

    I don't understand at which step does the code fail.
     
    7stud, Apr 18, 2007
    #16
  17. Steven D'Aprano

    Terry Reedy Guest

    An iterator is an object with a .__iter__ method that returns self and a
    ..next method that either returns an object or raises StopIteration.

    One very good way to get an iterator from an iterable is for .__iter__ to
    be a generator function. When called, it returns an generator with
    ..__iter__ and .next methods that work as specified.

    words = "Norwegian Blues have beautiful plumage!".split()
    print words

    prints
    ['Norwegian', 'Blues', 'have', 'beautiful', 'plumage!']

    class Parrot(object):
    def __init__(self, words):
    self.words = words
    def __iter__(self):
    for word in words:
    yield word

    for word in Parrot(words):
    print word

    prints

    Norwegian
    Blues
    have
    beautiful
    plumage!

    One can also make an iterable an (self) iterator by correctly writing the
    two needed methods.

    class Parrot2(object):
    def __init__(self, words):
    self.words = words
    def __iter__(self):
    self.it = -1
    return self
    def next(self):
    self.it += 1
    try:
    return self.words[self.it]
    except IndexError:
    raise StopIteration

    for word in Parrot2(words):
    print word

    which prints the same 5 lines. However, when an object is its own
    iterator, it can only be one iterator at a time (here, pointing at one
    place in the word list), whereas the first method allows multiple iterators
    (here, possibly pointing to different places in the list.

    | Can you explain some of the details of why this code fails:

    [snip all examples of bad code that violate the iterator rule by improperly
    writing .next as a generator function]

    I think most people should learn how to write iterators correctly, as I
    gave examples of above, rather than worry about the details of how and why
    mis-written code fails with a particular version of a particular
    implementation. So I won't go down that road.

    Terry Jan Reedy
     
    Terry Reedy, Apr 19, 2007
    #17
  18. 7stud <> wrote:
    ...
    > Can you explain some of the details of why this code fails:

    ...
    > def next(self):
    > for word in "Norwegian Blue's have beautiful
    > plumage!".split():
    > yield word


    Sure, easily: a loop like "for x in y:" binds an unnamed temporary
    variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
    be pedantic type(_t).next(t)] until that raises StopIteration.

    Calling a generator, such as this next method, returns an iterator
    object; calling it repeatedly returns many such iterator objects, and
    never raises StopIteration, thus obviously producing an unending loop.


    Alex
     
    Alex Martelli, Apr 19, 2007
    #18
  19. Steven D'Aprano

    7stud Guest

    Hi,

    Thanks for the responses.

    > 7stud <> wrote:
    > > Can you explain some of the details of why this code fails:

    > ---
    > class Parrot(object):
    > def __iter__(self):
    > return self
    > def __init__(self):
    > self.next = self.next().next
    > def next(self):
    > for word in "Norwegian Blue's have beautiful
    > plumage!".split():
    > yield word
    >
    > P = Parrot()
    > for word in P:
    > print word
    > ------


    On Apr 18, 8:45 pm, (Alex Martelli) wrote:
    >
    > ...a loop like "for x in y:" binds an unnamed temporary
    > variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
    > be pedantic type(_t).next(t)] until that raises StopIteration.



    Aiiii. Isn't this the crux:

    > repeatedly calls....[type(_t).next(t)]


    As far as I can tell, if the call was actually _t.next(), the code I
    asked about would work. However, the name look up for 'next' when you
    call:

    P.next()

    is entirely different from the name look up for 'next' when you call:

    type(P).next().

    In the first lookup, the instance attribute "next" hides the class
    attribute "next". In the second lookup, the "next" attribute of the
    class is returned, i.e. the next() method. Yet, a generator-function-
    call returns an iterator object that wraps the generator function, so
    when the for loop makes repeated calls to type(P).next(), an iterator
    object is repeatedly returned--what you want is i.next() to be
    returned.

    I suspected next() might be called through the class, but after
    carefully parsing the section on iterators (Python in a Nutshell, p.
    65) where it says iter() returns an object i, and then the for loop
    repeatedly calls i.next(), I dismissed that idea. I have to admit I'm
    still a little confused by why you only parenthetically noted that
    information and called it pedantic.


    On Apr 18, 8:38 pm, "Terry Reedy" <> wrote:
    >
    > One very good way to get an iterator from an iterable is for .__iter__ to
    > be a generator function.


    Ahhh. That eliminates having to deal with next().next constructs.
    Nice.

    > snip all examples of bad code that violate the iterator rule
    > by improperly writing .next as a generator function


    What iterator rule states that .next can't be a generator function?
    My book says an iterator is any object with a .next method that is
    callable without arguments (Python in a Nutshell(p.65) says the same
    thing). Also, my boos says an iterable is any object with an
    __iter__ method. As a result, next() and __iter__() don't have to
    be in the same object:

    lass MyIterator(object):
    def __init__(self, obj):
    self.toIterateOver = obj
    def next(self):
    for x in range(self.toIterateOver.age):
    print x
    raise StopIteration

    class Dog(object):
    def __init__(self, age):
    self.age = age
    def __iter__(self):
    return MyIterator(self)

    d = Dog(5)
    for year in d:
    print year


    I've read recommendations that an iterator should additionally contain
    an __iter__() method, but I'm not sure why that is. In particular PEP
    234 says:

    ----------
    Classes can define how they are iterated over by defining an
    __iter__() method; this should take no additional arguments and
    return a valid iterator object. A class that wants to be an
    iterator should implement two methods: a next() method that
    behaves
    as described above, and an __iter__() method that returns self.

    The two methods correspond to two distinct protocols:

    1. An object can be iterated over with "for" if it implements
    __iter__() or __getitem__().

    2. An object can function as an iterator if it implements next().

    Container-like objects usually support protocol 1. Iterators are
    currently required to support both protocols. The semantics of
    iteration come only from protocol 2; protocol 1 is present to make
    iterators behave like sequences; in particular so that code
    receiving an iterator can use a for-loop over the iterator.

    Classes can define how they are iterated over by defining an
    __iter__() method; this should take no additional arguments and
    return a valid iterator object. A class that wants to be an
    iterator should implement two methods: a next() method that
    behaves
    as described above, and an __iter__() method that returns self.

    The two methods correspond to two distinct protocols:

    1. An object can be iterated over with "for" if it implements
    __iter__() or __getitem__().

    2. An object can function as an iterator if it implements next().

    Container-like objects usually support protocol 1. Iterators are
    currently required to support both protocols. The semantics of
    iteration come only from protocol 2; protocol 1 is present to make
    iterators behave like sequences; in particular so that code
    receiving an iterator can use a for-loop over the iterator.
    --------


    >The semantics of
    > iteration come only from protocol 2


    My example demonstrates that.

    >protocol 1 is present to make
    > iterators behave like sequences; in particular so that code
    > receiving an iterator can use a for-loop over the iterator.


    I don't understand that part--it looks like my example is using a for
    loop over the iterator.
     
    7stud, Apr 19, 2007
    #19
  20. On Wed, 18 Apr 2007 01:45:10 -0700, Paul McGuire wrote:

    >> For the record, this is what I actually wanted: a four-line self-sorting
    >> dictionary:
    >>
    >> class SortedDict(dict):
    >> def __iter__(self):
    >> for key in sorted(self.keys()):
    >> yield key


    [snip]

    > Very neat. Why not this?
    >
    > class SortedDict(dict):
    > def __iter__(self):
    > return iter(sorted(self.keys()))



    Good question. I don't have a good answer except for "because I didn't
    think of it".


    --
    Steven.
     
    Steven D'Aprano, Apr 19, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Hendrik Maryns
    Replies:
    18
    Views:
    1,429
  2. greg
    Replies:
    6
    Views:
    460
    Dietmar Kuehl
    Jul 17, 2003
  3. Replies:
    6
    Views:
    653
    Jim Langston
    Oct 30, 2005
  4. David Bilsby
    Replies:
    5
    Views:
    2,052
    David Bilsby
    Oct 9, 2007
  5. Pranav
    Replies:
    3
    Views:
    711
    ManicQin
    Aug 8, 2008
Loading...

Share This Page