What makes an iterator an iterator?

S

Steven D'Aprano

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

But this is what I get:
.... print word
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: iter() returned non-iterator of type 'Parrot'

Why is my instance not an iterator?

But I can do this:
.... print word
....
Norwegian
Blue's
have
beautiful
plumage!

I find myself perplexed as to this behaviour.
 
I

I V

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method. ....
class Parrot(object): ....
def __init__(self):
self.next = self._next()

self.next isn't a method here, it's a generator. You could do:

def __init__(self):
self.next = self._next().next

But, having tested, that doesn't appear to work either - I think
this is because, again, self.next is not strictly a method, it's an
attribute that happens to be callable.

The python manual gives you a possible solution:

---QUOTE http://docs.python.org/lib/typeiter.html ---
Python's generators provide a convenient way to implement the iterator
protocol. If a container object's __iter__() method is implemented as a
generator, it will automatically return an iterator object (technically, a
generator object) supplying the __iter__() and next() methods.
---END QUOTE---

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.
 
D

Dennis Lee Bieber

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
.... def __iter__(self):
.... return self.next
.... def __init__(self):
.... self.next = self._next()
.... def _next(self):
.... for word in "Norwegian Blue's have beautiful
plumage!".split():
.... yield word
.... .... print word
....
Norwegian
Blue's
have
beautiful
plumage!--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 
B

Ben Finney

Steven D'Aprano said:
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

Clearly the problem is you've misused an apostrophe. Python doesn't
like the plural getting an apostrophe.

<URL:http://www.angryflower.com/bobsqu.gif>
 
S

Stefan Rank

I thought that an iterator was any object that follows the iterator

replace object with "instance of a class", i.e. the relevant methods are
looked up in the __class__ not in the instance (I think).
I had the same troubles trying to dynamically reassign a __call__ method...
protocol, that is, it has a next() method and an __iter__() method.

But I'm having problems writing a class that acts as an iterator. I have:

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self._next()
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

Try this::

class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.__class__.next = self._next().next # see post by I V
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word

This works but practically forces the class to be used as a singleton...
not very helpful :)

Better:

* use the '__iter__ returns/is a generator' way,
* or if you need the object to be the iterator, implement the next
method directly on the class::

class Parrot(object):
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word
def __iter__(self):
self.generator = self._next()
return self
def next(self):
return self.generator.next()

cheers,
stefan
 
S

Steven D'Aprano

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.
[snip]

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.


Thanks to all those who helped, this fixed my problem.

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key

Note that using sorted(self) does not work.

Iterating over a SortedDictionary returns the keys in sorted order. This
minimalist implementation doesn't sort the values, items or string
representation of the dict, but they should be easy to implement.
 
P

Paul McGuire

I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.
[snip]

i.e., just rename your _next function to __iter__ . Your class won't
itself be an iterator, but it will be usable in for statements and so one,
and convertable to an iterator with the iter builtin.

Thanks to all those who helped, this fixed my problem.

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key

Note that using sorted(self) does not work.

Iterating over a SortedDictionary returns the keys in sorted order. This
minimalist implementation doesn't sort the values, items or string
representation of the dict, but they should be easy to implement.

Very neat. Why not this?

class SortedDict(dict):
def __iter__(self):
return iter(sorted(self.keys()))

-- Paul
 
P

Peter Otten

Steven said:
class SortedDict(dict):
    def __iter__(self):
        for key in sorted(self.keys()):
            yield key

Note that using sorted(self) does not work.

That's because sorted() invokes __iter__() if present. To prevent the
recursion you can explicitly invoke dict.__iter__():
.... def __iter__(self):
.... return iter(sorted(super(SortedDict, self).__iter__()))
....['a', 'b', 'c']

Note that a list of keys is still built before the first key is yielded,
and, unlike dict, you can modify your SortedDict while iterating over it:
.... if k == "b": sd["x"] = 42
....{'a': 1, 'x': 42, 'c': 3, 'b': 2}

whereas:
.... if k == "b": d["x"] = 42
....
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

By the way, I think it would be worthwile to change super() to allow
e. g. super(SomeClass, self)[...] as an alternate spelling for
super(SomeClass, self).__getitem__(...) etc. With such an enhancement
SortedDict would become

class SortedDict(dict):
def __iter__(self):
# doesn't work in current Python
iter(sorted(super(SortedDict, self)))


Peter
 
G

Georg Brandl

Stefan said:
replace object with "instance of a class", i.e. the relevant methods are
looked up in the __class__ not in the instance (I think).
I had the same troubles trying to dynamically reassign a __call__ method...

This is correct.

It's not properly documented though, and not applied consistently, e.g.
__enter__ and __exit__ are looked up in the instance itself.

Georg
 
I

Isaac Rodriguez

class Parrot(object):
def __iter__(self):
return self
def __init__(self):


Typo right here!!!!
self.next = self._next()

write:
self.next = self._next

no parenthesis.
def _next(self):
for word in "Norwegian Blue's have beautiful plumage!".split():
yield word


See previous explanation.

thanks,

- Isaac.
 
I

Isaac Rodriguez

Sorry, my previous post was incomplete. I didn't realized that you
implemented _next() as a generator funcition. Besides changing
__init__() from

self.next = self._next()

to
self.next = self._next

you need to implement __iter__() as:

return self.next()
 
A

Alex Martelli

Steven D'Aprano said:
I thought that an iterator was any object that follows the iterator
protocol, that is, it has a next() method and an __iter__() method.

The special methods need to be on the type -- having attributes of those
names on the instance doesn't help (applies to all special methods in
the normal, aka newstyle, object model; legacy, aka classic, classes,
work by slightly different and not entirely self-consistent semantics).


Alex
 
7

7stud

The special methods need to be on the type -- having attributes of those
names on the instance doesn't help (applies to all special methods in
the normal, aka newstyle, object model; legacy, aka classic, classes,
work by slightly different and not entirely self-consistent semantics).

Alex

Can you explain some of the details of why this code fails:

---
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()
for word in P:
print word
------

It causes an infinite loop that just prints out the iterator object
returned when you call the generator function.

If I try this:

-----
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
print self.next()
print self.next().next
#self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()

'''
for word in P:
print word
'''
----------

the output is:

<generator object at 0x5b080>
<method-wrapper object at 0x55d30>

Based on that output, self.next() is the iterator object that wraps
the generator function, and it has a next() method. If 'i' is the
iterator object, then the statement:

self.next = self.next().next

is equivalent to:

self.next = i.next

and therefor calling P.next() is equivalent to i.next(), which appears
to be exactly what you want. As I understand it, this is how the for
loop works:

1) The for loop causes the built in, global iter() function to be
called with P as an argument: iter(P). That calls P's __iter__()
method, which just returns P.

2) The for loop repeatedly calls next() on whatever object is returned
by 1), so that results in calls to P.next().

3) P.next() is equivalent to i.next()

I don't understand at which step does the code fail.
 
T

Terry Reedy

An iterator is an object with a .__iter__ method that returns self and a
..next method that either returns an object or raises StopIteration.

One very good way to get an iterator from an iterable is for .__iter__ to
be a generator function. When called, it returns an generator with
..__iter__ and .next methods that work as specified.

words = "Norwegian Blues have beautiful plumage!".split()
print words

prints
['Norwegian', 'Blues', 'have', 'beautiful', 'plumage!']

class Parrot(object):
def __init__(self, words):
self.words = words
def __iter__(self):
for word in words:
yield word

for word in Parrot(words):
print word

prints

Norwegian
Blues
have
beautiful
plumage!

One can also make an iterable an (self) iterator by correctly writing the
two needed methods.

class Parrot2(object):
def __init__(self, words):
self.words = words
def __iter__(self):
self.it = -1
return self
def next(self):
self.it += 1
try:
return self.words[self.it]
except IndexError:
raise StopIteration

for word in Parrot2(words):
print word

which prints the same 5 lines. However, when an object is its own
iterator, it can only be one iterator at a time (here, pointing at one
place in the word list), whereas the first method allows multiple iterators
(here, possibly pointing to different places in the list.

| Can you explain some of the details of why this code fails:

[snip all examples of bad code that violate the iterator rule by improperly
writing .next as a generator function]

I think most people should learn how to write iterators correctly, as I
gave examples of above, rather than worry about the details of how and why
mis-written code fails with a particular version of a particular
implementation. So I won't go down that road.

Terry Jan Reedy
 
A

Alex Martelli

7stud said:
Can you explain some of the details of why this code fails: ...
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

Sure, easily: a loop like "for x in y:" binds an unnamed temporary
variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
be pedantic type(_t).next(t)] until that raises StopIteration.

Calling a generator, such as this next method, returns an iterator
object; calling it repeatedly returns many such iterator objects, and
never raises StopIteration, thus obviously producing an unending loop.


Alex
 
7

7stud

Hi,

Thanks for the responses.
7stud said:
Can you explain some of the details of why this code fails:
---
class Parrot(object):
def __iter__(self):
return self
def __init__(self):
self.next = self.next().next
def next(self):
for word in "Norwegian Blue's have beautiful
plumage!".split():
yield word

P = Parrot()
for word in P:
print word
------


...a loop like "for x in y:" binds an unnamed temporary
variable (say _t) to iter(y) and then repeatedly calls _t.next() [or to
be pedantic type(_t).next(t)] until that raises StopIteration.


Aiiii. Isn't this the crux:
repeatedly calls....[type(_t).next(t)]

As far as I can tell, if the call was actually _t.next(), the code I
asked about would work. However, the name look up for 'next' when you
call:

P.next()

is entirely different from the name look up for 'next' when you call:

type(P).next().

In the first lookup, the instance attribute "next" hides the class
attribute "next". In the second lookup, the "next" attribute of the
class is returned, i.e. the next() method. Yet, a generator-function-
call returns an iterator object that wraps the generator function, so
when the for loop makes repeated calls to type(P).next(), an iterator
object is repeatedly returned--what you want is i.next() to be
returned.

I suspected next() might be called through the class, but after
carefully parsing the section on iterators (Python in a Nutshell, p.
65) where it says iter() returns an object i, and then the for loop
repeatedly calls i.next(), I dismissed that idea. I have to admit I'm
still a little confused by why you only parenthetically noted that
information and called it pedantic.


One very good way to get an iterator from an iterable is for .__iter__ to
be a generator function.

Ahhh. That eliminates having to deal with next().next constructs.
Nice.
snip all examples of bad code that violate the iterator rule
by improperly writing .next as a generator function

What iterator rule states that .next can't be a generator function?
My book says an iterator is any object with a .next method that is
callable without arguments (Python in a Nutshell(p.65) says the same
thing). Also, my boos says an iterable is any object with an
__iter__ method. As a result, next() and __iter__() don't have to
be in the same object:

lass MyIterator(object):
def __init__(self, obj):
self.toIterateOver = obj
def next(self):
for x in range(self.toIterateOver.age):
print x
raise StopIteration

class Dog(object):
def __init__(self, age):
self.age = age
def __iter__(self):
return MyIterator(self)

d = Dog(5)
for year in d:
print year


I've read recommendations that an iterator should additionally contain
an __iter__() method, but I'm not sure why that is. In particular PEP
234 says:

----------
Classes can define how they are iterated over by defining an
__iter__() method; this should take no additional arguments and
return a valid iterator object. A class that wants to be an
iterator should implement two methods: a next() method that
behaves
as described above, and an __iter__() method that returns self.

The two methods correspond to two distinct protocols:

1. An object can be iterated over with "for" if it implements
__iter__() or __getitem__().

2. An object can function as an iterator if it implements next().

Container-like objects usually support protocol 1. Iterators are
currently required to support both protocols. The semantics of
iteration come only from protocol 2; protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.

Classes can define how they are iterated over by defining an
__iter__() method; this should take no additional arguments and
return a valid iterator object. A class that wants to be an
iterator should implement two methods: a next() method that
behaves
as described above, and an __iter__() method that returns self.

The two methods correspond to two distinct protocols:

1. An object can be iterated over with "for" if it implements
__iter__() or __getitem__().

2. An object can function as an iterator if it implements next().

Container-like objects usually support protocol 1. Iterators are
currently required to support both protocols. The semantics of
iteration come only from protocol 2; protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.
--------

The semantics of
iteration come only from protocol 2

My example demonstrates that.
protocol 1 is present to make
iterators behave like sequences; in particular so that code
receiving an iterator can use a for-loop over the iterator.

I don't understand that part--it looks like my example is using a for
loop over the iterator.
 
S

Steven D'Aprano

For the record, this is what I actually wanted: a four-line self-sorting
dictionary:

class SortedDict(dict):
def __iter__(self):
for key in sorted(self.keys()):
yield key
[snip]

Very neat. Why not this?

class SortedDict(dict):
def __iter__(self):
return iter(sorted(self.keys()))


Good question. I don't have a good answer except for "because I didn't
think of it".
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top