Use-cases for alternative iterator

[Steven D'Aprano]

The iter() built-in takes two different forms, the familiar
iter(iterable) we all know and love, and an alternative form:

iter(callable, sentinel)

E.g.:
.... global T
.... T += 1
.... return T
....Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration



I've never seen this second form in actual code. Does anyone use it, and
if so, what use-cases do you have?

 

[Hrvoje Niksic]

I've never seen this second form in actual code. Does anyone use it,
and if so, what use-cases do you have?

Since APIs that signal end-of-iteration by returning a sentinel have
fallen out of favor in Python (with good reason), this form is rare, but
still it's sometimes useful. I've used it in actual code for reading a
file in fixed-size chunks, like this:

for chunk in iter(lambda: f.read(CHUNK_SIZE), ''):
...

 

[Peter Otten]

The iter() built-in takes two different forms, the familiar
iter(iterable) we all know and love, and an alternative form:

iter(callable, sentinel)

I've never seen this second form in actual code. Does anyone use it, and
if so, what use-cases do you have?

I found 36 examples (7% of all iter() calls) on my Kubuntu:

/usr/lib/pymodules/python2.6/multiprocessing/pool.py
iter(taskqueue.get, None)
/usr/lib/pymodules/python2.6/multiprocessing/tests.py
iter(self.child_conn.recv, None)
iter(q.get, None)
iter(conn.recv_bytes, SENTINEL)
iter(conn.recv, None)
iter(conn.recv, 'STOP')
/usr/lib/pymodules/python2.6/multiprocessing/examples/mp_workers.py
iter(input.get, 'STOP')
/usr/lib/pymodules/python2.6/pychecker2/util.py
iter(Parents(obj), None)
/usr/lib/pymodules/python2.6/processing/pool.py
iter(inqueue.get, None)
iter(taskqueue.get, None)
iter(get, None)
/usr/lib/pymodules/python2.6/processing/tests/__init__.py
iter(self.child_conn.recv, None)
iter(conn.recvBytes, '')
iter(conn.recv, None)
iter(conn.recv, 'STOP')
/usr/lib/pymodules/python2.6/IPython/Extensions/ipipe.py
iter(self, obj)
iter(self, obj)
iter(self, obj)
/usr/lib/pymodules/python2.6/nose/plugins/multiprocess.py
iter(get, 'STOP')
/usr/lib/pymodules/python2.6/django/db/models/sql/query.py
iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
self.connection.features.empty_fetchmany_value)
iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)), sentinel)
/usr/lib/pymodules/python2.6/httplib2/__init__.py
iter(self, domain)
/usr/lib/pymodules/python2.6/MySQLdb/cursors.py
iter(self.fetchone, None)
/usr/lib/pymodules/python2.6/MoinMoin/server/server_wsgi.py
iter(lambda: fileobj.read(bufsize), '')
/usr/lib/python2.6/mailbox.py
iter(self.readline, "")
iter(self.next, None)
iter(self.next, None)
/usr/lib/python2.6/multiprocessing/pool.py
iter(taskqueue.get, None)
/usr/lib/python2.6/dist-packages/mutagen/id3.py
iter(iter(parts).next, None)
/usr/lib/python2.6/dist-packages/twisted/python/zipstream.py
iter(lambda : self.read(1), "")
/usr/lib/python2.6/dist-packages/twisted/web2/test/test_stream.py
iter(s.read, None)
/usr/lib/python2.6/dist-packages/twisted/test/test_compat.py
iter(IterableCounter(20).next, 10)
/usr/lib/python2.6/dist-packages/setuptools/command/sdist.py
iter(f.readline, '')
/usr/lib/python2.6/dist-packages/lxml/cssselect.py
iter(self.next, None)
/usr/lib/python2.6/dist-packages/bzrlib/smart/protocol.py
iter(_body_decoder.read_next_chunk, None)
/usr/lib/python2.6/dist-packages/bzrlib/tests/test_smart_transport.py
iter(decoder.read_next_chunk, None)
467 36
7.15705765408

 

[Terry Reedy]

The iter() built-in takes two different forms, the familiar
iter(iterable) we all know and love, and an alternative form:

iter(callable, sentinel)
... global T
... T += 1
... return T
...
Traceback (most recent call last):
File "<stdin>", line 1, in<module>
StopIteration



I've never seen this second form in actual code. Does anyone use it, and
if so, what use-cases do you have?

Looking through Peter's nice list of real use cases, I see two
categories: adapting an iterator-like function to Python's protocol;
stopping an iterator before its normal exhaustion.

An example of the latter: suppose a file consists of header and body
separated by a blank line. An opened file is an iterable, but its
iterator only stops at eof. To process the two parts separately (using
example from Peter's list):

for l in iter(f.readline,''):
<process header line>
for l in f:
<process body line>

(I am pretty sure that there are no buffer screwups with stop and
restart since f iterator should call f.readline internally.)
Alternative is explicit "if l == '': break" in first loop.

I suspect the two param form, still rather new and unusual, is used less
that is could, and perhaps should be. I am glad you raised the question
to get me thinking about it.