Idioms combining 'next(items)' and 'for item in items:'

[Terry Reedy]

Python's iterator protocol for an iterator 'items' allows combinations
of explicit "next(items)" calls with the implicit calls of a "for item
in items:" loop. There are at least three situations in which this can
be useful. (While the code posted here is not testable, being incomplete
or having pseudocode lines, I have tested full examples of each idiom
with 3.2.)


1. Process first item of an iterable separately.

A traditional solution is a flag variable that is tested for each item.

first = True
<other setup>
for item in iterable:
if first:
<process first>
first = False
else:
<process non-first>

(I have seen code like this posted on this list several times, including
today.)

Better, to me, is to remove the first item *before* the loop.

items = iter(iterable)
<set up with next(items)
for item in items:
<process non-first>

Sometimes <other setup> and <process first> can be combined in <setup
with next(items)>. For instance, "result = []" followed by
"result.append(process_first(item))" becomes "result =
[process_first(item)]".

The statement containing the explicit next(items) call can optionally be
wrapped to explicitly handle the case of an empty iterable in whatever
manner is desired.

try:
<set up with next(items)>
except StopIteration:
raise ValueError("iterable cannot be empty")


For an iterable known to have a small number of items, the first item
can be removed, with only a small copying penalty, with a simple assignment.

first, *rest = iterable

The older and harder to write version of this requires the iterable to
be a sequence?

first, rest = seq[0], seq[1:]


2. Process the last item of an iterable differently. As far as I know,
this cannot be done for a generic iterable with a flag. It requires a
look ahead.

items = iter(iterable)
current = next(items)
for item in items:
<process non-last current>
current = item
<process last current>

To treat both first and last differently, pull off the first before
binding 'current'. Wrap next() calls as desired.


3. Process the items of an iterable in pairs.

items = iter(iterable)
for first in items:
second = next(items)
<process first and second>

This time, StopIteration is raised for an odd number of items. Catch and
process as desired. One possibility is to raise ValueError("Iterable
must have an even number of items").

A useful example of just sometimes pairing is iterating a (unicode)
string by code points ('characters') rather than by code units. This
requires combining the surrogate pairs used for supplementary characters
when the units are 16 bits.

chars = iter(string)
for char in chars:
if is_first_surrogate(char):
char += next(chars)
yield char

 

[Terry Reedy]

Alternatively, if all you want is for an empty iterable to do nothing,

To do nothing, just pass above. If the function does nothing, it returns
None. In the fix_title function, it should return '', not None.

you could write it like this:

items = iter(iterable)
for first in items:
<process first>
break

I could, but I doubt I would ;-). Try...except StopIteration: pass is
more explicit and less roundabout.

for item in items:
<process non-first>

However, the issue I have with any of this pulling the first element out of
the loop is that if you want special processing for the first element you
are likely to also want it for the last,

Likely? I would say occasionally. Sentences have first words; file have
headers. Special processing for last items is only an issue if it
*replaces* the normal processing, rather than following the normal
processing.

and if it is a single item you need
to process that item with both bits of special code. I don't see how that works
unless you have all elements within the single loop and test for first/last.

Like so, with tests:

def first_last_special(iterable):
print("\nIterable is",repr(iterable))
items = iter(iterable)
try:
first = next(items)
except StopIteration:
print('Nothing'); return
print(first, 'is the first item')
try:
current = next(items)
except StopIteration:
current = first
else:
for item in items:
print(current, 'is a middle item')
current = item
print(current, 'is the last item')

first_last_special('')
first_last_special('1')
first_last_special('12')
first_last_special('123')
first_last_special('12345')