when an iterable object is exhausted or not

[Franck Ditter]

Two similar iterable objects but with a different behavior :

$$$ i = range(2,5)
$$$ for x in i : print(x,end=' ')

2 3 4
$$$ for x in i : print(x,end=' ') # i is not exhausted

2 3 4

--------- Compare with :

$$$ i = filter(lambda c : c.isdigit(), 'a1b2c3')
$$$ for x in i : print(x,end=' ')

1 2 3
$$$ for x in i : print(x,end=' ') # i is exhausted

$$$

IMHO, this should not happen in Py3k.
What is the rationale of this (bad ?) design, which forces the programmer
to memorize which one is exhaustable and which one is not ?...

franck

 

[Steven D'Aprano]

It's interesting that it DOESN'T happen in Python 2. The first "i" is
of type list, the second "i" is of type string, and both are
restartable.

What's the type of "i" in the second case in Python 3?

In Python 3, filter returns a lazy iterator, a "filter object". It
generates items on demand.

In Python 2, filter is eager, not lazy, and generates items all up-front.
If the input is a string, it generates a string; if the input is a tuple,
it generates a tuple; otherwise it generates a list.

 

[Steven D'Aprano]

Two similar iterable objects but with a different behavior : [...]
IMHO, this should not happen in Py3k. What is the rationale of this (bad
?) design, which forces the programmer to memorize which one is
exhaustable and which one is not ?...

What makes you say that they are "similar" iterable objects? Except that
they are both iterable, they are very different. You might as well say
that lists and dicts are "similar iterable objects".

filter objects are iterators, and so obey the intentionally simple
iterator protocol. range objects are iterable but not iterators, and do
not obey the iterator protocol.

py> it = filter(lambda x: x, set('abc'))
py> iter(it) is it
True
py> x = range(1, 15, 2)
py> iter(x) is x
False


filter relies on its input, which it consumes as it does. Since the input
may not be restartable, filter cannot be restartable either. For
simplicity, filter does not try to be "clever" and guess when the input
argument is restartable. Instead, it simply and consistently behaves the
same for any iterable input.

range is a specialist iterable data structure that does not consume
anything. It computes its output lazily, but that is the only similarity
with iterators. There's no need to limit range objects to the simple
iterator protocol, since they don't consume their input -- a single
object can easily compute its output as often as you want. range objects
are indexable and sliceable:

py> x = range(1, 15, 2)
py> x[4]
9
py> x[2:4]
range(5, 9, 2)


Why artificially make range objects unrestartable just to satisfy
compatibility with iterators?

The caller already has to remember that range and filter take different
arguments, do different things, and return different objects. Why is it
hard to remember that range is restartable and filter is not?