is parameter an iterable?

[Fredrik Lundh]

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:

Python 2.1.1 (#1, Aug 25 2001, 04:19:08)
[GCC 3.0.1] on sunos5
Type "copyright", "credits" or "license" for more
information.Traceback (most recent call last):
File "<stdin>", line 1, in ?
NameError: name 'iter' is not defined

What should I do when I can't rely on functions that
don't exist in older versions of Python?

python 2.1 doesn't support iterators, so that question doesn't
make much sense.

$ python2.1
.... def __iter__(self):
.... print "ITER"
.... return self
.... def next(self):
.... print "NEXT"
.... return 1
........ print i
....
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: iterable instance has no attribute '__getitem__'

if you want to write code that runs under 2.1, you have to write
your code in terms of what 2.1 supports. Python's compatibility
model means that code written for old versions still work in new
versions; it doesn't mean that code written for new versions will
always work properly (or raise proper exceptions) in old versions.

</F>

 

[Steven D'Aprano]

Steven D'Aprano wrote:

Alas and alack, I have to write code which is backwards
compatible with older versions of Python:

[snip]

What should I do when I can't rely on functions that
don't exist in older versions of Python?

python 2.1 doesn't support iterators, so that question doesn't
make much sense.

The _question_ doesn't make much sense? I could
understand you saying that backwards-compatibility is
"not important [to me]" but to say that the very
question of how to maintain backwards compatibility
makes little sense is a bit extreme, don't you think?

Fredrik, I bow to your superior knowledge about Python,
no sarcasm intended, and I've learnt a lot from your
posts, thank you. But this is not one of your shining
moments. Your attitude was utterly dismissive: the
"right way" to solve the problem of recognising
iterables was to use iter, and that's all that needs to
be said.

The problem of how to recognise iterables did not come
into existence with version 2.2, and backwards
compatibility is sometimes a real requirement. A few
months back I had to mangle some Py2.4 code so that it
would run under version 2.0, and wasn't that fun.

if you want to write code that runs under 2.1, you have to write
your code in terms of what 2.1 supports.

Do you think I don't know this?

I never imagined for an instant that Python 2.1 would
somehow magically be able to use features that didn't
exist in Python 2.1. But then it wasn't me saying that
there was nothing to discuss, the "right way" is to use
iter(), end of story.

If I have to write code that can't rely on iter()
existing in the language, what should I do?

Are there practical idioms for solving the metaproblem
"solve problem X using the latest features where
available, otherwise fall back on older, less powerful
features"?

For instance, perhaps I might do this:

try:
built_in_feature
except NameError:
# fall back on a work-around
from backwards_compatibility import \
feature as built_in_feature

Do people do this or is it a bad idea?

Are there other techniques to use? Obviously refusing
to run is a solution (for some meaning of "solution"),
it may even be a practical solution for some cases, but
is it the only one?

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?

 

[Alex Martelli]

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?

Not a problem I've had often, but when I did, if I recall correctly, I
did something like:

try:
iter
except NameError:
def isiterable(x):
try: x[0]
except Exception: return 0
else: return 1
else:
def isiterable(x):
try: iter(x)
except TypeError: return 0
else: return 1

Not True/False because they wouldn't be available in 2.0 any more than
iter would. "Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming". The tests here
are not 100% reliable, of course -- in 2.0, a dict which just happens to
have a 0 key would erroneously pass; but short of actually trying the
loop, there's no infallible way that I know of.

One possibility (that I haven't tried in real life) to deal with the
hellish situation of having to write code that's able to loop on an
iterable but fail softly otherwise, while not hiding errors in the
loop's body, would be the suitably hellish...:

class NotATypeError(Exception):
def __init__(self, e, t):
self.e = e
self.t = t

try:
try:
for item in whatever:
try:
...loop body here...
except TypeError, e, t:
raise NotATypeError(e, t)
except TypeError:
....fail-softly code...
except NotATypeError, x:
raise TypeError, x.e, x.t


This kind of spaghetti code is what gives backwards compatibility its
bad name, of course. Be sure that you're getting paid for this in
proportion to its ugliness, and your finances should be set for life.


Alex

 

[Steven D'Aprano]

"Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming".

Ah yes, of course. How quickly we forget: you couldn't
say "for line in file...". No generators either.

But there was still at least one object that was
consumable: xreadlines.

Still, I think using an xreadlines object is an unusual
enough case that I'm not going to lose any sleep about
it. Document it as a known issue and forget it *wink*

[snip]

This kind of spaghetti code is what gives backwards compatibility its
bad name, of course. Be sure that you're getting paid for this in
proportion to its ugliness, and your finances should be set for life.

Hah, I wish!


Thanks for the assistance, I learnt a lot. Let's hope I
don't have to use it ever again...

 

[Fredrik Lundh]

In the specific case of iter(), are there good
alternative ways of detecting an iterable without
consuming it?

Not a problem I've had often, but when I did, if I recall correctly, I
did something like:

try:
iter
except NameError:
def isiterable(x):
try: x[0]
except Exception: return 0
else: return 1
else:
def isiterable(x):
try: iter(x)
except TypeError: return 0
else: return 1

Not True/False because they wouldn't be available in 2.0 any more than
iter would. "Consuming" didn't really come into consideration for the
backwards compatibility part because only objects indexable with
integers, 0 and up (and raising IndexError at some point) were usable in
for statements in old Pythons, there was no "consuming".

Before the iterator protocol was added, using the sequence protocol
to implement "forward-only iterators" weren't uncommon. e.g.

class my_file_iterator:
def __init__(self, file):
self.file = file
def __getitem__(self, index):
line = self.file.readline()
if not line:
raise IndexError
return line

(for extra points, add code that makes sure that the index matches
the line number)

To test for this, you could look for __getitem__ methods on instance
objects, and use operator.isSequenceType on everything else, but that
doesn't distinguish between sequences and mappings.

short of actually trying the loop, there's no infallible way that I know of.

Exactly. And iter() can fail too, even if it's far less likely that you
stumble upon an iterable that misbehaves if you call __iter__ one
extra time. Which is why the portable pragmatic pythonic solution
is to design your program so it doesn't depend on type/interface
testing.

If you want to loop over things, loop over things, and leave it to
Python to raise an exception if you get something that doesn't
support looping.

And unless you have very good reasons, you should treat that
exception as a bug in your program.

</F>

 

[Tom Anderson]

Are there practical idioms for solving the metaproblem "solve problem X
using the latest features where available, otherwise fall back on older,
less powerful features"?

For instance, perhaps I might do this:

try:
built_in_feature
except NameError:
# fall back on a work-around
from backwards_compatibility import \
feature as built_in_feature

Do people do this or is it a bad idea?

From some code i wrote yesterday, which has to run under 2.2:

try:
True
except NameError:
True = 1 == 1
False = 1 == 0

Great minds think alike!

As for whether it's a bad idea, well, bad or not, it certainly seems like
the least worst.

Are there other techniques to use? Obviously refusing to run is a
solution (for some meaning of "solution"), it may even be a practical
solution for some cases, but is it the only one?

How about detecting which environment you're in, then running one of two
entirely different sets of code? Rather than trying to construct modern
features in the antique environment, write code for each, using the local
idioms. The trouble with this is that you end up with massive duplication;
you can try to factor out the common parts, but i suspect that the
differing parts will be a very large fraction of the codebase.

If I have to write code that can't rely on iter() existing in the
language, what should I do?

Can you implement your own iter()? I have no idea what python 2.0 was
like, but would something like this work:

class _iterator:
def __init__(self, x):
self.x = x
self.j = 0
def next(self):
self.j = self.j + 1
return self.x.next()
def __getitem__(self, i):
if (i != self.j):
raise ValueError, "out of order iteration"
try:
return self.next()
except StopIteration:
raise IndexError
def __iter__(self):
return self
# hopefully, we don't need this, but if we do ...
def __len__(self):
return sys.maxint # and rely on StopIteration to stop the loop

class _listiterator(_iterator):
def next(self):
try:
item = self.x[self.j]
self.j = self.j + 1
return item
except IndexError:
raise StopIteration
def __getitem__(self, i):
if (i != self.j):
raise ValueError, "out of order iteration"
self.j = self.j + 1
return self.x

import types

def iter(x):
# if there's no hasattr, use explicit access and try-except blocks
# handle iterators and iterables from the future
if hasattr(x, "__iter__"):
return _iterator(x.__iter__())
# if there's no __getitem__ on lists, try x[0] and catch the exception
# but leave the __getitem__ test to catch objects from the future
if hasattr(x, "__getitem__"):
return _listiterator(x)
if type(x) == types.FileType:
return _fileiterator(x) # you can imagine the implementation of this
# insert more tests for specific types here as you like
raise TypeError, "iteration over non-sequence"

?

NB haven't actually tried to run that code.

tom

 

[Fredrik Lundh]

How about detecting which environment you're in, then running one of two
entirely different sets of code? Rather than trying to construct modern
features in the antique environment, write code for each, using the local
idioms. The trouble with this is that you end up with massive duplication;
you can try to factor out the common parts, but i suspect that the
differing parts will be a very large fraction of the codebase.

That sounds like the worst possible way of writing portable code (unless you
have huge amounts of time and money, and wasting them don't really matter
to you).

In my experience, a more practical approach is to write code that targets the
2.0/2.1 platform, and selectively (based on feature tests, not version numbers)
replace portions with more efficient mechanisms from newer versions. (most
of those are found in the standard library, not in the language itself).

(this is related to another popular myth: that code that uses "the old way" is
automatically slower than code that uses newfangled stuff, also when running
on recent Python versions. that's very seldom true; new Python versions runs
"old" code faster too.)

If you want to support 1.5.2 or Unicode-less Python builds, you need to add
some fallbacks too, but that's mostly trivial. The easiest way is to pass 8-bit
strings through "as is", leaving encoding issues to the application.

Can you implement your own iter()? I have no idea what python 2.0 was
like, but would something like this work:

Python 2.0 had sequences. Python 2.4 has sequences. Iterators don't really
add anything. Emulating them in older versions is mostly pointless.

I've said it many times, and I'll say it again: the only fundamentally new concept
that has been added since Python 1.5.2 is generators. And a lot of the stuff you
want generators for can be emulated with sequences and extra buffer layers.

All the rest is just coloured frosting; you can save a line here and there (which is
a real benefit only if your return key is broken), but that's about it.

</F>

 

[Roy Smith]

I've said it many times, and I'll say it again: the only fundamentally
new concept that has been added since Python 1.5.2 is generators.
[...]
All the rest is just coloured frosting

In my mind, the biggest thing since 1.5.2 is string methods. They are
perhaps, as you say, just frosting, but they're very good tasting frosting