Iterators

[Chris Lyon]

coming from a Vb'ish sort of background, issues like iterators and
list comprehensions appear at first site to be a confusion. However
since I have a great deal of respect for the large brains in a jar
(LBIAJ) that post on this list and construct my favourite language I
realise I'm missing something.

So can anyone point me at some applications, tutorials, websites and
advantages that these mechanisms provide? that don't rely on having
done LBIAJ courses such that I dare use them in my own code that
thrives on the massive over application of lists and dictionaries that
I believe I finally got my head round.

Apologies to all for continually stirring up the mud at the bottom of
the crystal clear river that is python but I fear being left behind
when whatever is built on top of these constructs raises it's head in
the forseable future.

As ever I am attempting to use humour (badly) to voice issues that are
assumed by many but complicated to a few.

Chris

 

[Peter Hansen]

So can anyone point me at some applications, tutorials, websites and
advantages that these mechanisms provide? that don't rely on having
done LBIAJ courses such that I dare use them in my own code that
thrives on the massive over application of lists and dictionaries that
I believe I finally got my head round.

There is *nothing* wrong with continuing to use (or over-use lists
and dictionaries. I fully intend to do so myself!

Apologies to all for continually stirring up the mud at the bottom of
the crystal clear river that is python but I fear being left behind
when whatever is built on top of these constructs raises it's head in
the forseable future.

As ever I am attempting to use humour (badly) to voice issues that are
assumed by many but complicated to a few.

I understand where you're coming from, and I'm sure you'll get
some responses from others that will actually point you somewhere
that can help improve your understanding in this area.

I just wanted to reassure you that you don't need to be too
concerned about the whole thing. I didn't figure out how to
write a list comprehension for at least a year after they were
first added to the language... I had too much productive work
to do with lists. I have _not_ gone back to change any of the
list-based code(*), and still only rarely reach for list
comprehensions as my first option.

There's nothing wrong with not being a leader in adopting new
language features. If you try too hard to keep up, you'll
probably "churn" too much and waste time rewriting code or
something, when you could just be writing working code using
the features you know and understand well.

-Peter

* The fact that I no longer work at that the company would, of
course, mean that going back to change the code would result in
criminal trespassing and probably industrial sabotage charges,
but that's beside the point. ;-)

 

[Daniel Dittmar]

coming from a Vb'ish sort of background, issues like iterators and
list comprehensions appear at first site to be a confusion. However
since I have a great deal of respect for the large brains in a jar
(LBIAJ) that post on this list and construct my favourite language I
realise I'm missing something.

Perhaps it's best explained by the problem these constructs were intended to
solve:

= list comprehension =
There seems to be a lot of code like the following:

# transforming a list
result = []
for element in sourcelist:
result.append (transform (element))

# filtering a list
result = []
for element in sourcelist:
if passesFilter (element):
result.append (element)

# both filtering and transforming
result = []
for element in sourcelist:
if passesFilter (element):
result.append (transform (element))

Some thought that this is a lot of typing and possibly obfuscates the intent
of the code, so list comprehension syntax was borrowed from the Haskell
programming language:

result = [transform (element) for element in sourcelist]
result = [element for element in sourcelist if passesFilter (element)]
result = [transform (element) for element in sourcelist if passesFilter
(element)]

In the old days, this was done using map (for transformation) and filter
(for filtering):
result = map (transform, sourcelist)
result = filter (passesFilter, sourcelist)
result = map (transform, filter (passesFilter, sourcelist))

This looks actually shorter than list comprehension. But assume that
transform and passesFilter are member functions. Then it becomes
result = map (lambda e: e.transform (), sourcelist)
result = filter (lambda e: e.passesFilter (), sourcelist)
result = map (lambda e: e.transform (), filter (lambda e: e.passesFilter (),
sourcelist))
versus
result = [element.transform () for element in sourcelist]
result = [element for element in sourcelist if element.passesFilter ()]
result = [element.transform () for element in sourcelist if
element.passesFilter ()]

The main disadvantages of list comprehension:
- you can't insert print statements
- you can't transform first and filter last unless you nest list
comprehension
result = [outer for outer in [transform (inner) for inner in sourcelist]
if passesFilter (outer)]

2.4 will also introduce 'generator comprehension':
- no [] around the construct
- more efficient because the list isn't built in memory

= Iterators =
The code to read all lines of a file used to be as follows:
while 1:
line = stream.readline ()
if not line:
break # end of file
pass # do something with line

which is cumbersome to write.

Or
for line in stream.readlines ():
pass # do something with line

which will consume too much memory if the file is large.

So what was needed was some kind of object:
- that could be used in a for loop
- doesn't put everything in memory

Another example would be potentially large SQL queries, where you want to
loop over all the rows.

A simple implementation to iterate over the lines of a file
class LineIterator:
def __init__ (self, fname):
self.stream = open (fname, 'r')

def next (self):
line = self.stream.readline ()
if not line:
raise StopIteration
return line

def __iter__ (self):
return self

Disadvantage of using iterators:
- you can read them only once
- print doesn't write anything useful

Daniel