problems caused by very large for-loop

[sam]

hi all,

i am writing some software to model polymerisation kinetics and have
created a way of doing so which involves taking ever smaller slices of
time to model until some sort of convergence was observed in the
results.

so far so good, but i was using 'for i in the range(iterations):' a for
loop over each slice of time, where the number of iterations was
getting into the tens of millions. up until about 125,000,000 it
worked, then i got a MemoryError.

now i wasn't surprised to get a memory error eventually because what i
was doing was some serous number-crunching. but it didn't occur to me
where it was at first. i had to keep some results in memory to
calculate something at the end, but that was about a million integers,
not that hard with 512 Mb of RAM. then i thought it might be something
structural in the way python works, but i couldn't guess at what it
might be.

thinking about it a bit more, i remembered having read that the
for-loop declaration i used actually created a list of integers, which
would be eating up over half of my memory before i'd even started doing
anything. the program was creating a directory as instructed, then
doing nothing, just churning, which made it clear that the problem was
encountered right at the start of the program. that helped clear it up,
but i was surprised that something as innocuous as the for-loop was
creating such a problem. i changed it to a while-loop and it worked
fine.

has anyone else bumped up against this problem before? i suppose
for-loops with 250 million iterations are seldom used in most
applications. it was just the first time i'd ever solved a problem by
actually having some insight into how python works at a slightly lower
level.

anyway, sorry to be so long-winded. i'm just glad the damn thing's
working again...

sam

 

[true911]

hi all,
...

has anyone else bumped up against this problem before? i suppose
for-loops with 250 million iterations are seldom used in most
applications. it was just the first time i'd ever solved a problem by
actually having some insight into how python works at a slightly lower
level.

Sam,

The problem is not with the 'for' operator, but the fact that the
range() operator creates the iterable list (in this case, with 250
million elements) prior to beginning the loop.

Try using xrange() with the same syntax, which returns items from a
tuple (NOT a list, if it matters to you) just-in-time and discards
them, much like FOR as used in a BASIC context.

xrange() is somewhat less flexible, though, and may or may not suit
your needs. If not, use 'while' instead, and set your own loop exit
conditions.