for loop: range() result has too many items

[Peng Yu]

Hi,

The following code does not run because range() does not accept a big
number. Is there a way to make the code work. I'm wondering if there
is a way to write a for-loop in python similar to that of C style.

for(int i = 0; i < a_big_number; ++ i)

Regards,
Peng

$ cat for_loop.py
import sys

def foo():
for i in range(sys.maxint):
if i % 100 == 0:
print i

foo()
$ python for_loop.py
Traceback (most recent call last):
File "for_loop.py", line 8, in <module>
foo()
File "for_loop.py", line 4, in foo
for i in range(sys.maxint):
OverflowError: range() result has too many items

 

[Mel]

Hi,

The following code does not run because range() does not accept a big
number. Is there a way to make the code work. I'm wondering if there
is a way to write a for-loop in python similar to that of C style.

for(int i = 0; i < a_big_number; ++ i)

Regards,
Peng

$ cat for_loop.py
import sys

def foo():
for i in range(sys.maxint):
if i % 100 == 0:
print i

foo()
$ python for_loop.py
Traceback (most recent call last):
File "for_loop.py", line 8, in <module>
foo()
File "for_loop.py", line 4, in foo
for i in range(sys.maxint):
OverflowError: range() result has too many items

Are you using Python 2.x ??? If you are, range will accept your number, but
it will try to build a list holding sys.maxint integers, and you don't have
enough memory.

Try xrange instead.

Mel.

 

[Steven D'Aprano]

Hi,

The following code does not run because range() does not accept a big
number.
Incorrect.

[2147483649L, 2147483650L, 2147483651L]

Is there a way to make the code work. I'm wondering if there is
a way to write a for-loop in python similar to that of C style.

for(int i = 0; i < a_big_number; ++ i)

An exact translation of that would be a while loop:

i = 0
while i < a_big_number:
i += 1


but that's not actually your problem. Your actual problem is that you're
trying to generate a list containing 2147483647 (sys.maxint) items. Each
item requires 8 bytes, so this will need a single contiguous block of at
least 16 gigabytes.

xrange(2147483647)



def foo():
for i in range(sys.maxint):
if i % 100 == 0:
print i

Rather than iterating through every single integer between 0 and
2147483647, you can skip the ones you don't care about:

for i in xrange(0, sys.maxint, 100):
print i

This will run approximately 100 times faster than your code.

 

[Mark Dickinson]

Hi,

The following code does not run because range() does not accept a big
number.
Incorrect.

[2147483649L, 2147483650L, 2147483651L]

For what it's worth, there *is* a Python oddity lurking
under the surface here, at least for 64-bit. Here's
Python 2.6 on a 64-bit machine:
Traceback (most recent call last):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: range() result has too many items

The first call tries to allocate a list containing
2**31-1 integers. At 32 bytes per entry (8 for the
list pointer, 24 for the integer itself), that's not
surprising on a machine with < 64 Gb of memory.

The second call, however, doesn't even try to
allocate the memory, but decides that the range can't
be represented. That's not right: this is an LP64
machine, so the size of the list can be represented,
and the start, stop and step values are representable
as C longs (which is what CPython uses internally for
this purpose). On a machine with >64 Gb of memory,
this call should succeed. On an LP64 machine with
<64 Gb of memory, it should produce MemoryError, not
OverflowError.

Mark

 

[Mark Dickinson]

Is that a 64-bit Python as well?

Yes.

Mark