for loop: range() result has too many items

Discussion in 'Python' started by Peng Yu, Oct 13, 2009.

  1. Peng Yu

    Peng Yu Guest

    Hi,

    The following code does not run because range() does not accept a big
    number. Is there a way to make the code work. I'm wondering if there
    is a way to write a for-loop in python similar to that of C style.

    for(int i = 0; i < a_big_number; ++ i)

    Regards,
    Peng

    $ cat for_loop.py
    import sys

    def foo():
    for i in range(sys.maxint):
    if i % 100 == 0:
    print i

    foo()
    $ python for_loop.py
    Traceback (most recent call last):
    File "for_loop.py", line 8, in <module>
    foo()
    File "for_loop.py", line 4, in foo
    for i in range(sys.maxint):
    OverflowError: range() result has too many items
     
    Peng Yu, Oct 13, 2009
    #1
    1. Advertising

  2. Peng Yu

    Mel Guest

    Peng Yu wrote:

    > Hi,
    >
    > The following code does not run because range() does not accept a big
    > number. Is there a way to make the code work. I'm wondering if there
    > is a way to write a for-loop in python similar to that of C style.
    >
    > for(int i = 0; i < a_big_number; ++ i)
    >
    > Regards,
    > Peng
    >
    > $ cat for_loop.py
    > import sys
    >
    > def foo():
    > for i in range(sys.maxint):
    > if i % 100 == 0:
    > print i
    >
    > foo()
    > $ python for_loop.py
    > Traceback (most recent call last):
    > File "for_loop.py", line 8, in <module>
    > foo()
    > File "for_loop.py", line 4, in foo
    > for i in range(sys.maxint):
    > OverflowError: range() result has too many items


    Are you using Python 2.x ??? If you are, range will accept your number, but
    it will try to build a list holding sys.maxint integers, and you don't have
    enough memory.

    Try xrange instead.

    Mel.
     
    Mel, Oct 13, 2009
    #2
    1. Advertising

  3. On Tue, 13 Oct 2009 16:17:58 -0500, Peng Yu wrote:

    > Hi,
    >
    > The following code does not run because range() does not accept a big
    > number.


    Incorrect.

    >>> range(sys.maxint+2, sys.maxint+5)

    [2147483649L, 2147483650L, 2147483651L]


    > Is there a way to make the code work. I'm wondering if there is
    > a way to write a for-loop in python similar to that of C style.
    >
    > for(int i = 0; i < a_big_number; ++ i)


    An exact translation of that would be a while loop:

    i = 0
    while i < a_big_number:
    i += 1


    but that's not actually your problem. Your actual problem is that you're
    trying to generate a list containing 2147483647 (sys.maxint) items. Each
    item requires 8 bytes, so this will need a single contiguous block of at
    least 16 gigabytes.

    On my system, when I try it, I get MemoryError. You get OverflowError.

    Try instead using a lazy list:

    >>> xrange(sys.maxint)

    xrange(2147483647)



    > def foo():
    > for i in range(sys.maxint):
    > if i % 100 == 0:
    > print i



    Rather than iterating through every single integer between 0 and
    2147483647, you can skip the ones you don't care about:

    for i in xrange(0, sys.maxint, 100):
    print i

    This will run approximately 100 times faster than your code.



    --
    Steven
     
    Steven D'Aprano, Oct 13, 2009
    #3
  4. On Oct 13, 10:39 pm, Steven D'Aprano
    <> wrote:
    > On Tue, 13 Oct 2009 16:17:58 -0500, Peng Yu wrote:
    > > Hi,

    >
    > > The following code does not run because range() does not accept a big
    > > number.

    >
    > Incorrect.
    >
    > >>> range(sys.maxint+2, sys.maxint+5)

    >
    > [2147483649L, 2147483650L, 2147483651L]


    For what it's worth, there *is* a Python oddity lurking
    under the surface here, at least for 64-bit. Here's
    Python 2.6 on a 64-bit machine:

    >>> range(2**31-1)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    MemoryError
    >>> range(2**31)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    OverflowError: range() result has too many items

    The first call tries to allocate a list containing
    2**31-1 integers. At 32 bytes per entry (8 for the
    list pointer, 24 for the integer itself), that's not
    surprising on a machine with < 64 Gb of memory.

    The second call, however, doesn't even try to
    allocate the memory, but decides that the range can't
    be represented. That's not right: this is an LP64
    machine, so the size of the list can be represented,
    and the start, stop and step values are representable
    as C longs (which is what CPython uses internally for
    this purpose). On a machine with >64 Gb of memory,
    this call should succeed. On an LP64 machine with
    <64 Gb of memory, it should produce MemoryError, not
    OverflowError.

    Mark
     
    Mark Dickinson, Oct 14, 2009
    #4
  5. On Oct 15, 8:39 pm, David C. Ullrich <> wrote:
    > >For what it's worth, there *is* a Python oddity lurking
    > >under the surface here, at least for 64-bit.  Here's
    > >Python 2.6 on a 64-bit machine:

    >
    > Is that a 64-bit Python as well?


    Yes.

    Mark
     
    Mark Dickinson, Oct 15, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. peelman

    How many threads is too many?

    peelman, Jan 13, 2005, in forum: Java
    Replies:
    12
    Views:
    841
    Esmond Pitt
    Jan 15, 2005
  2. Michael Tan
    Replies:
    32
    Views:
    991
    Ara.T.Howard
    Jul 21, 2005
  3. Joey Zhou
    Replies:
    5
    Views:
    236
    Joey Zhou
    Apr 15, 2011
  4. Dag Sunde
    Replies:
    4
    Views:
    358
  5. Isaac Won
    Replies:
    9
    Views:
    382
    Ulrich Eckhardt
    Mar 4, 2013
Loading...

Share This Page