why I don't like range/xrange

Discussion in 'Python' started by stdazi, Feb 16, 2007.

  1. stdazi

    stdazi Guest

    Hello!

    Many times I was suggested to use xrange and range instead of the
    while constructs, and indeed, they are quite more elegant - but, after
    calculating the overhead (and losen flexibility) when working with
    range/xrange, and while loops, you get to the conclusion that it isn't
    really worth using range/xrange loops.

    I'd like to show some examples and I'll be glad if someone can suggest
    some other fixes than while a loop :)

    a) range overfllow :


    for i in range(0, 1 << len(S)) :
    ......
    OverflowError: range() result has too many items

    ok, so we fix this one with xrange !

    b) xrange long int overflow :

    for i in xrange(0, 1 << len(S)) :
    .........
    OverflowError: long int too large to convert to int

    Next thing I miss is the flexibility as in C for loops :

    for (i = 0; some_function() /* or other condition */ ; i++)

    or,

    for (i = 0 ; i < 10 ; i++)
    i = 10;


    I don't think range/xrange sucks, but I really think there should be
    some other constructs to improve the looping flexibility. Other thing
    may be, that I just miss an equally elegant alternative that's why I'd
    like to hear some suggestions on how to fix the above issues.. (btw,
    I've already browsed the archives related to my issue,but i don't see
    any good solution)

    Thanks

    Jernej.
     
    stdazi, Feb 16, 2007
    #1
    1. Advertising

  2. stdazi

    Chris Mellon Guest

    On 16 Feb 2007 07:30:15 -0800, stdazi <> wrote:
    > Hello!
    >
    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but, after
    > calculating the overhead (and losen flexibility) when working with
    > range/xrange, and while loops, you get to the conclusion that it isn't
    > really worth using range/xrange loops.
    >
    > I'd like to show some examples and I'll be glad if someone can suggest
    > some other fixes than while a loop :)
    >
    > a) range overfllow :
    >
    >
    > for i in range(0, 1 << len(S)) :
    > .....
    > OverflowError: range() result has too many items
    >
    > ok, so we fix this one with xrange !
    >
    > b) xrange long int overflow :
    >
    > for i in xrange(0, 1 << len(S)) :
    > ........
    > OverflowError: long int too large to convert to int
    >


    xrange should be able to handle this. It's probably worth a bug report.

    > Next thing I miss is the flexibility as in C for loops :
    >
    > for (i = 0; some_function() /* or other condition */ ; i++)
    >


    You'd use a regular for or a while loop here, without the loop index.

    > or,
    >
    > for (i = 0 ; i < 10 ; i++)
    > i = 10;
    >


    This doesn't do anything as written. For all reasonable uses of it,
    you can do it the same way in Python. Writing C code in Python is a
    waste of time and an exercise in frustration.

    >
    > I don't think range/xrange sucks, but I really think there should be
    > some other constructs to improve the looping flexibility. Other thing
    > may be, that I just miss an equally elegant alternative that's why I'd
    > like to hear some suggestions on how to fix the above issues.. (btw,
    > I've already browsed the archives related to my issue,but i don't see
    > any good solution)
    >


    Enumeration over range() and xrange() is rare in Python. loops like:

    data = [...]
    for ii in xrange(len(data)):
    datum = data[ii]

    are a wart. Enumerate over whatever you're going to work with.

    In some cases, you want the index and the item. Use enumerate for that:

    for index, datum in enumerate(data):
    ....
     
    Chris Mellon, Feb 16, 2007
    #2
    1. Advertising

  3. stdazi

    Paul McGuire Guest

    On Feb 16, 9:30 am, "stdazi" <> wrote:
    > Hello!
    >
    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but, after
    > calculating the overhead (and losen flexibility) when working with
    > range/xrange, and while loops, you get to the conclusion that it isn't
    > really worth using range/xrange loops.
    >
    > I'd like to show some examples and I'll be glad if someone can suggest
    > some other fixes than while a loop :)
    >
    > a) range overfllow :
    >
    > for i in range(0, 1 << len(S)) :
    > .....
    > OverflowError: range() result has too many items
    >
    > ok, so we fix this one with xrange !
    >
    > b) xrange long int overflow :
    >
    > for i in xrange(0, 1 << len(S)) :
    > ........
    > OverflowError: long int too large to convert to int
    >
    > Next thing I miss is the flexibility as in C for loops :
    >
    > for (i = 0; some_function() /* or other condition */ ; i++)
    >
    > or,
    >
    > for (i = 0 ; i < 10 ; i++)
    > i = 10;
    >
    > I don't think range/xrange sucks, but I really think there should be
    > some other constructs to improve the looping flexibility. Other thing
    > may be, that I just miss an equally elegant alternative that's why I'd
    > like to hear some suggestions on how to fix the above issues.. (btw,
    > I've already browsed the archives related to my issue,but i don't see
    > any good solution)
    >
    > Thanks
    >
    > Jernej.


    Very little of my own code uses range or xrange, most of my for loops
    iterate over a sequence or generator, as in "for item in blahList: do
    something with item". I think range/xrange are common beginner's
    constructs, since they reflect a more C-like looping method ("for i in
    range(len(blahList)): do something with blahList").

    I really would *not* encourage use of range/xrange, but feel that
    iteration over sequences and generators is the more elegant/Pythonic
    way to go - who is suggesting you use range/xrange?

    For that matter, just what is it you plan to do 2**len(S) times? If S
    is of any significant length, the sun may be a lump of coal before you
    are finished, regardless of what loop mechanism you use (although it
    would make sense to avoid range's implementation of creating a list of
    2**len(S) items - fortunately the implementation already resolves this
    problem by raising a "that's too many items" exception, and thankfully
    so).

    Maybe instead of working around range/xrange, you should think whether
    a 2**len(S) approach to your problem is feasible in the first place.

    -- Paul
     
    Paul McGuire, Feb 16, 2007
    #3
  4. stdazi

    Larry Bates Guest

    stdazi wrote:
    > Hello!
    >
    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but, after
    > calculating the overhead (and losen flexibility) when working with
    > range/xrange, and while loops, you get to the conclusion that it isn't
    > really worth using range/xrange loops.
    >
    > I'd like to show some examples and I'll be glad if someone can suggest
    > some other fixes than while a loop :)
    >
    > a) range overfllow :
    >
    >
    > for i in range(0, 1 << len(S)) :
    > .....
    > OverflowError: range() result has too many items
    >
    > ok, so we fix this one with xrange !
    >
    > b) xrange long int overflow :
    >
    > for i in xrange(0, 1 << len(S)) :
    > ........
    > OverflowError: long int too large to convert to int
    >
    > Next thing I miss is the flexibility as in C for loops :
    >
    > for (i = 0; some_function() /* or other condition */ ; i++)
    >
    > or,
    >
    > for (i = 0 ; i < 10 ; i++)
    > i = 10;
    >
    >
    > I don't think range/xrange sucks, but I really think there should be
    > some other constructs to improve the looping flexibility. Other thing
    > may be, that I just miss an equally elegant alternative that's why I'd
    > like to hear some suggestions on how to fix the above issues.. (btw,
    > I've already browsed the archives related to my issue,but i don't see
    > any good solution)
    >
    > Thanks
    >
    > Jernej.
    >


    Your example of for i in xrange(0, 1<<len(s)): must have resulted in
    a number greater than 1 billion. Are you really doing this much or
    are you just pointing out an edge case here?

    You can always use while loop:

    i=0
    while i < (1<<len(s)):
    i+=1


    or

    i=1
    while 1:
    if i == 10: break
    i+=1

    Personally I use a lot of for loops in my code where I iterate
    over lists or with a generator as my target that returns instances
    of my data until exhausted and find the code easy to read:

    for line in myfile:
    # do something with the line

    No need to even "think" about numbers here unless I want to keep
    track of the line numbers in which case I would do:

    for linenumber, line in enumerate(myfile):
    # do something with each linenumber, line

    I find that I use i, j, k pointers less and less as I write more code in
    Python. I guess it is just what you get accustomed to using.

    -Larry
     
    Larry Bates, Feb 16, 2007
    #4
  5. On Fri, 16 Feb 2007 07:30:15 -0800, stdazi wrote:

    > Hello!
    >
    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but, after
    > calculating the overhead (and losen flexibility) when working with
    > range/xrange, and while loops, you get to the conclusion that it isn't
    > really worth using range/xrange loops.


    I prefer to _measure_ the overhead instead of guessing.

    import timeit
    whileloop = """i = 0
    while i < N:
    i += 1
    pass
    """
    forloop = """for i in xrange(N):
    pass
    """

    Now let's see how fast the loops are.

    >>> timeit.Timer(whileloop, "N = 10000").repeat(3, 1000)

    [3.5716907978057861, 3.5263650417327881, 3.5975079536437988]
    >>> timeit.Timer(forloop, "N = 10000").repeat(3, 1000)

    [1.3608510494232178, 1.341961145401001, 1.3180010318756104]

    Looks to me that a for loop using xrange is more than twice as fast as a
    while loop. The advantage is about the same for small N:

    >>> timeit.Timer(whileloop, "N = 100").repeat(3, 1000)

    [0.052264213562011719, 0.049374103546142578, 0.041945934295654297]
    >>> timeit.Timer(forloop, "N = 100").repeat(3, 1000)

    [0.012259006500244141, 0.013512134552001953, 0.015196800231933594]


    What makes you think that a while loop has less overhead?



    > I'd like to show some examples and I'll be glad if someone can suggest
    > some other fixes than while a loop :)
    >
    > a) range overfllow :
    >
    >
    > for i in range(0, 1 << len(S)) :
    > .....
    > OverflowError: range() result has too many items
    >
    > ok, so we fix this one with xrange !



    By the way, you don't need to write range(0, N). You can just write
    range(N).

    Yes, you're correct, range(some_enormous_number) will fail if
    some_enormous_number is too big.



    > b) xrange long int overflow :
    >
    > for i in xrange(0, 1 << len(S)) :
    > ........
    > OverflowError: long int too large to convert to int



    Are you really doing something at least 2147483647 times? I'm guessing
    that you would be better off rethinking your algorithm.



    > Next thing I miss is the flexibility as in C for loops :
    >
    > for (i = 0; some_function() /* or other condition */ ; i++)


    This would be written in Python as:

    i = 0
    while some_function():
    i += 1

    A more flexible way would be to re-write some_function() as an iterator,
    then use it directly:

    for item in some_function():
    # do something with item



    > or,
    >
    > for (i = 0 ; i < 10 ; i++)
    > i = 10;



    This would be written in Python as:

    for i in xrange(10):
    i = 10



    > I don't think range/xrange sucks, but I really think there should be
    > some other constructs to improve the looping flexibility.


    Every loop can be turned into a while loop. If you have while, you don't
    _need_ anything else.

    But for elegance and ease of use, a small number of looping constructs is
    good. Python has:

    while condition:
    block
    else:
    # runs if while exits *without* hitting break statement
    block


    for item in any_sequence:
    block
    else:
    # runs if for exits *without* hitting break statement
    block


    Nice, clean syntax.

    any_sequence isn't limited to mere arithmetic sequences -- it can be any
    sequence of any objects. But if you need a C-style for loop, using
    integers, range/xrange([start, ] stop [, step]) is provided.


    --
    Steven.
     
    Steven D'Aprano, Feb 16, 2007
    #5
  6. On Feb 16, 4:30 pm, "stdazi" <> wrote:

    > for (i = 0; some_function() /* or other condition */ ; i++)


    C's "for(pre,cond,post) code" is nothing more, then shorthand form of
    "pre; while(cond) {code; post;}"
    Which translated to Python would be:

    pre
    while cond:
    code
    post
     
    Bart Ogryczak, Feb 16, 2007
    #6
  7. In <>, Steven
    D'Aprano wrote:

    >> for (i = 0 ; i < 10 ; i++)
    >> i = 10;

    >
    >
    > This would be written in Python as:
    >
    > for i in xrange(10):
    > i = 10


    Nope, in Python it's:

    for i in xrange(10):
    break

    I think his example should demonstrate that assigning to the loop variable
    has no effect on the "loop test". Which IMHO is a good thing.

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Feb 16, 2007
    #7
  8. stdazi wrote:

    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but,
    > after calculating the overhead (and losen flexibility) when
    > working with range/xrange, and while loops, you get to the
    > conclusion that it isn't really worth using range/xrange loops.


    How did you calculate that?

    > b) xrange long int overflow :
    >
    > for i in xrange(0, 1 << len(S)) :
    > ........
    > OverflowError: long int too large to convert to int


    Why do you bit shift a len result here? Just to get a huge number?
    Try the same with C. You'd get an integer overflow in the first
    place.

    But this long int => int issue should not exist in a future python
    version any more, IIRC int and long int is scheduled to be merged
    somehow. (Or isn't it?)

    Regards,


    Björn

    --
    BOFH excuse #203:

    Write-only-memory subsystem too slow for this machine. Contact your
    local dealer.
     
    Bjoern Schliessmann, Feb 16, 2007
    #8
  9. Eduardo \EdCrypt\ O. Padoan, Feb 16, 2007
    #9
  10. stdazi

    Neil Cerutti Guest

    On 2007-02-16, Bart Ogryczak <> wrote:
    > On Feb 16, 4:30 pm, "stdazi" <> wrote:
    >
    >> for (i = 0; some_function() /* or other condition */ ; i++)

    >
    > C's "for(pre,cond,post) code" is nothing more, then shorthand form of
    > "pre; while(cond) {code; post;}"
    > Which translated to Python would be:
    >
    > pre
    > while cond:
    > code
    > post


    No, when you consider the continue statement, which which Python
    also supports.

    for (pre; cond; post) {
    continue;
    }

    That's not an infinite loop in C, but the Python while-loop
    version would be.

    --
    Neil Cerutti
     
    Neil Cerutti, Feb 16, 2007
    #10
  11. stdazi

    Roger Miller Guest

    On Feb 16, 7:01 am, "Bart Ogryczak" <> wrote:
    > On Feb 16, 4:30 pm, "stdazi" <> wrote:
    >
    > > for (i = 0; some_function() /* or other condition */ ; i++)

    >
    > C's "for(pre,cond,post) code" is nothing more, then shorthand form of
    > "pre; while(cond) {code; post;}"


    I don't disagree with your basic point, but technically this is not
    quite true.
    Try it with
    for (i = 0; i < n; i++) {
    if (x == 0) continue;

    printf("%d\n", x);
    }
     
    Roger Miller, Feb 16, 2007
    #11
  12. Eduardo "EdCrypt" O. Padoan wrote:

    >> But this long int => int issue should not exist in a future
    >> python version any more, IIRC int and long int is scheduled to be
    >> merged somehow. (Or isn't it?)

    >
    > It is done.


    Thanks for the info.

    Please don't send mail copies!

    Regards,


    Björn

    --
    BOFH excuse #217:

    The MGs ran out of gas.
     
    Bjoern Schliessmann, Feb 16, 2007
    #12
  13. stdazi a écrit :
    > Hello!
    >
    > Many times I was suggested to use xrange and range instead of the
    > while constructs, and indeed, they are quite more elegant - but, after
    > calculating the overhead (and losen flexibility) when working with
    > range/xrange, and while loops, you get to the conclusion that it isn't
    > really worth using range/xrange loops.


    I've almost never had to use (x)range in a for loop. Usually, one just
    iterate over an iterable.

    (snip)


    > Next thing I miss is the flexibility as in C for loops :
    >
    > for (i = 0; some_function() /* or other condition */ ; i++)


    You may want to have a look at generators. The main idea here is to
    encapsulate the knowledge about how iteration is implemented, so the
    client code just have to iterate.

    > or,
    >
    > for (i = 0 ; i < 10 ; i++)
    > i = 10;


    for i in range(10):
    i = 10

    What's your point, exactly ?

    >
    > I don't think range/xrange sucks, but I really think there should be
    > some other constructs to improve the looping flexibility.


    Have a look at generators then.
     
    Bruno Desthuilliers, Feb 16, 2007
    #13
  14. Bruno Desthuilliers schreef:
    > stdazi a écrit :


    >> for (i = 0 ; i < 10 ; i++)
    >> i = 10;

    >
    > for i in range(10):
    > i = 10
    >
    > What's your point, exactly ?


    In the first iteration, i is set equal to 10. Then, before starting the
    second iteration, i is incremented to 11; then the loop condition is
    checked and results in false. So the loop terminates after the first
    iteration.

    So, the point is that in C you can influence the loop's behavior by
    modifying the loop variable, while you cannot do that in Python (at
    least not in a for-loop).

    In other words, the point is that you can't translate loops literally
    from C to Python. Which is nothing new, and I fail to see how that is
    supposed to be a disadvantage.

    --
    If I have been able to see further, it was only because I stood
    on the shoulders of giants. -- Isaac Newton

    Roel Schroeven
     
    Roel Schroeven, Feb 17, 2007
    #14
  15. Roel Schroeven <> on Sat, 17 Feb 2007
    01:31:13 GMT didst step forth and proclaim thus:

    ....
    > So, the point is that in C you can influence the loop's behavior by
    > modifying the loop variable, while you cannot do that in Python (at
    > least not in a for-loop).


    What's wrong with...

    for i in range(10):
    if condition: break

    ....?
    --
    Sam Peterson
    skpeterson At nospam ucdavis.edu
    "if programmers were paid to remove code instead of adding it,
    software would be much better" -- unknown
     
    Samuel Karl Peterson, Feb 17, 2007
    #15
  16. Roel Schroeven a écrit :
    > Bruno Desthuilliers schreef:
    >
    >> stdazi a écrit :

    >
    >
    >>> for (i = 0 ; i < 10 ; i++)
    >>> i = 10;

    >>
    >>
    >> for i in range(10):
    >> i = 10
    >>
    >> What's your point, exactly ?

    >
    >
    > In the first iteration, i is set equal to 10. Then, before starting the
    > second iteration, i is incremented to 11; then the loop condition is
    > checked and results in false. So the loop terminates after the first
    > iteration.


    oops - my bad. But why would one do so when a break would do the trick:
    for i in range(10):
    break

    >
    > So, the point is that in C you can influence the loop's behavior by
    > modifying the loop variable, while you cannot do that in Python (at
    > least not in a for-loop).


    A good thing IMHO.

    > In other words, the point is that you can't translate loops literally
    > from C to Python. Which is nothing new, and I fail to see how that is
    > supposed to be a disadvantage.
    >

    <aol />
     
    Bruno Desthuilliers, Feb 17, 2007
    #16
  17. stdazi

    Guest

    Bruno Desthuilliers wrote:
    > Roel Schroeven a ecrit :
    > > Bruno Desthuilliers schreef:
    > >
    > >> stdazi a ecrit :

    > >
    > >
    > >>> for (i = 0 ; i < 10 ; i++)
    > >>> i = 10;
    > >>
    > >>
    > >> for i in range(10):
    > >> i = 10
    > >>
    > >> What's your point, exactly ?

    > >
    > >
    > > In the first iteration, i is set equal to 10. Then, before starting the
    > > second iteration, i is incremented to 11; then the loop condition is
    > > checked and results in false. So the loop terminates after the first
    > > iteration.

    >
    > oops - my bad. But why would one do so when a break would do the trick:
    > for i in range(10):
    > break


    After the C loop finishes, i is 11. After the python loop-with-break
    finishes, i is 0. That may effect later code.
     
    , Feb 17, 2007
    #17
  18. a écrit :
    > Bruno Desthuilliers wrote:
    >
    >>Roel Schroeven a ecrit :
    >>
    >>>Bruno Desthuilliers schreef:
    >>>
    >>>
    >>>>stdazi a ecrit :
    >>>
    >>>
    >>>>>for (i = 0 ; i < 10 ; i++)
    >>>>> i = 10;
    >>>>
    >>>>
    >>>>for i in range(10):
    >>>> i = 10
    >>>>
    >>>>What's your point, exactly ?
    >>>
    >>>
    >>>In the first iteration, i is set equal to 10. Then, before starting the
    >>>second iteration, i is incremented to 11; then the loop condition is
    >>>checked and results in false. So the loop terminates after the first
    >>>iteration.

    >>
    >>oops - my bad. But why would one do so when a break would do the trick:
    >>for i in range(10):
    >> break

    >
    >
    > After the C loop finishes, i is 11. After the python loop-with-break
    > finishes, i is 0. That may effect later code.
    >

    Lord saves me from having to maintain programs based on such
    constructions...
     
    Bruno Desthuilliers, Feb 18, 2007
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steve R. Hastings

    efficiency of range() and xrange() in for loops

    Steve R. Hastings, Apr 5, 2006, in forum: Python
    Replies:
    29
    Views:
    821
    Fredrik Lundh
    Apr 9, 2006
  2. Mr. SweatyFinger
    Replies:
    2
    Views:
    2,139
    Smokey Grindel
    Dec 2, 2006
  3. Joe Goldthwaite
    Replies:
    2
    Views:
    271
    Chris Mellon
    Dec 7, 2007
  4. Joe Goldthwaite
    Replies:
    4
    Views:
    313
    Carl Banks
    Dec 8, 2007
  5. harrismh777
    Replies:
    11
    Views:
    1,486
    Steven D'Aprano
    Aug 4, 2011
Loading...

Share This Page