What is xrange?

[Billy Mays]

Is xrange not a generator? I know it doesn't return a tuple or list, so
what exactly is it? Y doesn't ever complete, but x does.

x = (i for i in range(10))
y = xrange(10)

print "===X==="
while True:
for i in x:
print i
break
else:
break

print "===Y==="
while True:
for i in y:
print i
break
else:
break

 

[harrismh777]

Is xrange not a generator? I know it doesn't return a tuple or list, so
what exactly is it? Y doesn't ever complete, but x does.

range(n) creates a list containing all the integers 0..n-1. This
is a problem if you do range(1000000), because you'll end up with a >4Mb
list. xrange deals with this by returning an object that pretends to be
a list, but just works out the number needed from the index asked for,
and returns that.

range() can actually be faster in some cases - eg. if iterating
over the same sequence multiple times. xrange has to reconstruct the
integer object every time, but range will have real integer objects. (It
will always perform worse in terms of memory however)

xrange isn't usable in all cases where a real list is needed. For
instance, it doesn't support slices, or any list methods.

 

[Thomas Jollans]

Is xrange not a generator? I know it doesn't return a tuple or list,
so what exactly is it? Y doesn't ever complete, but x does.

x = (i for i in range(10))
y = xrange(10)

print "===X==="
while True:
for i in x:
print i
break
else:
break

print "===Y==="
while True:
for i in y:
print i
break
else:
break

Every for loop calls gets a new iterator from the object you're
iterating over. (__iter__) -- Apparently, xrange is implemented in such
a way (as are lists) that you can iterate over the object many times,
while each generator object (and how could it be otherwise can only be
iterated over once. What is xrange(foo)? It is an object that supports
list-like indices, the iterator protocol, and probably a few other
things that you will find in the stdlib docs. Generators also support
the iterator protocol, but that's about as far as the similarity goes
(in general)

- Thomas

 

[Jerry Hill]

Is xrange not a generator?  I know it doesn't return a tuple or list, so
what exactly is it?  Y doesn't ever complete, but x does.

x = (i for i in range(10))
y = xrange(10)

xrange() does not return a generator. It returns an iterable xrange
object. If you want the iterator derived from the iterable xrange
object, you can get it like this: iterator = y.__iter__()

See http://docs.python.org/library/functions.html#xrange for the
definition of the xrange object.

http://www.learningpython.com/2009/02/23/iterators-iterables-and-generators-oh-my/
seems to cover the differences between iterables, iterators, and
generators pretty well.

Some more reading:
http://docs.python.org/howto/functional.html
http://www.python.org/dev/peps/pep-0255/
http://www.python.org/dev/peps/pep-0289/

 

[Steven D'Aprano]

Is xrange not a generator? I know it doesn't return a tuple or list, so
what exactly is it?

xrange pre-dates generators by approximately forever. It returns a
special "xrange object", which generates values suitable for use in
for-loops using the old __getitem__ protocol.

You can consider xrange to be implemented something vaguely like this:

class My_xrange:
def __init__(self, start, end=None, step=1):
if end is None:
end = start
start = 0
self.start = start
self.end = end
self.step = step
def __getitem__(self, index):
if self.step != 1:
raise NotImplementedError('too lazy to support step values')
start, end = self.start, self.end
if start <= index < end:
return start + index
raise IndexError('out of range')

Y doesn't ever complete, but x does.

x = (i for i in range(10))

That is better written as iter(range(10)).

y = xrange(10)

Y doesn't complete because xrange objects are restartable. The for-loop ends
up using the original iteration protocol:

i = y[0]
i = y[1]
i = y[2]
....

until IndexError is raised. You break after calling y[0], but the next loop
starts at y[0] again, and so you never progress beyond the first item in
the xrange object.

 

[Brian Blais]

xrange pre-dates generators by approximately forever. It returns a
special "xrange object", which generates values suitable for use in
for-loops using the old __getitem__ protocol.

interesting...I never really thought about this. Is there a reason that it wasn't reimplemented when generators came out? Is there a use-case for the current implementation that wouldn't work as a generator?


bb

 

[Steven D'Aprano]

interesting...I never really thought about this. Is there a reason that
it wasn't reimplemented when generators came out? Is there a use-case for
the current implementation that wouldn't work as a generator?

It certainly couldn't be re-implemented before Python 3, because that would
change the behaviour and therefore break people's code that expected to be
able to treat xrange objects as sequences.

And now that Python 3.2 is out, and 3.3 is being worked on, it can't be
changed now either, for the same reason. There was a very narrow window of
opportunity for the functionality to be changed, and it was missed.

Probably because nobody thought about it, but if it had been proposed to
change xrange into an iterator, I'm pretty confident that the suggestion
would have been rejected. xrange objects might be lazily generated, but
they're also sequence types: you can get their length, and you can index
them. (However you can't slice them.) Iterators are not sequence types:
they aren't indexable and you can't get their length.

So why bother *taking away* functionality from xrange just to make it a less
powerful and more restricted iterator? It works fine just the way it is,
there's no need to change it.

It isn't like the Python developers are sitting around bored, looking for
things to do. They are overworked with far too much to do and not enough
manpower or time to do it. There are a huge number of much more important
bug fixes and features that haven't been dealt with for them to bother with
something like this.

 

[Brian Blais]

xrange objects might be lazily generated, but
they're also sequence types: you can get their length, and you can index
them. (However you can't slice them.) Iterators are not sequence types:
they aren't indexable and you can't get their length.

ah! now that makes sense. I guess I never check their length, or index them, and only use them like generators.

thanks for the clarification!

It isn't like the Python developers are sitting around bored, looking for
things to do. They are overworked with far too much to do and not enough
manpower or time to do it. There are a huge number of much more important
bug fixes and features that haven't been dealt with for them to bother with
something like this.

Now, that seems a little harsh, and nothing that I was intending. I figured (before learning about getting its length and indexing) that if the xrange object was basically a pre-generator hack, that it would make sense from a code-clarity point of view to reimplement them as generators -- it would gave made the developer's life easier. Now that I realize that xrange has different functionality than a generator (which was the point of my question...just curious *what* different functionality) it clearly doesn't make sense to implement them as generators. I certainly wasn't trying to imply that the developers are lazy, or bored!


bb

 

[Dennis Lee Bieber]

Probably because nobody thought about it, but if it had been proposed to
change xrange into an iterator, I'm pretty confident that the suggestion
would have been rejected. xrange objects might be lazily generated, but
they're also sequence types: you can get their length, and you can index
them. (However you can't slice them.) Iterators are not sequence types:
they aren't indexable and you can't get their length.

Ah, but what did they change range() into -- I seem to recall
reading the Python 3.x turned the regular range() into something else...
(from Python <3.x returning a full list)

 

[Chris Angelico]

       Ah, but what did they change range() into -- I seem to recall
reading the Python 3.x turned the regular range() into something else...
(from Python <3.x returning a full list)

xrange got renamed to range, as I understand it.

ChrisA

 

[Gregory Ewing]

It's an iterable object.

There are advantages in having xrange (or range in python3) return
an iterable object, rather than creating an iterator directly.
One of them is that you can iterate over the same object more
than once.

This can be useful if you're iterating over a multi-dimensional
space of some kind. Some efficiency can be gained by pre-creating
an xrange object for each of the inner loops and re-using them.

 

[Ethan Furman]

Ah, but what did they change range() into -- I seem to recall
reading the Python 3.x turned the regular range() into something else...
(from Python <3.x returning a full list)

From the What's New docs, in the Views And Iterators Instead Of Lists
section:

range() now behaves like xrange() used to behave, except it works with
values of arbitrary size. The latter no longer exists.

~Ethan~