Efficiently iterating over part of a list

[Steven D'Aprano]

If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist

 

[James Stroud]

If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist

I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)


Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James


James

 

[James Stroud]

If I want to iterate over part of the list, the normal Python idiom is to
do something like this:

alist = range(50)
# first item is special
x = alist[0]
# iterate over the rest of the list
for item in alist[1:]
x = item

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do
people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist

I think this is a job for iterators:

listiter = iter(alist)

first_item_is_special = listiter.next()

for not_special_item in listiter:
do_stuff_with(not_special_item)


Other solutions might involve enumerators:

special = [i for i in xrange(50) if not i%13]

for i,item in alist:
if i in special:
do_something_special_with(item)
else:
do_other_stuff_with(item)

James


James

I mean

for i,item in enumerate(alist):

 

[Ziga Seilnacht]

Steven D'Aprano wrote:
[snip]

The important thing to notice is that alist[1:] makes a copy. What if the
list has millions of items and duplicating it is expensive? What do people
do in that case?

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist

for x in itertools.islice(alist, 1, len(alist)):


HTH
Ziga

 

[Peter Otten]

Are there better or more Pythonic alternatives to this obvious C-like
idiom?

for i in range(1, len(alist)):
x = alist

For small start values you can use itertools.islice(), e. g:

for x in islice(alist, 1, None):
# use x

You'd have to time at what point the C-like idiom (which I would have no
qualms using throughout) becomes faster.

Peter

 

[Duncan Booth]

The important thing to notice is that alist[1:] makes a copy. What if
the list has millions of items and duplicating it is expensive? What
do people do in that case?

I think you are worrying prematurely.

On my system slicing one element off the front of a 10,000,000 element list
takes 440mS. The same operation on 1,000,000 elements taks 41mS. Iterating
through the sliced list:

for x in r[1:]:
y = x+1

takes 1.8s and 157mS respectively, so the slicing is only a quarter of the
time for even this minimal loop. As soon as you do anything much inside the
loop you can forget the slice cost.

Remember that copying the list never copies the elements in the list, it
just copies pointers and bumps ref counts. Copying a list even if it has
millions of items is not usually expensive compared with the costs of
manipulating all the items in the list.

So the first thing you do is not to worry about this until you know it is
an issue. Once you know for a fact that it is a problem, then you can look
at optimising it with fancy lazy slicing techniques, but not before.

 

[Paul Rubin]

for i in range(1, len(alist)):
x = alist

a2 = iter(alist)
a2.next() # throw away first element
for x in a2:
...