# a sequence question

Discussion in 'Python' started by Chris Wright, Jan 28, 2005.

1. ### Chris WrightGuest

Hi,

1) I want to iterate over a list "N at a time"
sort of like:

# Two at a time... won't work, obviously

>>> for a, b in [1,2,3,4]:

.... print a,b
....
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: unpack non-sequence
>>>

Is there a nifty way to do with with list comprehensions,
or do I just have to loop over the list ?

cheers and thanks

chris wright

Chris Wright, Jan 28, 2005

2. ### Diez B. RoggischGuest

l = [1,2,3,4]

for a, b in zip(l[::2], l[1::2]):
print a,b

--
Regards,

Diez B. Roggisch

Diez B. Roggisch, Jan 28, 2005

3. ### Roy SmithGuest

In article <lzrKd.138680\$>,
Chris Wright <> wrote:

> Hi,
>
> 1) I want to iterate over a list "N at a time"

You could do it with slicing and zip:

>>> l = [1, 2, 3, 4, 5, 6, 7, 8]
>>> zip (l[::2], l[1::2])

[(1, 2), (3, 4), (5, 6), (7, 8)]

To my eyes, that's a bit cryptic, but it works and it's certainly
compact. I don't use either zip() or extended slicing a lot; perhaps if
I used them more often, the above would be more obvious to me if I read
it in somebody else's code.

The interesting thing would be generalizing this to the "N at a time"
case. I think this works:

def nzip (list0, n):
args = []
for i in range(n):
slice = list0[i::n]
args.append (slice)
return zip (*args)

l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
print nzip (l, 3)

Roy-Smiths-Computerlay\$ ./nzip.py
[(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)]

but I haven't given any thought to what happens if the length of the
list isn't a multiple of n (exercise for the reader). It's also
annoying that the above generates a bunch of temporary lists. It would
be cool if there was a way to have the intermediates be generator
expressions, but I'm not that good with that stuff, so I'll leave that
as an exercise for other readers

Roy Smith, Jan 28, 2005
4. ### F. PetitjeanGuest

Le Fri, 28 Jan 2005 13:59:45 GMT, Chris Wright a écrit :
> Hi,
>
> 1) I want to iterate over a list "N at a time"
>
>
> Is there a nifty way to do with with list comprehensions,
> or do I just have to loop over the list ?
>
> cheers and thanks

seq = xrange(1, 9) # an iterable [1, 2, ... 8]
N = 2
it = (iter(seq,)*N # a tuple containing N times the *same* iterator on
seq
print zip(*it) # the list you are after
from itertools import izip
help(izip)
it = (iter(seq),)*2
for tup in izip(*it):
print tup
>
> chris wright

F. Petitjean, Jan 28, 2005
5. ### Duncan BoothGuest

Chris Wright wrote:

> 1) I want to iterate over a list "N at a time"
> sort of like:
>
> # Two at a time... won't work, obviously
>
> >>> for a, b in [1,2,3,4]:

> ... print a,b
> ...

Try this:

l = [1, 2, 3, 4]
for a, b in zip(*[iter(l)]*2):
print a, b

zip(*[iter(seq)]*N) will group by N (but if there are any odd items at the
end it will ignore them).

map(None, *[iter(seq)]*N) will group by N padding the last item with None
if it needs to.

Duncan Booth, Jan 28, 2005
6. ### Michael HartlGuest

For problems like this I use a partition function defined in a utils.py
file that I use (based on Peter Norvig's utils file at
http://aima.cs.berkeley.edu/python/utils.py). Using partition, the
problem you posed can be solved by writing

#for a, b in partition([1, 2, 3, 4], 2):
# print a, b

The implementation of partition I use is simple-minded; the previous
posts in this thread suggest some more sophisticated ways to attack it
using generators.

#def partition(seq, partsize):
# """Partition a sequence into subsequences of length partsize."""
# ls = len(seq)
# assert ls % partsize == 0, ('length %s, partition size %s\n'
# % (ls, partsize))
# return [seq[ii+partsize)] for i in range(0, ls, partsize)]
Michael

Michael Hartl, Jan 28, 2005
7. ### Nick CoghlanGuest

Duncan Booth wrote:
> Try this:
>
> l = [1, 2, 3, 4]
> for a, b in zip(*[iter(l)]*2):
> print a, b
>
> zip(*[iter(seq)]*N) will group by N (but if there are any odd items at the
> end it will ignore them).
>
> map(None, *[iter(seq)]*N) will group by N padding the last item with None
> if it needs to.

For anyone else who was as bemused as I was that Duncan's and F. Petitjean's
suggestions actually *work*, this was what I had to do to figure out *why* they
work:

Py> l = [1, 2, 3, 4]
Py> itr = iter(l)
Py> zip(itr) # Put all items from iterator in position 1
[(1,), (2,), (3,), (4,)]
Py> itr = iter(l)
Py> zip(itr, itr) # Put every second item in position 2
[(1, 2), (3, 4)]

Using zip(*[iter(l)]*N) or zip(*(iter(l),)*N) simply extends the above to the
general case.

I'd definitely recommend hiding this trick inside a function. Perhaps something
like (using Michael's function name):

from itertools import izip, repeat, chain

def partition(seq, part_len):
return izip(*((iter(seq),) * part_len))

itr = iter(seq)
if (len(seq) % part_len != 0):
return izip(*((itr,) * part_len))

Py> list(partition(range(10), 2))
[(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
Py> list(partition(range(10), 3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8)]
[(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, False, False)]
[(0, 3, 6, 9), (1, 4, 7, None), (2, 5, 8, None)]

Not sure how useful that last example is, but I thought it was cute

Cheers,
Nick.

--
Nick Coghlan | | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Nick Coghlan, Jan 29, 2005
8. ### Guest

, Jan 31, 2005
9. ### Steven BethardGuest

Nick Coghlan wrote:
> I'd definitely recommend hiding this trick inside a function. Perhaps
> something like (using Michael's function name):
>
> from itertools import izip, repeat, chain
>
> def partition(seq, part_len):
> return izip(*((iter(seq),) * part_len))
>
> itr = iter(seq)
> if (len(seq) % part_len != 0):
> return izip(*((itr,) * part_len))

I think you can write that second one so that it works for iterables
without a __len__:

.... itr = itertools.chain(
.... iter(iterable), itertools.repeat(pad_val, part_len - 1))
.... return itertools.izip(*[itr]*part_len)
....
[(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]

I just unconditionally pad the iterable with 1 less than the partition
size... I think that works right, but I haven't tested it any more than
what's shown.

Steve

Steven Bethard, Feb 1, 2005
10. ### Guest

Chris Wright wrote:
> Hi,
>
> 1) I want to iterate over a list "N at a time"
> sort of like:
>
> # Two at a time... won't work, obviously
>
> >>> for a, b in [1,2,3,4]:

> ... print a,b
> ...
> Traceback (most recent call last):
> File "<interactive input>", line 1, in ?
> TypeError: unpack non-sequence
> >>>

>
>
> Is there a nifty way to do with with list comprehensions,
> or do I just have to loop over the list ?
>
> cheers and thanks
>
> chris wright

I wouldn't call this nifty, but it does use list comprehensions:
(n-(len(l)%n))%n is the amount of padding
(len(l)+(n-(len(l)%n))%n)/n is the number of groups (calculated by
adding the padding to the length of l and then dividing by n)

>>> l = range(10)
>>> n = 3
>>> [(l+[None]*((n-(len(l)%n))%n))[i*ni+1)*n] for i in

xrange((len(l)+(n-(len(l)%n))%n)/n)]
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, None, None]]

Regards,
Todd

, Feb 1, 2005
11. ### Nick CoghlanGuest

Steven Bethard wrote:
> I think you can write that second one so that it works for iterables
> without a __len__:
>
> ... itr = itertools.chain(
> ... iter(iterable), itertools.repeat(pad_val, part_len - 1))
> ... return itertools.izip(*[itr]*part_len)
> ...
> [(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
> [(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]
>
> I just unconditionally pad the iterable with 1 less than the partition
> size... I think that works right, but I haven't tested it any more than
> what's shown.

I think you're right - I was looking at padding unconditionally, but because I
was padding with the actual partition length, it didn't work correctly when the

Padding with one less than the partition length fixes that quite neatly.

Cheers,
Nick.

--
Nick Coghlan | | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Nick Coghlan, Feb 2, 2005
12. ### David IsaacGuest

"Nick Coghlan" <> wrote in message
news:...
> Using zip(*[iter(l)]*N) or zip(*(iter(l),)*N) simply extends the above to

the
> general case.

Clearly true.
But can you please go into much more detail for a newbie?
I see that [iter(l)]*N produces an N element list with each element being
the same iterator object, but after that
http://www.python.org/doc/2.3.5/lib/built-in-funcs.html
just didn't get me there.

Thanks,
Alan Isaac

David Isaac, Feb 11, 2005
13. ### Nick CoghlanGuest

David Isaac wrote:
> "Nick Coghlan" <> wrote in message
> news:...
>
>>Using zip(*[iter(l)]*N) or zip(*(iter(l),)*N) simply extends the above to

>
> the
>
>>general case.

>
>
> Clearly true.
> But can you please go into much more detail for a newbie?
> I see that [iter(l)]*N produces an N element list with each element being
> the same iterator object, but after that
> http://www.python.org/doc/2.3.5/lib/built-in-funcs.html
> just didn't get me there.

See if the following interactive examples clear things up at all:

# The unclear version
Py> itr = iter(range(10))
Py> zipped = zip(*(itr,)*3) # How does this bit work?
Py> print "\n".join(map(str, zipped))
(0, 1, 2)
(3, 4, 5)
(6, 7, 8)

# Manual zip, printing as we go
Py> itr = iter(range(10))
Py> try:
.... while 1: print (itr.next(), itr.next(), itr.next())
.... except StopIteration:
.... pass
....
(0, 1, 2)
(3, 4, 5)
(6, 7, 8)

# Manual zip, actually behaving somewhat like the real thing
Py> itr = iter(range(10))
Py> zipped = []
Py> try:
.... while 1: zipped.append((itr.next(), itr.next(), itr.next()))
.... except StopIteration:
.... pass
....
Py> print "\n".join(map(str, zipped))
(0, 1, 2)
(3, 4, 5)
(6, 7, 8)

Cheers,
Nick.

--
Nick Coghlan | | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Nick Coghlan, Feb 11, 2005
14. ### David IsaacGuest

> Alan Isaac wrote:
> > I see that [iter(l)]*N produces an N element list with each element

being
> > the same iterator object, but after that
> > http://www.python.org/doc/2.3.5/lib/built-in-funcs.html
> > just didn't get me there.

"Nick Coghlan" <> wrote in message
news:...
> Py> itr = iter(range(10))
> Py> zipped = zip(*(itr,)*3) # How does this bit work?
> # Manual zip, actually behaving somewhat like the real thing
> Py> itr = iter(range(10))
> Py> zipped = []
> Py> try:
> ... while 1: zipped.append((itr.next(), itr.next(), itr.next()))
> ... except StopIteration:
> ... pass

http://www.python.org/doc/2.3.5/lib/built-in-funcs.html says:

"This function returns a list of tuples,
where the i-th tuple contains the i-th element from each of the argument
sequences."

So an "argument sequence" can in fact be any iterable,
and these in turn are asked *in rotation* for their yield, right?
So we pass the (identical) iterables in a tuple or list,
thereby allowing a variable number of arguments.
We unpack the argument list with '*',
which means we have provided three iterables as arguments.
And then zip works as "expected",
once we have learned to expect zip to "rotate" through the arguments.

If that is right, I still cannot extract it from the doc cited above.
So where should I have looked?

Thanks,
Alan Isaac

David Isaac, Feb 11, 2005
15. ### Nick CoghlanGuest

David Isaac wrote:
> If that is right, I still cannot extract it from the doc cited above.
> So where should I have looked?

Ouch. The terminology's evolved, and it looks to me like the docs for the older
builtins haven't been updated to track it.

The terminology has pretty much settled to 'iterable' for anything which returns
a sensible result from iter(obj), 'iterator' for any iterable which returns
itself from iter(obj), 'reiterable' for any iterable which is not an iterator,
and 'sequence' for any reiterable which supports len(obj) and integer indexing.

That's not the terminology the older docs use, though, even in the most recent
versions of that page [1].

For most of them it's OK, since the text clarifies what the term means in
context (e.g. that 'sequence' actually means 'iterable' for some function
signatures). zip() doesn't do that though - it actually accepts iterables, but

A bug report on Sourceforge would help in getting the problem fixed for the 2.5
docs (possibly even the 2.4.1 docs if it happens soon). 2.3's a lost cause
though, since 2.3.5 is already out the door and only another security bug is
likely to trigger a new 2.3 release.

For the 'left-to-right' evaluation thing, that's technically an implementation
artifact of the CPython implementation, since the zip() docs don't make any
promises. So updating the docs to include that information would probably be a
bigger issue, as it involves behaviour which is currently not defined by the
library.

Cheers,
Nick.

[1] http://www.python.org/dev/doc/devel/lib/built-in-funcs.html
--
Nick Coghlan | | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Nick Coghlan, Feb 12, 2005
16. ### David IsaacGuest

"Nick Coghlan" <> wrote in message
news:...
> A bug report on Sourceforge would help in getting the problem fixed for

the 2.5
> docs

Done.

> For the 'left-to-right' evaluation thing, that's technically an

implementation
> artifact of the CPython implementation, since the zip() docs don't make

any
> promises. So updating the docs to include that information would probably

be a
> bigger issue, as it involves behaviour which is currently not defined by

the
> library.

OK, thanks.

Alan Isaac

David Isaac, Feb 12, 2005
17. ### Nick CoghlanGuest

David Isaac wrote:
> "Nick Coghlan" <> wrote in message
> news:...
>
>>A bug report on Sourceforge would help in getting the problem fixed for

>
> the 2.5
>
>>docs

>
>
> Done.

Bug 1121416, for anyone else interested. Looks Raymond agrees with me about the
left-to-right evaluation of iterables being an overspecification.

Anyway, that means the zip and izip based solutions are technically version and
implementation specific. Fortunately, the final versions are fairly easy to turn
into a custom generator that doesn't rely on izip:

from itertools import islice, chain, repeat

def partition(iterable, part_len):
itr = iter(iterable)
while 1:
item = tuple(islice(itr, part_len))
if len(item) < part_len:
raise StopIteration
yield item

return partition(itr, part_len)

Py> list(partition(range(10), 3))
[(0, 1, 2), (3, 4, 5), (6, 7, 8)]
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, True, True)]

Well spotted on the fact that the way we were using zip/izip was undocumented,
btw

Cheers,
Nick.

--
Nick Coghlan | | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Nick Coghlan, Feb 13, 2005