Iterating over several lists at once

G

Gal Diskin

Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...


I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:

for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3

Or maybe like this:

for x1 in l1, x2 in l2, x3 in l3:
print "do something with", x1, x2, x3

However, this code obviously doesn't work...


I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).

Thanks in advance,
Gal
 
F

Fredrik Lundh

Gal Diskin said:
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...

I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:

for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3

how about

for x1, x2, x3 in func(l1, l2, l3):
print x1, x2, x3

where func is defined as, say,

def func(l1, l2, l3):
return ((x1, x2, x3) for x1 in l1 for x2 in l2 for x3 in l3)

or if you prefer

def helper(l1, l2, l3):
for x1 in l1:
for x2 in l2:
for x3 in l3:
yield x1, x2, x3

</F>
 
R

Roberto Bonvallet

Gal said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What's wrong with this?

[...]
I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).

def cartesian_product(l1, l2, l3):
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)

for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
 
P

Paul Rubin

Gal Diskin said:
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3


This does look a little kludgy (untested):

for x1,x2,x3 in ((x1,x2,x3) for x1 in l1 for x2 in l2 for x3 in l3):
print "do something with", x1, x2, x3
 
P

Peter Otten

Gal said:
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3
I was wondering if one could write this more easily in some manner
using only 1 for loop.

def nested_loops(*args):
assert args
if len(args) == 1:
for item in args[0]:
yield (item,)
else:
gap = len(args) // 2
for left in nested_loops(*args[:gap]):
for right in nested_loops(*args[gap:]):
yield left + right

if __name__ == "__main__":
for i, k in nested_loops("abc", "12"):
print i, k
for i, j, k in nested_loops("ab", "123", "xyz"):
print i, j, k

Be prepared for a significant performance hit.

Peter

PS: Did anybody say macro? No? I must be hallucinating...
 
G

Gal Diskin

Nothing seriously wrong, but it's not too elegent. Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python). As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.

Thanks for replying me.


Gal said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
[...]

I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def cartesian_product(l1, l2, l3):
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)

for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
 
K

Kay Schluehr

Gal said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...

Heard about recursion?

def collect(L,*lists):
if not lists:
return [(x,) for x in L]
collection = []
for x in L:
for y in collect(lists[0],*lists[1:]):
collection.append((x,)+y)
return collection

for item in collect( l1, l2, l3):
func(*item)

Here is the same in generator form

def collect(L,*lists):
if not lists:
for x in L:
yield (x,)
else:
for x in L:
for y in collect(lists[0],*lists[1:]):
yield (x,)+y

( o.k - it required two nested for-loops in each implementation :)
 
G

Gal Diskin

Thanks, that's an improvment (your first way).
But I still wish I could find an even shorter (or more elegent) way of
doing it. (Well, I guess if I expect every wish I have to come true I
should at least wish for something more valuable.)

Thanks again,
Gal
 
M

Maksim Kasimov

Hi,

if you "needs to iterate over 3 lists at the same" and according your example
> for (x1,x2,x3) in (l1,l2,l3):
> print "do something with", x1, x2, x3


i has guessed that you need this (may be i was wrong):


a = (1,2,3, 1)
b = (4,5,6)
c = (7,8,9, 2, 3)

for x, y, z in zip(a, b, c):
print x, y, z


or this


for x, y, z in map(None, a, b, c):
print x,y,z


Try both examples with tuples that have various length, they have difference
 
C

Carl Banks

Gal said:
Gal said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
[...]

I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def cartesian_product(l1, l2, l3):
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)

for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k

Nothing seriously wrong, but it's not too elegent.

I wonder why you think that. Many people here would consider it the
height of elegance to take some complicated logic, writing a function
encompassing the complicated parts, and ending up with simpler logic.
That's what happens here.

The cartesian_product solution above is not much longer than this
syntax you proposed:

for (i,j,k) in (a,b,c):
use(i,j,k)

(You can make the function name shorter if you want.) What is it that
you find inelegant about this solution? Is it that you don't like
extra function sitting there? Just put it in a module and import it.

Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python).

The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.

def cartesian_product(*args):
if len(args) > 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()

As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.

Even if Python had a special syntax for it, it would still be looping
internally.

And there's almost no chance of there ever being a special syntax for
it. First of all, it's very straightforward to do with functional
solutions, as we've shown here. Second, compare the case of zip.
Iteration in parallel (such as you would do with zip) is a lot more
common than nested iteration (cartesian_product), but even parallel
iteration doesn't have a special syntax, and th BDFL had declared that
won't in Python 3.0.

The best you could hope for is a built in function like zip, and that's
doubtful. OTOH, something like this has decent shot of appearing in a
standard functional or iterator module of some sort.

Thanks for replying me.

You're welcome in advance. Also, one note about comp.lang.python
and/or python-list: it's conventional here to put replies beneath the
quoted text. This is for the benefit of future readers, so they don't
have read conversations in reverse order.


Carl Banks
 
A

at

Sorry for breaking into this thread, but I agree completely that any
unnecessary indentations should be avoided. For the same reason I advocate
that the following syntax should work:

for x in some_list if some_condition:
... code ...

in stead of

for x in some_list
if some_condition:
... code ...

All the best!

@

PS: maybe using 'sets' can help you out for a particular problem.


Gal said:
Nothing seriously wrong, but it's not too elegent. Especially when the
number of lists you want to iterate over gets bigger (especially
because of the indentation in python). As you noticed (an phrased
better than me), what I was wondering is if there is a way to iterate
over the cartesian product, but without actually doing all n for loops
but using a single "for" loop.

Thanks for replying me.


Gal said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:
for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3What's wrong with this?
[...]

I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).def
cartesian_product(l1, l2, l3):
for i in l1:
for j in l2:
for k in l3:
yield (i, j, k)

for (i, j, k) in cartesian_product(l1, l2, l3):
print "do something with", i, j, k
 
M

Mike Erickson

* at ([email protected]) said:
Sorry for breaking into this thread, but I agree completely that any
unnecessary indentations should be avoided. For the same reason I advocate
that the following syntax should work:

for x in some_list if some_condition:
... code ...

in stead of

for x in some_list
if some_condition:
... code ...

It is possible to avoid the extra level of indentaion, but I think it's
much less readable than the 2-level verbose expresion:
a [1, 2, 3, 4, 5, 6, 7]
for odd in (num for num in a if num % 2 == 1):
.... print odd
....
1
3
5
7

there is also continue, which I think is a good compromise:
.... if num % 2 == 0:
.... continue
.... print num
....
1
3
5
7

HTH (and not lead astray),

mike
 
J

John Henry

Carl Banks wrote:
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.

def cartesian_product(*args):
if len(args) > 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()

Very nice.
 
M

Michael Spencer

John said:
Carl Banks wrote:
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.

def cartesian_product(*args):
if len(args) > 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()

Very nice.
another implementation of cartesian_product using 'reduce' to add
mystery ;-)

def star(outer, inner):
for rest in outer:
for item in inner:
yield rest + (item,)

def cartesian_product(*args):
return reduce(star, args, ((),))
[('0', '0', '0'), ('0', '0', '1'), ('0', '1', '0'), ('0', '1', '1'),
('1', '0', '0'), ('1', '0', '1'), ('1', '1', '0'), ('1', '1', '1')]
Michael
 
J

John Henry

Is this suppose to be a brain teaser or something?

Michael said:
John said:
Carl Banks wrote:
The function can be extended to allow arbitrary arguments. Here's a
non-minmal recursive version.

def cartesian_product(*args):
if len(args) > 1:
for item in args[0]:
for rest in cartesian_product(*args[1:]):
yield (item,) + rest
elif len(args) == 1:
for item in args[0]:
yield (item,)
else:
yield ()

Very nice.
another implementation of cartesian_product using 'reduce' to add
mystery ;-)

def star(outer, inner):
for rest in outer:
for item in inner:
yield rest + (item,)

def cartesian_product(*args):
return reduce(star, args, ((),))
[('0', '0', '0'), ('0', '0', '1'), ('0', '1', '0'), ('0', '1', '1'),
('1', '0', '0'), ('1', '0', '1'), ('1', '1', '0'), ('1', '1', '1')]
Michael
 
G

Gal Diskin

Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...

I was wondering if one could write this more easily in some manner
using only 1 for loop.
What I mean is something like this:

for (x1,x2,x3) in (l1,l2,l3):
print "do something with", x1, x2, x3

Or maybe like this:

for x1 in l1, x2 in l2, x3 in l3:
print "do something with", x1, x2, x3

However, this code obviously doesn't work...

I'd be very happy to receive ideas about how to do this in one loop and
with minimal initialization (if at all required).

Thanks in advance,Gal



Sorry for bumping this thread up. I just got back from a vacation and I
had to say thanks to all the people that took the time to answer me.

To Maksim - that's not what I meant, I want to iterate over the
cartesian product of the lists (all ordered tuples, _not_ the tuple
created from the first item in each list, then the tubple created from
the second item in each list... as your example does).

Thanks,
Gal
 
E

Erik Johnson

I don't see your previous article. Not sure if you are still looking for
a solution, but here's one:
[(x, y, z) for x in [1,2,3] for y in list('abc') for z in list('XY')]
[(1, 'a', 'X'), (1, 'a', 'Y'), (1, 'b', 'X'), (1, 'b', 'Y'), (1, 'c', 'X'),
(1, 'c', 'Y'), (2, 'a', 'X'), (2, 'a', 'Y'), (2, 'b', 'X'), (2, 'b', 'Y'),
(2, 'c', 'X'), (2, 'c', 'Y'), (3, 'a', 'X'), (3, 'a', 'Y'), (3, 'b', 'X'),
(3, 'b', 'Y'), (3, 'c', 'X'), (3, 'c', 'Y')]

This is a bit more compact, but I don't see anything wrong with what you
have.
 
R

Roy Smith

"Gal Diskin said:
Hi,
I am writing a code that needs to iterate over 3 lists at the same
time, i.e something like this:

for x1 in l1:
for x2 in l2:
for x3 in l3:
print "do something with", x1, x2, x3

What I need to do is go over all n-tuples where the first argument is
from the first list, the second from the second list, and so on...

Take a look at
http://mail.python.org/pipermail/python-list/2001-September/104983.html

or

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/159975

There's nothing magic about either -- fundamentally, you're still doing an
N^3 operation and it's going to be slow. You might want to stop and think
if there's some better algorithm than an exhaustive search of the entire
domain space for whatever it is that you're trying to do.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top