list comprehensions

[Elaine Jackson]

List comprehensions don't work the way you intuitively expect them to work. I
realize many people have no intuitions about how list comprehensions 'should'
work, so if you recognize yourself in this description, feel free to go back to
whatever you were doing before. If you're still here, though, I invite you to
consider the following definition:

partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
range(1,n)]

As defined above, the function raises an exception when you call it ('k'
referenced before assignment). For the sake of clarity, here is workable code
expressing the same intention:

def partitions(n):
reVal=[[n]]
for k in range(1,n):
for x in partitions(n-k):
reVal.append([k]+x)
return reVal

So I guess what I want to ask is: Can somebody explain the semantics of list
comprehensions to me? Or even better: Can somebody tell me where to look in the
documentation to find out about list comprehensions? All donations gratefully
received.

Peace

 

[Daniel Dittmar]

List comprehensions don't work the way you intuitively expect them to work. I

How can you say such a thing. 100 Haskell programmers have been asked
about Python list comprehension and all found it intuitive.

partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
range(1,n)]

As defined above, the function raises an exception when you call it ('k'
referenced before assignment).

Try a simpler example and you'll see the order of execution with nested
loops:

>>> [(a, x) for a in "ab" for x in "xy"]

[('a', 'x'), ('a', 'y'), ('b', 'x'), ('b', 'y')]

So I guess what I want to ask is: Can somebody explain the semantics of list
comprehensions to me? Or even better: Can somebody tell me where to look in the
documentation to find out about list comprehensions? All donations gratefully
received.

http://www.python.org/doc/2.3.3/ref/lists.html, especially "each of the
for or if clauses a block, nesting from left to right".

>>> partitions = lambda n: [[n]]+[[k]+x for k in range(1,n) for x in partitions(n-k)]
>>> partitions (3)

[[3], [1, 2], [1, 1, 1], [2, 1]]

Daniel

 

[Terry Reedy]

List comprehensions don't work the way you intuitively expect them to

work.

One of my pet posting peeves is when people ascribe their own
characteristics to others, as when they write 'you' when they really mean,
or should have meant, 'I'. But nevermind.

I realize many people have no intuitions about how list comprehensions 'should'
work,

List comprehensions are sufficiently 'artificial' that I would advise
anyone to avoid the temptation to intuit how they work without consulting
the manual, tutorials, or other written explanations.

The Python Reference Manual Index, section L, entry List..comprehensions
takes one to subsection 5.2.4 List displays. The last two sentences
therein read

"When a list comprehension is supplied, it consists of a single expression
followed by at least one for clause and zero or more for or if clauses. In
this case, the elements of the new list are those that would be produced by
considering each of the for or if clauses a block, nesting from left to
right, and evaluating the expression to produce a list element each time
the innermost block is reached."

OK, takes a couple of readings and maybe some working examples.

partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
range(1,n)]

As defined above, the function raises an exception when you call it ('k'
referenced before assignment).

Of course, you have the two for clauses reversed, as the error message
might suggest to an experienced Pythoneer. As the manual specifies, for/if
clauses nest *left to right*. To match the double loop below, you want
instead

[[k]+x for k in range(1,n) for x in partitions(n-k)]

def partitions(n):
reVal=[[n]]
for k in range(1,n):
for x in partitions(n-k):
reVal.append([k]+x)
return reVal

Terry J. Reedy

 

[Paul McGuire]

List comprehensions don't work the way you intuitively expect them to work. I
realize many people have no intuitions about how list comprehensions 'should'
work, so if you recognize yourself in this description, feel free to go back to
whatever you were doing before. If you're still here, though, I invite you to
consider the following definition:

partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
range(1,n)]

As defined above, the function raises an exception when you call it ('k'
referenced before assignment). For the sake of clarity, here is workable code
expressing the same intention:

def partitions(n):
reVal=[[n]]
for k in range(1,n):
for x in partitions(n-k):
reVal.append([k]+x)
return reVal

So I guess what I want to ask is: Can somebody explain the semantics of list
comprehensions to me? Or even better: Can somebody tell me where to look in the
documentation to find out about list comprehensions? All donations gratefully
received.

Peace

Elaine -

I suspect you wrote your listcomp by thinking this is an inversion of the
nested for loops. That is, if you think of some arbitrary nesting of loops
as:

for i in firstRange:
for j in secondRange:
for k in thirdRange:
...<as many more fors as you like>...
<build up list using i,j,k,...>

then you looked at the list comp as working its way inside out, since it
starts with the <build up list> part.

But as the other posters have already stated, even though the listcomp
starts with the most deeply nested body, the remaining 'for... for ...
for... ' clauses match the nested loops, as if you had just removed the
colons and copied them all onto one line. So the listcomp ends up looking
like:

[<build up list using i,j,k,...> for i in firstRange for j in
secondRange for k in thirdRange...]

If you want some kind of model for how to conceptualize this, then I'd say
that a listcomp starts with its most important part being the actual list
element assembly/definition, followed by the iteration specifiers in
successive outer-to-inner order, that will get the loop variables to the
necessary values to satisfy the list assembly code.

You also asked for doc references. The Python Tutorial includes an obscure
example (you have do a bit of in-your-head multiplication to get it), but
try these other articles describing listcomps when they were a new feature:
http://www-106.ibm.com/developerworks/linux/library/l-py20.html
http://python.org/2.0/new-python.html#SECTION000600000000000000000
Perhaps the tutorial could lift some of the examples from these other pages,
as I'm sure people don't necessarily go back to the "What's New" pages for
version x-2, x-3, etc.

HTH,
-- Paul

 

[Elaine Jackson]

I still need to digest it, but yes, I think it will help. Thanks.

| | >
| > List comprehensions don't work the way you intuitively expect them to
| work. I
| > realize many people have no intuitions about how list comprehensions
| 'should'
| > work, so if you recognize yourself in this description, feel free to go
| back to
| > whatever you were doing before. If you're still here, though, I invite you
| to
| > consider the following definition:
| >
| > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
| > range(1,n)]
| >
| > As defined above, the function raises an exception when you call it ('k'
| > referenced before assignment). For the sake of clarity, here is workable
| code
| > expressing the same intention:
| >
| > def partitions(n):
| > reVal=[[n]]
| > for k in range(1,n):
| > for x in partitions(n-k):
| > reVal.append([k]+x)
| > return reVal
| >
| > So I guess what I want to ask is: Can somebody explain the semantics of
| list
| > comprehensions to me? Or even better: Can somebody tell me where to look
| in the
| > documentation to find out about list comprehensions? All donations
| gratefully
| > received.
| >
| > Peace
| >
| >
| Elaine -
|
| I suspect you wrote your listcomp by thinking this is an inversion of the
| nested for loops. That is, if you think of some arbitrary nesting of loops
| as:
|
| for i in firstRange:
| for j in secondRange:
| for k in thirdRange:
| ...<as many more fors as you like>...
| <build up list using i,j,k,...>
|
| then you looked at the list comp as working its way inside out, since it
| starts with the <build up list> part.
|
| But as the other posters have already stated, even though the listcomp
| starts with the most deeply nested body, the remaining 'for... for ...
| for... ' clauses match the nested loops, as if you had just removed the
| colons and copied them all onto one line. So the listcomp ends up looking
| like:
|
| [<build up list using i,j,k,...> for i in firstRange for j in
| secondRange for k in thirdRange...]
|
| If you want some kind of model for how to conceptualize this, then I'd say
| that a listcomp starts with its most important part being the actual list
| element assembly/definition, followed by the iteration specifiers in
| successive outer-to-inner order, that will get the loop variables to the
| necessary values to satisfy the list assembly code.
|
| You also asked for doc references. The Python Tutorial includes an obscure
| example (you have do a bit of in-your-head multiplication to get it), but
| try these other articles describing listcomps when they were a new feature:
| http://www-106.ibm.com/developerworks/linux/library/l-py20.html
| http://python.org/2.0/new-python.html#SECTION000600000000000000000
| Perhaps the tutorial could lift some of the examples from these other pages,
| as I'm sure people don't necessarily go back to the "What's New" pages for
| version x-2, x-3, etc.
|
| HTH,
| -- Paul
|
|
|
|

 

[Elaine Jackson]

OK, it's pretty embarassing that I gave up when it wasn't in the contents and
never even thought of the index. And so far everybody's been too polite to point
out that I left the base case out of my recursion. But I stand by my complaint
about unintuitiveness, because I've discovered that you get an error from

x = [(i,j) for i in range(7-j) for j in range(3)]

while

y = [[(i,j) for i in range(7-j)] for j in range(3)]

works fine. In other words, my intuition was that x would be
reduce(list.__add__,y). And in fact I'm still a little confused, because I would
describe the syntax of y as going, as you say, from left to right (ie: a
left-branching tree). I suppose I'm going from the inside out, and you're
talking about going from the outside inward. Or something like that. Anyway,
thanks for your help. Now that I know what to read, I'll read it.

Peace


|
| | >
| > List comprehensions don't work the way you intuitively expect them to
| work.
|
| One of my pet posting peeves is when people ascribe their own
| characteristics to others, as when they write 'you' when they really mean,
| or should have meant, 'I'. But nevermind.
|
| > I realize many people have no intuitions about how list comprehensions
| 'should'
| > work,
|
| List comprehensions are sufficiently 'artificial' that I would advise
| anyone to avoid the temptation to intuit how they work without consulting
| the manual, tutorials, or other written explanations.
|
| The Python Reference Manual Index, section L, entry List..comprehensions
| takes one to subsection 5.2.4 List displays. The last two sentences
| therein read
|
| "When a list comprehension is supplied, it consists of a single expression
| followed by at least one for clause and zero or more for or if clauses. In
| this case, the elements of the new list are those that would be produced by
| considering each of the for or if clauses a block, nesting from left to
| right, and evaluating the expression to produce a list element each time
| the innermost block is reached."
|
| OK, takes a couple of readings and maybe some working examples.
|
| > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
| > range(1,n)]
| >
| > As defined above, the function raises an exception when you call it ('k'
| > referenced before assignment).
|
| Of course, you have the two for clauses reversed, as the error message
| might suggest to an experienced Pythoneer. As the manual specifies, for/if
| clauses nest *left to right*. To match the double loop below, you want
| instead
|
| [[k]+x for k in range(1,n) for x in partitions(n-k)]
|
| > def partitions(n):
| > reVal=[[n]]
| > for k in range(1,n):
| > for x in partitions(n-k):
| > reVal.append([k]+x)
| > return reVal
|
| Terry J. Reedy
|
|
|
|

 

[Mikael Olofsson]

[snip] I've discovered that you get an error from

x = [(i,j) for i in range(7-j) for j in range(3)]

while

y = [[(i,j) for i in range(7-j)] for j in range(3)]

works fine. [snip]

I will not argue about intuitiveness, but FYI:

z = [(i,j) for j in range(3) for i in range(7-j)]

works fine. Others can explain why it is one way and not the other.

/Mikael Olofsson
Universitetslektor (Associate professor)
Linköpings universitet

 

[Terry Reedy]

But I stand by my complaint about unintuitiveness,

Which I already agreed with, which I why I said, "Don't try to intuit!";-)

because I've discovered that you get an error from

x = [(i,j) for i in range(7-j) for j in range(3)]

because the i loop is outside/before the j loop. The problem with trying
to intuit is that list comps move the appended expression from inside to
outtermost while otherwise leaving the order outside-in. You were
expecting the order to be uniformly reversed, and it is not. If the syntax
had specified the above to be written as

[for i in range(7-j): for j in range(3): (i,j)]

which more obviously abbreviates the corresponding statements, the error
would be more obvious. Moving the expression to the front was, of course,
intentional -- to make it stand out more -- but I can be somewhat confusing
at first until one gets used to it.

> while

y = [[(i,j) for i in range(7-j)] for j in range(3)]

works fine.

because now the i-loop, as part of the appended expression, gets executed
inside the j loop.

y=[]
for j in range(3):
y.append([(i,j) for i in range(7-j)])

Terry J. Reedy

 

[Aahz]

List comprehensions don't work the way you intuitively expect them
to work. I realize many people have no intuitions about how list
comprehensions 'should' work, so if you recognize yourself in this
description, feel free to go back to whatever you were doing before. If
you're still here, though, I invite you to consider the following
definition:

Short response: don't write complex listcomps.

 

[Dave Benjamin]

Short response: don't write complex listcomps.

Would you consider a 2D loop too complex for a listcomp?

 

[Aahz]

Would you consider a 2D loop too complex for a listcomp?

It depends. If it's something simple, go ahead. If you've got an
``if`` or a complex expression, I'd think real hard.