list comprehensions

Discussion in 'Python' started by Elaine Jackson, Apr 7, 2004.

  1. List comprehensions don't work the way you intuitively expect them to work. I
    realize many people have no intuitions about how list comprehensions 'should'
    work, so if you recognize yourself in this description, feel free to go back to
    whatever you were doing before. If you're still here, though, I invite you to
    consider the following definition:

    partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    range(1,n)]

    As defined above, the function raises an exception when you call it ('k'
    referenced before assignment). For the sake of clarity, here is workable code
    expressing the same intention:

    def partitions(n):
    reVal=[[n]]
    for k in range(1,n):
    for x in partitions(n-k):
    reVal.append([k]+x)
    return reVal

    So I guess what I want to ask is: Can somebody explain the semantics of list
    comprehensions to me? Or even better: Can somebody tell me where to look in the
    documentation to find out about list comprehensions? All donations gratefully
    received.

    Peace
    Elaine Jackson, Apr 7, 2004
    #1
    1. Advertising

  2. Elaine Jackson wrote:
    > List comprehensions don't work the way you intuitively expect them to work. I


    How can you say such a thing. 100 Haskell programmers have been asked
    about Python list comprehension and all found it intuitive.

    > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    > range(1,n)]
    >
    > As defined above, the function raises an exception when you call it ('k'
    > referenced before assignment).


    Try a simpler example and you'll see the order of execution with nested
    loops:
    >>> [(a, x) for a in "ab" for x in "xy"]

    [('a', 'x'), ('a', 'y'), ('b', 'x'), ('b', 'y')]

    > So I guess what I want to ask is: Can somebody explain the semantics of list
    > comprehensions to me? Or even better: Can somebody tell me where to look in the
    > documentation to find out about list comprehensions? All donations gratefully
    > received.


    http://www.python.org/doc/2.3.3/ref/lists.html, especially "each of the
    for or if clauses a block, nesting from left to right".

    >>> partitions = lambda n: [[n]]+[[k]+x for k in range(1,n) for x in

    partitions(n-k)]
    >>> partitions (3)

    [[3], [1, 2], [1, 1, 1], [2, 1]]

    Daniel
    Daniel Dittmar, Apr 7, 2004
    #2
    1. Advertising

  3. Elaine Jackson

    Terry Reedy Guest

    "Elaine Jackson" <> wrote in message
    news:yjZcc.42823$Pk3.9547@pd7tw1no...
    >
    > List comprehensions don't work the way you intuitively expect them to

    work.

    One of my pet posting peeves is when people ascribe their own
    characteristics to others, as when they write 'you' when they really mean,
    or should have meant, 'I'. But nevermind.

    > I realize many people have no intuitions about how list comprehensions

    'should'
    > work,


    List comprehensions are sufficiently 'artificial' that I would advise
    anyone to avoid the temptation to intuit how they work without consulting
    the manual, tutorials, or other written explanations.

    The Python Reference Manual Index, section L, entry List..comprehensions
    takes one to subsection 5.2.4 List displays. The last two sentences
    therein read

    "When a list comprehension is supplied, it consists of a single expression
    followed by at least one for clause and zero or more for or if clauses. In
    this case, the elements of the new list are those that would be produced by
    considering each of the for or if clauses a block, nesting from left to
    right, and evaluating the expression to produce a list element each time
    the innermost block is reached."

    OK, takes a couple of readings and maybe some working examples.

    > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    > range(1,n)]
    >
    > As defined above, the function raises an exception when you call it ('k'
    > referenced before assignment).


    Of course, you have the two for clauses reversed, as the error message
    might suggest to an experienced Pythoneer. As the manual specifies, for/if
    clauses nest *left to right*. To match the double loop below, you want
    instead

    [[k]+x for k in range(1,n) for x in partitions(n-k)]

    > def partitions(n):
    > reVal=[[n]]
    > for k in range(1,n):
    > for x in partitions(n-k):
    > reVal.append([k]+x)
    > return reVal


    Terry J. Reedy
    Terry Reedy, Apr 7, 2004
    #3
  4. Elaine Jackson

    Paul McGuire Guest

    "Elaine Jackson" <> wrote in message
    news:yjZcc.42823$Pk3.9547@pd7tw1no...
    >
    > List comprehensions don't work the way you intuitively expect them to

    work. I
    > realize many people have no intuitions about how list comprehensions

    'should'
    > work, so if you recognize yourself in this description, feel free to go

    back to
    > whatever you were doing before. If you're still here, though, I invite you

    to
    > consider the following definition:
    >
    > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    > range(1,n)]
    >
    > As defined above, the function raises an exception when you call it ('k'
    > referenced before assignment). For the sake of clarity, here is workable

    code
    > expressing the same intention:
    >
    > def partitions(n):
    > reVal=[[n]]
    > for k in range(1,n):
    > for x in partitions(n-k):
    > reVal.append([k]+x)
    > return reVal
    >
    > So I guess what I want to ask is: Can somebody explain the semantics of

    list
    > comprehensions to me? Or even better: Can somebody tell me where to look

    in the
    > documentation to find out about list comprehensions? All donations

    gratefully
    > received.
    >
    > Peace
    >
    >

    Elaine -

    I suspect you wrote your listcomp by thinking this is an inversion of the
    nested for loops. That is, if you think of some arbitrary nesting of loops
    as:

    for i in firstRange:
    for j in secondRange:
    for k in thirdRange:
    ...<as many more fors as you like>...
    <build up list using i,j,k,...>

    then you looked at the list comp as working its way inside out, since it
    starts with the <build up list> part.

    But as the other posters have already stated, even though the listcomp
    starts with the most deeply nested body, the remaining 'for... for ...
    for... ' clauses match the nested loops, as if you had just removed the
    colons and copied them all onto one line. So the listcomp ends up looking
    like:

    [<build up list using i,j,k,...> for i in firstRange for j in
    secondRange for k in thirdRange...]

    If you want some kind of model for how to conceptualize this, then I'd say
    that a listcomp starts with its most important part being the actual list
    element assembly/definition, followed by the iteration specifiers in
    successive outer-to-inner order, that will get the loop variables to the
    necessary values to satisfy the list assembly code.

    You also asked for doc references. The Python Tutorial includes an obscure
    example (you have do a bit of in-your-head multiplication to get it), but
    try these other articles describing listcomps when they were a new feature:
    http://www-106.ibm.com/developerworks/linux/library/l-py20.html
    http://python.org/2.0/new-python.html#SECTION000600000000000000000
    Perhaps the tutorial could lift some of the examples from these other pages,
    as I'm sure people don't necessarily go back to the "What's New" pages for
    version x-2, x-3, etc.

    HTH,
    -- Paul
    Paul McGuire, Apr 8, 2004
    #4
  5. I still need to digest it, but yes, I think it will help. Thanks.

    "Paul McGuire" <._bogus_.com> wrote in message
    news:ku0dc.74742$...
    | "Elaine Jackson" <> wrote in message
    | news:yjZcc.42823$Pk3.9547@pd7tw1no...
    | >
    | > List comprehensions don't work the way you intuitively expect them to
    | work. I
    | > realize many people have no intuitions about how list comprehensions
    | 'should'
    | > work, so if you recognize yourself in this description, feel free to go
    | back to
    | > whatever you were doing before. If you're still here, though, I invite you
    | to
    | > consider the following definition:
    | >
    | > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    | > range(1,n)]
    | >
    | > As defined above, the function raises an exception when you call it ('k'
    | > referenced before assignment). For the sake of clarity, here is workable
    | code
    | > expressing the same intention:
    | >
    | > def partitions(n):
    | > reVal=[[n]]
    | > for k in range(1,n):
    | > for x in partitions(n-k):
    | > reVal.append([k]+x)
    | > return reVal
    | >
    | > So I guess what I want to ask is: Can somebody explain the semantics of
    | list
    | > comprehensions to me? Or even better: Can somebody tell me where to look
    | in the
    | > documentation to find out about list comprehensions? All donations
    | gratefully
    | > received.
    | >
    | > Peace
    | >
    | >
    | Elaine -
    |
    | I suspect you wrote your listcomp by thinking this is an inversion of the
    | nested for loops. That is, if you think of some arbitrary nesting of loops
    | as:
    |
    | for i in firstRange:
    | for j in secondRange:
    | for k in thirdRange:
    | ...<as many more fors as you like>...
    | <build up list using i,j,k,...>
    |
    | then you looked at the list comp as working its way inside out, since it
    | starts with the <build up list> part.
    |
    | But as the other posters have already stated, even though the listcomp
    | starts with the most deeply nested body, the remaining 'for... for ...
    | for... ' clauses match the nested loops, as if you had just removed the
    | colons and copied them all onto one line. So the listcomp ends up looking
    | like:
    |
    | [<build up list using i,j,k,...> for i in firstRange for j in
    | secondRange for k in thirdRange...]
    |
    | If you want some kind of model for how to conceptualize this, then I'd say
    | that a listcomp starts with its most important part being the actual list
    | element assembly/definition, followed by the iteration specifiers in
    | successive outer-to-inner order, that will get the loop variables to the
    | necessary values to satisfy the list assembly code.
    |
    | You also asked for doc references. The Python Tutorial includes an obscure
    | example (you have do a bit of in-your-head multiplication to get it), but
    | try these other articles describing listcomps when they were a new feature:
    | http://www-106.ibm.com/developerworks/linux/library/l-py20.html
    | http://python.org/2.0/new-python.html#SECTION000600000000000000000
    | Perhaps the tutorial could lift some of the examples from these other pages,
    | as I'm sure people don't necessarily go back to the "What's New" pages for
    | version x-2, x-3, etc.
    |
    | HTH,
    | -- Paul
    |
    |
    |
    |
    Elaine Jackson, Apr 8, 2004
    #5
  6. OK, it's pretty embarassing that I gave up when it wasn't in the contents and
    never even thought of the index. And so far everybody's been too polite to point
    out that I left the base case out of my recursion. But I stand by my complaint
    about unintuitiveness, because I've discovered that you get an error from

    x = [(i,j) for i in range(7-j) for j in range(3)]

    while

    y = [[(i,j) for i in range(7-j)] for j in range(3)]

    works fine. In other words, my intuition was that x would be
    reduce(list.__add__,y). And in fact I'm still a little confused, because I would
    describe the syntax of y as going, as you say, from left to right (ie: a
    left-branching tree). I suppose I'm going from the inside out, and you're
    talking about going from the outside inward. Or something like that. Anyway,
    thanks for your help. Now that I know what to read, I'll read it.

    Peace


    "Terry Reedy" <> wrote in message
    news:...
    |
    | "Elaine Jackson" <> wrote in message
    | news:yjZcc.42823$Pk3.9547@pd7tw1no...
    | >
    | > List comprehensions don't work the way you intuitively expect them to
    | work.
    |
    | One of my pet posting peeves is when people ascribe their own
    | characteristics to others, as when they write 'you' when they really mean,
    | or should have meant, 'I'. But nevermind.
    |
    | > I realize many people have no intuitions about how list comprehensions
    | 'should'
    | > work,
    |
    | List comprehensions are sufficiently 'artificial' that I would advise
    | anyone to avoid the temptation to intuit how they work without consulting
    | the manual, tutorials, or other written explanations.
    |
    | The Python Reference Manual Index, section L, entry List..comprehensions
    | takes one to subsection 5.2.4 List displays. The last two sentences
    | therein read
    |
    | "When a list comprehension is supplied, it consists of a single expression
    | followed by at least one for clause and zero or more for or if clauses. In
    | this case, the elements of the new list are those that would be produced by
    | considering each of the for or if clauses a block, nesting from left to
    | right, and evaluating the expression to produce a list element each time
    | the innermost block is reached."
    |
    | OK, takes a couple of readings and maybe some working examples.
    |
    | > partitions = lambda n: [[n]]+[[k]+x for x in partitions(n-k) for k in
    | > range(1,n)]
    | >
    | > As defined above, the function raises an exception when you call it ('k'
    | > referenced before assignment).
    |
    | Of course, you have the two for clauses reversed, as the error message
    | might suggest to an experienced Pythoneer. As the manual specifies, for/if
    | clauses nest *left to right*. To match the double loop below, you want
    | instead
    |
    | [[k]+x for k in range(1,n) for x in partitions(n-k)]
    |
    | > def partitions(n):
    | > reVal=[[n]]
    | > for k in range(1,n):
    | > for x in partitions(n-k):
    | > reVal.append([k]+x)
    | > return reVal
    |
    | Terry J. Reedy
    |
    |
    |
    |
    Elaine Jackson, Apr 8, 2004
    #6
  7. "Elaine Jackson" <> wrote:
    > [snip] I've discovered that you get an error from
    >
    > x = [(i,j) for i in range(7-j) for j in range(3)]
    >
    > while
    >
    > y = [[(i,j) for i in range(7-j)] for j in range(3)]
    >
    > works fine. [snip]


    I will not argue about intuitiveness, but FYI:

    z = [(i,j) for j in range(3) for i in range(7-j)]

    works fine. Others can explain why it is one way and not the other.

    /Mikael Olofsson
    Universitetslektor (Associate professor)
    Linköpings universitet

    -----------------------------------------------------------------------
    E-Mail:
    WWW: http://www.dtr.isy.liu.se/en/staff/mikael
    Phone: +46 - (0)13 - 28 1343
    Telefax: +46 - (0)13 - 28 1339
    -----------------------------------------------------------------------
    Linköpings kammarkör: www.kammarkoren.com Vi söker tenorer och basar!
    Mikael Olofsson, Apr 8, 2004
    #7
  8. Elaine Jackson

    Terry Reedy Guest

    "Elaine Jackson" <> wrote in message
    news:i56dc.46224$Pk3.1562@pd7tw1no...
    > But I stand by my complaint about unintuitiveness,


    Which I already agreed with, which I why I said, "Don't try to intuit!";-)

    > because I've discovered that you get an error from
    >
    > x = [(i,j) for i in range(7-j) for j in range(3)]


    because the i loop is outside/before the j loop. The problem with trying
    to intuit is that list comps move the appended expression from inside to
    outtermost while otherwise leaving the order outside-in. You were
    expecting the order to be uniformly reversed, and it is not. If the syntax
    had specified the above to be written as

    [for i in range(7-j): for j in range(3): (i,j)]

    which more obviously abbreviates the corresponding statements, the error
    would be more obvious. Moving the expression to the front was, of course,
    intentional -- to make it stand out more -- but I can be somewhat confusing
    at first until one gets used to it.

    > while
    >
    > y = [[(i,j) for i in range(7-j)] for j in range(3)]
    >
    > works fine.


    because now the i-loop, as part of the appended expression, gets executed
    inside the j loop.

    y=[]
    for j in range(3):
    y.append([(i,j) for i in range(7-j)])

    Terry J. Reedy
    Terry Reedy, Apr 8, 2004
    #8
  9. Elaine Jackson

    Aahz Guest

    In article <yjZcc.42823$Pk3.9547@pd7tw1no>,
    Elaine Jackson <> wrote:
    >
    >List comprehensions don't work the way you intuitively expect them
    >to work. I realize many people have no intuitions about how list
    >comprehensions 'should' work, so if you recognize yourself in this
    >description, feel free to go back to whatever you were doing before. If
    >you're still here, though, I invite you to consider the following
    >definition:


    Short response: don't write complex listcomps.
    --
    Aahz () <*> http://www.pythoncraft.com/

    Why is this newsgroup different from all other newsgroups?
    Aahz, Apr 9, 2004
    #9
  10. In article <c56n0j$9r8$>, Aahz wrote:
    > In article <yjZcc.42823$Pk3.9547@pd7tw1no>,
    > Elaine Jackson <> wrote:
    >>
    >>List comprehensions don't work the way you intuitively expect them
    >>to work. I realize many people have no intuitions about how list
    >>comprehensions 'should' work, so if you recognize yourself in this
    >>description, feel free to go back to whatever you were doing before. If
    >>you're still here, though, I invite you to consider the following
    >>definition:

    >
    > Short response: don't write complex listcomps.


    Would you consider a 2D loop too complex for a listcomp?

    --
    ..:[ dave benjamin: ramen/[sp00] -:- spoomusic.com -:- ramenfest.com ]:.
    : please talk to your son or daughter about parametric polymorphism. :
    Dave Benjamin, Apr 12, 2004
    #10
  11. Elaine Jackson

    Aahz Guest

    In article <>,
    Dave Benjamin <> wrote:
    >In article <c56n0j$9r8$>, Aahz wrote:
    >> In article <yjZcc.42823$Pk3.9547@pd7tw1no>,
    >> Elaine Jackson <> wrote:
    >>>
    >>>List comprehensions don't work the way you intuitively expect them
    >>>to work. I realize many people have no intuitions about how list
    >>>comprehensions 'should' work, so if you recognize yourself in this
    >>>description, feel free to go back to whatever you were doing before. If
    >>>you're still here, though, I invite you to consider the following
    >>>definition:

    >>
    >> Short response: don't write complex listcomps.

    >
    >Would you consider a 2D loop too complex for a listcomp?


    It depends. If it's something simple, go ahead. If you've got an
    ``if`` or a complex expression, I'd think real hard.
    --
    Aahz () <*> http://www.pythoncraft.com/

    Why is this newsgroup different from all other newsgroups?
    Aahz, Apr 13, 2004
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. seguso
    Replies:
    9
    Views:
    379
    seguso
    Dec 22, 2004
  2. Dave Kuhlman

    Re: Style in list comprehensions

    Dave Kuhlman, Aug 15, 2003, in forum: Python
    Replies:
    1
    Views:
    323
    Alex Martelli
    Aug 16, 2003
  3. Frank Millman

    Simple db using list comprehensions

    Frank Millman, Apr 5, 2004, in forum: Python
    Replies:
    1
    Views:
    278
    Paddy McCarthy
    Apr 16, 2004
  4. Mahesh Padmanabhan

    Generator expressions v/s list comprehensions

    Mahesh Padmanabhan, Aug 28, 2004, in forum: Python
    Replies:
    24
    Views:
    637
    Raymond Hettinger
    Sep 1, 2004
  5. Steven Bethard
    Replies:
    7
    Views:
    390
    Rocco Moretti
    Jan 20, 2006
Loading...

Share This Page