do you master list comprehensions?

Discussion in 'Python' started by Will Stuyvesant, Dec 13, 2004.

  1. Here is a question about list comprehensions [lc]. The
    question is dumb because I can do without [lc]; but I am
    posing the question because I am curious.

    This:

    >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> result = []
    >>> for d in data:

    .... for w in d:
    .... result.append(w)
    >>> print result

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

    puts all the words in a list, like I want.

    How to do this with [lc] instead of for-loops?

    I tried funnies like [[w for w in L] for L in data],
    that is correct syntax, but you'd never guess.

    I know, silly! No need for [lc]! So there's my
    question. I am sure a one-liner using [lc] will be very
    enlightening. Like studying LISP.


    --
    I wish there was a knob on the TV to turn up the intelligence.
    There's a knob called `brightness', but it doesn't work.
    -- Gallagher
    Will Stuyvesant, Dec 13, 2004
    #1
    1. Advertising

  2. Will Stuyvesant

    Max M Guest

    Will Stuyvesant wrote:

    > I tried funnies like [[w for w in L] for L in data],
    > that is correct syntax, but you'd never guess.


    That is absolutely correct. It's not a funnie at all. If you find it odd
    it's only because you are not used to list comprehensiones.

    In that case you might be more comfortable with:

    data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    result = []
    for l in data:
    result += l


    --

    hilsen/regards Max M, Denmark

    http://www.mxm.dk/
    IT's Mad Science
    Max M, Dec 13, 2004
    #2
    1. Advertising

  3. Will Stuyvesant

    Peter Otten Guest

    Will Stuyvesant wrote:

    >>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>>> result = []
    >>>> for d in data:

    > ... for w in d:
    > ... result.append(w)
    >>>> print result

    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >
    > puts all the words in a list, like I want.
    >
    > How to do this with [lc] instead of for-loops?


    >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> [w for d in data for w in d]

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

    See how the for expressions in the list comprehension exactly match your
    nested for loops? That's all there is to it.

    Peter
    Peter Otten, Dec 13, 2004
    #3
  4. Will Stuyvesant wrote:
    >>>>data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>>>result = []
    >>>>for d in data:

    >
    > ... for w in d:
    > ... result.append(w)
    >
    >>>>print result

    >
    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >


    Take advantage of the fact that you can have more than one 'for' in a
    list comprehension:

    >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> [item for item_list in data for item in item_list]

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

    Steve
    Steven Bethard, Dec 13, 2004
    #4
  5. >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> [e for l in data for e in l]

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    --
    Regards,

    Diez B. Roggisch
    Diez B. Roggisch, Dec 13, 2004
    #5
  6. Max M wrote:

    >> I tried funnies like [[w for w in L] for L in data],
    >> that is correct syntax, but you'd never guess.

    >
    > That is absolutely correct. It's not a funnie at all. If you find it odd it's only because you are
    > not used to list comprehensiones.


    well, syntactically correct or not, it doesn't do what he want...

    > In that case you might be more comfortable with:
    >
    > data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    > result = []
    > for l in data:
    > result += l


    how about (slightly evil):

    result = []; map(result.extend, data)

    </F>
    Fredrik Lundh, Dec 13, 2004
    #6
  7. Will Stuyvesant

    Max M Guest

    Fredrik Lundh wrote:
    > Max M wrote:
    >
    >
    >>>I tried funnies like [[w for w in L] for L in data],

    >>
    >>That is absolutely correct. It's not a funnie at all.

    >
    > well, syntactically correct or not, it doesn't do what he want...


    Doh! *I* might not be used to list comprehensions then... You are right.

    That example could have been expressed more clearly as:

    result = data

    ;-)

    --

    hilsen/regards Max M, Denmark

    http://www.mxm.dk/
    IT's Mad Science
    Max M, Dec 13, 2004
    #7
  8. I guess the simplest to do it is like this:

    >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> result=[w for d in data for w in d]
    >>> result

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >>>
    Luis M. Gonzalez, Dec 14, 2004
    #8
  9. I guess the simplest way to do it is like this:

    >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>> result=[w for d in data for w in d]
    >>> result

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >>>
    Luis M. Gonzalez, Dec 14, 2004
    #9
  10. Will Stuyvesant

    James Stroud Guest

    Here is one for arbitrary depth:

    def unroll(ary):
    unrolled = []
    for item in ary:
    # add test for your favorite sequence type
    if ( type(item) == types.ListType or \
    type(item) == types.TupleType \
    ):
    unrolled.extend(unroll(item))
    else:
    unrolled.append(item)
    return unrolled



    >>> unroll([[1, 2, 3], ('fred', 'barney', ['wilma', 'betty']), 'dino'])

    [1, 2, 3, 'fred', 'barney', 'wilma', 'betty', 'dino']




    On Monday 13 December 2004 12:51 pm, Will Stuyvesant wrote:
    > Here is a question about list comprehensions [lc]. The
    > question is dumb because I can do without [lc]; but I am
    > posing the question because I am curious.
    >
    > This:
    > >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    > >>> result = []
    > >>> for d in data:

    >
    > ... for w in d:
    > ... result.append(w)
    >
    > >>> print result

    >
    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >
    > puts all the words in a list, like I want.
    >
    > How to do this with [lc] instead of for-loops?
    >
    > I tried funnies like [[w for w in L] for L in data],
    > that is correct syntax, but you'd never guess.
    >
    > I know, silly! No need for [lc]! So there's my
    > question. I am sure a one-liner using [lc] will be very
    > enlightening. Like studying LISP.


    --
    James Stroud, Ph.D.
    UCLA-DOE Institute for Genomics and Proteomics
    611 Charles E. Young Dr. S.
    MBI 205, UCLA 951570
    Los Angeles CA 90095-1570
    http://www.jamesstroud.com/
    James Stroud, Dec 14, 2004
    #10
  11. On Tue, 14 Dec 2004 00:41:36 +0100, Max M wrote:

    > Fredrik Lundh wrote:
    >> Max M wrote:
    >>
    >>
    >>>>I tried funnies like [[w for w in L] for L in data],
    >>>
    >>>That is absolutely correct. It's not a funnie at all.

    >>
    >> well, syntactically correct or not, it doesn't do what he want...

    >
    > Doh! *I* might not be used to list comprehensions then... You are right.
    >
    > That example could have been expressed more clearly as:
    >
    > result = data


    result = data[:]

    :)
    Jeremy Bowers, Dec 14, 2004
    #11
  12. James Stroud wrote:

    > Here is one for arbitrary depth:
    >
    > def unroll(ary):
    > unrolled = []
    > for item in ary:
    > # add test for your favorite sequence type
    > if ( type(item) == types.ListType or \
    > type(item) == types.TupleType \
    > ):
    > unrolled.extend(unroll(item))
    > else:
    > unrolled.append(item)
    > return unrolled
    >


    >>>> unroll([[1, 2, 3], ('fred', 'barney', ['wilma', 'betty']), 'dino'])

    > [1, 2, 3, 'fred', 'barney', 'wilma', 'betty', 'dino']


    or, shorter:

    >>> from Tkinter import _flatten as unroll
    >>> (1, 2, 3, 'fred', 'wilma', 'betty', 'dino')


    (alright, it returns a tuple, but that can be easily fixed, if necessary)

    </F>
    Fredrik Lundh, Dec 14, 2004
    #12
  13. Will Stuyvesant wrote:

    >>>>data = [['foo','bar','baz'],['my','your'],['holy','grail']]


    sum(data, [])

    ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

    The second parameter passed to sum is just to overrride default
    initial value "zero".

    --
    Timothy Babytch
    Timothy Babytch, Dec 14, 2004
    #13
  14. Okay that was fun. Enlightening as I hoped. unroll() in Python, for
    arbitrary depth, _flatten in Tkinter (what else is in Tkinter!), sum()
    abuse.

    The sum(data,[]) was funniest, it works like ((['foo','bar'] + []) +
    ['my','your']) + ['holy','grail']. Before I think of such things I
    have already coded an algorithm in imperative style. Guess I have not
    been exposed to functional programming enough.
    Will Stuyvesant, Dec 14, 2004
    #14
  15. Timothy Babytch wrote:
    > Will Stuyvesant wrote:
    >
    >>>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]

    >
    >
    > sum(data, [])
    >
    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >
    > The second parameter passed to sum is just to overrride default
    > initial value "zero".
    >


    It's worth keeping in mind that this solution has the same efficiency
    problems that a loop that =+ strings does:

    > python -m timeit -s "data = [range(10) for _ in range(100)]"

    "sum(data, [])"
    1000 loops, best of 3: 530 usec per loop

    > python -m timeit -s "data = [range(10) for _ in range(100)]" "[w for

    d in data for w in d]"
    10000 loops, best of 3: 151 usec per loop

    > python -m timeit -s "data = [range(10) for _ in range(1000)]"

    "sum(data, [])"
    10 loops, best of 3: 54.2 msec per loop

    > python -m timeit -s "data = [range(10) for _ in range(1000)]" "[w for

    d in data for w in d]"
    100 loops, best of 3: 1.75 msec per loop

    The sum function used in this way (or a loop with a +=) is O(N**2) while
    the LC is O(N).

    Steve
    Steven Bethard, Dec 14, 2004
    #15
  16. Steven Bethard wrote:

    > > python -m timeit -s "data = [range(10) for _ in range(1000)]"

    > "sum(data, [])"
    > 10 loops, best of 3: 54.2 msec per loop
    >
    > > python -m timeit -s "data = [range(10) for _ in range(1000)]" "[w for

    > d in data for w in d]"
    > 100 loops, best of 3: 1.75 msec per loop
    >
    > The sum function used in this way (or a loop with a +=) is O(N**2) while the LC is O(N).


    also:

    timeit -s "data = [range(10) for _ in range(1000)]" "L = sum(data, [])"
    10 loops, best of 3: 4.02e+004 usec per loop

    timeit -s "data = [range(10) for _ in range(1000)]" "L = [w for d in data for w in d]"
    1000 loops, best of 3: 1.12e+003 usec per loop

    timeit -s "data = [range(10) for _ in range(1000)]" "L = []; map(L.extend, data)"
    1000 loops, best of 3: 283 usec per loop

    timeit -s "data = [range(10) for _ in range(1000)]; from Tkinter import _flatten" "L =
    _flatten(data)"
    1000 loops, best of 3: 308 usec per loop

    (the last one supports arbitrary nestings, the others don't)

    </F>
    Fredrik Lundh, Dec 14, 2004
    #16
  17. Will Stuyvesant

    Nick Coghlan Guest

    Will Stuyvesant wrote:
    > Here is a question about list comprehensions [lc]. The
    > question is dumb because I can do without [lc]; but I am
    > posing the question because I am curious.
    >
    > This:
    >
    >
    >>>>data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>>>result = []
    >>>>for d in data:

    >
    > ... for w in d:
    > ... result.append(w)
    >
    >>>>print result

    >
    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']
    >
    > puts all the words in a list, like I want.


    There's a way to avoid generating the intermediate list if you don't actually
    need it (e.g. you want to feed the sequence to another method):

    ..>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    ..>>> from itertools import chain
    ..>>> result = "".join(chain(*data))
    'foobarbazmyyourholygrail'


    Some timing with integers:

    C:\>python -m timeit -s "data = [range(x) for x in range(1000)]" "L= []; map(L.e
    xtend, data); max(L)"
    10 loops, best of 3: 78.5 msec per loop

    C:\>python -m timeit -s "data = [range(x) for x in range(1000)]; from Tkinter im
    port _flatten" "max(_flatten(data))"
    10 loops, best of 3: 58.4 msec per loop

    C:\>python -m timeit -s "data = [range(x) for x in range(1000)]; from itertools
    import chain" "max(chain(*data))"
    10 loops, best of 3: 43 msec per loop

    And with strings:

    C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]" "L= [
    ]; map(L.extend, data); ''.join(L)"
    10 loops, best of 3: 106 msec per loop

    C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
    Tkinter import _flatten" "''.join(_flatten(data))"
    10 loops, best of 3: 85.4 msec per loop

    C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
    itertools import chain" "''.join(chain(*data))"
    10 loops, best of 3: 1.2 sec per loop ****** OUCH!!

    C:\>python -m timeit -s "data = [map(str, range(x)) for x in range(1000)]; from
    itertools import chain" "''.join(list(chain(*data)))"
    10 loops, best of 3: 107 msec per loop


    Yikes - looks like chain() really sucks for str.join. However, the addition of
    the 'list' call makes a big difference. Maybe PySequence_Fast should be called
    PySequence_SlowAsADeadDingo() in this case ;)

    (FYI, I filed bug report #1085744 on SF about this)

    Cheers,
    Nick.

    --
    Nick Coghlan | | Brisbane, Australia
    ---------------------------------------------------------------
    http://boredomandlaziness.skystorm.net
    Nick Coghlan, Dec 15, 2004
    #17
  18. Will Stuyvesant

    Matthew Moss Guest

    Diez B. Roggisch wrote:
    > >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    > >>> [e for l in data for e in l]

    > ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']



    Okay, I tried this in an interactive Python session and it works as
    stated. My question is, why? How is the interpreter parsing that list
    expression that it makes sense?
    Matthew Moss, Dec 15, 2004
    #18
  19. Matthew Moss wrote:

    >> >>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >> >>> [e for l in data for e in l]

    >> ['foo', 'bar', 'baz', 'my', 'your', 'holy', 'grail']

    >
    > Okay, I tried this in an interactive Python session and it works as
    > stated. My question is, why? How is the interpreter parsing that list
    > expression that it makes sense?


    the language reference has the answer:

    http://www.python.org/doc/2.3.4/ref/lists.html

    "... the elements of the new list are those that would be produced
    by considering each of the 'for' or 'if' clauses a block, nesting
    from left to right, and evaluating the expression to produce a list
    element each time the innermost block is reached."

    or, conceptually, the compiler strips off the expression, and then
    inserts newlines, colons and indentation, and wraps the expression
    in an append statement.

    so

    [e for l in data for e in l]

    behaves like:

    result = []
    for l in data:
    for e in l:
    result.append(e)

    except that it's an expression, and the result variable is hidden. (and
    the compiler and the runtime is of course free to implement this in a
    more efficient way)

    </F>
    Fredrik Lundh, Dec 15, 2004
    #19
  20. ".>>>" is a good idea! (OT, was: Re: do you master list comprehensions?)

    Nick Coghlan schrieb:
    >>>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    >>>>> result = []
    >>>>> for d in data:

    >
    > .>>> data = [['foo','bar','baz'],['my','your'],['holy','grail']]
    > .>>> from itertools import chain
    > .>>> result = "".join(chain(*data))
    > 'foobarbazmyyourholygrail'


    This is the first time I see that and I totally like the idea of writing
    ".>>>" instead of ">>>" at the beginning of a line. Thank you Dr. Dobb!
    It's unfortunate for c.l.py that Python uses ">>>" as the default prompt
    as it messes up the display on mail/news readers that provide "syntax
    highlighting" for quotes.

    I wish everyone would write examples that way! On the other hand - three
    levels of quoting are really rare, maybe it would be easier to change that
    in the mail readers...

    .... or in Py3k ?

    Stefan
    Stefan Behnel, Dec 16, 2004
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Doug McCann
    Replies:
    1
    Views:
    987
    William Brogden
    Aug 5, 2004
  2. seguso
    Replies:
    9
    Views:
    376
    seguso
    Dec 22, 2004
  3. Dave Kuhlman

    Re: Style in list comprehensions

    Dave Kuhlman, Aug 15, 2003, in forum: Python
    Replies:
    1
    Views:
    322
    Alex Martelli
    Aug 16, 2003
  4. Frank Millman

    Simple db using list comprehensions

    Frank Millman, Apr 5, 2004, in forum: Python
    Replies:
    1
    Views:
    276
    Paddy McCarthy
    Apr 16, 2004
  5. Steven Bethard
    Replies:
    7
    Views:
    387
    Rocco Moretti
    Jan 20, 2006
Loading...

Share This Page