list comprehension question

Discussion in 'Python' started by Ross, May 1, 2009.

  1. Ross

    Ross Guest

    If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    return a new list of each individual element in these tuples, I can do
    it with a nested for loop but when I try to do it using the list
    comprehension b = [j for j in i for i in a], my output is b =
    [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    doing wrong?
    Ross, May 1, 2009
    #1
    1. Advertising

  2. Ross

    Chris Rebert Guest

    On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
    > If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    > return a new list of each individual element in these tuples, I can do
    > it with a nested for loop but when I try to do it using the list
    > comprehension b = [j for j in i for i in a], my output is b =
    > [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    > doing wrong?


    Your comprehension is the identity comprehension (i.e. it effectively
    just copies the list as-is).
    What you're trying to do is difficult if not impossible to do as a
    comprehension.

    Here's another approach:
    b = list(itertools.chain.from_iterable(a))

    And without using a library function:
    b = []
    for pair in a:
    for item in pair:
    b.append(item)

    Cheers,
    Chris
    --
    http://blog.rebertia.com
    Chris Rebert, May 1, 2009
    #2
    1. Advertising

  3. Ross wrote:
    > If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    > return a new list of each individual element in these tuples, I can do
    > it with a nested for loop but when I try to do it using the list
    > comprehension b = [j for j in i for i in a], my output is b =
    > [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    > doing wrong?
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >



    >>> a = [(1,2), (3,4), (5,6)]
    >>> [i for t in a for i in t]

    [1, 2, 3, 4, 5, 6]

    Or, with different spacing to make the nesting clearer:

    >>> [i

    ... for t in a
    ... for i in t]
    [1, 2, 3, 4, 5, 6]
    >>>


    Michael
    Michael Spencer, May 1, 2009
    #3
  4. Ross <> writes:

    > If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    > return a new list of each individual element in these tuples, I can do
    > it with a nested for loop but when I try to do it using the list
    > comprehension b = [j for j in i for i in a], my output is b =
    > [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    > doing wrong?


    When writing nested list comprehension, the for loops are in the same
    order as you would write a normal nested for loop (which is not
    necessarily intuitive when you first find out but is very handy in the
    long run I think).

    So write:

    [j for i in a for j in i]

    --
    Arnaud
    Arnaud Delobelle, May 1, 2009
    #4
  5. Ross

    thor Guest

    On May 1, 2:28 pm, Arnaud Delobelle <> wrote:
    > Ross <> writes:
    > > If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    > > return a new list of each individual element in these tuples, I can do
    > > it with a nested for loop but when I try to do it using the list
    > > comprehension b = [j for j in i for i in a], my output is b =
    > > [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    > > doing wrong?

    >
    > When writing nested list comprehension, the for loops are in the same
    > order as you would write a normal nested for loop (which is not
    > necessarily intuitive when you first find out but is very handy in the
    > long run I think).
    >
    > So write:
    >
    >     [j for i in a for j in i]
    >
    > --
    > Arnaud

    an trick
    >>> a

    [(1, 2), (3, 4), (5, 6)]
    >>> sum(a, ())

    (1, 2, 3, 4, 5, 6)
    >>>

    you may search the maillist , somebody questioned before
    thor, May 1, 2009
    #5
  6. Chris Rebert <> writes:

    > On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
    >> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    >> return a new list of each individual element in these tuples, I can do
    >> it with a nested for loop but when I try to do it using the list
    >> comprehension b = [j for j in i for i in a], my output is b =
    >> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    >> doing wrong?

    >
    > Your comprehension is the identity comprehension (i.e. it effectively
    > just copies the list as-is).
    > What you're trying to do is difficult if not impossible to do as a
    > comprehension.
    >
    > Here's another approach:
    > b = list(itertools.chain.from_iterable(a))
    >
    > And without using a library function:
    > b = []
    > for pair in a:
    > for item in pair:
    > b.append(item)


    This is much more clear than a nested comprehension.

    I love comprehensions, but abusing them can lead to really dense and
    difficult to read code.
    J Kenneth King, May 1, 2009
    #6
  7. On 5/1/2009 7:31 AM J Kenneth King said...
    > Chris Rebert <> writes:
    >> b = []
    >> for pair in a:
    >> for item in pair:
    >> b.append(item)

    >
    > This is much more clear than a nested comprehension.
    >
    > I love comprehensions, but abusing them can lead to really dense and
    > difficult to read code.


    I disagree on dense and difficult, although I'll leave open the question
    of abuse.

    b = [ item for pair in a for item in pair ]

    This is exactly the code above expressed in comprehension form.

    It's worth knowing that a list comprehension is structured identically
    to the equivalent for loop. So it really is neither more dense nor more
    difficult to read. Further, you can tell immediately from the start of
    the list comprehension what you've got -- in this case a list of item(s).

    Here with some slight changes...

    >>> a = [(1, 2), (3, 4, 7), (5, 6)]
    >>> [ item for j in a if len(j)==2 for item in j if item % 2 ]

    [1, 5]

    ....opposed to...

    >>> for j in a:

    .... if len(j)==2:
    .... for item in j:
    .... if item % 2:
    .... b.append(item)
    ....
    >>> b

    [1, 5]
    >>>


    YMMV,

    Emile
    Emile van Sebille, May 1, 2009
    #7
  8. Emile van Sebille <> writes:

    > On 5/1/2009 7:31 AM J Kenneth King said...
    >> Chris Rebert <> writes:
    >>> b = []
    >>> for pair in a:
    >>> for item in pair:
    >>> b.append(item)

    >>
    >> This is much more clear than a nested comprehension.
    >>
    >> I love comprehensions, but abusing them can lead to really dense and
    >> difficult to read code.

    >
    > I disagree on dense and difficult, although I'll leave open the
    > question of abuse.
    >
    > b = [ item for pair in a for item in pair ]
    >
    > This is exactly the code above expressed in comprehension form.


    If the comprehension above is an abuse, then every nested list
    comprehension is an abuse of comprehensions so they might as well not be
    in the language...

    --
    Arnaud
    Arnaud Delobelle, May 1, 2009
    #8
  9. Ross

    Shane Geiger Guest

    from goopy.functional import flatten #
    http://sourceforge.net/projects/goog-goopy/
    b = [(1,2), (3,4), (5,6)]
    print flatten(b)



    #from goopy.functional import flatten #
    http://sourceforge.net/projects/goog-goopy/

    def flatten(seq):
    """
    Returns a list of the contents of seq with sublists and tuples "exploded".
    The resulting list does not contain any sequences, and all inner sequences
    are exploded. For example:

    >>> flatten([7,(6,[5,4],3),2,1])

    [7,6,5,4,3,2,1]
    """
    lst = []
    for el in seq:
    if type(el) == list or type(el) is tuple:
    lst.extend(flatten(el))
    else:
    lst.append(el)
    return lst



    Chris Rebert wrote:
    > On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
    >
    >> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
    >> return a new list of each individual element in these tuples, I can do
    >> it with a nested for loop but when I try to do it using the list
    >> comprehension b = [j for j in i for i in a], my output is b =
    >> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
    >> doing wrong?
    >>

    >
    > Your comprehension is the identity comprehension (i.e. it effectively
    > just copies the list as-is).
    > What you're trying to do is difficult if not impossible to do as a
    > comprehension.
    >
    > Here's another approach:
    > b = list(itertools.chain.from_iterable(a))
    >
    > And without using a library function:
    > b = []
    > for pair in a:
    > for item in pair:
    > b.append(item)
    >
    > Cheers,
    > Chris
    >



    --
    Shane Geiger, IT Director
    Council For Economic Education / www.councilforeconed.org
    / 402-438-8958

    Teaching Opportunity
    Shane Geiger, May 1, 2009
    #9
  10. Ross

    John Posner Guest

    Re: Re: list comprehension question

    Shane Geiger wrote:
    > if type(el) == list or type(el) is tuple:

    A tiny improvement:

    if type(el) in (list, tuple):
    John Posner, May 1, 2009
    #10
  11. On 5/1/2009 9:19 AM Arnaud Delobelle said...
    > Emile van Sebille <> writes:
    >> On 5/1/2009 7:31 AM J Kenneth King said...
    >>> I love comprehensions, but abusing them can lead to really dense and
    >>> difficult to read code.

    >> I disagree on dense and difficult, although I'll leave open the
    >> question of abuse.
    >>
    >> b = [ item for pair in a for item in pair ]
    >>
    >> This is exactly the code above expressed in comprehension form.

    >
    > If the comprehension above is an abuse, then every nested list
    > comprehension is an abuse of comprehensions so they might as well not be
    > in the language...
    >


    I hope you didn't understand me to say that this usage is abuse. That's
    certainly not what I mean. I have, however, examples in my code base
    that _would_ qualify as abuse (mostly from pre dictionary comprehension
    vintage where I would build a dict from within a list comp), and I have
    seen enough from this group to know that abuse abounds, with only slight
    provocation. :)

    Fondly-remembering-byte-code-hacks-and-introspection-ly y'rs,

    Emile
    Emile van Sebille, May 1, 2009
    #11
  12. Emile van Sebille <> writes:

    > On 5/1/2009 9:19 AM Arnaud Delobelle said...
    >> Emile van Sebille <> writes:
    >>> On 5/1/2009 7:31 AM J Kenneth King said...
    >>>> I love comprehensions, but abusing them can lead to really dense and
    >>>> difficult to read code.
    >>> I disagree on dense and difficult, although I'll leave open the
    >>> question of abuse.
    >>>
    >>> b = [ item for pair in a for item in pair ]
    >>>
    >>> This is exactly the code above expressed in comprehension form.

    >>
    >> If the comprehension above is an abuse, then every nested list
    >> comprehension is an abuse of comprehensions so they might as well not be
    >> in the language...
    >>

    >
    > I hope you didn't understand me to say that this usage is abuse.


    Emile: sorry, the way I was quoting earlier posts was unclear. I was
    refering to the post you quoted in your message, which describe the
    nested list comprehension as abuse. As supporting evidence for my reply
    I included your example of nested list comprehension, which is one of
    the simplest possible ones one can think of.

    --
    Arnaud
    Arnaud Delobelle, May 1, 2009
    #12
  13. On 5/5/2009 9:15 AM J Kenneth King said...

    > List comprehensions can make a reader of your code apprehensive
    > because it can read like a run-on sentence and thus be difficult to
    > parse. The Python documentation discourages their use and I believe
    > for good reason.


    Can you provide a link for this? I'd like to see specifically what's
    being discouraged, as I'd be surprised to find routine usage frowned upon.

    Emile
    Emile van Sebille, May 5, 2009
    #13
  14. Re: Re: list comprehension question

    On Fri, 2009-05-01 at 13:00 -0400, John Posner wrote:
    > Shane Geiger wrote:
    > > if type(el) == list or type(el) is tuple:

    > A tiny improvement:
    >
    > if type(el) in (list, tuple):
    >


    Another alternative, which might be useful in some cases:

    if hasattr(el, '__iter__'):

    This covers all iterables, not just lists and tuples.

    So:

    >>> flatten([1,2, xrange(3,15), 15, 16])

    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]

    The downside, of course, is that some iterables might be infinite (such
    as count), which would cause hidden breakage. But if you have one of
    those in your data structure, why are you trying to flatten it anyway?


    Cheers,
    Cliff
    J. Cliff Dyer, May 5, 2009
    #14
  15. On Tue, 2009-05-05 at 12:15 -0400, J Kenneth King wrote:
    > Emile van Sebille <> writes:
    >
    > > On 5/1/2009 7:31 AM J Kenneth King said...
    > >> Chris Rebert <> writes:
    > >>> b = []
    > >>> for pair in a:
    > >>> for item in pair:
    > >>> b.append(item)
    > >>
    > >> This is much more clear than a nested comprehension.
    > >>
    > >> I love comprehensions, but abusing them can lead to really dense and
    > >> difficult to read code.

    > >
    > > I disagree on dense and difficult, although I'll leave open the
    > > question of abuse.

    >
    > Dense and difficult may be subjective to people like you or I, but you
    > left out the "to read" part of that sentence. I was referring to the
    > negative effect nested or complex list comprehensions can have on
    > readability.
    >
    > > b = [ item for pair in a for item in pair ]

    >
    > It's a clever statement, but use it once and its gone. Why not use the
    > expanded form in a function definition and get an even shorter, but
    > clearer statement?
    >


    It's also not obvious what it means. I would have thought the proper
    incantation would be:

    [item for item in pair for pair in a]

    but that returns [5,5,5,6,6,6]. I still haven't figured out why. The
    way you have to bounce your eyes back and forth in the comprehension
    makes it hard to read the logic. With the loop, on the other hand it is
    blatantly obvious which way the nesting occurs.



    >
    > >>> b = flatten(a)

    >
    > Boom, done.
    >
    > > This is exactly the code above expressed in comprehension form.
    > >
    > > It's worth knowing that a list comprehension is structured identically
    > > to the equivalent for loop. So it really is neither more dense nor
    > > more difficult to read. Further, you can tell immediately from the
    > > start of the list comprehension what you've got -- in this case a list
    > > of item(s).


    Empirically, I'd have to disagree with you. The structure is changed by
    the fact that "item" gets put at the beginning of the comprehension.
    With the loop, each variable gets introduced in a for statement, and
    then used. With the list comprehension, "item" gets used at the
    beginning, and then assigned to in the for expression at the end.
    Meanwhile, pair gets introduced as the loop variable in the first for
    expression, and iterated over in the second for expression. So do you
    read it left to right or right to left? Neither. You have to hold the
    whole expression in your mind at once. Good for short and simple
    comprehensions, bad as they get more complicated.


    > >
    > > Here with some slight changes...
    > >
    > >>>> a = [(1, 2), (3, 4, 7), (5, 6)]
    > >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]

    > > [1, 5]
    > >


    This, to me, looks like line noise, approaching perl in its
    unreadability.


    > > ...opposed to...
    > >
    > >>>> for j in a:

    > > ... if len(j)==2:
    > > ... for item in j:
    > > ... if item % 2:
    > > ... b.append(item)
    > > ...
    > >>>> b

    > > [1, 5]
    > >>>>

    >


    Much nicer. Thank you.

    > Thanks for the lesson in list comprehensions, but I'm well aware of
    > how they work.
    >
    > List comprehensions can make a reader of your code apprehensive
    > because it can read like a run-on sentence and thus be difficult to
    > parse. The Python documentation discourages their use and I believe
    > for good reason. It's much easier to explain complex processes with a
    > function rather than a single nested statement.
    >
    > >
    > > YMMV,
    > >
    > > Emile

    >
    > It will apparently vary greatly. Depending on how much coffee I've
    > had.
    >
    > ;)
    >
    > J
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >
    J. Cliff Dyer, May 5, 2009
    #15
  16. On Tue, 05 May 2009 12:15:15 -0400, J Kenneth King wrote:

    > List comprehensions can make a reader of your code apprehensive because
    > it can read like a run-on sentence and thus be difficult to parse. The
    > Python documentation discourages their use


    Do you have a URL for such an astonishing claim?

    > and I believe for good
    > reason. It's much easier to explain complex processes with a function
    > rather than a single nested statement.


    Sure, if you treat the function as a black box: "Input goes in, output
    comes out, and I trust it does the right thing". But if you can do that
    for a function, with no care about its internal complexity, then you can
    do the same thing for a list comp.

    What's the difference between these?

    results = [complicated nested list comp]

    versus

    def func():
    return [complicated nested list comp]

    results = func()


    The function call isn't *actually* easier to understand, it just allows
    you to treat it as a black box and pretend you understand it. Once you
    look inside the function, there's no promise that it will be simpler than
    a nested list comp. Maybe it will be, and maybe it won't be.

    Besides, not all list comprehensions are equivalent to nested loops. Why
    label list comps as difficult to parse when only *some* of them are
    difficult? That's like insisting that 1+3*x must be hard to read and
    scary-looking to the average programmer, just because arithmetic
    operators allow one to also write

    1/(1+1/(2*x+3/(7*x+x/(5*x**2-3*x+4/(9**x+8/(-3+6/(2*x+1)))))))




    --
    Steven
    Steven D'Aprano, May 6, 2009
    #16
  17. On Tue, 05 May 2009 13:43:32 -0400, J. Cliff Dyer wrote:

    > On Fri, 2009-05-01 at 13:00 -0400, John Posner wrote:
    >> Shane Geiger wrote:
    >> > if type(el) == list or type(el) is tuple:

    >> A tiny improvement:
    >>
    >> if type(el) in (list, tuple):
    >>
    >>

    > Another alternative, which might be useful in some cases:
    >
    > if hasattr(el, '__iter__'):
    >
    > This covers all iterables, not just lists and tuples.


    Except for the ones that it doesn't cover, like strings:

    >>> hasattr('abcd', '__iter__')

    False
    >>> list(iter('abcd'))

    ['a', 'b', 'c', 'd']


    And classes that follow the sequence protocol:


    >>> class Spam:

    .... def __getitem__(self, index):
    .... if 0 <= index < 5:
    .... return "spam"
    .... raise IndexError
    ....
    >>> hasattr(Spam(), '__iter__')

    False
    >>> list(iter(Spam()))

    ['spam', 'spam', 'spam', 'spam', 'spam']




    --
    Steven
    Steven D'Aprano, May 6, 2009
    #17
  18. Ross

    alex23 Guest

    On May 6, 4:01 am, "J. Cliff Dyer" <> wrote:
    > The way you have to bounce your eyes back and forth in the comprehension
    > makes it hard to read the logic.  With the loop, on the other hand it is
    > blatantly obvious which way the nesting occurs.


    > > >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]
    > > > ...opposed to...
    > > >>>> for j in a:
    > > > ...     if len(j)==2:
    > > > ...         for item in j:
    > > > ...             if item % 2:
    > > > ...                 b.append(item)
    > > > ...


    > Much nicer.  Thank you.


    Apart from the presence of 'item' at the beginning of the list
    comprehension as opposed to 'b.append(item)' at the end of the for-
    loop, how exactly does the listcomp force you to "bounce [..] back and
    forth" to follow the logic? The only other difference between the two
    is in the layout - the indentation & colons - otherwise they're
    structurally identical.
    alex23, May 6, 2009
    #18
  19. On Tue, 05 May 2009 20:36:37 -0700, alex23 wrote:

    > On May 6, 4:01 am, "J. Cliff Dyer" <> wrote:
    >> The way you have to bounce your eyes back and forth in the
    >> comprehension makes it hard to read the logic.  With the loop, on the
    >> other hand it is blatantly obvious which way the nesting occurs.

    >
    >> > >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]
    >> > > ...opposed to...
    >> > >>>> for j in a:
    >> > > ...     if len(j)==2:
    >> > > ...         for item in j:
    >> > > ...             if item % 2:
    >> > > ...                 b.append(item) ...

    >
    >> Much nicer.  Thank you.

    >
    > Apart from the presence of 'item' at the beginning of the list
    > comprehension as opposed to 'b.append(item)' at the end of the for-
    > loop, how exactly does the listcomp force you to "bounce [..] back and
    > forth" to follow the logic? The only other difference between the two is
    > in the layout - the indentation & colons - otherwise they're
    > structurally identical.



    It's precisely the indentation and colons (plus newlines) that makes
    nested for-loops easier to read than list-comps with multiple fors.

    for a in alist:
    for b in blist:
    for c in clist:
    if c:
    parrot(a, b, c)


    is much easier to read than the equivalent one-liner:

    [parrot(a, b, c) for a in alist for b in blist for c in clist if c]

    You can get back *nearly* all the readability by splitting the list comp
    into multiple lines:

    [parrot(a, b, c)
    for a in alist
    for b in blist
    for c in clist if c
    ]



    --
    Steven
    Steven D'Aprano, May 6, 2009
    #19
  20. Ross

    Terry Reedy Guest

    Steven D'Aprano wrote:
    > On Tue, 05 May 2009 13:43:32 -0400, J. Cliff Dyer wrote:


    > Except for the ones that it doesn't cover, like strings:
    >
    >>>> hasattr('abcd', '__iter__')

    > False


    True in Python3
    Terry Reedy, May 6, 2009
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark Elston

    List Comprehension question

    Mark Elston, Dec 11, 2003, in forum: Python
    Replies:
    1
    Views:
    312
    Francis Avila
    Dec 11, 2003
  2. Eelco Hoekema

    [Newby question] List comprehension

    Eelco Hoekema, Aug 6, 2004, in forum: Python
    Replies:
    7
    Views:
    272
    Eelco Hoekema
    Aug 6, 2004
  3. Shane Geiger
    Replies:
    4
    Views:
    374
    bullockbefriending bard
    Mar 25, 2007
  4. Debajit Adhikary
    Replies:
    17
    Views:
    672
    Debajit Adhikary
    Oct 18, 2007
  5. Vedran Furac(
    Replies:
    4
    Views:
    317
    Marc 'BlackJack' Rintsch
    Dec 19, 2008
Loading...

Share This Page