# list comprehension question

Discussion in 'Python' started by Ross, May 1, 2009.

1. ### RossGuest

If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
return a new list of each individual element in these tuples, I can do
it with a nested for loop but when I try to do it using the list
comprehension b = [j for j in i for i in a], my output is b =
[5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
doing wrong?

Ross, May 1, 2009

2. ### Chris RebertGuest

On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
> return a new list of each individual element in these tuples, I can do
> it with a nested for loop but when I try to do it using the list
> comprehension b = [j for j in i for i in a], my output is b =
> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
> doing wrong?

Your comprehension is the identity comprehension (i.e. it effectively
just copies the list as-is).
What you're trying to do is difficult if not impossible to do as a
comprehension.

Here's another approach:
b = list(itertools.chain.from_iterable(a))

And without using a library function:
b = []
for pair in a:
for item in pair:
b.append(item)

Cheers,
Chris
--
http://blog.rebertia.com

Chris Rebert, May 1, 2009

3. ### Michael SpencerGuest

Ross wrote:
> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
> return a new list of each individual element in these tuples, I can do
> it with a nested for loop but when I try to do it using the list
> comprehension b = [j for j in i for i in a], my output is b =
> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
> doing wrong?
> --
> http://mail.python.org/mailman/listinfo/python-list
>

>>> a = [(1,2), (3,4), (5,6)]
>>> [i for t in a for i in t]

[1, 2, 3, 4, 5, 6]

Or, with different spacing to make the nesting clearer:

>>> [i

... for t in a
... for i in t]
[1, 2, 3, 4, 5, 6]
>>>

Michael

Michael Spencer, May 1, 2009
4. ### Arnaud DelobelleGuest

Ross <> writes:

> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
> return a new list of each individual element in these tuples, I can do
> it with a nested for loop but when I try to do it using the list
> comprehension b = [j for j in i for i in a], my output is b =
> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
> doing wrong?

When writing nested list comprehension, the for loops are in the same
order as you would write a normal nested for loop (which is not
necessarily intuitive when you first find out but is very handy in the
long run I think).

So write:

[j for i in a for j in i]

--
Arnaud

Arnaud Delobelle, May 1, 2009
5. ### thorGuest

On May 1, 2:28 pm, Arnaud Delobelle <> wrote:
> Ross <> writes:
> > If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
> > return a new list of each individual element in these tuples, I can do
> > it with a nested for loop but when I try to do it using the list
> > comprehension b = [j for j in i for i in a], my output is b =
> > [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
> > doing wrong?

>
> When writing nested list comprehension, the for loops are in the same
> order as you would write a normal nested for loop (which is not
> necessarily intuitive when you first find out but is very handy in the
> long run I think).
>
> So write:
>
>     [j for i in a for j in i]
>
> --
> Arnaud

an trick
>>> a

[(1, 2), (3, 4), (5, 6)]
>>> sum(a, ())

(1, 2, 3, 4, 5, 6)
>>>

you may search the maillist , somebody questioned before

thor, May 1, 2009
6. ### J Kenneth KingGuest

Chris Rebert <> writes:

> On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
>> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
>> return a new list of each individual element in these tuples, I can do
>> it with a nested for loop but when I try to do it using the list
>> comprehension b = [j for j in i for i in a], my output is b =
>> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
>> doing wrong?

>
> Your comprehension is the identity comprehension (i.e. it effectively
> just copies the list as-is).
> What you're trying to do is difficult if not impossible to do as a
> comprehension.
>
> Here's another approach:
> b = list(itertools.chain.from_iterable(a))
>
> And without using a library function:
> b = []
> for pair in a:
> for item in pair:
> b.append(item)

This is much more clear than a nested comprehension.

I love comprehensions, but abusing them can lead to really dense and

J Kenneth King, May 1, 2009
7. ### Emile van SebilleGuest

On 5/1/2009 7:31 AM J Kenneth King said...
> Chris Rebert <> writes:
>> b = []
>> for pair in a:
>> for item in pair:
>> b.append(item)

>
> This is much more clear than a nested comprehension.
>
> I love comprehensions, but abusing them can lead to really dense and

I disagree on dense and difficult, although I'll leave open the question
of abuse.

b = [ item for pair in a for item in pair ]

This is exactly the code above expressed in comprehension form.

It's worth knowing that a list comprehension is structured identically
to the equivalent for loop. So it really is neither more dense nor more
difficult to read. Further, you can tell immediately from the start of
the list comprehension what you've got -- in this case a list of item(s).

Here with some slight changes...

>>> a = [(1, 2), (3, 4, 7), (5, 6)]
>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]

[1, 5]

....opposed to...

>>> for j in a:

.... if len(j)==2:
.... for item in j:
.... if item % 2:
.... b.append(item)
....
>>> b

[1, 5]
>>>

YMMV,

Emile

Emile van Sebille, May 1, 2009
8. ### Arnaud DelobelleGuest

Emile van Sebille <> writes:

> On 5/1/2009 7:31 AM J Kenneth King said...
>> Chris Rebert <> writes:
>>> b = []
>>> for pair in a:
>>> for item in pair:
>>> b.append(item)

>>
>> This is much more clear than a nested comprehension.
>>
>> I love comprehensions, but abusing them can lead to really dense and

>
> I disagree on dense and difficult, although I'll leave open the
> question of abuse.
>
> b = [ item for pair in a for item in pair ]
>
> This is exactly the code above expressed in comprehension form.

If the comprehension above is an abuse, then every nested list
comprehension is an abuse of comprehensions so they might as well not be
in the language...

--
Arnaud

Arnaud Delobelle, May 1, 2009
9. ### Shane GeigerGuest

from goopy.functional import flatten #
http://sourceforge.net/projects/goog-goopy/
b = [(1,2), (3,4), (5,6)]
print flatten(b)

#from goopy.functional import flatten #
http://sourceforge.net/projects/goog-goopy/

def flatten(seq):
"""
Returns a list of the contents of seq with sublists and tuples "exploded".
The resulting list does not contain any sequences, and all inner sequences
are exploded. For example:

>>> flatten([7,(6,[5,4],3),2,1])

[7,6,5,4,3,2,1]
"""
lst = []
for el in seq:
if type(el) == list or type(el) is tuple:
lst.extend(flatten(el))
else:
lst.append(el)
return lst

Chris Rebert wrote:
> On Thu, Apr 30, 2009 at 5:56 PM, Ross <> wrote:
>
>> If I have a list of tuples a = [(1,2), (3,4), (5,6)], and I want to
>> return a new list of each individual element in these tuples, I can do
>> it with a nested for loop but when I try to do it using the list
>> comprehension b = [j for j in i for i in a], my output is b =
>> [5,5,5,6,6,6] instead of the correct b = [1,2,3,4,5,6]. What am I
>> doing wrong?
>>

>
> Your comprehension is the identity comprehension (i.e. it effectively
> just copies the list as-is).
> What you're trying to do is difficult if not impossible to do as a
> comprehension.
>
> Here's another approach:
> b = list(itertools.chain.from_iterable(a))
>
> And without using a library function:
> b = []
> for pair in a:
> for item in pair:
> b.append(item)
>
> Cheers,
> Chris
>

--
Shane Geiger, IT Director
Council For Economic Education / www.councilforeconed.org
/ 402-438-8958

Teaching Opportunity

Shane Geiger, May 1, 2009
10. ### John PosnerGuest

Re: Re: list comprehension question

Shane Geiger wrote:
> if type(el) == list or type(el) is tuple:

A tiny improvement:

if type(el) in (list, tuple):

John Posner, May 1, 2009
11. ### Emile van SebilleGuest

On 5/1/2009 9:19 AM Arnaud Delobelle said...
> Emile van Sebille <> writes:
>> On 5/1/2009 7:31 AM J Kenneth King said...
>>> I love comprehensions, but abusing them can lead to really dense and

>> I disagree on dense and difficult, although I'll leave open the
>> question of abuse.
>>
>> b = [ item for pair in a for item in pair ]
>>
>> This is exactly the code above expressed in comprehension form.

>
> If the comprehension above is an abuse, then every nested list
> comprehension is an abuse of comprehensions so they might as well not be
> in the language...
>

I hope you didn't understand me to say that this usage is abuse. That's
certainly not what I mean. I have, however, examples in my code base
that _would_ qualify as abuse (mostly from pre dictionary comprehension
vintage where I would build a dict from within a list comp), and I have
seen enough from this group to know that abuse abounds, with only slight
provocation.

Fondly-remembering-byte-code-hacks-and-introspection-ly y'rs,

Emile

Emile van Sebille, May 1, 2009
12. ### Arnaud DelobelleGuest

Emile van Sebille <> writes:

> On 5/1/2009 9:19 AM Arnaud Delobelle said...
>> Emile van Sebille <> writes:
>>> On 5/1/2009 7:31 AM J Kenneth King said...
>>>> I love comprehensions, but abusing them can lead to really dense and
>>> I disagree on dense and difficult, although I'll leave open the
>>> question of abuse.
>>>
>>> b = [ item for pair in a for item in pair ]
>>>
>>> This is exactly the code above expressed in comprehension form.

>>
>> If the comprehension above is an abuse, then every nested list
>> comprehension is an abuse of comprehensions so they might as well not be
>> in the language...
>>

>
> I hope you didn't understand me to say that this usage is abuse.

Emile: sorry, the way I was quoting earlier posts was unclear. I was
refering to the post you quoted in your message, which describe the
nested list comprehension as abuse. As supporting evidence for my reply
I included your example of nested list comprehension, which is one of
the simplest possible ones one can think of.

--
Arnaud

Arnaud Delobelle, May 1, 2009
13. ### Emile van SebilleGuest

On 5/5/2009 9:15 AM J Kenneth King said...

> because it can read like a run-on sentence and thus be difficult to
> parse. The Python documentation discourages their use and I believe
> for good reason.

Can you provide a link for this? I'd like to see specifically what's
being discouraged, as I'd be surprised to find routine usage frowned upon.

Emile

Emile van Sebille, May 5, 2009
14. ### J. Cliff DyerGuest

Re: Re: list comprehension question

On Fri, 2009-05-01 at 13:00 -0400, John Posner wrote:
> Shane Geiger wrote:
> > if type(el) == list or type(el) is tuple:

> A tiny improvement:
>
> if type(el) in (list, tuple):
>

Another alternative, which might be useful in some cases:

if hasattr(el, '__iter__'):

This covers all iterables, not just lists and tuples.

So:

>>> flatten([1,2, xrange(3,15), 15, 16])

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]

The downside, of course, is that some iterables might be infinite (such
as count), which would cause hidden breakage. But if you have one of
those in your data structure, why are you trying to flatten it anyway?

Cheers,
Cliff

J. Cliff Dyer, May 5, 2009
15. ### J. Cliff DyerGuest

On Tue, 2009-05-05 at 12:15 -0400, J Kenneth King wrote:
> Emile van Sebille <> writes:
>
> > On 5/1/2009 7:31 AM J Kenneth King said...
> >> Chris Rebert <> writes:
> >>> b = []
> >>> for pair in a:
> >>> for item in pair:
> >>> b.append(item)
> >>
> >> This is much more clear than a nested comprehension.
> >>
> >> I love comprehensions, but abusing them can lead to really dense and
> >> difficult to read code.

> >
> > I disagree on dense and difficult, although I'll leave open the
> > question of abuse.

>
> Dense and difficult may be subjective to people like you or I, but you
> left out the "to read" part of that sentence. I was referring to the
> negative effect nested or complex list comprehensions can have on
>
> > b = [ item for pair in a for item in pair ]

>
> It's a clever statement, but use it once and its gone. Why not use the
> expanded form in a function definition and get an even shorter, but
> clearer statement?
>

It's also not obvious what it means. I would have thought the proper
incantation would be:

[item for item in pair for pair in a]

but that returns [5,5,5,6,6,6]. I still haven't figured out why. The
way you have to bounce your eyes back and forth in the comprehension
makes it hard to read the logic. With the loop, on the other hand it is
blatantly obvious which way the nesting occurs.

>
> >>> b = flatten(a)

>
> Boom, done.
>
> > This is exactly the code above expressed in comprehension form.
> >
> > It's worth knowing that a list comprehension is structured identically
> > to the equivalent for loop. So it really is neither more dense nor
> > more difficult to read. Further, you can tell immediately from the
> > start of the list comprehension what you've got -- in this case a list
> > of item(s).

Empirically, I'd have to disagree with you. The structure is changed by
the fact that "item" gets put at the beginning of the comprehension.
With the loop, each variable gets introduced in a for statement, and
then used. With the list comprehension, "item" gets used at the
beginning, and then assigned to in the for expression at the end.
Meanwhile, pair gets introduced as the loop variable in the first for
expression, and iterated over in the second for expression. So do you
read it left to right or right to left? Neither. You have to hold the
whole expression in your mind at once. Good for short and simple
comprehensions, bad as they get more complicated.

> >
> > Here with some slight changes...
> >
> >>>> a = [(1, 2), (3, 4, 7), (5, 6)]
> >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]

> > [1, 5]
> >

This, to me, looks like line noise, approaching perl in its

> > ...opposed to...
> >
> >>>> for j in a:

> > ... if len(j)==2:
> > ... for item in j:
> > ... if item % 2:
> > ... b.append(item)
> > ...
> >>>> b

> > [1, 5]
> >>>>

>

Much nicer. Thank you.

> Thanks for the lesson in list comprehensions, but I'm well aware of
> how they work.
>
> because it can read like a run-on sentence and thus be difficult to
> parse. The Python documentation discourages their use and I believe
> for good reason. It's much easier to explain complex processes with a
> function rather than a single nested statement.
>
> >
> > YMMV,
> >
> > Emile

>
> It will apparently vary greatly. Depending on how much coffee I've
>
>
>
> J
> --
> http://mail.python.org/mailman/listinfo/python-list
>

J. Cliff Dyer, May 5, 2009
16. ### Steven D'ApranoGuest

On Tue, 05 May 2009 12:15:15 -0400, J Kenneth King wrote:

> List comprehensions can make a reader of your code apprehensive because
> it can read like a run-on sentence and thus be difficult to parse. The
> Python documentation discourages their use

Do you have a URL for such an astonishing claim?

> and I believe for good
> reason. It's much easier to explain complex processes with a function
> rather than a single nested statement.

Sure, if you treat the function as a black box: "Input goes in, output
comes out, and I trust it does the right thing". But if you can do that
for a function, with no care about its internal complexity, then you can
do the same thing for a list comp.

What's the difference between these?

results = [complicated nested list comp]

versus

def func():
return [complicated nested list comp]

results = func()

The function call isn't *actually* easier to understand, it just allows
you to treat it as a black box and pretend you understand it. Once you
look inside the function, there's no promise that it will be simpler than
a nested list comp. Maybe it will be, and maybe it won't be.

Besides, not all list comprehensions are equivalent to nested loops. Why
label list comps as difficult to parse when only *some* of them are
difficult? That's like insisting that 1+3*x must be hard to read and
scary-looking to the average programmer, just because arithmetic
operators allow one to also write

1/(1+1/(2*x+3/(7*x+x/(5*x**2-3*x+4/(9**x+8/(-3+6/(2*x+1)))))))

--
Steven

Steven D'Aprano, May 6, 2009
17. ### Steven D'ApranoGuest

On Tue, 05 May 2009 13:43:32 -0400, J. Cliff Dyer wrote:

> On Fri, 2009-05-01 at 13:00 -0400, John Posner wrote:
>> Shane Geiger wrote:
>> > if type(el) == list or type(el) is tuple:

>> A tiny improvement:
>>
>> if type(el) in (list, tuple):
>>
>>

> Another alternative, which might be useful in some cases:
>
> if hasattr(el, '__iter__'):
>
> This covers all iterables, not just lists and tuples.

Except for the ones that it doesn't cover, like strings:

>>> hasattr('abcd', '__iter__')

False
>>> list(iter('abcd'))

['a', 'b', 'c', 'd']

And classes that follow the sequence protocol:

>>> class Spam:

.... def __getitem__(self, index):
.... if 0 <= index < 5:
.... return "spam"
.... raise IndexError
....
>>> hasattr(Spam(), '__iter__')

False
>>> list(iter(Spam()))

['spam', 'spam', 'spam', 'spam', 'spam']

--
Steven

Steven D'Aprano, May 6, 2009
18. ### alex23Guest

On May 6, 4:01 am, "J. Cliff Dyer" <> wrote:
> The way you have to bounce your eyes back and forth in the comprehension
> makes it hard to read the logic.  With the loop, on the other hand it is
> blatantly obvious which way the nesting occurs.

> > >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]
> > > ...opposed to...
> > >>>> for j in a:
> > > ...     if len(j)==2:
> > > ...         for item in j:
> > > ...             if item % 2:
> > > ...                 b.append(item)
> > > ...

> Much nicer.  Thank you.

Apart from the presence of 'item' at the beginning of the list
comprehension as opposed to 'b.append(item)' at the end of the for-
loop, how exactly does the listcomp force you to "bounce [..] back and
forth" to follow the logic? The only other difference between the two
is in the layout - the indentation & colons - otherwise they're
structurally identical.

alex23, May 6, 2009
19. ### Steven D'ApranoGuest

On Tue, 05 May 2009 20:36:37 -0700, alex23 wrote:

> On May 6, 4:01Â am, "J. Cliff Dyer" <> wrote:
>> The way you have to bounce your eyes back and forth in the
>> comprehension makes it hard to read the logic. Â With the loop, on the
>> other hand it is blatantly obvious which way the nesting occurs.

>
>> > >>>> [ item for j in a if len(j)==2 for item in j if item % 2 ]
>> > > ...opposed to...
>> > >>>> for j in a:
>> > > ... Â  Â  if len(j)==2:
>> > > ... Â  Â  Â  Â  for item in j:
>> > > ... Â  Â  Â  Â  Â  Â  if item % 2:
>> > > ... Â  Â  Â  Â  Â  Â  Â  Â  b.append(item) ...

>
>> Much nicer. Â Thank you.

>
> Apart from the presence of 'item' at the beginning of the list
> comprehension as opposed to 'b.append(item)' at the end of the for-
> loop, how exactly does the listcomp force you to "bounce [..] back and
> forth" to follow the logic? The only other difference between the two is
> in the layout - the indentation & colons - otherwise they're
> structurally identical.

It's precisely the indentation and colons (plus newlines) that makes
nested for-loops easier to read than list-comps with multiple fors.

for a in alist:
for b in blist:
for c in clist:
if c:
parrot(a, b, c)

is much easier to read than the equivalent one-liner:

[parrot(a, b, c) for a in alist for b in blist for c in clist if c]

You can get back *nearly* all the readability by splitting the list comp
into multiple lines:

[parrot(a, b, c)
for a in alist
for b in blist
for c in clist if c
]

--
Steven

Steven D'Aprano, May 6, 2009
20. ### Terry ReedyGuest

Steven D'Aprano wrote:
> On Tue, 05 May 2009 13:43:32 -0400, J. Cliff Dyer wrote:

> Except for the ones that it doesn't cover, like strings:
>
>>>> hasattr('abcd', '__iter__')

> False

True in Python3

Terry Reedy, May 6, 2009