is list comprehension necessary?

Discussion in 'Python' started by Xah Lee, Oct 26, 2010.

  1. Xah Lee

    Xah Lee Guest

    recently wrote a article based on a debate here. (can't find the
    original thread on Google at the moment)

    • 〈What's List Comprehension and Why is it Harmful?〉
    http://xahlee.org/comp/list_comprehension.html

    it hit reddit.
    http://www.reddit.com/r/programming/comments/dw8op/whats_list_comprehension_and_why_is_it_harmful/

    though, i don't find the argument there informative.

    For python, i can understand that it might be preferred, due to the
    special syntax, being more in sync with python because of the
    imperative hints in keywords. (e.g. those “forâ€, “if†in it.) But for
    more pure functional lang (e.g. haskell), i think lc is pretty bad.

    here's the plain text version of my essay
    --------------------------------
    What's List Comprehension and Why is it Harmful?

    Xah Lee, 2010-10-14

    This page explains what is List Comprehension, with examples from
    several languages, with my opinion on why the jargon and concept of
    “list comprehension†are unnecessary, and harmful to functional
    programing.
    What is List Comprehension?

    Here's a example of List Comprehension (LC) in python:

    S = [2*n for n in range(0,9) if ( (n % 2) == 0)]
    print S
    # prints [0, 4, 8, 12, 16]

    It generates a list from 0 to 9 by 「range(0,9)ã€, then remove the odd
    numbers by 「( (n % 2) == 0)ã€, then multiply each element by 2 in
    「2*nã€, then returns a list.

    Python's LC syntax has this form:

    [myExpression for myVar in myList if myPredicateExpression]

    In summary, it is a special syntax for generating a list, and allows
    programers to also filter and apply a function to the list, but all
    done using expressions.

    In functional notation, list comprehension is doing this:

    map( f, filter(list, predicate))

    Other languages's LC are similiar. Here are some examples from
    Wikipedia. In the following, the filter used is 「x^2 > 3ã€, and the
    「2*x〠is applied to the result.
    Haskell

    s = [ 2*x | x <- [0..], x^2 > 3 ]

    F#

    seq { for x in 0..100 do if x*x > 3 then yield 2*x } ;;

    OCaml

    [? 2 * x | x <- 0 -- max_int ; x * x > 3 ?];;

    Clojure

    (take 20 (for [x (iterate inc 0) :when (> (* x x) 3)] (* 2 x)))

    Common Lisp

    (loop for x from 1 to 20 when (> (* x x) 3) collect (* 2 x))

    Erlang

    S = [2*X || X <- lists:seq(0,100), X*X > 3].

    Scala

    val s = for (x <- Stream.from(0); if x*x > 3) yield 2*x

    Here's how Wikipedia explains List comprehension. Quote:

    A list comprehension is a syntactic construct available in some
    programming languages for creating a list based on existing lists.

    The following features makes up LC:

    * (1) A flat list generator, with the ability to do filtering and
    applying a function.
    * (2) A special syntax in the language.
    * (3) The syntax uses expressions, not functions.

    Why is List Comprehension Harmful?

    • List Comprehension is a opaque jargon; It hampers communication, and
    encourage mis-understanding.

    • List Comprehension is a redundant concept in programing. It is a
    very simple list generator. It can be easily expressed in existing
    functional form 「map(func, filter(list, predicate))〠or imperative
    form e.g. perl: 「for (0..9) { if ( ($_ % 2) == 0) {push @result,
    $_*2 }}ã€.

    • The special syntax of List Comprehension as it exists in many langs,
    are not necessary. If a special purpose function is preferred, then it
    can simply be a plain function, e.g 「LC(function, list, predicate)ã€.
    Map + Filter = List Comprehension Semantics

    The LC's semantics is not necessary. A better way and more in sync
    with functional lang spirit, is simply to combine plain functions:

    map( f, filter(list, predicate))

    Here's the python syntax:

    map(lambda x: 2*x , filter( lambda x:x%2==0, range(9) ) )
    # result is [0, 4, 8, 12, 16]

    In Mathematica, this can be written as:

    Map[ #*2 &, Select[Range@9, EvenQ]]

    In Mathematica, arithemetic operations can be applied to list
    directely without using Map explicitly, so the above can be written
    as:

    Select[Range@9, EvenQ] * 2

    in my coding style, i usually write it in the following syntactically
    equivalent forms:

    (#*2 &) @ (Select[#, EvenQ]&) @ Range @ 9

    or

    9 // Range // (Select[#, EvenQ]&) // (#*2 &)

    In the above, we sequence functions together, as in unix pipe. We
    start with 9, then apply “Range†to it to get a list from 1 to 9, then
    apply a function that filters out odd numbers, then we apply a
    function to multiply each number by 2. The “//†sign is a postfix
    notation, analogous to bash's “|â€, and “@†is a prefix notation that's
    the reverse of “|â€.

    (See: Short Intro of Mathematica For Lisp Programers.)
    List Comprehension Function Without Special Syntax

    Suppose we want some “list comprehension†feature in a functional
    lang. Normally, by default this can be done by

    map(func, filter(inputList, Predicate))

    but perhaps this usage is so frequent that we want to create a new
    function for it, to make it more convenient, and perhaps easier to
    make the compiler to optimize more. e.g.

    LC(func, inputList, Predicate)

    this is about whether a lang should create a new convenient function
    that otherwise require 3 function combinations. Common Lisp vs Scheme
    Lisp are the typical example of extreme opposites.

    Note, there's no new syntax involved.

    Suppose, someone argues that

    For instance, this is far more convenient:

    [x+1 for x in [1,2,3,4,5] if x%2==0]

    than this:

    map(lambda x:x+1,filter(lambda x:x%2==0,[1,2,3,4,5]))

    How about this:

    LC(func, inputList, P)

    compared to

    [func for myVar in inputList if P]

    the functional form is:

    * Shorter
    * Not another idiosyncratic new syntax

    Issues and Decisions on Creating a New Function

    Suppose we decided that generating list by a filter is so frequently
    used that it worth it to create a new func for it.

    LC(func, inputList, Predicate)

    Now, in functional langs, in general a design principle is that you
    want to reduce the number of function unless you really need it.
    Because, any combination of list related functions could potentially
    be a new function in your lang. So, if we really think LC is useful,
    we might want to generalize it. e.g. in

    LC(func, inputList, Predicate)

    is it worthwhile say to add a 4th param, that says return just the
    first n? (here we presume the lang doesn't support list of infinite
    elements) e.g.

    LC(func, inputList, Predicate, n)

    what about partition the list to m sublists?

    LC(func, inputList, Predicate, n, m)

    what about actually more generalized partition, by m sublist then by
    m1 sublist then by m2 sublist?

    LC(func, inputList, Predicate, n, list(m,m1,m2,...))

    what about sorting? maybe that's always used together when you need a
    list?

    LC(func, inputList, Predicate, n, list(m,m1,m2,...), sortPredcate)

    what if actually frequently we want LC to map parallel to branches?
    e.g.

    LC(func, inputList, Predicate, n, list(m,m1,m2,...), sortPredcate,
    mapBranch:True)

    what if ...

    you see, each of these or combination of these can be done by default
    in the lang by sequencing one or more functions (i.e. composition).
    But when we create a new function, we really should think a lot about
    its justification, because otherwise the lang becomes a bag of
    functions that are non-essential, confusing.

    So the question is, is generating a list really that much needed? And,
    if so, why should we create a special syntax such as 「[ expr for var
    in list if P]〠than 「 LC(func, list, P)�

    Also note, that LC is not capable of generating arbitrary nested list.
    For a example of a much powerful list generator that can generate
    arbitrary nested tree, see:

    * http://reference.wolfram.com/mathematica/ref/Table.html
    * Tree Functions: Table

    For those who find imperative lang good, then perhaps “list
    comprehension†is good, because it adds another idiosyncratic syntax
    to the lang, but such is with the tradition of imperative langs. The
    ad hoc syntax aids in reading code by various syntactical forms and
    hint words such as “[... for ... in ...]â€.
    Bad Jargon and How To Judge a Jargon

    Someone wrote:

    The term “list comprehension†is intuitive, it's based on math set
    notation.

    The jargon “list comprehension†is opaque. It hampers communication
    and increases misunderstanding. A better name is simply “list
    generatorâ€.

    What's your basis in saying that “List Comprehension†is intuitive?
    Any statics, survey, research, references?

    To put this in context, are you saying that lambda, is also intuitive?
    “let†is intuitive? “for†is intuitive? “when†is intuitive? I mean,
    give your evaluation of some common computer language terminologies,
    and tell us which you think are good and which are bad, so we have
    some context to judge your claim.

    For example, let us know, in your view, how good are terms: currying,
    lisp1 lisp2, tail recursion, closure, subroutine, command, object. Or,
    perhaps expound on the comparative merits and meaning on the terms
    module vs package vs add-on vs library. I would like to see your view
    on this with at least few paragraphs of analysis on each. If you, say,
    write a essay that's at least 1k words on this topic, then we all can
    make some judgment of your familiarity and understanding in this area.

    Also, “being intuitive†is not the only aspect to consider whether a
    term is good or bad. For example, emacs's uses the term “frameâ€. It's
    quite intuitive, because frame is a common english word, everyone
    understands. You know, door frame, window frame, picture frame, are
    all analogous to emacs's “frame†on a computer. However, by some turn
    of history, in computer software we call such as “window†now, and by
    happenstance the term “window†also has a technical meaning in emacs,
    what we call “split window†or “frame†today. So, in emacs, the term
    “frame†and “window†is confusing, because emacs's “frame†is what we
    call “windowâ€, while emacs's “window†is what we call a frame. So
    here, is a example, that even when a term is intuitive, it can still
    be bad.

    As another example, common understanding by the target group the term
    is to be used is also a important aspect. For example, the term
    “lambdaâ€, which is a name of greek char, does not convey well what we
    use it for. The word's meaning by itself has no connection to the
    concept of function. The char happens to be used by a logician as a
    shorthand notation in his study of what's called “lambda
    calculus†(the “calculus†part is basically 1700's terminology for a
    systematic science, especially related to mechanical reasoning).
    However, the term “lambda†used in this way in computer science and
    programing has been long and wide, around 50 years in recent history
    (and more back if we trace origins). So, because of established use,
    here it may decrease the level of what we might think of it as a bad
    jargon, by the fact that it already become a standard usage or
    understanding. Even still, note that just because a term has establish
    use, if the term itself is very bad in many other aspects, it may
    still warrant a need for change. For one example of a reason, the
    argon will be a learning curve problem for all new generations.

    You see, when you judge a terminology, you have to consider many
    aspects. It is quite involved. When judging a jargon, some question
    you might ask are:

    • Does the jargon convey its meaning by the word itself? (i.e. whether
    the jargon as a word is effective in communication)

    • How long has been the jargon in use?

    • Do people in the community understand the jargon? (e.g. more
    scientifically: what percentage?)

    Each of these sample questions can get quite involved. For example, it
    calls for expertise in linguistics (many sub-fields are relevant:
    pragmatics, history of language, etymology), practical experience in
    the field (programing or computer science), educational expertise
    (e.g. educators, professors, programing book authors/teachers),
    scientific survey, social science of communication...

    Also, you may not know, there are bodies of professional scientists
    who work on terminologies for publication. It is not something like “O
    think it's good, becus it is intuitive to me.â€.
    Acknowledgement

    Thanks to w_a_x_man for this ruby code:

    (0..9).select{|n| n.even?}.map{|n| 2*n}

    Not list comprehension since it does not use a special syntax. But it
    captures the 「map(f, filter(list,predicate))〠in ruby style..

    Xah ∑ http://xahlee.org/ ☄
    Xah Lee, Oct 26, 2010
    #1
    1. Advertising

  2. Xah Lee

    John Nagle Guest

    On 10/26/2010 2:31 AM, Xah Lee wrote:
    > recently wrote a article based on a debate here. (can't find the
    > original thread on Google at the moment)
    >
    > • 〈What's List Comprehension and Why is it Harmful?〉
    > http://xahlee.org/comp/list_comprehension.html
    >
    > it hit reddit.
    > http://www.reddit.com/r/programming/comments/dw8op/whats_list_comprehension_and_why_is_it_harmful/
    >
    > though, i don't find the argument there informative.
    >
    > For python, i can understand that it might be preferred, due to the
    > special syntax, being more in sync with python because of the
    > imperative hints in keywords. (e.g. those “forâ€, “if†in it.) But for
    > more pure functional lang (e.g. haskell), i think lc is pretty bad.


    That's from the functional programming crowd.

    Python isn't a functional language. It has some minimal
    functional capabilities, and there's a lobby that would like
    more. So far, that's mostly been resisted. Attempts to allow
    multiline lambdas have been averted. The weird "functional if"
    syntax additions were a cave-in to the functional crowd, and may
    have been a mistake.

    John Nagle
    John Nagle, Oct 26, 2010
    #2
    1. Advertising

  3. Hello,

    I occasionally use LCs, if they seem useful. However, what I don't like
    about LCs is that they 'look-like' being a closed scope, while actually
    they are in the scope of there call. Example:

    >>> i = 5
    >>> l = [i**2 for i in range(3)]
    >>> i

    2

    Regards


    Andre
    Andre Alexander Bell, Oct 26, 2010
    #3
  4. Xah Lee

    Paul Rudin Guest

    Andre Alexander Bell <> writes:

    > I occasionally use LCs, if they seem useful. However, what I don't like
    > about LCs is that they 'look-like' being a closed scope, while actually
    > they are in the scope of there call. Example:
    >
    >>>> i = 5
    >>>> l = [i**2 for i in range(3)]
    >>>> i

    > 2
    >


    Although:

    paul@sleeper-service:~$ python3
    Python 3.1.2 (r312:79147, Sep 27 2010, 09:57:50)
    [GCC 4.4.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> i = 5
    >>> l = [i**2 for i in range(3)]
    >>> i

    5
    Paul Rudin, Oct 26, 2010
    #4
  5. On 10/26/2010 07:22 PM, Ian Kelly wrote:
    > >>> i = 5
    > >>> l = [i**2 for i in range(3)]
    > >>> i

    > 2
    >
    >
    > This has been corrected in Python 3.


    Sorry. You're right. I forgot to mention that...


    Andre
    Andre Alexander Bell, Oct 26, 2010
    #5
  6. Xah Lee

    Chris Rebert Guest

    On 10/26/10, Mikael B <> wrote:
    >> That's from the functional programming crowd.
    >>
    >> Python isn't a functional language.

    >
    > A noob question: what is a functional language? What does it meen?


    A language which supports the functional programming paradigm:
    http://en.wikipedia.org/wiki/Functional_programming

    Google+Wikipedia are your friends as always.

    Cheers,
    Chris
    Chris Rebert, Oct 26, 2010
    #6
  7. Xah Lee

    rantingrick Guest

    On Oct 26, 11:29 am, John Nagle <> wrote:
    > On 10/26/2010 2:31 AM, Xah Lee wrote:
    >
    > > recently wrote a article based on a debate here. (can't find the
    > > original thread on Google at the moment)

    >
    > > • 〈What's List Comprehension and Why is it Harmful?〉
    > >http://xahlee.org/comp/list_comprehension.html

    >
    > > it hit reddit.
    > >http://www.reddit.com/r/programming/comments/dw8op/whats_list_compreh...

    >
    > > though, i don't find the argument there informative.

    >
    > > For python, i can understand that it might be preferred, due to the
    > > special syntax, being more in sync with python because of the
    > > imperative hints in keywords. (e.g. those “forâ€, “if†in it.) But for
    > > more pure functional lang (e.g. haskell), i think lc is pretty bad.

    >
    >     That's from the functional programming crowd.
    >
    >     Python isn't a functional language.  It has some minimal
    > functional capabilities, and there's a lobby that would like
    > more.  So far, that's mostly been resisted.  Attempts to allow
    > multiline lambdas have been averted.  The weird "functional if"
    > syntax additions were a cave-in to the functional crowd, and may
    > have been a mistake.
    >
    >                                 John Nagle


    I think if you look at LC's (Python's LC's that is) from an esoteric
    and mainstream (almost haughty) point of view then yes they will seem
    offensive to you. However i think Guido and his cast of extras (if I
    may speak on behalf of this fine group of folks!) intended LC's to be
    like any other construct we have come to love about Python. Syntactic
    simplicity coupled with elegant phrasing whist never forgetting to
    drop a good joke when the situation permits (as is the case for the
    easter eggs and whatnot). So my point is that Python LC's are
    different from "mainstream" LC's and that is a good thing. I find
    Pythons map and lambda far more atrocious than LC's , really.
    rantingrick, Oct 27, 2010
    #7
  8. Xah Lee

    rantingrick Guest

    On Oct 26, 12:07 pm, Andre Alexander Bell <> wrote:
    > Hello,
    >
    > I occasionally use LCs, if they seem useful. However, what I don't like
    > about LCs is that they 'look-like' being a closed scope, while actually
    > they are in the scope of there call. Example:
    >
    > >>> i = 5
    > >>> l = [i**2 for i in range(3)]
    > >>> i

    >
    > 2



    I must admit you make a good point here however the only time that
    will slip you up is when you first experienced the Python LC syntax.
    After a few "hello world" LC's you'll begin to love and understand
    them completely.
    rantingrick, Oct 27, 2010
    #8
  9. Xah Lee

    Gary Herron Guest

    On 10/26/2010 09:28 PM, rantingrick wrote:
    > On Oct 26, 12:07 pm, Andre Alexander Bell<> wrote:
    >> Hello,
    >>
    >> I occasionally use LCs, if they seem useful. However, what I don't like
    >> about LCs is that they 'look-like' being a closed scope, while actually
    >> they are in the scope of there call. Example:
    >>
    >>>>> i = 5
    >>>>> l = [i**2 for i in range(3)]
    >>>>> i

    >> 2



    That (very small) issue has been fixed in Python3:

    >>> l = [i**2 for i in range(3)]
    >>> i

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    NameError: name 'i' is not defined



    Gary Herron



    > I must admit you make a good point here however the only time that
    > will slip you up is when you first experienced the Python LC syntax.
    > After a few "hello world" LC's you'll begin to love and understand
    > them completely.
    Gary Herron, Oct 27, 2010
    #9
  10. Xah Lee

    Roy Smith Guest

    Andre Alexander Bell <> wrote:
    >> I occasionally use LCs, if they seem useful. However, what I
    >> don't like about LCs is that they 'look-like' being a closed scope
    >> [...]

    >


    antingrick <> wrote:
    > I must admit you make a good point here however the only time that
    > will slip you up is when you first experienced the Python LC syntax.
    > After a few "hello world" LC's you'll begin to love and understand
    > them completely.


    I agree. I resisted LCs when they first came out, passing them off as
    unnecessary, confusing, etc. Eventually, I came to be comfortable with
    them and use them often.

    I did not realize just how far my mind-change had gone until just
    yesterday when I was hacking on some PHP code. I wrote:

    $values = array();
    foreach ($ids as $id) {
    $values[] = $id;
    }

    and found myself asking my office-mate, "Hey, Chris, does PHP have list
    comprehensions?" I was bummed to discover that it does not.
    Roy Smith, Oct 27, 2010
    #10
  11. Xah Lee

    rantingrick Guest

    On Oct 26, 4:31 am, Xah Lee <> wrote:
    > recently wrote a article based on a debate here. (can't find the
    > original thread on Google at the moment)



    Hey all you numbskulls who are contributing the annoying off-topic
    chatter about Report Lab need to...

    1) GET A LIFE
    2) START A NEW THREAD!

    This thread is about Python LC's and if you cannot stay on topic then
    stay of the discussion. Yes xah has made a few enemies within this
    (and quite a few other newsgroups!) however this thread is very much
    on topic and qualifies for discussion within the Python community by
    those who may have an opinion.

    Now I know a few "little girls" out there wish to do anything within
    their power to silence any voices of dissent. However your bulling,
    propaganda, booby traps, and diversions have no effect on me or anyone
    else who has an honest opinion. So why don't you save the hot air,
    stop flapping your gums, and get a life already!

    If you would invest 1/10 of this negative energy into something more
    positive the world might just be a better place for all of us. How
    many diseases could be cured, or problems solved in the time you spent
    sabotaging others over something so ridiculous as differing points of
    view. How many self realizations has been missed, how many sunsets
    have been missed? really! Look, i know your life sucks and you need to
    justify your existence, but ask yourself a simple question...

    "Does my life suck because of them, or because of me?".

    ....I'll let you marinate on that for a while then we'll have some
    retrospective Q&A time.
    rantingrick, Oct 28, 2010
    #11
  12. Xah Lee

    Xah Lee Guest

    On Oct 27, 5:46 pm, rantingrick <> wrote:
    > On Oct 26, 4:31 am, Xah Lee <> wrote:
    >
    > > recently wrote a article based on a debate here. (can't find the
    > > original thread on Google at the moment)

    >
    > Hey all you numbskulls who are contributing the annoying off-topic
    > chatter about Report Lab need to...
    >
    > 1) GET A LIFE
    > 2) START A NEW THREAD!


    :)

    when i saw the thread got quietly hijacked to about PDF... it was
    funny. But yeah, they need to get a life.

    Xah
    Xah Lee, Oct 28, 2010
    #12
  13. In message <>, Andre
    Alexander Bell wrote:

    > >>> i = 5
    > >>> l = [i**2 for i in range(3)]
    > >>> i

    > 2


    The last line comes out as 5 in Python 3.1.
    Lawrence D'Oliveiro, Oct 29, 2010
    #13
  14. In message <4cc701e7$0$1606$>, John Nagle wrote:

    > The weird "functional if" syntax additions were a cave-in to the
    > functional crowd, and may have been a mistake.


    The only mistake was not putting functional-if into the language in the
    first place, and having to use that strange inside-out syntax to add it.
    Lawrence D'Oliveiro, Oct 29, 2010
    #14
  15. Xah Lee

    Aahz Guest

    In article <4cc701e7$0$1606$>,
    John Nagle <> wrote:
    >
    >Python isn't a functional language. It has some minimal functional
    >capabilities, and there's a lobby that would like more. So far, that's
    >mostly been resisted. Attempts to allow multiline lambdas have been
    >averted. The weird "functional if" syntax additions were a cave-in to
    >the functional crowd, and may have been a mistake.


    Did you actually read the PEP explanation for *why* Guido decided to add
    conditional expressions?
    --
    Aahz () <*> http://www.pythoncraft.com/

    "Look, it's your affair if you want to play with five people, but don't
    go calling it doubles." --John Cleese anticipates Usenet
    Aahz, Nov 21, 2010
    #15
  16. On 10/27/2010 05:37 AM, Roy Smith wrote:
    > I agree. I resisted LCs when they first came out, passing them off as
    > unnecessary, confusing, etc. Eventually, I came to be comfortable with
    > them and use them often.


    I do use LCs fairly often, but only then a for loop would be ugly or
    inefficient. In many cases, I find a normal for loop often is clearer.
    Basically I'm saying that you can over-use LCs. And sometimes LCs only
    serve to make code harder to read. Just because you can use an LC
    doesn't always mean you should. But of course that's just my opinion.
    Michael Torrie, Nov 22, 2010
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Peter Barth

    Mix lambda and list comprehension?

    Peter Barth, Jul 15, 2003, in forum: Python
    Replies:
    4
    Views:
    405
    Michele Simionato
    Jul 17, 2003
  2. Shane Geiger
    Replies:
    4
    Views:
    381
    bullockbefriending bard
    Mar 25, 2007
  3. Debajit Adhikary
    Replies:
    17
    Views:
    678
    Debajit Adhikary
    Oct 18, 2007
  4. Vedran Furac(
    Replies:
    4
    Views:
    322
    Marc 'BlackJack' Rintsch
    Dec 19, 2008
  5. MooMaster
    Replies:
    11
    Views:
    421
    MooMaster
    Apr 9, 2009
Loading...

Share This Page