List comprehension vs filter()

Discussion in 'Python' started by Chris Angelico, Apr 20, 2011.

  1. Context: Embedded Python interpreter, version 2.6.6

    I have a list of dictionaries, where each dictionary has a "type"
    element which is a string. I want to reduce the list to just the
    dictionaries which have the same "type" as the first one.

    lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]

    I'm seeing a weird difference between two otherwise-equivalent-looking
    ways of doing the job.

    type=lst[0]["type"].lower()

    lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one type

    lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one type

    If I use the filter() method, the resulting list is completely empty.
    If I use the list comprehension, it works perfectly. Oddly, either
    version works in the stand-alone interpreter.

    I have no idea where to start looking for the problem. Hints, please!

    Chris Angelico
    Chris Angelico, Apr 20, 2011
    #1
    1. Advertising

  2. On Wed, 20 Apr 2011 13:10:21 +1000, Chris Angelico wrote:

    > Context: Embedded Python interpreter, version 2.6.6
    >
    > I have a list of dictionaries, where each dictionary has a "type"
    > element which is a string. I want to reduce the list to just the
    > dictionaries which have the same "type" as the first one.


    [snip discussion and code]

    It should, and does, work as expected, both in the global scope:


    >>> lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},

    .... {42: None, "type": "CALC"}]
    >>> t = lst[0]["type"].lower()
    >>>
    >>> filter(lambda x: x["type"].lower() == t, lst)

    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]
    >>> [i for i in lst if i["type"].lower() == t]

    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]

    and in a function:

    >>> def test():

    .... lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},
    .... {42: None, "type": "CALC"}]
    .... t = lst[0]["type"].lower()
    .... print filter(lambda x: x["type"].lower() == t, lst)
    .... print [i for i in lst if i["type"].lower() == t]
    ....
    >>> test()

    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]
    [{'type': 'calc'}, {42: None, 'type': 'CALC'}]


    [...]
    > If I use the filter() method, the resulting list is completely empty. If
    > I use the list comprehension, it works perfectly. Oddly, either version
    > works in the stand-alone interpreter.


    Let me guess... you're using an IDE?

    There's your problem. IDEs often play silly buggers with the environment
    in order to be "clever". You've probably found a bug in whatever IDE
    you're using.

    And this is why I won't touch the buggers with a 30 ft pole, at least not
    for anything serious.



    --
    Steven
    Steven D'Aprano, Apr 20, 2011
    #2
    1. Advertising

  3. Chris Angelico

    Peter Otten Guest

    Chris Angelico wrote:

    > Context: Embedded Python interpreter, version 2.6.6
    >
    > I have a list of dictionaries, where each dictionary has a "type"
    > element which is a string. I want to reduce the list to just the
    > dictionaries which have the same "type" as the first one.
    >
    > lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]
    >
    > I'm seeing a weird difference between two otherwise-equivalent-looking
    > ways of doing the job.
    >
    > type=lst[0]["type"].lower()
    >
    > lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one
    > type
    >
    > lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one
    > type
    >
    > If I use the filter() method, the resulting list is completely empty.
    > If I use the list comprehension, it works perfectly. Oddly, either
    > version works in the stand-alone interpreter.
    >
    > I have no idea where to start looking for the problem. Hints, please!
    >
    > Chris Angelico


    The assignment writes to the local namespace, the lambda function reads from
    the global namespace; this will only work as expected if the two namespaces
    are the same:

    >>> exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}

    []
    >>> ns = {}
    >>> exec """type = 42; print filter(lambda x: x == type, [42])""" in ns

    [42]

    The list comprehension doesn't introduce another namespace, at least in 2.x:

    $ python2.7 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
    {})'
    [42]
    $ python3.2 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
    {})'
    []
    Peter Otten, Apr 20, 2011
    #3
  4. On Wed, Apr 20, 2011 at 8:16 PM, Steven D'Aprano
    <> wrote:
    > There's your problem. IDEs often play silly buggers with the environment
    > in order to be "clever". You've probably found a bug in whatever IDE
    > you're using.
    >
    > And this is why I won't touch the buggers with a 30 ft pole, at least not
    > for anything serious.


    Not an IDE, but it's running in an embedded interpreter. Which is why
    I looked there for issues, but I can't find any. (Checking sys.version
    and friends shows that it's the same version of Python either way.) I
    described the embed environment above, in the hopes that someone would
    spot an "obvious error", and if your instant suspicion is an IDE, then
    that's probably it.

    Chris Angelico
    Chris Angelico, Apr 20, 2011
    #4
  5. Chris Angelico

    Ian Kelly Guest

    On Wed, Apr 20, 2011 at 4:41 AM, Peter Otten <> wrote:
    > The assignment writes to the local namespace, the lambda function reads from
    > the global namespace; this will only work as expected if the two namespaces
    > are the same:
    >
    >>>> exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {}

    > []
    >>>> ns = {}
    >>>> exec """type = 42; print filter(lambda x: x == type, [42])""" in ns

    > [42]


    That must be a quirk of exec, because it works just fine without using
    exec, both in and out of functions, either from the interactive
    interpreter or from a script:

    Python 2.7.1 (r271:86832, Apr 2 2011, 19:44:19)
    [GCC 4.4.5] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> def f():

    .... t = 42
    .... print filter(lambda x: x == t, [42])
    ....
    >>> f()

    [42]
    >>> t = 43
    >>> print filter(lambda x: x == t, [43])

    [43]

    So, the question for the OP: Is this file being run with execfile?

    Cheers,
    Ian
    Ian Kelly, Apr 20, 2011
    #5
  6. On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly <> wrote:
    > So, the question for the OP:  Is this file being run with execfile?
    >


    Not execfile per se; the code is fetched from the database and then
    executed with:

    PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

    Is Py_file_input the problem?

    Chris Angelico
    Chris Angelico, Apr 20, 2011
    #6
  7. Chris Angelico

    Ian Kelly Guest

    On Wed, Apr 20, 2011 at 12:03 PM, Chris Angelico <> wrote:
    > On Thu, Apr 21, 2011 at 12:44 AM, Ian Kelly <> wrote:
    >> So, the question for the OP:  Is this file being run with execfile?
    >>

    >
    > Not execfile per se; the code is fetched from the database and then
    > executed with:
    >
    > PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);
    >
    > Is Py_file_input the problem?


    Not specifically. The problem is that you're execing the code, and
    the locals and globals are two different namespaces, as Peter
    suggested. Since the locals namespace is not associated with a
    function, the compiler evidently doesn't generate closures for it, and
    thus the lambda (which has its own locals namespace) can't see it.

    That's my take, anyway. Somebody else may have better insight than me.

    Cheers,
    Ian
    Ian Kelly, Apr 20, 2011
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shane Geiger
    Replies:
    4
    Views:
    384
    bullockbefriending bard
    Mar 25, 2007
  2. Debajit Adhikary
    Replies:
    17
    Views:
    683
    Debajit Adhikary
    Oct 18, 2007
  3. zax75
    Replies:
    1
    Views:
    1,096
  4. Vedran Furac(
    Replies:
    4
    Views:
    327
    Marc 'BlackJack' Rintsch
    Dec 19, 2008
  5. Terry Reedy
    Replies:
    14
    Views:
    311
    Tim Golden
    Aug 25, 2012
Loading...

Share This Page