List comprehension vs filter()

[Chris Angelico]

Context: Embedded Python interpreter, version 2.6.6

I have a list of dictionaries, where each dictionary has a "type"
element which is a string. I want to reduce the list to just the
dictionaries which have the same "type" as the first one.

lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]

I'm seeing a weird difference between two otherwise-equivalent-looking
ways of doing the job.

type=lst[0]["type"].lower()

lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one type

lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one type

If I use the filter() method, the resulting list is completely empty.
If I use the list comprehension, it works perfectly. Oddly, either
version works in the stand-alone interpreter.

I have no idea where to start looking for the problem. Hints, please!

Chris Angelico

 

[Steven D'Aprano]

Context: Embedded Python interpreter, version 2.6.6

I have a list of dictionaries, where each dictionary has a "type"
element which is a string. I want to reduce the list to just the
dictionaries which have the same "type" as the first one.

[snip discussion and code]

It should, and does, work as expected, both in the global scope:

lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"}, .... {42: None, "type": "CALC"}]
t = lst[0]["type"].lower()

filter(lambda x: x["type"].lower() == t, lst) [{'type': 'calc'}, {42: None, 'type': 'CALC'}]
[i for i in lst if i["type"].lower() == t]

[{'type': 'calc'}, {42: None, 'type': 'CALC'}]

and in a function:
.... lst = [{"type": "calc"}, {"type": "fixed"}, {"type": "spam"},
.... {42: None, "type": "CALC"}]
.... t = lst[0]["type"].lower()
.... print filter(lambda x: x["type"].lower() == t, lst)
.... print [i for i in lst if i["type"].lower() == t]
....[{'type': 'calc'}, {42: None, 'type': 'CALC'}]
[{'type': 'calc'}, {42: None, 'type': 'CALC'}]


[...]

If I use the filter() method, the resulting list is completely empty. If
I use the list comprehension, it works perfectly. Oddly, either version
works in the stand-alone interpreter.

Let me guess... you're using an IDE?

There's your problem. IDEs often play silly buggers with the environment
in order to be "clever". You've probably found a bug in whatever IDE
you're using.

And this is why I won't touch the buggers with a 30 ft pole, at least not
for anything serious.

 

[Peter Otten]

Context: Embedded Python interpreter, version 2.6.6

I have a list of dictionaries, where each dictionary has a "type"
element which is a string. I want to reduce the list to just the
dictionaries which have the same "type" as the first one.

lst=[{"type":"calc",...},{"type":"fixed",...},{"type":"calc",...},...]

I'm seeing a weird difference between two otherwise-equivalent-looking
ways of doing the job.

type=lst[0]["type"].lower()

lst=filter(lambda x: x["type"].lower()==type,lst) # Restrict to that one
type

lst=[i for i in lst if i["type"].lower()==type] # Restrict to that one
type

If I use the filter() method, the resulting list is completely empty.
If I use the list comprehension, it works perfectly. Oddly, either
version works in the stand-alone interpreter.

I have no idea where to start looking for the problem. Hints, please!

Chris Angelico

The assignment writes to the local namespace, the lambda function reads from
the global namespace; this will only work as expected if the two namespaces
are the same:

exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {} []
ns = {}
exec """type = 42; print filter(lambda x: x == type, [42])""" in ns

[42]

The list comprehension doesn't introduce another namespace, at least in 2.x:

$ python2.7 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
{})'
[42]
$ python3.2 -c 'exec("type = 42; print([x for x in [42] if x == type])", {},
{})'
[]

 

[Chris Angelico]

There's your problem. IDEs often play silly buggers with the environment
in order to be "clever". You've probably found a bug in whatever IDE
you're using.

And this is why I won't touch the buggers with a 30 ft pole, at least not
for anything serious.

Not an IDE, but it's running in an embedded interpreter. Which is why
I looked there for issues, but I can't find any. (Checking sys.version
and friends shows that it's the same version of Python either way.) I
described the embed environment above, in the hopes that someone would
spot an "obvious error", and if your instant suspicion is an IDE, then
that's probably it.

Chris Angelico

 

[Ian Kelly]

The assignment writes to the local namespace, the lambda function reads from
the global namespace; this will only work as expected if the two namespaces
are the same:

exec """type = 42; print filter(lambda x: x == type, [42])""" in {}, {} []
ns = {}
exec """type = 42; print filter(lambda x: x == type, [42])""" in ns

[42]

That must be a quirk of exec, because it works just fine without using
exec, both in and out of functions, either from the interactive
interpreter or from a script:

Python 2.7.1 (r271:86832, Apr 2 2011, 19:44:19)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information..... t = 42
.... print filter(lambda x: x == t, [42])
....

f() [42]
t = 43
print filter(lambda x: x == t, [43])

[43]

So, the question for the OP: Is this file being run with execfile?

Cheers,
Ian

 

[Chris Angelico]

So, the question for the OP:  Is this file being run with execfile?

Not execfile per se; the code is fetched from the database and then
executed with:

PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

Is Py_file_input the problem?

Chris Angelico

 

[Ian Kelly]

Not execfile per se; the code is fetched from the database and then
executed with:

PyObject *v=PyRun_StringFlags(code,Py_file_input,py_globals,locals,0);

Is Py_file_input the problem?

Not specifically. The problem is that you're execing the code, and
the locals and globals are two different namespaces, as Peter
suggested. Since the locals namespace is not associated with a
function, the compiler evidently doesn't generate closures for it, and
thus the lambda (which has its own locals namespace) can't see it.

That's my take, anyway. Somebody else may have better insight than me.

Cheers,
Ian