exec(), execfile() and local-variable binding?

[Jonathan]

I'm puzzled by Python's behavior when binding local variables which
are introduced within exec() or execfile() statements. First, consider
this simple Python program:

# main.py
def f() :
x = 1
print "x:", x
f()

This just prints "x: 1" when run. Now, what happens if we move the
assignment into a separate file "assign.py", and attempt to read it in
with execfile()?

# assign.py
x = 1
print "x in assign.py:", x

# main_execfile.py
def f () :
execfile("assign.py")
print "x in main_execfile.py:", x
f()

So, you might assume that if execfile() worked like a C-style
#include, this would work fine. But it doesn't. Instead, when
"main_execfile.py" is compiled, the "x = 1" assignment in "assign.py"
is never seen, so the 'x' in the print statement is assumed to be a
global variable. Then at run-time, we get an error about 'x' being an
undefined global variable:

x in assign.py: 1
x in main_execfile.py:
Traceback (most recent call last):
File "main_execfile.py", line 4, in ?
f()
File "main_execfile.py", line 3, in f
print "x in main_execfile.py:", x
NameError: global name 'x' is not defined

Well, that's understandable. But what I find strange is that exec()
*doesn't* work in this way. Consider this version:

# main_exec.py
def f() :
exec("x = 1")
print "x:", x
f()

I would have thought that, just like the execfile() version, the "x =
1" string would not be interpreted at parse-time, so that the 'x' in
the "print" statement would again be taken to be a global variable.
But no, this version runs fine, producing the output "x: 1".

And this fancier version also runs:

# main_exec2.py
def f(v) :
exec(v + " = 1")
print "x:", x
print "Type 'x':"
f(input())

Here, if the user enters 'x', the assignment string "x = 1" is made
up, and the program runs as before.

So, why do these exec() versions work, when the execfile() one didn't?
Specifically, why aren't the 'x' variables in the "print" statements
taken as global variables? AFAIK, this should be a compile-time
decision, but the "x = 1" assignment strings can't have been
interpreted at that time.

Thanks for any help,

-- Jonathan

 

[Jonathan]

I'm puzzled by Python's behavior when binding local variables which
are introduced within exec() or execfile() statements. [...]

I now think I understand the gist of the problem. A Google group
search turned up some very similar past discussions. Essentially,
execfile() is a function, and so can't (reliably) change the
local-variable dictionary (locals()) which is passed into it. So any
local-variable settings which happen in the file which execfile()
executes are lost when it returns.

In contrast, exec() is a *statement*, not a function. Therefore, the
locals() dictionary passed into it is mutable. So variable assignments
in the string of an exec() *will* affect the local-variable dictionary
of the function which calls it. Python's usual static compile-time
variable binding can't cope with these potential dynamically-defined
variables. So therefore, if any exec() statement is seen in a function
body at compile-time, static binding is switched off, and a slower
form of run-time binding is used. This dynamically searches the
enclosing variable scopes (dictionaries) for variable definitions.

The run-time binding triggered by exec() explains a weird effect which
I stumbled on after I posted my original message. That is, the
execfile() version of the code *will* work if an exec() appears
anywhere before or after it in the f() function definition. This is
because run-time binding is now used for the whole function - even for
the execfile(). That's why I said above that execfile() can't
*reliably* change the calling function's local-variable dictionary -
it can if this run-time-binding mode is in effect.

Such sneaky insertion of a dummy exec() is not great form, though. The
recommended way to allow execfile() code to affect current variables
is to pass in an explicit context dictionary as an argument to
execfile(). Then you are free to use that dictionary as you see fit
(notably, using it for other exec()'s or execfile()'s). Another way to
implement more of a #include-type effect is to replace this:
execfile("blah.py")
with this:
exec open("blah.py").read()

However, once again, this solution will trigger the use of the slower
run-time binding for the whole enclosing function.

I trust the experts out there will comment if I misrepresented any of
these issues.

-- Jonathan

 

[Steven Taschuk]

Quoth Jonathan:
[...]

Well, that's understandable. But what I find strange is that exec()
*doesn't* work in this way. Consider this version:

# main_exec.py
def f() :
exec("x = 1")
print "x:", x

Note that exec is a keyword, not a function; you might as well
write
exec 'x = 1'
I'm not just picking a nit here -- it is important for your
question that exec is a keyword, since this means it is possible
to determine at compile-time whether exec is used in the body of a
function. This is quite unlike calls to built-in functions such
as execfile(), which cannot in general be identified as such at
compile-time.

As you noted, in the presence of exec statements, the compiler
abandons the optimization by which LOAD_NAMEs are replaced with
LOAD_GLOBALs. This optimization normally speeds up variable
access by skipping a futile name lookup in the locals; abandoning
this optimization when there are exec statements makes your
example above work in the obvious and desired way.