exec and locals

[Steven D'Aprano]

I have to dynamically generate some code inside a function using exec,
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py> def spam():
.... exec( """x = 23""" )
.... return x
....
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should
actually work, and yet it appears to. The documentation says:

The default locals act as described for function locals()
below: modifications to the default locals dictionary should
not be attempted. Pass an explicit locals dictionary if you
need to see effects of the code on locals after function
exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable x
is created by exec, I need to do this instead:

py> def eggs():
.... mylocals = {}
.... exec( """x = 23""", globals(), mylocals)
.... x = mylocals['x']
.... return x
....
py> eggs()
23

The fact that it works in spam() above is perhaps an accident of
implementation? Yes no maybe?

 

[Chris Angelico]

py> def spam():
... exec( """x = 23""" )
... return x
...
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should
actually work, and yet it appears to.

Doesn't work for me, in IDLE in 3.4.0b2, nor in command-line Python on
3.4.0rc1+ (from hg a couple of weeks ago). But it did (happen to?)
work in 2.7. What version did you use?

ChrisA

 

[Peter Otten]

I have to dynamically generate some code inside a function using exec,
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py> def spam():
... exec( """x = 23""" )
... return x
...
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should
actually work, and yet it appears to. The documentation says:

The default locals act as described for function locals()
below: modifications to the default locals dictionary should
not be attempted. Pass an explicit locals dictionary if you
need to see effects of the code on locals after function
exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable x
is created by exec, I need to do this instead:

py> def eggs():
... mylocals = {}
... exec( """x = 23""", globals(), mylocals)
... x = mylocals['x']
... return x
...
py> eggs()
23

The fact that it works in spam() above is perhaps an accident of
implementation? Yes no maybe?

eggs() should work in Python 2 and 3,
spam() should work in Python 2, but not in Python 3.

Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL) to
make spam() work:
.... return x
....2 0 LOAD_GLOBAL 0 (x)
3 RETURN_VALUE.... exec ""
.... return x
....2 0 LOAD_CONST 1 ('')
3 LOAD_CONST 0 (None)
6 DUP_TOP
7 EXEC_STMT

3 8 LOAD_NAME 0 (x)
11 RETURN_VALUE

 

[Peter Otten]

I have to dynamically generate some code inside a function using exec,
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py> def spam():
... exec( """x = 23""" )
... return x
...
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should
actually work, and yet it appears to. The documentation says:

The default locals act as described for function locals()
below: modifications to the default locals dictionary should
not be attempted. Pass an explicit locals dictionary if you
need to see effects of the code on locals after function
exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable x
is created by exec, I need to do this instead:

py> def eggs():
... mylocals = {}
... exec( """x = 23""", globals(), mylocals)
... x = mylocals['x']
... return x
...
py> eggs()
23

The fact that it works in spam() above is perhaps an accident of
implementation? Yes no maybe?

eggs() should work in Python 2 and 3,
spam() should work in Python 2, but not in Python 3.

Fun fact: Python 2 tweaks the bytecode (LOAD_NAME instead of LOAD_GLOBAL)
to make spam() work:
... return x
...2 0 LOAD_GLOBAL 0 (x)
3 RETURN_VALUE... exec ""
... return x
...2 0 LOAD_CONST 1 ('')
3 LOAD_CONST 0 (None)
6 DUP_TOP
7 EXEC_STMT

3 8 LOAD_NAME 0 (x)
11 RETURN_VALUE

Some more bytcode fun, because it just occured to me that you can optimize
away the code that triggered the modification:
.... return x
.... if 0: exec ""
....2 0 LOAD_NAME 0 (x)
3 RETURN_VALUE

 

[Steven D'Aprano]

I have to dynamically generate some code inside a function using exec,
but I'm not sure if it is working by accident or if I can rely on it.

Here is a trivial example:


py> def spam():
... exec( """x = 23""" )
... return x
...
py> spam()
23


(My real example is more complex than this.)

According to the documentation of exec, I don't think this should
actually work, and yet it appears to. The documentation says:

The default locals act as described for function locals() below:
modifications to the default locals dictionary should not be
attempted. Pass an explicit locals dictionary if you need to see
effects of the code on locals after function exec() returns.

http://docs.python.org/3.4/library/functions.html#exec


I *think* this means that if I want to guarantee that a local variable
x is created by exec, I need to do this instead:

py> def eggs():
... mylocals = {}
... exec( """x = 23""", globals(), mylocals) ... x =
mylocals['x']
... return x
...
py> eggs()
23

The fact that it works in spam() above is perhaps an accident of
implementation? Yes no maybe?

eggs() should work in Python 2 and 3, spam() should work in Python 2,
but not in Python 3.

Aha! That explains it -- I was reading the 3.x docs and testing in Python
2.7.

Thanks everyone for answering.

By the way, if anyone cares what my actual use-case is, I have a function
that needs to work under Python 2.4 through 3.4, and it uses a with
statement. With statements are not available in 2.4 (or 2.5, unless you
give a from __future__ import). So after messing about for a while with
circular imports and dependency injections, I eventually settled on some
code that works something like this:


def factory():
blah blah blah
try:
exec("""def inner():
with something:
return something
""", globals(), mylocals)
inner = mylocals['inner']
except SyntaxError:
def inner():
# manually operate the context manager
call context manager __enter__
try:
try:
return something
except: # Yes, a bare except. Catch EVERYTHING.
blah blah blah
finally:
call context manager __exit__
blah blah blah
return inner


(By the way, yes, I have to use a bare except, not just "except
BaseException". Python 2.4 and 2.5 still have string exceptions.)

 

[Steven D'Aprano]

I have to dynamically generate some code inside a function using exec,
but I'm not sure if it is working by accident or if I can rely on it.

[...]

I have no idea but as exec is generally considered to be a bad idea are
you absolutely sure this is the correct way to achieve your end goal?

perhaps if you detailed your requirement someone may be able to suggest
a safer solution.

Thanks for your concern, but what I'm doing is perfectly safe. The string
being exec'ed is a string literal known at compile-time and written by me
(see my previous email for details) and the only reason I'm running it
with exec at runtime rather than treating it as normal source code is
that it relies on a feature that may not be available (with statement).

The joys of writing code that has to run under multiple incompatible
versions of Python.

 

[Gregory Ewing]

except SyntaxError:
def inner():
# manually operate the context manager
call context manager __enter__
try:
try:
return something
except: # Yes, a bare except. Catch EVERYTHING.
blah blah blah
finally:
call context manager __exit__

Why not just use this version all the time? It should
work in both 2.x and 3.x.

 

[Dave Angel]

code that works something like this:


def factory():
blah blah blah
try:
exec("""def inner():

Before I would use exec, I'd look hard at either generating a
source file to import, or using a preprocessor. And if this
code was to be installed, make the version choice or the
preprocess step happen at install time.

I once implemented a system that generated 20k lines of C++ header
and sources. And the generated code was properly indented and
fairly well commented.

 

[Steven D'Aprano]

Why not just use this version all the time? It should work in both 2.x
and 3.x.

Because that's yucky. It's an aesthetic thing: when supported, I want the
Python interpreter to manage the context manager.

The exec part is only half a dozen lines, only three lines of source
code. It's no burden to keep it for the cases where it works (that is, at
least 2.6 onwards).

 

[Steven D'Aprano]

Before I would use exec, I'd look hard at either generating a
source file to import,

Yes, I went through the process of pulling out the code into a separate
module, but that just made more complexity and was pretty nasty. If the
function was stand-alone, it might have worked, but it needed access to
other code in the module, so there were circular dependencies.

or using a preprocessor.

I don't think that it's easier/better to write a custom Python
preprocessor and run the entire module through it, just to avoid a three-
line call to exec.

Guys, I know that exec is kinda dangerous and newbies should be
discouraged from throwing every string they see at it, but this isn't my
second day Python programming, and it's not an accident that Python
supports the dynamic compilation and execution of source code at runtime.
It's a deliberate language feature. We're allowed to use it

And if this code was
to be installed, make the version choice or the preprocess step happen
at install time.

Completely inappropriate in my case. This is a module which can be called
from multiple versions of Python from a single installation.

 

[Chris Angelico]

Guys, I know that exec is kinda dangerous and newbies should be
discouraged from throwing every string they see at it, but this isn't my
second day Python programming, and it's not an accident that Python
supports the dynamic compilation and execution of source code at runtime.
It's a deliberate language feature. We're allowed to use it

Code smell means "look at this". It doesn't mean "don't use this
feature ever". Steven's looked into this thoroughly, I'm sure, and
exec is important.

ChrisA

 

[Gregory Ewing]

Because that's yucky. It's an aesthetic thing: when supported, I want the
Python interpreter to manage the context manager.

More yucky than wrapping the Py3 version in an
exec? To my way of thinking, that cancels out any
elegance that might have been gained from using
a with-statement.

Do you really need to use the context manager
at all? Could you just write the try-statement
that you would have written in Py2 if you
didn't have a context manager?

 

[Chris Angelico]

More yucky than wrapping the Py3 version in an
exec? To my way of thinking, that cancels out any
elegance that might have been gained from using
a with-statement.

Do you really need to use the context manager
at all? Could you just write the try-statement
that you would have written in Py2 if you
didn't have a context manager?

If I have to support two vastly different versions, I would prefer
(when possible) to write the code so that dropping the old version's
support is simply a matter of deleting stuff. Write the code for the
new version, then warp it as little as possible to support the old
version as well, and keep it clear which bits are for the old. Writing
code that avoids 'with' altogether goes against that.

ChrisA

 

[Steven D'Aprano]

More yucky than wrapping the Py3 version in an exec? To my way of
thinking, that cancels out any elegance that might have been gained from
using a with-statement.

Do you really need to use the context manager at all? Could you just
write the try-statement that you would have written in Py2 if you didn't
have a context manager?

I don't *have* to do it any particular way, but the way it works now, the
version of the inner function (which does eventually get exposed to the
caller) is simple with statement wrapping a function call. So for seven
of the eight versions of Python supported (2.5 through 3.4) the function
is the simplest it can be. Even if the code creating that function is a
tiny bit more complex, since it is wrapped inside an exec. For the other
two versions (2.4 and 2.5), I have to fall back on a more complex chunk
of code. (And yes, it's deliberate that 2.5 gets counted in both groups.)

I'm not saying that I have objective reasons for preferring this way over
the manual alternative, or at least not *strong* objective reasons. It's
mostly subjective. I don't expect to convince you my way is better, and I
doubt that you'll convince me your way is better. But if I were to try,
I'd put it this way:

At a cost of six (by memory) extra lines of code, including one call to
exec, I have an inner function which is *guaranteed* to use the exact
same semantics and efficiency of a with-statement when possible, because
it *is* a with-statement. Any differences between with-statement and my
manual handling will only affect 2.4 and sometimes 2.5, rather than
everything. The only differences that I know of are insignificant -- the
byte-code will be different, there may be trivial performance differences
-- but if it turns out to be some meaningful difference, I have three
choices:

1) deny that the difference is meaningful;

2) accept that the difference is meaningful, and fix it; or

3) accept that the difference is meaningful, but say that it
won't be fixed for 2.4 and 2.5.


If I use the same manual handling for all versions, then I don't have
that last option, no matter how unlikely it is that I will need it.

But really, it's a subjective choice of what feels right to me in this
instance. If the body of the with-statement was bigger, or if the feature
in question was something else, I might choose a different approach.