func_code vs. string problem

[Filip Dreger]

Each function has a func_code property that is suposed to contain the
pure bytecode of the function. All the context (including reference to
relevant namespaces) is stored in different fields of the function
object. Since 'exec' is able to execute any string or bytecode in the
current scope, it would seem possible to execute code of any function
in any namespace. But no matter how I tried, I could not do it. There
must be something I am missing.
Here's what I do: (if anyone wants to help, I placed the source
under http://www.bajobongo.net/foo.py - tested on Python 2.4.1)

1. I declare a function. In the next steps I will try to run its code
from inside a class:

def myfunction():
print abc
self.test()

2. I declare a class foo, with two methods. The first one tries to
reach some local variables from a string passed to exec. The other one
tries to do the same from inside a bytecode (from myfunction). IMHE
this should make no difference to 'exec' - [spoiler: it does].

class foo:
def test(self):
print "ABC"
def checkfunction(self):
abc=10
exec myfunction.func_code
def checkstring(self):
abc=10
exec "print abc;self.test()"

3. I test the both methods. Sadly, the 'checkfunction' fails to find
the correct namespace (id does not see 'abc' nor 'self'). Adding
things like:
"exec myfunction.func_code in globals(),locals()" does not help.

i=foo()
i.checkstring()
i.checkfunction() # this throws exception; why???

4. I try to find some help here, and hope to also gain better
undesrtanding of how Python works

Thanks for any suggestions,
regards,
Filip Dreger

 

[Filip Dreger]

I came up with a simpler description of the problem.
It's all in the simple source:

# we define 'print b' in three different ways: as a string,
# a bytecode and a function
string="print b"
code=compile(string,'string','exec')
def function():
print b

# now we make functions that test if it is possible to execute 'print
b'
# in some local scope

def execstring():
b=5
exec string

def execfunction():
b=5
exec function.func_code

def execcode():
b=5
exec code

execstring() # works
execcode() # works
execfunction() # throws name error exception...

My problem is that both 'string' and 'code' are references to code
objects, so they _should_ behave in the same way... I am close to
giving up...
I am trying to find a way of executing functions without creating a
nested scope, so they can share local and global namespace (even if
they are declared in some other module). I _could_ turn them into
strings and pass around as compiled objects, butthis would be very
ugly. I am sure Python has some better, cleaner way to do this.

regards,
Filip Dreger

 

[Steven Bethard]

Each function has a func_code property that is suposed to contain the
pure bytecode of the function. All the context (including reference to
relevant namespaces) is stored in different fields of the function
object. Since 'exec' is able to execute any string or bytecode in the
current scope, it would seem possible to execute code of any function
in any namespace. But no matter how I tried, I could not do it. There
must be something I am missing.
Here's what I do: (if anyone wants to help, I placed the source
under http://www.bajobongo.net/foo.py - tested on Python 2.4.1)

1. I declare a function. In the next steps I will try to run its code
from inside a class:

def myfunction():
print abc
self.test()

2. I declare a class foo, with two methods. The first one tries to
reach some local variables from a string passed to exec. The other one
tries to do the same from inside a bytecode (from myfunction). IMHE
this should make no difference to 'exec' - [spoiler: it does].

class foo:
def test(self):
print "ABC"
def checkfunction(self):
abc=10
exec myfunction.func_code
def checkstring(self):
abc=10
exec "print abc;self.test()"

3. I test the both methods. Sadly, the 'checkfunction' fails to find
the correct namespace (id does not see 'abc' nor 'self'). Adding
things like:
"exec myfunction.func_code in globals(),locals()" does not help.

i=foo()
i.checkstring()
i.checkfunction() # this throws exception; why???

See the documentation:

http://docs.python.org/ref/dynamic-features.html

"""The eval(), execfile(), and input() functions and the exec statement
do not have access to the full environment for resolving names. Names
may be resolved in the local and global namespaces of the caller. Free
variables are not resolved in the nearest enclosing namespace, but in
the global namespace."""

Note the last sentence, which tells you that your free variable, 'abc',
will be resolved in the *global* namespace. In your particular problem,
you can solve this by substituting your local namespace for the global
namespace:

py> def myfunction():
.... print abc
.... self.test()
....
py> class foo:
.... def test(self):
.... print "ABC"
.... def checkfunction(self):
.... abc=10
.... exec myfunction.func_code in locals()
.... def checkstring(self):
.... abc=10
.... exec "print abc;self.test()"
....
py> foo().checkstring()
10
ABC
py> foo().checkfunction()
10
ABC

But note that if your code actually needs access to any globals, it's
out of luck:

py> def myfunction():
.... print foo
.... print abc
.... self.test()
....
py> foo().checkfunction()
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
File "<interactive input>", line 6, in checkfunction
File "<interactive input>", line 2, in myfunction
NameError: global name 'foo' is not defined

One possible workaround might be:

py> class foo:
.... def test(self):
.... print "ABC"
.... def checkfunction(self):
.... abc=10
.... l = locals()
.... l.update(globals())
.... exec myfunction.func_code in l
.... def checkstring(self):
.... abc=10
.... exec "print abc;self.test()"
....
py> foo().checkfunction()
__main__.foo
10
ABC

But I'd have to know what your real use case is to tell you whether or
not this is worth the trouble. Why do you want to exec the func_code
anyway? Why can't you just call the function?

STeVe

 

[Steven Bethard]

I am trying to find a way of executing functions without creating a
nested scope, so they can share local and global namespace (even if
they are declared in some other module).

Why? Can you explain what the use case is?

STeVe

 

[Filip Dreger]

Why? Can you explain what the use case is?

I have a large set of objects of a single class - 'actor'. They are
created all at once, and they all live in a sort of container written
in C. The methods of the functions are few and general. One of them is
simply 'act()', that is called by the C program every X seconds (tics,
whatever).
Each object can play 0..N roles, and the roles played by an object
often change during its lifetime. The roles should be played in the
act() method. Also, the roles should be defined in some simple way,
and in an external python file (so adding/removing roles does not
require recompiling of the parts embedded in C).

If I had a magic function 'exec in current scope' I would implement it
like this:

class actor:
def __init__():
self.roles=[]
def act():
for i in self.roles:
exec i in current scope

then the roles would simply be functions defined in any importable
file. For example creating an actor that logs each time it is called
would be as simple as:

import actor

def log():
self.counter+=1
print "called %d times"%self.counter

a=actor.actor()
a.counter=0
a.roles.append(log)

Let me recapitulate (is that the right English word?):
1. I need to keep the definitions of roles in a separate file (they
can not simply be additional methods)
2. I need the roles to have full access to global and local namespace
of the actor object (sometimes they have to change it, and sometimes
they have to use some external modules) - the roles should work like
plugins.
3. I guess I also need a fresh look at the problem. Maybe I am simply
going the wrong way? I am more than willing to change my design, if
someone shows me the light

regards,
Filip Dreger

 

[Steven Bethard]

If I had a magic function 'exec in current scope' I would implement it
like this:

class actor:
def __init__():
self.roles=[]
def act():
for i in self.roles:
exec i in current scope

then the roles would simply be functions defined in any importable
file. For example creating an actor that logs each time it is called
would be as simple as:

import actor

def log():
self.counter+=1
print "called %d times"%self.counter

a=actor.actor()
a.counter=0
a.roles.append(log)

Any reason you can't define it like:

class actor(object):
def __init__(self):
self.roles = []
def act(self):
for role_func in self.roles:
role_func(self)

And then write your other modules like:

import actor

def log(self):
self.counter += 1
print "called %d times"%self.counter

a = actor.actor()
a.counter = 0
a.roles.append(log)

The only real difference here is that log is basically declared as an
instance method. So if you need to update actor.actor state, you simply
modify the self object.

2. I need the roles to have full access to global and local namespace
of the actor object (sometimes they have to change it, and sometimes
they have to use some external modules) - the roles should work like
plugins.

By importing actor, they have full access to the global namespace of the
actor.actor object, by simply accessing the actor module attributes.

So the issue here is really about full *local* namespace access. Do you
really need *full* local namespace access? Why isn't access to the
actor.actor instance sufficient?

STeVe

 

[Filip Dreger]

See the documentation:

http://docs.python.org/ref/dynamic-features.html

"""The eval(), execfile(), and input() functions and the exec
statement do not have access to the full environment for resolving
names. Names may be resolved in the local and global namespaces of
the caller. Free variables are not resolved in the nearest enclosing
namespace, but in the global namespace."""

Thank you! I feel silly I have not found the piece of instruction
myself. And even worse, as I still do not understand: if exec()
resolves free bariable names in the global namespace, how come it
works well with code in a string? Or with a code in a compiled string?

Note the last sentence, which tells you that your free variable,
'abc', will be resolved in the *global* namespace. In your
particular problem, you can solve this by substituting your local
namespace for the global namespace.

This particular problem - sadly - is a simplified version of the real
one. I need access to both: global and local namespaces, so this
solution is not for me.
The manual says I can pass two dictionaries to exec, one for global
namespace and one for local. But, strange as it seems, I could not get
it to work. In the example I gave, changing exec f.func_code to exec
f.func_code in globals(),locals() does not help a bit.

One possible workaround might be:

py> class foo:
... def test(self):
... print "ABC"
... def checkfunction(self):
... abc=10
... l = locals()
... l.update(globals())
... exec myfunction.func_code in l
... def checkstring(self):
... abc=10
... exec "print abc;self.test()"
...
py> foo().checkfunction()
__main__.foo
10
ABC

I thought about that... I don't know why, but it seems wrong. Maybe
the updating dictionaries is not very expensive, but there must be
some better way. This code has a 'workaround - fixme' written all over
it.
Up to now, each time I encountered such a workaroundish solution, it
turned out Python has a cleaner way to do it. I sure hope this is also
the case here .

But I'd have to know what your real use case is to tell you whether
or not this is worth the trouble. Why do you want to exec the
func_code anyway? Why can't you just call the function?

I put the description in the other post. Perhaps it's jkust my design
that's broken.
Thanks again.

regards,
Filip Dreger

 

[Steven Bethard]

Thank you! I feel silly I have not found the piece of instruction
myself. And even worse, as I still do not understand: if exec()
resolves free bariable names in the global namespace, how come it
works well with code in a string? Or with a code in a compiled string?

Well, probably the best thing to do is to check out what byte-code is
generated in each case:

py> s = compile('print b', 'string', 'exec')
py> def f():
.... print b
....
py> import dis
py> dis.dis(s)
1 0 LOAD_NAME 0 (b)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE
py> dis.dis(f)
2 0 LOAD_GLOBAL 0 (b)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE

So, the code is basically identical, except that with the string, the
code uses LOAD_NAME, while with the function, the code uses LOAD_GLOBAL.
Note that this is because Python can tell that b is not defined in the
function, and so it assumes that is must be a global lookup. Look what
happens if I give f a parameter:

py> def f(b):
.... print b
....
py> dis.dis(f)
2 0 LOAD_FAST 0 (b)
3 PRINT_ITEM
4 PRINT_NEWLINE
5 LOAD_CONST 0 (None)
8 RETURN_VALUE

Now f is using LOAD_FAST, which looks at the locals. I believe the
point of LOAD_GLOBAL and LOAD_FAST (instead of always using LOAD_NAME)
is to increase performance in the common case, where exec is not used.

Seems like the documentation is being a little conservative -- it does
look like some statements will do the lookup in the locals too... If I
was more confident about what goes on here, I'd probably try to file a
documentation bug...

The manual says I can pass two dictionaries to exec, one for global
namespace and one for local. But, strange as it seems, I could not get
it to work. In the example I gave, changing exec f.func_code to exec
f.func_code in globals(),locals() does not help a bit.

That's because, as the dis.dis output above shows, the function you've
defined is only searching the global namespace. So the locals() dict
you pass won't ever be looked at.

[snip example using locals as globals]

I thought about that... I don't know why, but it seems wrong. Maybe
the updating dictionaries is not very expensive, but there must be
some better way. This code has a 'workaround - fixme' written all over
it.

Yeah, I agree. On the other hand, whenever I see exec, it has
'workaround - fixme' written all over it too.

STeVe

 

[Filip Dreger]

Any reason you can't define it like:

class actor(object):
def __init__(self):
self.roles = []
def act(self):
for role_func in self.roles:
role_func(self) [snip]
By importing actor, they have full access to the global namespace of
the actor.actor object, by simply accessing the actor module
attributes.

So the issue here is really about full *local* namespace access. Do
you really need *full* local namespace access? Why isn't access to
the actor.actor instance sufficient?

!!! Yep, of course it is sufficient. Abondoning the obvious
role_func() must have had some good reasons some time ago, but now I
can not even remember them, probably they were not so important
Thanks a million,
Filip