Injecting code into a function

G

George Sakkis

Is there a general way of injecting code into a function, typically
before and/or after the existing code ? I know that for most purposes,
an OO solution, such as the template pattern, is a cleaner way to get
the same effect, but it's not always applicable (e.g. if you have no
control over the design and you are given a function to start with). In
particular, I want to get access to the function's locals() just before
it exits, i.e. something like:

def analyzeLocals(func):
func_locals = {}
def probeFunc():
# insert func's code here
sys._getframe(1).f_locals["func_locals"].update(locals())
probeFunc()
# func_locals now contains func's locals

So, how can I add func's code in probeFunc so that the injected code
(the update line here) is always called before the function exits ?
That is, don't just inject it lexically in the end of the function if
there are more than one exit points. I guess a solution will involve a
good deal bytecode hacking, on which i know very little; if there's a
link to a (relatively) simple HOWTO, it would be very useful.

Thanks,
George
 
R

Ron

George said:
Is there a general way of injecting code into a function, typically
before and/or after the existing code ? I know that for most purposes,
an OO solution, such as the template pattern, is a cleaner way to get
the same effect, but it's not always applicable (e.g. if you have no
control over the design and you are given a function to start with). In
particular, I want to get access to the function's locals() just before
it exits, i.e. something like:

def analyzeLocals(func):
func_locals = {}
def probeFunc():
# insert func's code here
sys._getframe(1).f_locals["func_locals"].update(locals())
probeFunc()
# func_locals now contains func's locals

So, how can I add func's code in probeFunc so that the injected code
(the update line here) is always called before the function exits ?
That is, don't just inject it lexically in the end of the function if
there are more than one exit points. I guess a solution will involve a
good deal bytecode hacking, on which i know very little; if there's a
link to a (relatively) simple HOWTO, it would be very useful.

Thanks,
George

I'd like to know this as well. :)

I think you will have to modify the function func in some way to get
locals when it exits.

def func():
x = 20
y = 40
func.locals = locals() # inserted line

func()
print func.locals


On a related note, I'd like to know how to import locals into a function.

Cheers,
Ron
 
S

Steve Holden

George said:
Is there a general way of injecting code into a function, typically
before and/or after the existing code ? I know that for most purposes,
an OO solution, such as the template pattern, is a cleaner way to get
the same effect, but it's not always applicable (e.g. if you have no
control over the design and you are given a function to start with). In
particular, I want to get access to the function's locals() just before
it exits, i.e. something like:

def analyzeLocals(func):
func_locals = {}
def probeFunc():
# insert func's code here
sys._getframe(1).f_locals["func_locals"].update(locals())
probeFunc()
# func_locals now contains func's locals

So, how can I add func's code in probeFunc so that the injected code
(the update line here) is always called before the function exits ?
That is, don't just inject it lexically in the end of the function if
there are more than one exit points. I guess a solution will involve a
good deal bytecode hacking, on which i know very little; if there's a
link to a (relatively) simple HOWTO, it would be very useful.

Thanks,
George
A decorator would seem to be the sensible way to do this, assuming you
are using Python 2.4.

def decorated(func):
def wrapper(arg1, arg2, arg3):
print "Arg2:", arg2
func(arg1)
print "Arg3:", arg3
return wrapper

@decorated
def f1(x):
print "F1:", x

f1('ARG1', 'ARG2', 'ARG3')

Arg2: ARG2
F1: ARG1
Arg3: ARG3

All the decorator really does is compute one function from another.
There's been enough discussion on the list recently that I won't repeat
the theory.

regards
Steve
 
L

Lonnie Princehouse

I expect you could combine the following with decorators as an easy way
to grab a function's locals just before it exits... you could also use
exec or eval to execute code in the function's local namespace.
---------------------------------------

# Try it:

def trace_returns(frame, event, arg):
if event == 'return':
print "[frame locals just before return: %s]" % frame.f_locals
return trace_returns

def foo(a, b):
return a + b

import sys
sys.settrace(trace_returns)

foo(1,2)
 
G

George Sakkis

Lonnie Princehouse said:
I expect you could combine the following with decorators as an easy way
to grab a function's locals just before it exits... you could also use
exec or eval to execute code in the function's local namespace.
---------------------------------------

# Try it:

def trace_returns(frame, event, arg):
if event == 'return':
print "[frame locals just before return: %s]" % frame.f_locals
return trace_returns

def foo(a, b):
return a + b

import sys
sys.settrace(trace_returns)

foo(1,2)

Thanks, that's the closest to what I wanted. A minor point I didn't
quite get from the documentation is how to set a local trace instead of
a global (sys) trace. Also, there's no sys.gettrace() to return the
current tracer; is there a way around this ?

George
 
K

Kay Schluehr

George said:
Thanks, that's the closest to what I wanted. A minor point I didn't
quite get from the documentation is how to set a local trace instead of
a global (sys) trace.

You never do. "Local" in this context only means that those local trace
functions are called inside the one global trace-function and return
the global trace function again.

In Lonnies example there are no local-trace functions at all.
Also, there's no sys.gettrace() to return the
current tracer; is there a way around this ?

The default tracer is None i.e. no debugging. The programmer has to
control his tracer which might not be to hard:


class Analyzer:
def trace_returns(self, frame, event, arg):
if event == 'return':
self.func_locals = frame.f_locals
return self.trace_returns

def analyzeLocals(self, func,*args,**kw):
sys.settrace(self.trace_returns)
func(*args,**kw)
sys.settrace(None)

Ciao,
Kay
 
L

Lonnie Princehouse

I don't know of a way to get the current global trace function. This
could certainly cause trouble if you're trying to be compatible with
other packages that want to use their own trace functions (like psyco,
or debuggers). Does anyone know how to get the global trace?

On the other hand, the local trace function is in the f_trace attribute
of a frame.

It looks like global trace functions only get the "call" event, and are
expected to return a local trace function that will receive "line" and
"return" events, so you will need a global trace in order to set local
traces (setting myframe.f_trace explicitly doesn't seem to do it).
 
K

Kay Schluehr

Lonnie said:
I don't know of a way to get the current global trace function. This
could certainly cause trouble if you're trying to be compatible with
other packages that want to use their own trace functions (like psyco,
or debuggers). Does anyone know how to get the global trace?

On the other hand, the local trace function is in the f_trace attribute
of a frame.

Oh, I overlooked this. Then the solution becomes simple:

sys._getframe().f_trace

Test:
<bound method Analyzer.trace_returns of <__main__.Analyzer instance at
0x010015D0>>

It looks like global trace functions only get the "call" event, and are
expected to return a local trace function that will receive "line" and
"return" events, so you will need a global trace in order to set local
traces (setting myframe.f_trace explicitly doesn't seem to do it).

I think that Your trace_returns function is actually a global trace
that returns itself and does not handle the 'call' event.

If You look at the code in the debug-module bdb.py the 'call' event
gets handled by the local trace function dispactch_call():

def trace_dispatch(self, frame, event, arg):
if self.quitting:
return # None
if event == 'line':
return self.dispatch_line(frame)
if event == 'call':
return self.dispatch_call(frame, arg)
if event == 'return':
return self.dispatch_return(frame, arg)
if event == 'exception':
return self.dispatch_exception(frame, arg)
print 'bdb.Bdb.dispatch: unknown debugging event:', `event`
return self.trace_dispatch

Ciao,
Kay
 
B

Bengt Richter

Is there a general way of injecting code into a function, typically
before and/or after the existing code ? I know that for most purposes,
an OO solution, such as the template pattern, is a cleaner way to get
the same effect, but it's not always applicable (e.g. if you have no
control over the design and you are given a function to start with). In
particular, I want to get access to the function's locals() just before
it exits, i.e. something like:

def analyzeLocals(func):
func_locals = {}
def probeFunc():
# insert func's code here
sys._getframe(1).f_locals["func_locals"].update(locals())
probeFunc()
# func_locals now contains func's locals

So, how can I add func's code in probeFunc so that the injected code
(the update line here) is always called before the function exits ?
That is, don't just inject it lexically in the end of the function if
there are more than one exit points. I guess a solution will involve a
good deal bytecode hacking, on which i know very little; if there's a
link to a (relatively) simple HOWTO, it would be very useful.
I'm not clear on what your real goal is, but if you just want a snapshot
of what locals() is just before exiting func, that could be done with
a byte-code-hacking decorator with usage looking something like

#func defined before this
func_locals = {}
@getlocals(func, func_locals)
def probefunc(): pass

which would make a probefunc function that would be identical to func
except that before exiting, it would do func_locals.update(locals()).
(you might want func_locals to be a list and do func_locals.append(locals())
in case func is recursive and you are interested in the all the locals).

Alternatively, if this is a debugging thing, you might want to look into
sys.settrace -- as in this thing I cobbled together (not tested beyond what you see ;-):

----< tracelocals.py >----------------------------------------------------
class TraceLocals(object):
from sys import settrace
def __init__(self, *names, **kw):
self.names = set(names)
self.func_locals = kw.get('func_locals', []) # [(name,locals()), ...] tuples
def _gwatch(self, frame, event, arg):
"""
Global scope watcher. When a new scope is entered, returns the local
scope watching method _lwatch to do the work for that.
"""
if event=='call':
name = frame.f_code.co_name # name of executing scope
if name in self.names: return self._lwatch # name is one whose locals we want

def _lwatch(self, frame, event, arg):
if event == 'return':
self.func_locals.append((frame.f_code.co_name,frame.f_locals))
else:
return self._lwatch # keep watching for return event

def on(self):
"""Set the system trace hook to enable tracing. """
self.settrace(self._gwatch)
def off(self):
"""Reset the system trace hook to disable tracing. """
self.settrace(None)

def main(modname, entry, *names):
print 'main(', modname, entry, names,')'
tr = TraceLocals(*names)
mod = __import__(modname)
try:
tr.on()
getattr(mod, entry)()
finally:
tr.off()
return tr.func_locals

def test():
tr = TraceLocals(*'t1 t2 t3'.split())
def t1():
x ='t1'
def t2(y=123):
y*=2
def t3():
t1()
t2()
t2('hello ')
try:
tr.on()
t3()
finally:
tr.off()
for name, loc in tr.func_locals: print '%5s: %s' %(name, loc)

if __name__=='__main__':
import sys
args = sys.argv[1:]
if not args:
raise SystemExit(
'Usage: python tracelocals.py (-test | module entry name+)\n'
)
if args[0]=='-test': test()
else:
print args
func_locals = main(args[0], args[1], *args[2:])
for name, loc in func_locals: print '%5s: %s' %(name, loc)
--------------------------------------------------------------------------
Test result:

[22:37] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py -test
t1: {'x': 't1'}
t2: {'y': 246}
t2: {'y': 'hello hello '}
t3: {'t2': <function t2 at 0x02EE8ED4>, 't1': <function t1 at 0x02EE8E9C>}

Note that t3 is seeing t1 and t2 in its locals -- I think because they're
visible as cell vars in test. If you put t1-t3 in a separate module, you don't see it:

----< tmod.py >---------------
def t1():
print '-- t1'
x ='t1'
def t2(y=123):
print '-- t2'
y*=2
def t3():
print '-- t3'
t1()
t2()
t2('hello ')
-----------------------------

[22:42] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py tmod t3 t1 t2 t3
['tmod', 't3', 't1', 't2', 't3']
main( tmod t3 ('t1', 't2', 't3') )
-- t3
-- t1
-- t2
-- t2
t1: {'x': 't1'}
t2: {'y': 246}
t2: {'y': 'hello hello '}
t3: {}

[22:46] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py tmod t3 t2
['tmod', 't3', 't2']
main( tmod t3 ('t2',) )
-- t3
-- t1
-- t2
-- t2
t2: {'y': 246}
t2: {'y': 'hello hello '}

Notice that the -- side effects from all being called in the last, but only t2 being captured.

Maybe this will give you something to expand to your needs.

Regards,
Bengt Richter
 
G

George Sakkis

Oh, I overlooked this. Then the solution becomes simple:
sys._getframe().f_trace

Test:

<bound method Analyzer.trace_returns of <__main__.Analyzer instance at
0x010015D0>>

Does this work for you non-interactively ? I tried running it from a
script or importing it from a module but it returns None. Very
strange...

George
 
G

George Sakkis

I'm not clear on what your real goal is, but if you just want a
snapshot
of what locals() is just before exiting func, that could be done with
a byte-code-hacking decorator with usage looking something like

#func defined before this
func_locals = {}
@getlocals(func, func_locals)
def probefunc(): pass

which would make a probefunc function that would be identical to func
except that before exiting, it would do func_locals.update(locals()).
(you might want func_locals to be a list and do func_locals.append(locals())
in case func is recursive and you are interested in the all the
locals).

That's all good, at least if I knew how to poke with bytecodes ;-)
What's a good starting point to look at ?

By the way, the original problem was yet another property packager,
which I posted as recipe at the cookbook:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410698.

George
 
K

Kay Schluehr

George said:
Does this work for you non-interactively ? I tried running it from a
script or importing it from a module but it returns None. Very
strange...

George

You are right. The expression was context-dependent :-/

I had not yet the time to analyze the output of the following function
but it returns stable values:

def gettracefunc():
import sys
i = 0
while 1:
try:
f_trace = sys._getframe(i).f_trace
if f_trace:
return f_trace
i+=1
except ValueError:
break

Ciao,
Kay
 
B

Bengt Richter

locals).

That's all good, at least if I knew how to poke with bytecodes ;-)
What's a good starting point to look at ?

By the way, the original problem was yet another property packager,
which I posted as recipe at the cookbook:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410698.
I made a couple of byte code hacking decorators based on the techniques
in Raymond Hettinger's optimization decorators, which optimize access to
globals and fold constant expressions and I forgot what else.

I stripped all the optimization out since I was interested in injecting
code to preset values for selected named local variables, to avoid using
the default argument value hack. Another variant used this plus modifying
the call signature to do a version of currying. Also modified the line
number map. You will have some trickier stuff to work out though, I think,
since there may be many returns, and they can happen in try/except/finally
context, and you'll have to see how that translates (you can see with
disassembler), and then if you can't just overwrite some returns to jump
to a common place, you may have to widen a place to put code and adjust
any jumps across modified stuff.

It would be much nicer if you can get the source and do it via AST mods.
I am toying with the concept of a custom importer that does import-time
"decoration" of statements or statement sequences (with their suites)

I am thinking of using '@@' something like

# ... statments
@@module.decorator(nstatements, args) # how many statements, and whatever other args
# decorated statement(s)
...

Where module has to be imported already at the time of the decorating-importer's importation
of the above (simplest way to guarantee that it's available to execute, but not the only way).

Of course this means that a .py file has to be available for the code you are interesting in
modifying, and you have to catch the import. (Since there's a '@@' you will know if you don't ;-)

Anyway, I am not going to have time for that for some time, so I'll mention what I had in mind,
in case someone wants to do something with it:

The decorating-importer would find the @@ lines and e.g. change

@@decomodule.decorator(nstatements, <anything legal>) # how many statements, and whatever other args

to

__AT_AT_DECO__ = decomodule.decorator(nstatementsliteral, <anything legal> ...)

which should just be a matter of src.replace('@@','__AT_AT_DECO__ = ') on lines where
line.lstrip().startswith('@@') is true, thus making valid python source.

It would then call compiler.parse with the modified source for an AST to work with.
It would walk the AST to find the __AT_AT_DECO__ assignments (saving a reference to the containing statement
list and index of the __AT_AT_DECO__ = ... statement in the list, and extract the nstatementsliteral
and then call the decomodule.decorator with a slice of statements to be replaced, including the __AT_AT_DECO__
statement, which is passed as the first statement AST node in the slice, and may have interesting things
in the <anything legal> part, which the decorating-importer doesn't need to know about. It just needs the
two names decomodule and decorator and the number of statements, replacing them something like

containing_statement_list[index:index+int(nstatementsliteral)+1] = getattr(__import__(decomodule), decorator)(
containing_statement_list[index:index+int(nstatementsliteral)+1])

IOW pass the slice of statement list nodes to the decorator and expect it to get additional info from the
__AT_AT_DECO__ = statement that is included, and expect it to return a replacement list of AST statment nodes to
replace what it got, which might be minor or major modifications.

The walk would continue until all the __AT_AT_DECO__ regions were replaced, and then the whole AST would be
compiled and made into a module delivered as if it were an ordinary import.

Note that this is not a macro preprocessor, but depending on what is in decomodule.decorator, one could mess with
legal source (after src.replace('@@','__AT_AT_DECO__ = ') it has to satisfy compiler.parse, and later check also)
pretty seriously, but it has to be pretty legal to start. Of course one could abuse magic-content strings, but
metaclasses can already do a lot of that. That's not what I'm hoping to do. Just super-duper-decoration ;-)

Anyway, that's a sketch of the idea. I won't have time for it or much else for a longish while now.
If someone's interested, they can quote this and start another thread ;-)

Regards,
Bengt Richter
 
L

Lonnie Princehouse

I think that Your trace_returns function is actually a global trace
that returns itself and does not handle the 'call' event.

By returning itself, the global trace also becomes the local trace.
The code in bdb.py does the same thing--- self.trace_dispatch is set
as the global trace, and returns self.trace_dispatch via
self.dispatch_call(frame, arg)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top