Injecting code into a function

Discussion in 'Python' started by George Sakkis, Apr 25, 2005.

  1. Is there a general way of injecting code into a function, typically
    before and/or after the existing code ? I know that for most purposes,
    an OO solution, such as the template pattern, is a cleaner way to get
    the same effect, but it's not always applicable (e.g. if you have no
    control over the design and you are given a function to start with). In
    particular, I want to get access to the function's locals() just before
    it exits, i.e. something like:

    def analyzeLocals(func):
    func_locals = {}
    def probeFunc():
    # insert func's code here
    sys._getframe(1).f_locals["func_locals"].update(locals())
    probeFunc()
    # func_locals now contains func's locals

    So, how can I add func's code in probeFunc so that the injected code
    (the update line here) is always called before the function exits ?
    That is, don't just inject it lexically in the end of the function if
    there are more than one exit points. I guess a solution will involve a
    good deal bytecode hacking, on which i know very little; if there's a
    link to a (relatively) simple HOWTO, it would be very useful.

    Thanks,
    George
     
    George Sakkis, Apr 25, 2005
    #1
    1. Advertising

  2. George Sakkis

    Guest

    use eval.
    eval will accept any string and treat the string as a code.

    pujo
     
    , Apr 25, 2005
    #2
    1. Advertising

  3. =?iso-8859-1?q?Steffen_Gl=FCckselig?=, Apr 25, 2005
    #3
  4. George Sakkis

    Ron Guest

    George Sakkis wrote:

    > Is there a general way of injecting code into a function, typically
    > before and/or after the existing code ? I know that for most purposes,
    > an OO solution, such as the template pattern, is a cleaner way to get
    > the same effect, but it's not always applicable (e.g. if you have no
    > control over the design and you are given a function to start with). In
    > particular, I want to get access to the function's locals() just before
    > it exits, i.e. something like:
    >
    > def analyzeLocals(func):
    > func_locals = {}
    > def probeFunc():
    > # insert func's code here
    > sys._getframe(1).f_locals["func_locals"].update(locals())
    > probeFunc()
    > # func_locals now contains func's locals
    >
    > So, how can I add func's code in probeFunc so that the injected code
    > (the update line here) is always called before the function exits ?
    > That is, don't just inject it lexically in the end of the function if
    > there are more than one exit points. I guess a solution will involve a
    > good deal bytecode hacking, on which i know very little; if there's a
    > link to a (relatively) simple HOWTO, it would be very useful.
    >
    > Thanks,
    > George


    I'd like to know this as well. :)

    I think you will have to modify the function func in some way to get
    locals when it exits.

    def func():
    x = 20
    y = 40
    func.locals = locals() # inserted line

    func()
    print func.locals


    On a related note, I'd like to know how to import locals into a function.

    Cheers,
    Ron
     
    Ron, Apr 25, 2005
    #4
  5. George Sakkis

    Paddy Guest

    Try searching for: 'python aspect-oriented' as aspect oriented
    programming is about modifying existing class-methods (not exactly
    functions which is what you asked for).
    You might also do a search for "AOP considered harmful"
    http://www.infosun.fmi.uni-passau.de/st/papers/EIWAS04/stoerzer04aop_harmful.pdf

    The main point is that when you are reading the source you don't know
    what the code is as it may be augmented by an "external" change.
     
    Paddy, Apr 25, 2005
    #5
  6. George Sakkis

    Steve Holden Guest

    George Sakkis wrote:
    > Is there a general way of injecting code into a function, typically
    > before and/or after the existing code ? I know that for most purposes,
    > an OO solution, such as the template pattern, is a cleaner way to get
    > the same effect, but it's not always applicable (e.g. if you have no
    > control over the design and you are given a function to start with). In
    > particular, I want to get access to the function's locals() just before
    > it exits, i.e. something like:
    >
    > def analyzeLocals(func):
    > func_locals = {}
    > def probeFunc():
    > # insert func's code here
    > sys._getframe(1).f_locals["func_locals"].update(locals())
    > probeFunc()
    > # func_locals now contains func's locals
    >
    > So, how can I add func's code in probeFunc so that the injected code
    > (the update line here) is always called before the function exits ?
    > That is, don't just inject it lexically in the end of the function if
    > there are more than one exit points. I guess a solution will involve a
    > good deal bytecode hacking, on which i know very little; if there's a
    > link to a (relatively) simple HOWTO, it would be very useful.
    >
    > Thanks,
    > George
    >

    A decorator would seem to be the sensible way to do this, assuming you
    are using Python 2.4.

    def decorated(func):
    def wrapper(arg1, arg2, arg3):
    print "Arg2:", arg2
    func(arg1)
    print "Arg3:", arg3
    return wrapper

    @decorated
    def f1(x):
    print "F1:", x

    f1('ARG1', 'ARG2', 'ARG3')

    Arg2: ARG2
    F1: ARG1
    Arg3: ARG3

    All the decorator really does is compute one function from another.
    There's been enough discussion on the list recently that I won't repeat
    the theory.

    regards
    Steve
    --
    Steve Holden +1 703 861 4237 +1 800 494 3119
    Holden Web LLC http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
     
    Steve Holden, Apr 25, 2005
    #6
  7. I expect you could combine the following with decorators as an easy way
    to grab a function's locals just before it exits... you could also use
    exec or eval to execute code in the function's local namespace.
    ---------------------------------------

    # Try it:

    def trace_returns(frame, event, arg):
    if event == 'return':
    print "[frame locals just before return: %s]" % frame.f_locals
    return trace_returns

    def foo(a, b):
    return a + b

    import sys
    sys.settrace(trace_returns)

    foo(1,2)
     
    Lonnie Princehouse, Apr 25, 2005
    #7
  8. "Lonnie Princehouse" wrote:

    > I expect you could combine the following with decorators as an easy

    way
    > to grab a function's locals just before it exits... you could also

    use
    > exec or eval to execute code in the function's local namespace.
    > ---------------------------------------
    >
    > # Try it:
    >
    > def trace_returns(frame, event, arg):
    > if event == 'return':
    > print "[frame locals just before return: %s]" % frame.f_locals
    > return trace_returns
    >
    > def foo(a, b):
    > return a + b
    >
    > import sys
    > sys.settrace(trace_returns)
    >
    > foo(1,2)
    >


    Thanks, that's the closest to what I wanted. A minor point I didn't
    quite get from the documentation is how to set a local trace instead of
    a global (sys) trace. Also, there's no sys.gettrace() to return the
    current tracer; is there a way around this ?

    George
     
    George Sakkis, Apr 25, 2005
    #8
  9. George Sakkis

    Kay Schluehr Guest

    George Sakkis wrote:

    > Thanks, that's the closest to what I wanted. A minor point I didn't
    > quite get from the documentation is how to set a local trace instead

    of
    > a global (sys) trace.


    You never do. "Local" in this context only means that those local trace
    functions are called inside the one global trace-function and return
    the global trace function again.

    In Lonnies example there are no local-trace functions at all.

    > Also, there's no sys.gettrace() to return the
    > current tracer; is there a way around this ?


    The default tracer is None i.e. no debugging. The programmer has to
    control his tracer which might not be to hard:


    class Analyzer:
    def trace_returns(self, frame, event, arg):
    if event == 'return':
    self.func_locals = frame.f_locals
    return self.trace_returns

    def analyzeLocals(self, func,*args,**kw):
    sys.settrace(self.trace_returns)
    func(*args,**kw)
    sys.settrace(None)

    Ciao,
    Kay
     
    Kay Schluehr, Apr 25, 2005
    #9
  10. I don't know of a way to get the current global trace function. This
    could certainly cause trouble if you're trying to be compatible with
    other packages that want to use their own trace functions (like psyco,
    or debuggers). Does anyone know how to get the global trace?

    On the other hand, the local trace function is in the f_trace attribute
    of a frame.

    It looks like global trace functions only get the "call" event, and are
    expected to return a local trace function that will receive "line" and
    "return" events, so you will need a global trace in order to set local
    traces (setting myframe.f_trace explicitly doesn't seem to do it).
     
    Lonnie Princehouse, Apr 26, 2005
    #10
  11. George Sakkis

    Kay Schluehr Guest

    Lonnie Princehouse wrote:
    > I don't know of a way to get the current global trace function. This
    > could certainly cause trouble if you're trying to be compatible with
    > other packages that want to use their own trace functions (like

    psyco,
    > or debuggers). Does anyone know how to get the global trace?
    >
    > On the other hand, the local trace function is in the f_trace

    attribute
    > of a frame.


    Oh, I overlooked this. Then the solution becomes simple:

    sys._getframe().f_trace

    Test:

    >>> an = Analyzer()
    >>> sys.settrace(an.trace_returns)
    >>> sys._getframe().f_trace

    <bound method Analyzer.trace_returns of <__main__.Analyzer instance at
    0x010015D0>>


    > It looks like global trace functions only get the "call" event, and

    are
    > expected to return a local trace function that will receive "line"

    and
    > "return" events, so you will need a global trace in order to set

    local
    > traces (setting myframe.f_trace explicitly doesn't seem to do it).


    I think that Your trace_returns function is actually a global trace
    that returns itself and does not handle the 'call' event.

    If You look at the code in the debug-module bdb.py the 'call' event
    gets handled by the local trace function dispactch_call():

    def trace_dispatch(self, frame, event, arg):
    if self.quitting:
    return # None
    if event == 'line':
    return self.dispatch_line(frame)
    if event == 'call':
    return self.dispatch_call(frame, arg)
    if event == 'return':
    return self.dispatch_return(frame, arg)
    if event == 'exception':
    return self.dispatch_exception(frame, arg)
    print 'bdb.Bdb.dispatch: unknown debugging event:', `event`
    return self.trace_dispatch

    Ciao,
    Kay
     
    Kay Schluehr, Apr 26, 2005
    #11
  12. On 25 Apr 2005 03:32:38 -0700, "George Sakkis" <> wrote:

    >Is there a general way of injecting code into a function, typically
    >before and/or after the existing code ? I know that for most purposes,
    >an OO solution, such as the template pattern, is a cleaner way to get
    >the same effect, but it's not always applicable (e.g. if you have no
    >control over the design and you are given a function to start with). In
    >particular, I want to get access to the function's locals() just before
    >it exits, i.e. something like:
    >
    >def analyzeLocals(func):
    > func_locals = {}
    > def probeFunc():
    > # insert func's code here
    > sys._getframe(1).f_locals["func_locals"].update(locals())
    > probeFunc()
    > # func_locals now contains func's locals
    >
    >So, how can I add func's code in probeFunc so that the injected code
    >(the update line here) is always called before the function exits ?
    >That is, don't just inject it lexically in the end of the function if
    >there are more than one exit points. I guess a solution will involve a
    >good deal bytecode hacking, on which i know very little; if there's a
    >link to a (relatively) simple HOWTO, it would be very useful.
    >

    I'm not clear on what your real goal is, but if you just want a snapshot
    of what locals() is just before exiting func, that could be done with
    a byte-code-hacking decorator with usage looking something like

    #func defined before this
    func_locals = {}
    @getlocals(func, func_locals)
    def probefunc(): pass

    which would make a probefunc function that would be identical to func
    except that before exiting, it would do func_locals.update(locals()).
    (you might want func_locals to be a list and do func_locals.append(locals())
    in case func is recursive and you are interested in the all the locals).

    Alternatively, if this is a debugging thing, you might want to look into
    sys.settrace -- as in this thing I cobbled together (not tested beyond what you see ;-):

    ----< tracelocals.py >----------------------------------------------------
    class TraceLocals(object):
    from sys import settrace
    def __init__(self, *names, **kw):
    self.names = set(names)
    self.func_locals = kw.get('func_locals', []) # [(name,locals()), ...] tuples
    def _gwatch(self, frame, event, arg):
    """
    Global scope watcher. When a new scope is entered, returns the local
    scope watching method _lwatch to do the work for that.
    """
    if event=='call':
    name = frame.f_code.co_name # name of executing scope
    if name in self.names: return self._lwatch # name is one whose locals we want

    def _lwatch(self, frame, event, arg):
    if event == 'return':
    self.func_locals.append((frame.f_code.co_name,frame.f_locals))
    else:
    return self._lwatch # keep watching for return event

    def on(self):
    """Set the system trace hook to enable tracing. """
    self.settrace(self._gwatch)
    def off(self):
    """Reset the system trace hook to disable tracing. """
    self.settrace(None)

    def main(modname, entry, *names):
    print 'main(', modname, entry, names,')'
    tr = TraceLocals(*names)
    mod = __import__(modname)
    try:
    tr.on()
    getattr(mod, entry)()
    finally:
    tr.off()
    return tr.func_locals

    def test():
    tr = TraceLocals(*'t1 t2 t3'.split())
    def t1():
    x ='t1'
    def t2(y=123):
    y*=2
    def t3():
    t1()
    t2()
    t2('hello ')
    try:
    tr.on()
    t3()
    finally:
    tr.off()
    for name, loc in tr.func_locals: print '%5s: %s' %(name, loc)

    if __name__=='__main__':
    import sys
    args = sys.argv[1:]
    if not args:
    raise SystemExit(
    'Usage: python tracelocals.py (-test | module entry name+)\n'
    )
    if args[0]=='-test': test()
    else:
    print args
    func_locals = main(args[0], args[1], *args[2:])
    for name, loc in func_locals: print '%5s: %s' %(name, loc)
    --------------------------------------------------------------------------
    Test result:

    [22:37] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py -test
    t1: {'x': 't1'}
    t2: {'y': 246}
    t2: {'y': 'hello hello '}
    t3: {'t2': <function t2 at 0x02EE8ED4>, 't1': <function t1 at 0x02EE8E9C>}

    Note that t3 is seeing t1 and t2 in its locals -- I think because they're
    visible as cell vars in test. If you put t1-t3 in a separate module, you don't see it:

    ----< tmod.py >---------------
    def t1():
    print '-- t1'
    x ='t1'
    def t2(y=123):
    print '-- t2'
    y*=2
    def t3():
    print '-- t3'
    t1()
    t2()
    t2('hello ')
    -----------------------------

    [22:42] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py tmod t3 t1 t2 t3
    ['tmod', 't3', 't1', 't2', 't3']
    main( tmod t3 ('t1', 't2', 't3') )
    -- t3
    -- t1
    -- t2
    -- t2
    t1: {'x': 't1'}
    t2: {'y': 246}
    t2: {'y': 'hello hello '}
    t3: {}

    [22:46] C:\pywk\clp\sakkis\tracelocals>py24 tracelocals.py tmod t3 t2
    ['tmod', 't3', 't2']
    main( tmod t3 ('t2',) )
    -- t3
    -- t1
    -- t2
    -- t2
    t2: {'y': 246}
    t2: {'y': 'hello hello '}

    Notice that the -- side effects from all being called in the last, but only t2 being captured.

    Maybe this will give you something to expand to your needs.

    Regards,
    Bengt Richter
     
    Bengt Richter, Apr 26, 2005
    #12
  13. > Oh, I overlooked this. Then the solution becomes simple:
    >
    > sys._getframe().f_trace
    >
    > Test:
    >
    > >>> an = Analyzer()
    > >>> sys.settrace(an.trace_returns)
    > >>> sys._getframe().f_trace

    > <bound method Analyzer.trace_returns of <__main__.Analyzer instance

    at
    > 0x010015D0>>


    Does this work for you non-interactively ? I tried running it from a
    script or importing it from a module but it returns None. Very
    strange...

    George
     
    George Sakkis, Apr 26, 2005
    #13
  14. > I'm not clear on what your real goal is, but if you just want a
    snapshot
    > of what locals() is just before exiting func, that could be done with
    > a byte-code-hacking decorator with usage looking something like
    >
    > #func defined before this
    > func_locals = {}
    > @getlocals(func, func_locals)
    > def probefunc(): pass
    >
    > which would make a probefunc function that would be identical to func
    > except that before exiting, it would do func_locals.update(locals()).
    > (you might want func_locals to be a list and do

    func_locals.append(locals())
    > in case func is recursive and you are interested in the all the

    locals).

    That's all good, at least if I knew how to poke with bytecodes ;-)
    What's a good starting point to look at ?

    By the way, the original problem was yet another property packager,
    which I posted as recipe at the cookbook:
    http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410698.

    George
     
    George Sakkis, Apr 26, 2005
    #14
  15. George Sakkis

    Kay Schluehr Guest

    George Sakkis wrote:
    > > Oh, I overlooked this. Then the solution becomes simple:
    > >
    > > sys._getframe().f_trace
    > >
    > > Test:
    > >
    > > >>> an = Analyzer()
    > > >>> sys.settrace(an.trace_returns)
    > > >>> sys._getframe().f_trace

    > > <bound method Analyzer.trace_returns of <__main__.Analyzer instance

    > at
    > > 0x010015D0>>

    >
    > Does this work for you non-interactively ? I tried running it from a
    > script or importing it from a module but it returns None. Very
    > strange...
    >
    > George


    You are right. The expression was context-dependent :-/

    I had not yet the time to analyze the output of the following function
    but it returns stable values:

    def gettracefunc():
    import sys
    i = 0
    while 1:
    try:
    f_trace = sys._getframe(i).f_trace
    if f_trace:
    return f_trace
    i+=1
    except ValueError:
    break

    Ciao,
    Kay
     
    Kay Schluehr, Apr 26, 2005
    #15
  16. George Sakkis

    Kay Schluehr Guest

    Kay Schluehr, Apr 26, 2005
    #16
  17. On 25 Apr 2005 23:31:29 -0700, "George Sakkis" <> wrote:

    >> I'm not clear on what your real goal is, but if you just want a

    >snapshot
    >> of what locals() is just before exiting func, that could be done with
    >> a byte-code-hacking decorator with usage looking something like
    >>
    >> #func defined before this
    >> func_locals = {}
    >> @getlocals(func, func_locals)
    >> def probefunc(): pass
    >>
    >> which would make a probefunc function that would be identical to func
    >> except that before exiting, it would do func_locals.update(locals()).
    >> (you might want func_locals to be a list and do

    >func_locals.append(locals())
    >> in case func is recursive and you are interested in the all the

    >locals).
    >
    >That's all good, at least if I knew how to poke with bytecodes ;-)
    >What's a good starting point to look at ?
    >
    >By the way, the original problem was yet another property packager,
    >which I posted as recipe at the cookbook:
    >http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/410698.
    >

    I made a couple of byte code hacking decorators based on the techniques
    in Raymond Hettinger's optimization decorators, which optimize access to
    globals and fold constant expressions and I forgot what else.

    I stripped all the optimization out since I was interested in injecting
    code to preset values for selected named local variables, to avoid using
    the default argument value hack. Another variant used this plus modifying
    the call signature to do a version of currying. Also modified the line
    number map. You will have some trickier stuff to work out though, I think,
    since there may be many returns, and they can happen in try/except/finally
    context, and you'll have to see how that translates (you can see with
    disassembler), and then if you can't just overwrite some returns to jump
    to a common place, you may have to widen a place to put code and adjust
    any jumps across modified stuff.

    It would be much nicer if you can get the source and do it via AST mods.
    I am toying with the concept of a custom importer that does import-time
    "decoration" of statements or statement sequences (with their suites)

    I am thinking of using '@@' something like

    # ... statments
    @@module.decorator(nstatements, args) # how many statements, and whatever other args
    # decorated statement(s)
    ...

    Where module has to be imported already at the time of the decorating-importer's importation
    of the above (simplest way to guarantee that it's available to execute, but not the only way).

    Of course this means that a .py file has to be available for the code you are interesting in
    modifying, and you have to catch the import. (Since there's a '@@' you will know if you don't ;-)

    Anyway, I am not going to have time for that for some time, so I'll mention what I had in mind,
    in case someone wants to do something with it:

    The decorating-importer would find the @@ lines and e.g. change

    @@decomodule.decorator(nstatements, <anything legal>) # how many statements, and whatever other args

    to

    __AT_AT_DECO__ = decomodule.decorator(nstatementsliteral, <anything legal> ...)

    which should just be a matter of src.replace('@@','__AT_AT_DECO__ = ') on lines where
    line.lstrip().startswith('@@') is true, thus making valid python source.

    It would then call compiler.parse with the modified source for an AST to work with.
    It would walk the AST to find the __AT_AT_DECO__ assignments (saving a reference to the containing statement
    list and index of the __AT_AT_DECO__ = ... statement in the list, and extract the nstatementsliteral
    and then call the decomodule.decorator with a slice of statements to be replaced, including the __AT_AT_DECO__
    statement, which is passed as the first statement AST node in the slice, and may have interesting things
    in the <anything legal> part, which the decorating-importer doesn't need to know about. It just needs the
    two names decomodule and decorator and the number of statements, replacing them something like

    containing_statement_list[index:index+int(nstatementsliteral)+1] = getattr(__import__(decomodule), decorator)(
    containing_statement_list[index:index+int(nstatementsliteral)+1])

    IOW pass the slice of statement list nodes to the decorator and expect it to get additional info from the
    __AT_AT_DECO__ = statement that is included, and expect it to return a replacement list of AST statment nodes to
    replace what it got, which might be minor or major modifications.

    The walk would continue until all the __AT_AT_DECO__ regions were replaced, and then the whole AST would be
    compiled and made into a module delivered as if it were an ordinary import.

    Note that this is not a macro preprocessor, but depending on what is in decomodule.decorator, one could mess with
    legal source (after src.replace('@@','__AT_AT_DECO__ = ') it has to satisfy compiler.parse, and later check also)
    pretty seriously, but it has to be pretty legal to start. Of course one could abuse magic-content strings, but
    metaclasses can already do a lot of that. That's not what I'm hoping to do. Just super-duper-decoration ;-)

    Anyway, that's a sketch of the idea. I won't have time for it or much else for a longish while now.
    If someone's interested, they can quote this and start another thread ;-)

    Regards,
    Bengt Richter
     
    Bengt Richter, Apr 26, 2005
    #17
  18. > I think that Your trace_returns function is actually a global trace
    > that returns itself and does not handle the 'call' event.


    By returning itself, the global trace also becomes the local trace.
    The code in bdb.py does the same thing--- self.trace_dispatch is set
    as the global trace, and returns self.trace_dispatch via
    self.dispatch_call(frame, arg)
     
    Lonnie Princehouse, Apr 26, 2005
    #18
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. amine  zejli
    Replies:
    1
    Views:
    427
    Tom Yates
    Oct 20, 2003
  2. amine  zejli
    Replies:
    0
    Views:
    402
    amine zejli
    Oct 19, 2003
  3. Amine Zejli
    Replies:
    1
    Views:
    440
    William Ryan
    Oct 19, 2003
  4. Brian W
    Replies:
    10
    Views:
    825
    Brian W
    Jul 2, 2003
  5. Cong Ma
    Replies:
    0
    Views:
    275
    Cong Ma
    Jan 15, 2009
Loading...

Share This Page