Deep Black Magic in Python: please help

Discussion in 'Python' started by Jan Burgy, Aug 13, 2004.

  1. Jan Burgy

    Jan Burgy Guest

    Hi everyone,

    I am trying to convince my managers that python can replace the outdated and
    soon no-longer maintained proprietary system (Tool for Calculator Design) we
    use here. In order to achieve this, I need to analyze python code which will
    look somethink like this:

    def foo(arg_dict):
    return arg_dict["one"] + bar(arg_dict)

    def bar(other_dict):
    return other_dict["two"] + other_dict[True and "three" or "four"]

    The result of my analysis should return a list

    ['one', 'two', 'three']

    Alright, that doesn't sound so bad. With a little RTFM-ing and hacking, I got

    -- black magic starts here --

    import compiler

    def _evaluate(partial, name=""):
    from compiler.ast import Expression
    from compiler.misc import set_filename
    from compiler.pycodegen import ExpressionCodeGenerator

    tree = Expression(partial)
    set_filename(name, tree)
    gen = ExpressionCodeGenerator(tree)
    return eval(gen.getCode())

    class RecursiveVisitor(compiler.visitor.ASTVisitor):
    def __init__(self, name):
    self.name = "Name(%s)" % repr(name)
    self.names = {}
    def visitSubscript(self, node, *args):
    if repr(node.expr) == self.name:
    try:
    name = _evaluate(node.subs[0])
    except:
    name = str(node.subs[0])
    self.names[name] = 1
    def visitCallFunc(self, node, *args):
    try:
    from inspect import getsource, getargspec
    from compiler import parse, walk
    func = _evaluate(node.node)
    pos = map(repr, node.args).index(self.name)
    src, arg = getsource(func), getargspec(func)[0]
    tmp, self.name = self.name, "Name(%s)" % repr(arg[pos])
    walk(parse(src), self)
    self.name = tmp
    except Exception, e:
    print str(e)

    if __name__ == "__main__":

    from inspect import getsource
    from compiler import parse, walk

    src = getsource(foo)
    mod = parse(src)
    visitor = RecursiveVisitor("kw")
    walk(mod, visitor)
    print visitor.names.keys()

    -- black magic ends here, ouch --

    Once again, I'm in total awe at the possibilities python offers us lowly
    users. Unfortunately we're all the same: you give us a finger, we want the
    whole arm! I want this method to generalize to method calls as in

    class baz:

    def foo(self, arg_dict):
    return arg_dict["one"] + self.bar(arg_dict)

    def bar(self, other_dict):
    return other_dict["two"] + other_dict[True and "three" or "four"]

    It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
    the AST it looks something like

    CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None, None)

    How do I find the corresponding function? Anybody feels like wrapping
    their head on this?

    Cheers,

    Jan Burgy
     
    Jan Burgy, Aug 13, 2004
    #1
    1. Advertising

  2. Jan Burgy

    Terry Reedy Guest

    "Jan Burgy" <> wrote in message
    news:...
    > return arg_dict["one"] + self.bar(arg_dict)


    > It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
    > the AST it looks something like
    >
    > CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None,

    None)
    >
    > How do I find the corresponding function?


    The lookup returns a bound method, whose structure is an implementation
    detail.

    Completely untested suggestion, possibly not correct or possible:

    Write an implementation-specific unwrap function. Then either augment the
    source code

    ..... + unwrap(self.bar)(self, arg_dict) # self now an explict arg

    or the analysis code to do the unwrapping. For the latter, recognize
    Name('self') and replace the result of Getattr(Name('self'), 'bar') with
    unwrap(thereof) and, again, augment the arg list.

    Terry J. Reedy
     
    Terry Reedy, Aug 13, 2004
    #2
    1. Advertising

  3. Jan Burgy

    Jan Burgy Guest

    "Terry Reedy" <> wrote in message news:<>...
    > "Jan Burgy" <> wrote in message
    > news:...
    > > return arg_dict["one"] + self.bar(arg_dict)

    >
    > > It shouldn't be all that hard. My problem is the lookup of 'self.bar'. In
    > > the AST it looks something like
    > >
    > > CallFunc(Getattr(Name('self'), 'bar'), [Name('arg_dict')], None,

    > None)
    > >
    > > How do I find the corresponding function?

    >
    > The lookup returns a bound method, whose structure is an implementation
    > detail.
    >
    > Completely untested suggestion, possibly not correct or possible:
    >
    > Write an implementation-specific unwrap function. Then either augment the
    > source code
    >
    > .... + unwrap(self.bar)(self, arg_dict) # self now an explict arg
    >
    > or the analysis code to do the unwrapping. For the latter, recognize
    > Name('self') and replace the result of Getattr(Name('self'), 'bar') with
    > unwrap(thereof) and, again, augment the arg list.
    >
    > Terry J. Reedy


    Hi Terry,

    Thank you very much for your suggestion. I am not sure how it would
    help though. What is preventing me now from already replacing the
    result of Getattr(Name('self'), 'bar') with what I need? That's
    precisely my problem however. Because Python is a dynamically typed
    language, I have no information about 'self' at compile-time. Short of
    saving context information while I am visiting the parse tree, I don't
    know how to tell what class I am in.

    Would-Meta-Classes-help-me-in-this-case-ly yours?

    Jan
     
    Jan Burgy, Aug 16, 2004
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Spencer

    Black Magic - Currying using __get__

    Michael Spencer, Mar 24, 2005, in forum: Python
    Replies:
    0
    Views:
    408
    Michael Spencer
    Mar 24, 2005
  2. Andrew Robert

    regex/lambda black magic

    Andrew Robert, May 25, 2006, in forum: Python
    Replies:
    5
    Views:
    426
    John Machin
    May 25, 2006
  3. fdm
    Replies:
    18
    Views:
    723
    Balog Pal
    Oct 5, 2009
  4. Tom Willis

    Ruby black magic? Meta Programming

    Tom Willis, Mar 12, 2005, in forum: Ruby
    Replies:
    4
    Views:
    374
    Mathieu Bouchard
    Mar 13, 2005
  5. Jon
    Replies:
    5
    Views:
    136
    Brian Candler
    Mar 31, 2007
Loading...

Share This Page