Import a module without executing it?

Discussion in 'Python' started by Jay O'Connor, Dec 7, 2004.

  1. Jay O'Connor

    Jay O'Connor Guest

    Is there a good way to import python files without executing their content?

    I'm trying some relfection based stuff and I want to be able to import a
    module dynamically to check it's contents (class ad functions defined)
    but without having any of the content executed.

    For example:
    ----------------------
    def test(var):
    print var


    #main
    test(3)
    ----------------------

    I want to be able to import this module so I can see "ah ha, this module
    defines a function called 'test'", but I don't want the code at the
    bottom executed during the import.

    Thanks

    Take care,
    Jay
     
    Jay O'Connor, Dec 7, 2004
    #1
    1. Advertisements

  2. Jay O'Connor

    Andy Gross Guest

    You'll want to use the "compiler" package. compiler.parseFile will
    return an AST that you can inspect (which is not really 'reflection',
    btw).

    /arg
     
    Andy Gross, Dec 7, 2004
    #2
    1. Advertisements

  3. Jay O'Connor

    Andy Gross Guest

    Here's a quick example that will pull out all functions defined in the
    top-level of a module:

    ---
    #/usr/bin/env python
    from compiler import parse, walk
    from compiler.visitor import ASTVisitor

    testdata = r'''
    def aFunction(anArg):
    return anArg + 1
    '''
    class SimpleVisitor(ASTVisitor):
    def visitFunction(self, parsedFunc):
    print "Function %(name)s at %(lineno)s takes %(argnames)s " \
    " with code %(code)s" % parsedFunc.__dict__

    if __name__ == "__main__":
    ast = parse(testdata)
    walk(ast, SimpleVisitor(), verbose=True)
    ---

    [email protected]:~$ ./test.py
    Function aFunction at 2 takes ['anArg'] with code
    Stmt([Return(Add((Name('anArg'), Const(1))))])

    HTH,

    /arg


     
    Andy Gross, Dec 7, 2004
    #3
  4. If you have source control over this file, you could write it with the
    more standard idiom:

    def test(var):
    print var

    if __name__ == "__main__":
    test(3)

    Then when the module is imported, only the def statement gets executed,
    not the 'test(3)'. Of course, if you don't have source control over the
    file, you can't do this...


    Also note that code that is not protected by an
    if __name__ == "__main__":
    test may be part of the module definition, so examining a module without
    executing this code may be misleading, e.g.:

    def test(var):
    print var

    globals()['test'] = type('C', (object,), dict(
    f=lambda self, var, test=test: test(var)))

    if __name__ == "__main__":
    test().f(3)

    The module above turns 'test' from a function into a class, so if all
    you do is look at the def statements, you may misinterpret the module.

    Steve
     
    Steven Bethard, Dec 7, 2004
    #4
  5. Jay O'Connor

    Andy Gross Guest

    I should have mentioned this first. If you're just trying to avoid
    existing top-level code from being executed, use the if __name__ ==
    "__main__" idiom.
     
    Andy Gross, Dec 7, 2004
    #5
  6. Functions and classes are created during the very execution you're
    trying to skip so there's no precise way to do what you want.

    That said, you can parse the code without executing it, and that will
    give you some information about defined functions and classes. It will
    _not_ give you actual function objects; the only way to get those is to
    execute the code. It will also not catch anything created "on the
    fly"*, e.g. "exec 'def func(x): pass'"


    # example:

    def defined_functions(code_text):
    module_ast = parse(code_text)
    return [statement for statement in
    module_ast.node.nodes if
    isinstance(statement, ast.Function)]

    # read the module's source code
    test_string = """
    def foo(x):
    pass
    def bar(y):
    pass
    """

    # Which functions are defined in the test string?
    function_asts = defined_functions(test_string)
    print "Defined functions:", [f.name for f in function_asts]




    * - Okay, everything in Python happens on-the-fly. But you know what I
    mean.

    Footnote: Wow. The new, "improved" google groups 2 beta has totally
    annihilated my indentation. Sorry about that. Hopefully you can still
    figure it out.
     
    finite.automaton, Dec 7, 2004
    #6
  7. Jay O'Connor

    Jay O'Connor Guest


    OK, some more information on what I'm tryng to do would help.

    I come from a strong Smalltalk background and in most Smalltalk IDEs is
    the capability to browse "Implementors" (what classes implement a method
    with a given name or like a given name) and "Senders" (from what methods
    is a given method name called)

    Given a set of code, not all of which I am familiar with, I wanted to
    have similar explorative capabilities in understanding the code. grep
    in Linux does a decent job, but under Windows, the file search (look for
    a particular string in a py file) does not seem to be too..useful (I
    don't get back the results I should)

    I shied away from just hand-parsing the code because knowing that a file
    has a string, or even a function with a given name, in it is not hard,
    knowing that the module has a class that has the function (and knowing
    the class as well) is more useful

    So, in Smalltalk, the way you find out about what's in the system is
    reflection...you ask the classes what methods they implement (if looking
    for 'implementors'), you ask the methods for the source and can string
    search the code (if looking for senders).

    Knowing that both modules and classes (and functions) had reality enough
    to be queried, that was my first line of thought, to load the module and
    query the module and query the classes in the module, etc... and do
    various matches against whatever I was looking for.

    It worked fine for a file that just defined functions and classes.
    However, files that had code not contained in classes or functions
    caused a bit of a problem because loading the module would execute that
    code and that's not what I wanted.

    So therefore, my original question.

    The real question, I suppose, is "what is a good technique to find what
    modules and classes implement or refer to particular names"

    Take care,
    Jay
     
    Jay O'Connor, Dec 7, 2004
    #7
  8. Jay O'Connor

    Andy Gross Guest

    So, in Smalltalk, the way you find out about what's in the system is
    There's no real way to get the source code from a compiled code object
    , although there are some tools, like decompyle, which will try for
    you. I think some people on this list were discussing the potential
    utility of having a __source__ attribute for code objects, but Python
    currently doesn't do this. You can probably also get some mileage out
    of the inspect module (try 'pydoc inspect'), but that *does* compile
    the code you're inspecting.
    To get at top-level module code like this, you'll have to parse the
    file and walk the AST - there's no other way. In a well implemented
    library or app, however, there typically shouldn't be much important
    code being executed at the top level. Typically the top-level code
    will be initialization of singleton globals or decorator-type function
    wrappers, and you can get at most of the important names reflectively
    through the inspect module.

    /arg
     
    Andy Gross, Dec 7, 2004
    #8
  9. The real question, I suppose, is "what is a good technique to find
    what
    I think your best bet is still to import the module and introspect it.
    It will execute some code, but (by convention) simply importing a
    module doesn't usually unleash any spectacular computation. Scripts
    meant to be executed from the command line will almost always have an
    (if __name__ == '__main__' ) clause to prevent just this.

    A search of this newsgroup for "duck" or "duck typing" will probably
    turn up more info on how people approximate the "X implements Y"
    relationship in Python.

    On the other hand, if you're just trying to figure out what a black-box
    object does from the interpreter's command line, you can use help(),
    dir() or pydoc amongst others. "import pydoc; pydoc.gui()"

    There are some functions that will be handy for determining properties
    of an unknown object:
    callable, eval, hasattr, isinstance, type, super

    And most objects will have a few magic attributes-['__call__', '__class__', '__delattr__', '__dict__', '__doc__',
    '__get__', '__getattribute__', '__hash__', '__init__', '__module__',
    '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
    '__setattr__', '__str__', 'func_closure', 'func_code', 'func_defaults',
    'func_dict', 'func_doc', 'func_globals', 'func_name']('x','y')
     
    Lonnie Princehouse, Dec 8, 2004
    #9
  10. Jay O'Connor

    Jay O'Connor Guest

    Well, I sorta ran into that trouble testing some testfiles of
    mine...little scripts for testing specific things that did'nt follow
    convension in that manner

    Thanks

    Take care,
    Jay
     
    Jay O'Connor, Dec 8, 2004
    #10
  11. Jay O'Connor

    Kent Johnson Guest

    You might like to try ctags. I have had a good experience with it. It's not as automatic as I would
    like - you have to build a cross-reference table before you can use it - and it only finds
    implementors, not referrers. It integrates with lots of editors, I have used it with TextPad.
    http://ctags.sourceforge.net/

    Kent
     
    Kent Johnson, Dec 8, 2004
    #11
  12. Hi

    You could just parse the model file. Off the top of my head

    ***
    f = open('ModuleYouWantToExamine.py','r')
    for i in f:
    if i.find('def ') > -1:
    print 'Found a function!: '+i.replace('def ','')

    f.close()
    ***

    You would have to build this up for a more complete examination. Of
    course, one of the guru's around here should be able to give you guidance
    regarding actually parsing the file with the interpreter (not executing)
    and building a dict or something with all the different types of
    constructs. That's not me :)
     
    Caleb Hattingh, Dec 8, 2004
    #12
  13. Andy

    thx for that. I had a file called 'tktest.py' lying around, and I did:

    '>>> a = compiler.parseFile('tktest.py')

    And "a" looks something like this:

    ***
    Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
    'titles', 'rows'], [], 0, None, Stmt([Discard(CallFunc(Getattr(Name('w'),
    'configure'), [Keyword('state', Const('normal'))], None, None)),
    For(AssName('r', 'OP_ASSIGN'), Name('rows'),
    Stmt([For(AssTuple([AssName('t', 'OP_ASSIGN'), AssName('v',
    'OP_ASSIGN')]), CallFunc(Name('zip'), [Name('titles'), Name('r')], None,
    None), Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
    Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None, None))]),
    None), Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
    Const('\n')], None, None))]), None), Discard(CallFunc(Getattr(Name('w'),
    'configure'), [Keyword('state', Const('disabled'))], None, None))])),
    Assign([AssName('app', 'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'),
    'Tk'), [], None, None)), Assign([AssName('t', 'OP_ASSIGN')],
    CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'), Keyword('state',
    Const('disabled'))], None, None)), Discard(CallFunc(Getattr(Name('t'),
    'pack'), [], None, None)), Assign([AssName('info', 'OP_ASSIGN')],
    List([List([Const('Ali'), Const(18)]), List([Const('Zainab'), Const(16)]),
    List([Const('Khalid'), Const(18)])])), Discard(CallFunc(Name('add_rows'),
    [Name('t'), List([Const('Name'), Const('Age')]), Name('info')], None,
    None)), Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None,
    None))])
    ***

    Pretty impressive :)

    Do you know of more batteries that can process this stuff further, for
    interest sake (and maybe the OP)?

    thx again
    Caleb
     
    Caleb Hattingh, Dec 8, 2004
    #13
  14. Jay O'Connor

    Kent Johnson Guest

    Andy, this is a nice example. It prompted me to look at the docs for compiler.visitor. The docs are,
    um, pretty bad. I'm going to attempt to clean them up a little. Would you mind if I include this
    example?

    Thanks,
    Kent

     
    Kent Johnson, Dec 8, 2004
    #14
  15. Jay O'Connor

    Andy Gross Guest

    Be my guest!

    /arg
     
    Andy Gross, Dec 8, 2004
    #15
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.