Import a module without executing it?

[Jay O'Connor]

Is there a good way to import python files without executing their content?

I'm trying some relfection based stuff and I want to be able to import a
module dynamically to check it's contents (class ad functions defined)
but without having any of the content executed.

For example:
----------------------
def test(var):
print var


#main
test(3)
----------------------

I want to be able to import this module so I can see "ah ha, this module
defines a function called 'test'", but I don't want the code at the
bottom executed during the import.

Thanks

Take care,
Jay

 

[Andy Gross]

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg

 

[Andy Gross]

Here's a quick example that will pull out all functions defined in the
top-level of a module:

---
#/usr/bin/env python
from compiler import parse, walk
from compiler.visitor import ASTVisitor

testdata = r'''
def aFunction(anArg):
return anArg + 1
'''
class SimpleVisitor(ASTVisitor):
def visitFunction(self, parsedFunc):
print "Function %(name)s at %(lineno)s takes %(argnames)s " \
" with code %(code)s" % parsedFunc.__dict__

if __name__ == "__main__":
ast = parse(testdata)
walk(ast, SimpleVisitor(), verbose=True)
---

andy@alyosha:~$ ./test.py
Function aFunction at 2 takes ['anArg'] with code
Stmt([Return(Add((Name('anArg'), Const(1))))])

HTH,

/arg

Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None,
Stmt([Discard(CallFunc(Getattr(Name('w'), 'configure'),
[Keyword('state', Const('normal'))], None, None)), For(AssName('r',
'OP_ASSIGN'), Name('rows'), Stmt([For(AssTuple([AssName('t',
'OP_ASSIGN'), AssName('v', 'OP_ASSIGN')]), CallFunc(Name('zip'),
[Name('titles'), Name('r')], None, None),
Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None,
None))]), None), Discard(CallFunc(Getattr(Name('w'), 'insert'),
[Const('end'), Const('\n')], None, None))]), None),
Discard(CallFunc(Getattr(Name('w'), 'configure'), [Keyword('state',
Const('disabled'))], None, None))])), Assign([AssName('app',
'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'), 'Tk'), [], None,
None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'),
Keyword('state', Const('disabled'))], None, None)),
Discard(CallFunc(Getattr(Name('t'), 'pack'), [], None, None)),
Assign([AssName('info', 'OP_ASSIGN')], List([List([Const('Ali'),
Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])),
Discard(CallFunc(Name('add_rows'), [Name('t'), List([Const('Name'),
Const('Age')]), Name('info')], None, None)),
Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None, None))])
***

Pretty impressive

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg

 

[Steven Bethard]

----------------------
def test(var):
print var


#main
test(3)
----------------------

I want to be able to import this module so I can see "ah ha, this module
defines a function called 'test'", but I don't want the code at the
bottom executed during the import.

If you have source control over this file, you could write it with the
more standard idiom:

def test(var):
print var

if __name__ == "__main__":
test(3)

Then when the module is imported, only the def statement gets executed,
not the 'test(3)'. Of course, if you don't have source control over the
file, you can't do this...


Also note that code that is not protected by an
if __name__ == "__main__":
test may be part of the module definition, so examining a module without
executing this code may be misleading, e.g.:

def test(var):
print var

globals()['test'] = type('C', (object,), dict(
f=lambda self, var, test=test: test(var)))

if __name__ == "__main__":
test().f(3)

The module above turns 'test' from a function into a class, so if all
you do is look at the def statements, you may misinterpret the module.

Steve

 

[Andy Gross]

If you have source control over this file, you could write it with the
more standard idiom...

I should have mentioned this first. If you're just trying to avoid
existing top-level code from being executed, use the if __name__ ==
"__main__" idiom.

 

[finite.automaton]

Functions and classes are created during the very execution you're
trying to skip so there's no precise way to do what you want.

That said, you can parse the code without executing it, and that will
give you some information about defined functions and classes. It will
_not_ give you actual function objects; the only way to get those is to
execute the code. It will also not catch anything created "on the
fly"*, e.g. "exec 'def func(x): pass'"


# example:

def defined_functions(code_text):
module_ast = parse(code_text)
return [statement for statement in
module_ast.node.nodes if
isinstance(statement, ast.Function)]

# read the module's source code
test_string = """
def foo(x):
pass
def bar(y):
pass
"""

# Which functions are defined in the test string?
function_asts = defined_functions(test_string)
print "Defined functions:", [f.name for f in function_asts]




* - Okay, everything in Python happens on-the-fly. But you know what I
mean.

Footnote: Wow. The new, "improved" google groups 2 beta has totally
annihilated my indentation. Sorry about that. Hopefully you can still
figure it out.

 

[Jay O'Connor]

Functions and classes are created during the very execution you're
trying to skip so there's no precise way to do what you want.

That said, you can parse the code without executing it, and that will
give you some information about defined functions and classes. It will
_not_ give you actual function objects; the only way to get those is to
execute the code. It will also not catch anything created "on the
fly"*, e.g. "exec 'def func(x): pass'"

OK, some more information on what I'm tryng to do would help.

I come from a strong Smalltalk background and in most Smalltalk IDEs is
the capability to browse "Implementors" (what classes implement a method
with a given name or like a given name) and "Senders" (from what methods
is a given method name called)

Given a set of code, not all of which I am familiar with, I wanted to
have similar explorative capabilities in understanding the code. grep
in Linux does a decent job, but under Windows, the file search (look for
a particular string in a py file) does not seem to be too..useful (I
don't get back the results I should)

I shied away from just hand-parsing the code because knowing that a file
has a string, or even a function with a given name, in it is not hard,
knowing that the module has a class that has the function (and knowing
the class as well) is more useful

So, in Smalltalk, the way you find out about what's in the system is
reflection...you ask the classes what methods they implement (if looking
for 'implementors'), you ask the methods for the source and can string
search the code (if looking for senders).

Knowing that both modules and classes (and functions) had reality enough
to be queried, that was my first line of thought, to load the module and
query the module and query the classes in the module, etc... and do
various matches against whatever I was looking for.

It worked fine for a file that just defined functions and classes.
However, files that had code not contained in classes or functions
caused a bit of a problem because loading the module would execute that
code and that's not what I wanted.

So therefore, my original question.

The real question, I suppose, is "what is a good technique to find what
modules and classes implement or refer to particular names"

Take care,
Jay

 

[Andy Gross]

So, in Smalltalk, the way you find out about what's in the system is

reflection...you ask the classes what methods they implement (if
looking for 'implementors'), you ask the methods for the source and
can string search the code (if looking for senders).

There's no real way to get the source code from a compiled code object
, although there are some tools, like decompyle, which will try for
you. I think some people on this list were discussing the potential
utility of having a __source__ attribute for code objects, but Python
currently doesn't do this. You can probably also get some mileage out
of the inspect module (try 'pydoc inspect'), but that *does* compile
the code you're inspecting.

It worked fine for a file that just defined functions and classes.
However, files that had code not contained in classes or functions
caused a bit of a problem because loading the module would execute
that code and that's not what I wanted.

To get at top-level module code like this, you'll have to parse the
file and walk the AST - there's no other way. In a well implemented
library or app, however, there typically shouldn't be much important
code being executed at the top level. Typically the top-level code
will be initialization of singleton globals or decorator-type function
wrappers, and you can get at most of the important names reflectively
through the inspect module.

/arg

 

[Lonnie Princehouse]

The real question, I suppose, is "what is a good technique to find
what

modules and classes implement or refer to particular names"

I think your best bet is still to import the module and introspect it.
It will execute some code, but (by convention) simply importing a
module doesn't usually unleash any spectacular computation. Scripts
meant to be executed from the command line will almost always have an
(if __name__ == '__main__' ) clause to prevent just this.

A search of this newsgroup for "duck" or "duck typing" will probably
turn up more info on how people approximate the "X implements Y"
relationship in Python.

On the other hand, if you're just trying to figure out what a black-box
object does from the interpreter's command line, you can use help(),
dir() or pydoc amongst others. "import pydoc; pydoc.gui()"

There are some functions that will be handy for determining properties
of an unknown object:
callable, eval, hasattr, isinstance, type, super

And most objects will have a few magic attributes-['__call__', '__class__', '__delattr__', '__dict__', '__doc__',
'__get__', '__getattribute__', '__hash__', '__init__', '__module__',
'__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__str__', 'func_closure', 'func_code', 'func_defaults',
'func_dict', 'func_doc', 'func_globals', 'func_name']('x','y')

 

[Jay O'Connor]

I think your best bet is still to import the module and introspect it.
It will execute some code, but (by convention) simply importing a
module doesn't usually unleash any spectacular computation. Scripts
meant to be executed from the command line will almost always have an
(if __name__ == '__main__' ) clause to prevent just this.

Well, I sorta ran into that trouble testing some testfiles of
mine...little scripts for testing specific things that did'nt follow
convension in that manner

Thanks

Take care,
Jay

 

[Kent Johnson]

The real question, I suppose, is "what is a good technique to find what
modules and classes implement or refer to particular names"

You might like to try ctags. I have had a good experience with it. It's not as automatic as I would
like - you have to build a cross-reference table before you can use it - and it only finds
implementors, not referrers. It integrates with lots of editors, I have used it with TextPad.
http://ctags.sourceforge.net/

Kent

 

[Caleb Hattingh]

Hi

You could just parse the model file. Off the top of my head

***
f = open('ModuleYouWantToExamine.py','r')
for i in f:
if i.find('def ') > -1:
print 'Found a function!: '+i.replace('def ','')

f.close()
***

You would have to build this up for a more complete examination. Of
course, one of the guru's around here should be able to give you guidance
regarding actually parsing the file with the interpreter (not executing)
and building a dict or something with all the different types of
constructs. That's not me

 

[Caleb Hattingh]

Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None, Stmt([Discard(CallFunc(Getattr(Name('w'),
'configure'), [Keyword('state', Const('normal'))], None, None)),
For(AssName('r', 'OP_ASSIGN'), Name('rows'),
Stmt([For(AssTuple([AssName('t', 'OP_ASSIGN'), AssName('v',
'OP_ASSIGN')]), CallFunc(Name('zip'), [Name('titles'), Name('r')], None,
None), Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None, None))]),
None), Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Const('\n')], None, None))]), None), Discard(CallFunc(Getattr(Name('w'),
'configure'), [Keyword('state', Const('disabled'))], None, None))])),
Assign([AssName('app', 'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'),
'Tk'), [], None, None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'), Keyword('state',
Const('disabled'))], None, None)), Discard(CallFunc(Getattr(Name('t'),
'pack'), [], None, None)), Assign([AssName('info', 'OP_ASSIGN')],
List([List([Const('Ali'), Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])), Discard(CallFunc(Name('add_rows'),
[Name('t'), List([Const('Name'), Const('Age')]), Name('info')], None,
None)), Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None,
None))])
***

Pretty impressive

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb

 

[Kent Johnson]

Andy, this is a nice example. It prompted me to look at the docs for compiler.visitor. The docs are,
um, pretty bad. I'm going to attempt to clean them up a little. Would you mind if I include this
example?

Thanks,
Kent

Here's a quick example that will pull out all functions defined in the
top-level of a module:

---
#/usr/bin/env python
from compiler import parse, walk
from compiler.visitor import ASTVisitor

testdata = r'''
def aFunction(anArg):
return anArg + 1
'''
class SimpleVisitor(ASTVisitor):
def visitFunction(self, parsedFunc):
print "Function %(name)s at %(lineno)s takes %(argnames)s " \
" with code %(code)s" % parsedFunc.__dict__

if __name__ == "__main__":
ast = parse(testdata)
walk(ast, SimpleVisitor(), verbose=True)
---

andy@alyosha:~$ ./test.py
Function aFunction at 2 takes ['anArg'] with code
Stmt([Return(Add((Name('anArg'), Const(1))))])

HTH,

/arg

Andy

thx for that. I had a file called 'tktest.py' lying around, and I did:

'>>> a = compiler.parseFile('tktest.py')

And "a" looks something like this:

***
Stmt([Import([('Tkinter', None)]), Function(None, 'add_rows', ['w',
'titles', 'rows'], [], 0, None,
Stmt([Discard(CallFunc(Getattr(Name('w'), 'configure'),
[Keyword('state', Const('normal'))], None, None)), For(AssName('r',
'OP_ASSIGN'), Name('rows'), Stmt([For(AssTuple([AssName('t',
'OP_ASSIGN'), AssName('v', 'OP_ASSIGN')]), CallFunc(Name('zip'),
[Name('titles'), Name('r')], None, None),
Stmt([Discard(CallFunc(Getattr(Name('w'), 'insert'), [Const('end'),
Mod((Const('%s:\t%s\n'), Tuple([Name('t'), Name('v')])))], None,
None))]), None), Discard(CallFunc(Getattr(Name('w'), 'insert'),
[Const('end'), Const('\n')], None, None))]), None),
Discard(CallFunc(Getattr(Name('w'), 'configure'), [Keyword('state',
Const('disabled'))], None, None))])), Assign([AssName('app',
'OP_ASSIGN')], CallFunc(Getattr(Name('Tkinter'), 'Tk'), [], None,
None)), Assign([AssName('t', 'OP_ASSIGN')],
CallFunc(Getattr(Name('Tkinter'), 'Text'), [Name('app'),
Keyword('state', Const('disabled'))], None, None)),
Discard(CallFunc(Getattr(Name('t'), 'pack'), [], None, None)),
Assign([AssName('info', 'OP_ASSIGN')], List([List([Const('Ali'),
Const(18)]), List([Const('Zainab'), Const(16)]),
List([Const('Khalid'), Const(18)])])),
Discard(CallFunc(Name('add_rows'), [Name('t'), List([Const('Name'),
Const('Age')]), Name('info')], None, None)),
Discard(CallFunc(Getattr(Name('app'), 'mainloop'), [], None, None))])
***

Pretty impressive

Do you know of more batteries that can process this stuff further, for
interest sake (and maybe the OP)?

thx again
Caleb

You'll want to use the "compiler" package. compiler.parseFile will
return an AST that you can inspect (which is not really 'reflection',
btw).

/arg

 

[Andy Gross]

Andy, this is a nice example. It prompted me to look at the docs for
compiler.visitor. The docs are, um, pretty bad. I'm going to attempt
to clean them up a little. Would you mind if I include this example?

Be my guest!

/arg