Finding a module's sub modules at runtime

J

Joshua J. Kugler

[If this is documented somewhere, please just point me there. I googled on
the terms that made sense to me, and didn't find anything.]

So, I have:

ModTest
__init__.py
AModule.py
BModule.py
CModule.py

All works fine. However, when I import ModTest, I would like it to discover
and store the names of the modules beneath it, and construct a list, say
mod_list, that I can access later to find the names of the sub-modules in
this module. Kind of setting __all__ at run time, I guess (yes, I'm aware
of the case caveats).

I figured __init__.py coudl take its own __path__ and walk the directory to
find all .py files other than __init__.py, but that seemed hackish. Is
there an "official" way to do this? Or a better way?

To give "context:" all the modules will have classes that have the same
name, same methods etc. One of the modules will be picked depending on
which implementation is needed.

Thanks!

j
 
K

kyosohma

[If this is documented somewhere, please just point me there. I googled on
the terms that made sense to me, and didn't find anything.]

So, I have:

ModTest
__init__.py
AModule.py
BModule.py
CModule.py

All works fine. However, when I import ModTest, I would like it to discover
and store the names of the modules beneath it, and construct a list, say
mod_list, that I can access later to find the names of the sub-modules in
this module. Kind of setting __all__ at run time, I guess (yes, I'm aware
of the case caveats).

I figured __init__.py coudl take its own __path__ and walk the directory to
find all .py files other than __init__.py, but that seemed hackish. Is
there an "official" way to do this? Or a better way?

To give "context:" all the modules will have classes that have the same
name, same methods etc. One of the modules will be picked depending on
which implementation is needed.

Thanks!

j

I think you need to research how to create documentation. When I
import a module/package, I can then type help(moduleName) and it'll
give me the module or package's contents.

http://mail.python.org/pipermail/python-list/2003-February/192069.html

You may be able to figure out how to do this just be studying the
"help" module itself.

Mike
 
J

Joshua J. Kugler

I think you need to research how to create documentation. When I
import a module/package, I can then type help(moduleName) and it'll
give me the module or package's contents.

http://mail.python.org/pipermail/python-list/2003-February/192069.html

You may be able to figure out how to do this just be studying the
"help" module itself.

Mike

Well, it seems the "help" module is a built in, and has no .py file
anywhere. The epydoc package uses the imports.file_modules(dirname)
function, which just walks the directory tree. Thanks for the
pointer...just confirmed that I have to do something I wanted to avoid.
But I guess if I use Python's os module and the __path__ string, it should
still be nicely portable. It just seems that since Python is gathering
that information anyway, it should make it available without me having to
walk the directory tree.

j
 
A

Alex Martelli

Joshua J. Kugler said:
still be nicely portable. It just seems that since Python is gathering
that information anyway, it should make it available without me having to
walk the directory tree.

Sorry, where is Python "gathering that information anyway"? Unless I'm
mistaken, Python by default does not walk directory trees of subpackages
-- what makes you think it does?


Alex
 
J

Joshua J. Kugler

Sorry, where is Python "gathering that information anyway"? Unless I'm
mistaken, Python by default does not walk directory trees of subpackages
-- what makes you think it does?

Hmm...right: dynamic language, runtime binding. It would go out and find
whether or not Module1.Module2 existed only when import was called. Sorry,
my bad. I am assuming, however, that the help function walks the directory
tree, or else I would not get output like this:
Help on package ModTest:

NAME
ModTest

FILE
/usr/local/lib/python2.4/site-packages/ModTest/__init__.py

PACKAGE CONTENTS
AModule
BModule
CModule

Thanks for the clarification.

j
 
A

Alex Martelli

Joshua J. Kugler said:
Hmm...right: dynamic language, runtime binding. It would go out and find
whether or not Module1.Module2 existed only when import was called. Sorry,
my bad. I am assuming, however, that the help function walks the directory
tree, or else I would not get output like this:

Help on package ModTest:

NAME
ModTest

FILE
/usr/local/lib/python2.4/site-packages/ModTest/__init__.py

PACKAGE CONTENTS
AModule
BModule
CModule

Thanks for the clarification.

Sure, pydoc (which help calls under the code) does that, with a nice mix
of inspect, os, and pkgutil.iter_modules calls. pkgutil.iter_modules
may in fact be most of what you need:
Help on function iter_modules in module pkgutil:

iter_modules(path=None, prefix='')
Yields (module_loader, name, ispkg) for all submodules on path,
or, if path is None, all top-level modules on sys.path.

'path' should be either None or a list of paths to look for
modules in.

'prefix' is a string to output on the front of every module name
on output.

You will still have to call it recursively, though, I believe.


Alex
 
J

Joshua J. Kugler

Sure, pydoc (which help calls under the code) does that, with a nice mix
of inspect, os, and pkgutil.iter_modules calls. pkgutil.iter_modules
may in fact be most of what you need:

Help on function iter_modules in module pkgutil:

iter_modules(path=None, prefix='')
Yields (module_loader, name, ispkg) for all submodules on path,
or, if path is None, all top-level modules on sys.path.

'path' should be either None or a list of paths to look for
modules in.

'prefix' is a string to output on the front of every module name
on output.

OK, that looks nice...but what version of Python is that?
http://docs.python.org/lib/module-pkgutil.html only shows one function (and
that's 2.5) and my python 2.4 installation is similarly lacking an
iter_modules() function for the pkgutil module. Is this a 2.6 thing?

Suppose I could just copy the code from here:
http://mail.python.org/pipermail/python-checkins/2006-April/051452.html and
add it to my module. :)

Thanks for the pointer!

j
 
R

Robert Kern

Joshua said:
OK, that looks nice...but what version of Python is that?
http://docs.python.org/lib/module-pkgutil.html only shows one function (and
that's 2.5) and my python 2.4 installation is similarly lacking an
iter_modules() function for the pkgutil module. Is this a 2.6 thing?

No, 2.5. The documentation is not up to date. Read the source.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
J

Joshua J. Kugler

No, 2.5. The documentation is not up to date. Read the source.

Gotcha. Thanks...well, since we're using 2.4, that will have to wait. For
the archives, here is what I've come up with.

Contents of the __init__.py for a module.

import os
_myDir = __path__[0]

def mod_list():
"""
A quick hack that retrieves all the sub modules in a directory that has
an __init__.py file. I could use pkgutil.iter_modules, but that is
Python 2.5 only, and this should work with several versions of Python.
"""
modList = []
modHash = {}
isModule = False
for ii in os.walk(_myDir):
if ii[0] == _myDir:
for f in ii[2]:
# If there is no __init__ file, then the directory
# upon which mod_list() is operating is not a module
if f[0:8] == '__init__':
isModule = True
elif f[-3:] == '.py':
modHash[f[:-3]] = True
elif f[-4:] == '.pyc' or f[-4:] == '.pyo':
modHash[f[:-4]] = True
if isModule:
modList = modHash.keys()
modList.sort()
return(modList)
else:
return(None)

Hope that helps someone!

j
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top