Finding a module's sub modules at runtime

Discussion in 'Python' started by Joshua J. Kugler, Mar 28, 2007.

  1. [If this is documented somewhere, please just point me there. I googled on
    the terms that made sense to me, and didn't find anything.]

    So, I have:

    ModTest
    __init__.py
    AModule.py
    BModule.py
    CModule.py

    All works fine. However, when I import ModTest, I would like it to discover
    and store the names of the modules beneath it, and construct a list, say
    mod_list, that I can access later to find the names of the sub-modules in
    this module. Kind of setting __all__ at run time, I guess (yes, I'm aware
    of the case caveats).

    I figured __init__.py coudl take its own __path__ and walk the directory to
    find all .py files other than __init__.py, but that seemed hackish. Is
    there an "official" way to do this? Or a better way?

    To give "context:" all the modules will have classes that have the same
    name, same methods etc. One of the modules will be picked depending on
    which implementation is needed.

    Thanks!

    j

    --
    Joshua Kugler
    Lead System Admin -- Senior Programmer
    http://www.eeinternet.com
    PGP Key: http://pgp.mit.edu/  ID 0xDB26D7CE

    --
    Posted via a free Usenet account from http://www.teranews.com
    Joshua J. Kugler, Mar 28, 2007
    #1
    1. Advertising

  2. Joshua J. Kugler

    Guest

    On Mar 28, 2:44 pm, "Joshua J. Kugler" <> wrote:
    > [If this is documented somewhere, please just point me there. I googled on
    > the terms that made sense to me, and didn't find anything.]
    >
    > So, I have:
    >
    > ModTest
    > __init__.py
    > AModule.py
    > BModule.py
    > CModule.py
    >
    > All works fine. However, when I import ModTest, I would like it to discover
    > and store the names of the modules beneath it, and construct a list, say
    > mod_list, that I can access later to find the names of the sub-modules in
    > this module. Kind of setting __all__ at run time, I guess (yes, I'm aware
    > of the case caveats).
    >
    > I figured __init__.py coudl take its own __path__ and walk the directory to
    > find all .py files other than __init__.py, but that seemed hackish. Is
    > there an "official" way to do this? Or a better way?
    >
    > To give "context:" all the modules will have classes that have the same
    > name, same methods etc. One of the modules will be picked depending on
    > which implementation is needed.
    >
    > Thanks!
    >
    > j
    >
    > --
    > Joshua Kugler
    > Lead System Admin -- Senior Programmerhttp://www.eeinternet.com
    > PGP Key:http://pgp.mit.edu/ ID 0xDB26D7CE
    >
    > --
    > Posted via a free Usenet account fromhttp://www.teranews.com


    I think you need to research how to create documentation. When I
    import a module/package, I can then type help(moduleName) and it'll
    give me the module or package's contents.

    http://mail.python.org/pipermail/python-list/2003-February/192069.html

    You may be able to figure out how to do this just be studying the
    "help" module itself.

    Mike
    , Mar 28, 2007
    #2
    1. Advertising

  3. On Wednesday 28 March 2007 12:04, wrote:
    >> All works fine. However, when I import ModTest, I would like it to
    >> discover and store the names of the modules beneath it, and construct a
    >> list, say mod_list, that I can access later to find the names of the
    >> sub-modules in
    >> this module. Kind of setting __all__ at run time, I guess (yes, I'm
    >> aware of the case caveats).
    >>
    >> I figured __init__.py coudl take its own __path__ and walk the directory
    >> to
    >> find all .py files other than __init__.py, but that seemed hackish. Is
    >> there an "official" way to do this? Or a better way?
    >>
    >> To give "context:" all the modules will have classes that have the same
    >> name, same methods etc. One of the modules will be picked depending on
    >> which implementation is needed.

    >
    > I think you need to research how to create documentation. When I
    > import a module/package, I can then type help(moduleName) and it'll
    > give me the module or package's contents.
    >
    > http://mail.python.org/pipermail/python-list/2003-February/192069.html
    >
    > You may be able to figure out how to do this just be studying the
    > "help" module itself.
    >
    > Mike


    Well, it seems the "help" module is a built in, and has no .py file
    anywhere. The epydoc package uses the imports.file_modules(dirname)
    function, which just walks the directory tree. Thanks for the
    pointer...just confirmed that I have to do something I wanted to avoid.
    But I guess if I use Python's os module and the __path__ string, it should
    still be nicely portable. It just seems that since Python is gathering
    that information anyway, it should make it available without me having to
    walk the directory tree.

    j

    --
    Joshua Kugler
    Lead System Admin -- Senior Programmer
    http://www.eeinternet.com
    PGP Key: http://pgp.mit.edu/  ID 0xDB26D7CE

    --
    Posted via a free Usenet account from http://www.teranews.com
    Joshua J. Kugler, Mar 28, 2007
    #3
  4. Joshua J. Kugler <> wrote:

    > still be nicely portable. It just seems that since Python is gathering
    > that information anyway, it should make it available without me having to
    > walk the directory tree.


    Sorry, where is Python "gathering that information anyway"? Unless I'm
    mistaken, Python by default does not walk directory trees of subpackages
    -- what makes you think it does?


    Alex
    Alex Martelli, Mar 29, 2007
    #4
  5. On Thursday 29 March 2007 07:33, Alex Martelli wrote:

    > Joshua J. Kugler <> wrote:
    >
    >> still be nicely portable. It just seems that since Python is gathering
    >> that information anyway, it should make it available without me having to
    >> walk the directory tree.

    >
    > Sorry, where is Python "gathering that information anyway"? Unless I'm
    > mistaken, Python by default does not walk directory trees of subpackages
    > -- what makes you think it does?


    Hmm...right: dynamic language, runtime binding. It would go out and find
    whether or not Module1.Module2 existed only when import was called. Sorry,
    my bad. I am assuming, however, that the help function walks the directory
    tree, or else I would not get output like this:

    >>> import ModTest
    >>> help(ModTest)

    Help on package ModTest:

    NAME
    ModTest

    FILE
    /usr/local/lib/python2.4/site-packages/ModTest/__init__.py

    PACKAGE CONTENTS
    AModule
    BModule
    CModule

    Thanks for the clarification.

    j

    --
    Joshua Kugler
    Lead System Admin -- Senior Programmer
    http://www.eeinternet.com
    PGP Key: http://pgp.mit.edu/  ID 0xDB26D7CE

    --
    Posted via a free Usenet account from http://www.teranews.com
    Joshua J. Kugler, Mar 29, 2007
    #5
  6. Joshua J. Kugler <> wrote:

    > On Thursday 29 March 2007 07:33, Alex Martelli wrote:
    >
    > > Joshua J. Kugler <> wrote:
    > >
    > >> still be nicely portable. It just seems that since Python is gathering
    > >> that information anyway, it should make it available without me having to
    > >> walk the directory tree.

    > >
    > > Sorry, where is Python "gathering that information anyway"? Unless I'm
    > > mistaken, Python by default does not walk directory trees of subpackages
    > > -- what makes you think it does?

    >
    > Hmm...right: dynamic language, runtime binding. It would go out and find
    > whether or not Module1.Module2 existed only when import was called. Sorry,
    > my bad. I am assuming, however, that the help function walks the directory
    > tree, or else I would not get output like this:
    >
    > >>> import ModTest
    > >>> help(ModTest)

    > Help on package ModTest:
    >
    > NAME
    > ModTest
    >
    > FILE
    > /usr/local/lib/python2.4/site-packages/ModTest/__init__.py
    >
    > PACKAGE CONTENTS
    > AModule
    > BModule
    > CModule
    >
    > Thanks for the clarification.


    Sure, pydoc (which help calls under the code) does that, with a nice mix
    of inspect, os, and pkgutil.iter_modules calls. pkgutil.iter_modules
    may in fact be most of what you need:

    >>> help(pkgutil.iter_modules)

    Help on function iter_modules in module pkgutil:

    iter_modules(path=None, prefix='')
    Yields (module_loader, name, ispkg) for all submodules on path,
    or, if path is None, all top-level modules on sys.path.

    'path' should be either None or a list of paths to look for
    modules in.

    'prefix' is a string to output on the front of every module name
    on output.

    You will still have to call it recursively, though, I believe.


    Alex
    Alex Martelli, Mar 30, 2007
    #6
  7. On Thursday 29 March 2007 17:58, Alex Martelli wrote:
    > Sure, pydoc (which help calls under the code) does that, with a nice mix
    > of inspect, os, and pkgutil.iter_modules calls. pkgutil.iter_modules
    > may in fact be most of what you need:
    >
    >>>> help(pkgutil.iter_modules)

    > Help on function iter_modules in module pkgutil:
    >
    > iter_modules(path=None, prefix='')
    > Yields (module_loader, name, ispkg) for all submodules on path,
    > or, if path is None, all top-level modules on sys.path.
    >
    > 'path' should be either None or a list of paths to look for
    > modules in.
    >
    > 'prefix' is a string to output on the front of every module name
    > on output.


    OK, that looks nice...but what version of Python is that?
    http://docs.python.org/lib/module-pkgutil.html only shows one function (and
    that's 2.5) and my python 2.4 installation is similarly lacking an
    iter_modules() function for the pkgutil module. Is this a 2.6 thing?

    Suppose I could just copy the code from here:
    http://mail.python.org/pipermail/python-checkins/2006-April/051452.html and
    add it to my module. :)

    Thanks for the pointer!

    j

    --
    Joshua Kugler
    Lead System Admin -- Senior Programmer
    http://www.eeinternet.com
    PGP Key: http://pgp.mit.edu/  ID 0xDB26D7CE

    --
    Posted via a free Usenet account from http://www.teranews.com
    Joshua J. Kugler, Apr 3, 2007
    #7
  8. Joshua J. Kugler

    Robert Kern Guest

    Joshua J. Kugler wrote:
    > On Thursday 29 March 2007 17:58, Alex Martelli wrote:
    >> Sure, pydoc (which help calls under the code) does that, with a nice mix
    >> of inspect, os, and pkgutil.iter_modules calls. pkgutil.iter_modules
    >> may in fact be most of what you need:
    >>
    >>>>> help(pkgutil.iter_modules)

    >> Help on function iter_modules in module pkgutil:
    >>
    >> iter_modules(path=None, prefix='')
    >> Yields (module_loader, name, ispkg) for all submodules on path,
    >> or, if path is None, all top-level modules on sys.path.
    >>
    >> 'path' should be either None or a list of paths to look for
    >> modules in.
    >>
    >> 'prefix' is a string to output on the front of every module name
    >> on output.

    >
    > OK, that looks nice...but what version of Python is that?
    > http://docs.python.org/lib/module-pkgutil.html only shows one function (and
    > that's 2.5) and my python 2.4 installation is similarly lacking an
    > iter_modules() function for the pkgutil module. Is this a 2.6 thing?


    No, 2.5. The documentation is not up to date. Read the source.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Apr 3, 2007
    #8
  9. On Monday 02 April 2007 16:33, Robert Kern wrote:
    >>>>>> help(pkgutil.iter_modules)
    >>> Help on function iter_modules in module pkgutil:
    >>>
    >>> iter_modules(path=None, prefix='')
    >>> Yields (module_loader, name, ispkg) for all submodules on path,
    >>> or, if path is None, all top-level modules on sys.path.
    >>>
    >>> 'path' should be either None or a list of paths to look for
    >>> modules in.
    >>>
    >>> 'prefix' is a string to output on the front of every module name
    >>> on output.

    >>
    >> OK, that looks nice...but what version of Python is that?
    >> http://docs.python.org/lib/module-pkgutil.html only shows one function
    >> (and that's 2.5) and my python 2.4 installation is similarly lacking an
    >> iter_modules() function for the pkgutil module. Is this a 2.6 thing?

    >
    > No, 2.5. The documentation is not up to date. Read the source.
    >


    Gotcha. Thanks...well, since we're using 2.4, that will have to wait. For
    the archives, here is what I've come up with.

    Contents of the __init__.py for a module.

    import os
    _myDir = __path__[0]

    def mod_list():
    """
    A quick hack that retrieves all the sub modules in a directory that has
    an __init__.py file. I could use pkgutil.iter_modules, but that is
    Python 2.5 only, and this should work with several versions of Python.
    """
    modList = []
    modHash = {}
    isModule = False
    for ii in os.walk(_myDir):
    if ii[0] == _myDir:
    for f in ii[2]:
    # If there is no __init__ file, then the directory
    # upon which mod_list() is operating is not a module
    if f[0:8] == '__init__':
    isModule = True
    elif f[-3:] == '.py':
    modHash[f[:-3]] = True
    elif f[-4:] == '.pyc' or f[-4:] == '.pyo':
    modHash[f[:-4]] = True
    if isModule:
    modList = modHash.keys()
    modList.sort()
    return(modList)
    else:
    return(None)

    Hope that helps someone!

    j

    --
    Joshua Kugler
    Lead System Admin -- Senior Programmer
    http://www.eeinternet.com
    PGP Key: http://pgp.mit.edu/  ID 0xDB26D7CE

    --
    Posted via a free Usenet account from http://www.teranews.com
    Joshua J. Kugler, Apr 3, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. THY
    Replies:
    1
    Views:
    389
    Steve C. Orr, MCSD
    Aug 19, 2003
  2. Ben
    Replies:
    2
    Views:
    866
  3. Replies:
    2
    Views:
    333
  4. ImpalerCore
    Replies:
    0
    Views:
    824
    ImpalerCore
    Mar 10, 2011
  5. Lawrence D'Oliveiro

    Death To Sub-Sub-Sub-Directories!

    Lawrence D'Oliveiro, May 5, 2011, in forum: Java
    Replies:
    92
    Views:
    1,979
    Lawrence D'Oliveiro
    May 20, 2011
Loading...

Share This Page