Listing modules from all installed packages

Discussion in 'Python' started by Julien Phalip, Jun 9, 2013.

  1. Hi,

    I'm trying to write a function that programmatically obtains and returns the exact location of all first-level modules for all installed packages.

    For example, if the packages named 'django' and 'django-debug-toolbar' are installed, I'd like this function to return something like:/Users/my_user/.virtualenvs/my_venv/lib/python2.6/site-packages/django
    /Users/my_user/.virtualenvs/my_venv/src/debug_toolbar

    That is, this function needs to consider all installed packages, including those that have been installed in "edit" mode (i.e. in the src/ folder). Note also that the main module for the 'django-debug-toolbar' is in fact named 'debug_toolbar'.

    So far the closest I've been to retrieving the list of first-level modules is as follows:

    import os
    import pkg_resources
    import setuptools

    pkgs = set()

    for dist in pkg_resources.working_set:
    if os.path.isdir(dist.location):
    for pkg in setuptools.find_packages(dist.location):
    if '.' not in pkg:
    pkgs.add(pkg)

    The idea is then to loop through that list of modules, import them and get their exact locations by fetching their __file__ attribute values.

    However, this feels very hackish and I don't think it's actually quite correct either. I'm sure there must be a better way. If possible I'd also like to avoid having to use setuptools.

    Do you have any tips on how to achieve this?

    Many thanks!

    Julien
     
    Julien Phalip, Jun 9, 2013
    #1
    1. Advertisements

  2. print '\n'.join([re.findall("from '(.*)'",str(v))[0] for k,v in sys.modules.items() if str(v).find('from')>-1])
     
    Carlos Nepomuceno, Jun 9, 2013
    #2
    1. Advertisements

  3. Just realized that you've asked for installed packages. Perhaps the following will do the trick. I don't know why the 'lib-tk' isn't included. Why not?

    toplevel_packages = ['%s\\%s'%(ml.path,name)for ml,name,ispkg in pkgutil.iter_modules() if ispkg]
    print '\n'.join(toplevel_packages)
     
    Carlos Nepomuceno, Jun 9, 2013
    #3
  4. Julien Phalip

    cclauss Guest

    Adding : python -c 'help("modules") to the other two suggestions:

    #!/usr/bin/env python

    import commands, pkgutil, re, sys

    print('sys.modules.items()...')
    print('\n'.join(sorted([re.findall("from '(.*)'",str(v))[0] for k,v in sys.modules.items() if str(v).find('from')>-1])))

    print('\npkgutil.iter_modules()...')
    toplevel_packages = ['%s\\%s'%(ml.path,name)for ml,name,ispkg in sorted(pkgutil.iter_modules()) if ispkg]
    print '\n'.join(toplevel_packages)

    theCommand = "python -c 'help(\"modules\")'"
    print('\n{} # this may take a few seconds...'.format(theCommand))
    print(commands.getstatusoutput(theCommand)[1]) # help() only works in the python interpreter...
     
    cclauss, Jun 9, 2013
    #4
  5. Carlos Nepomucenoæ–¼ 2013å¹´6月9日星期日UTC+8下åˆ1時23分15秒寫é“:
    Please use a dictionary to store a tree first.
    Then it is trivial to walk through all nodes of the tree.
     
    88888 Dihedral, Jun 9, 2013
    #5
  6. Thanks a lot Carlos, this gives me exactly what I needed!

    Best wishes,

    Julien
     
    Julien Phalip, Jun 14, 2013
    #6
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.