class / module introspection?

Discussion in 'Python' started by Brian Munroe, Apr 2, 2008.

  1. Brian Munroe

    Brian Munroe Guest

    I'm struggling with an architectural problem and could use some
    advice.

    I'm writing an application that will gather statuses from disparate
    systems. Because new systems show up all the time, I'm trying to
    design a plugin architecture that will allow people to contribute new
    backends by just dropping a package/module into a specific directory.
    The object methods in these backends will conform to a documented API
    that the main application will call.

    Currently I have something that looks like this:

    src/
    backends/
    system1/
    __init__.py
    system2/
    __init__.py
    ...

    Then from my application (main.py) I can simply call:

    from backends import system1

    be1 = system1.Backend()
    be1.getStatus()

    This would work great if I knew about system1 and system2 ahead of
    time, but that isn't the case. Having to rewrite main.py every time a
    new backend module comes along is obviously a stupid idea too. I've
    been thinking I need some kind of introspection, but I've been reading
    about it and a whole mess of design pattern stuff, so my head is
    swimming and I am totally unsure of what the best approach is.

    My guess is that I need to load all the backends at runtime - then
    introspect the loaded classes?

    Any suggestions would be greatly appreciated.
     
    Brian Munroe, Apr 2, 2008
    #1
    1. Advertising

  2. Brian Munroe

    Guest

    On 2 avr, 18:03, Brian Munroe <> wrote:
    > I'm struggling with an architectural problem and could use some
    > advice.
    >
    > I'm writing an application that will gather statuses from disparate
    > systems. Because new systems show up all the time, I'm trying to
    > design a plugin architecture that will allow people to contribute new
    > backends by just dropping a package/module into a specific directory.
    > The object methods in these backends will conform to a documented API
    > that the main application will call.
    >
    > Currently I have something that looks like this:
    >
    > src/
    > backends/
    > system1/
    > __init__.py
    > system2/
    > __init__.py
    > ...
    >
    > Then from my application (main.py) I can simply call:
    >
    > from backends import system1
    >
    > be1 = system1.Backend()
    > be1.getStatus()
    >
    > This would work great if I knew about system1 and system2 ahead of
    > time, but that isn't the case. Having to rewrite main.py every time a
    > new backend module comes along is obviously a stupid idea too. I've
    > been thinking I need some kind of introspection, but I've been reading
    > about it and a whole mess of design pattern stuff, so my head is
    > swimming and I am totally unsure of what the best approach is.
    >
    > My guess is that I need to load all the backends at runtime


    Anyway, almost everything happens at runtime in Python !-)

    More seriously: the answer is in the doc.
    http://www.python.org/doc/2.3.5/lib/built-in-funcs.html

    read about the __import__ function, experiment in your interactive
    python shell, and you should be done in a couple minutes.
     
    , Apr 2, 2008
    #2
    1. Advertising

  3. Brian Munroe

    Brian Munroe Guest

    On Apr 2, 11:04 am, ""
    <> wrote:

    >
    > More seriously: the answer is in the doc.http://www.python.org/doc/2.3.5/lib/built-in-funcs.html
    >
    > read about the __import__ function, experiment in your interactive
    > python shell, and you should be done in a couple minutes.


    Well, If I understand the docs correctly, that would work great if
    backends/ was a module and not a package? I need to keep backends/
    system1/ and backends/system2 as separate directory structures to make
    things fairly isolated from each other (for neatness sake)

    Currently I'm building the backends/__init__.py __all__ list
    dynamically, such as:

    backends/__init__.py
    --------------------

    import os

    __all__ = []

    for module in os.listdir(__path__[0]):
    if not module.startswith("__"):
    __all__.append(module)

    then from my main application, I can do the following

    main.py
    -------

    import backends

    print backends.__all__

    This gives me ['system1','system2'] - which I can then use __import__
    on.

    -- brian
     
    Brian Munroe, Apr 2, 2008
    #3
  4. Brian Munroe

    Brian Munroe Guest

    On Apr 2, 12:07 pm, Brian Munroe <> wrote:

    >
    > This gives me ['system1','system2'] - which I can then use __import__
    > on.
    >


    Addendum, thanks Bruno!

    I also required the helper function (my_import) from the Python docs
    you pointed me to, that actually was the key to getting everything
    working!
     
    Brian Munroe, Apr 2, 2008
    #4
  5. Brian Munroe

    Guest

    On 2 avr, 21:07, Brian Munroe <> wrote:
    > On Apr 2, 11:04 am, ""
    >
    > <> wrote:
    >
    > > More seriously: the answer is in the doc.http://www.python.org/doc/2.3.5/lib/built-in-funcs.html

    >
    > > read about the __import__ function, experiment in your interactive
    > > python shell, and you should be done in a couple minutes.

    >
    > Well, If I understand the docs correctly, that would work great if
    > backends/ was a module and not a package?


    Not necessarily.

    > I need to keep backends/
    > system1/ and backends/system2 as separate directory structures to make
    > things fairly isolated from each other (for neatness sake)
    >
    > Currently I'm building the backends/__init__.py __all__ list
    > dynamically, such as:
    >
    > backends/__init__.py
    > --------------------
    >
    > import os
    >
    > __all__ = []
    >
    > for module in os.listdir(__path__[0]):
    > if not module.startswith("__"):
    > __all__.append(module)


    Why not do the import here, so you store a real module instead of a
    name ?

    ie (not tested):

    for module_name in os.listdir(__path__[0]):
    if not module_name.startswith("__"):
    __all__.append(__import__(module_name, globals(), locals()))

    My 2 cents...
     
    , Apr 2, 2008
    #5
  6. Brian Munroe

    Brian Munroe Guest

    On Apr 2, 12:33 pm, ""
    <> wrote:


    >
    > Why not do the import here, so you store a real module instead of a
    > name ?
    >


    Right now I'm still in the prototyping phase and haven't really
    thought everything through. I needed the names because I am
    populating a GUI selection list element. I also assumed that I could
    then lazy load the modules I needed...

    Thanks for the help though, you've gotten me past my first of many
    (I'm sure) hurdles!

    -- brian
     
    Brian Munroe, Apr 2, 2008
    #6
  7. Brian Munroe

    7stud Guest

    On Apr 2, 1:27 pm, Brian Munroe <> wrote:
    > On Apr 2, 12:07 pm, Brian Munroe <> wrote:
    >
    >
    >
    > > This gives me ['system1','system2'] - which I can then use __import__
    > > on.

    >
    > Addendum, thanks Bruno!
    >
    > I also required the helper function (my_import) from the Python docs
    >


    You don't need that helper function, which is a little tricky to
    follow:

    If you have a module name as a string, you can use the builtin
    function __import__ to import the module, e.g.

    x = __import__("mymod")
    x.myfunc()

    Note that you have to assign the return value of __import__ to a
    variable. Thereafter you use the variable to access the module's
    functions.

    However, when you use __import__ with a package, it behaves in a
    peculiar manner: it returns the top level package object. For
    example, if you write:

    x = __import__("mypackage.subpack.myfuncs")


    then x is a reference to mypackage--not mypackage.subpack.myfuncs. In
    order to execute a function in the module myfuncs, you need to write:

    x.subpack.myfuncs.f().

    Or, if you check the docs, you can make __import__ return the
    rightmost module in the module string by writing:

    x = __import__("mypackage.subpack.myfuncs", globals(), locals(),
    ["subpack.myfuncs"])

    The last argument is what makes __import__ return a lower level module
    in the package hierarchy.


    To solve your problem, you can do something like the following:

    Directory structure
    -------------------

    ../mypackage
    __init__.py

    /system1
    __init__.py
    myfuncs.py

    /system2
    __init__.py
    myfuncs.py


    system1/myfuncs.py
    ------------
    def f():
    print "system1 here"


    system2/myfuncs.py
    -----------------
    def f():
    print "system2 here"




    import sys
    import os

    #Get all the file names contained in mypackage:
    fnames = os.listdir("../mypackage")
    print fnames

    #Separate out the directory names(which are the sub packages):
    os.chdir("../mypackage") #so abspath() below can construct full paths
    package_dirs = []

    for fname in fnames:
    abs_path = os.path.abspath(fname) #need full path for isdir()
    print abs_path,

    if os.path.isdir(abs_path):
    print 'dir'
    package_dirs.append(fname)
    else:
    print 'file'

    print package_dirs

    sys.path.append("../") #directs python to search the specified
    #dir for any modules that are imported

    #Import the sub packages:
    for dir in package_dirs:
    modname = "%s.%s.%s" % ("mypackage", dir, "myfuncs")
    modname_minus_toplevel_packname = "%s.%s" % (dir, "myfuncs")

    mod = __import__(modname, globals(), locals(),
    [modname_minus_toplevel_packname])
    mod.f()



    --output:--
    ['__init__.py', '__init__.pyc', 'system1', 'system2']
    /Users/me/2testing/mypackage/__init__.py file
    /Users/me/2testing/mypackage/__init__.pyc file
    /Users/me/2testing/mypackage/system1 dir
    /Users/me/2testing/mypackage/system2 dir
    ['system1', 'system2']
    system1 here
    system2 here
     
    7stud, Apr 2, 2008
    #7
  8. Brian Munroe

    Guest

    On 2 avr, 22:04, Brian Munroe <> wrote:
    > On Apr 2, 12:33 pm, ""
    >
    > <> wrote:
    >
    > > Why not do the import here, so you store a real module instead of a
    > > name ?

    >
    > Right now I'm still in the prototyping phase and haven't really
    > thought everything through. I needed the names because I am
    > populating a GUI selection list element.



    import os
    print os.__name__
    'os'

    > I also assumed that I could
    > then lazy load the modules I needed...


    Ok, that's something else. I had (wrongly) assumed you wanted to load
    all modules anyway.

    > Thanks for the help though, you've gotten me past my first of many
    > (I'm sure) hurdles!


    You're welcome.
     
    , Apr 2, 2008
    #8
  9. Brian Munroe

    Brian Munroe Guest

    On Apr 2, 2:26 pm, 7stud <> wrote:

    >
    > You don't need that helper function, which is a little tricky to
    > follow:
    >


    Well, I appreciate the code sharing you did, but the helper function
    is nice and compact, and I didn't have much trouble following it. I
    ended up doing the following in the backends/__init__.py:

    import os
    availableBackendsList = []

    for module in os.listdir(__path__[0]):
    if not module.startswith("__") and not module.startswith("."):
    availableBackendsList.append("backends." + module)

    def subpackage_import(name):
    mod = __import__(name)
    components = name.split('.')
    for comp in components[1:]:
    mod = getattr(mod, comp)
    return mod

    def get_available_backends():
    return map(subpackage_import, availableBackendsList)

    In then in my application code, it becomes a simple matter of:

    import backends

    for x in backends.get_available_backends():
    print x.PLUGIN_NAME
    be = x.Backend()
    print be.getStatus()

    Basically, I borrowed a page out of Dive into Python[1] and mapped
    each module object (system1, system2, ... systemN) to a list.


    [1] - Thanks Mark! - http://www.diveintopython.org/functional_programming/all_together.html
     
    Brian Munroe, Apr 3, 2008
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Harrison

    Re: class introspection

    John Harrison, Jul 23, 2003, in forum: C++
    Replies:
    4
    Views:
    376
    John Harrison
    Jul 24, 2003
  2. Roy Smith

    Introspection at the module level?

    Roy Smith, Mar 6, 2004, in forum: Python
    Replies:
    6
    Views:
    297
    Roy Smith
    Mar 7, 2004
  3. Holger Joukl
    Replies:
    1
    Views:
    326
    Jeremy Yallop
    May 7, 2004
  4. Andy Leszczynski
    Replies:
    1
    Views:
    321
    Michele Simionato
    Apr 28, 2005
  5. Benjamin Rutt
    Replies:
    4
    Views:
    396
    Mike Meyer
    Jul 8, 2005
Loading...

Share This Page