Best practices for dynamically loading plugins at startup

[Christoph Haas]

Dear coders...

I'm working on an application that is supposed to support "plugins".
The idea is to use the plugins as packages like this:

Plugins/
__init__.py
Plugin1.py
Plugin2.py
Plugin3.py

When the application starts up I want to have these modules loaded
dynamically. Users can put their own plugin modules into the
Plugins/ directory and the application should know about it.

Since I don't know which plugins have been put into that directory
I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.py".
So I need to find out the *.py there and load them during startup.
I could do that with a "walk" over that directory.

Each plugin is supposed to be a class derived from a general
"Plugin" superclass. I just don't know how to 'register' every
plugin. The main application needs to know which plugin classes
there are. On IRC I was recommended walking through all objects
and finding out if the class is a subclass of "Plugin". Another
person recommended using metaclasses to automatically register
the plugin in a global list.

Since I have only little real-life Python knowledge I wonder what the
best practice for this kind of problem is.

I looked at the "supybot" IRC bot to get an idea how plugins are handled
there. Unfortunately it was still a bit over my (python) head.

Regards
Christoph

 

[Jarek Zgoda]

Christoph Haas napisa³(a):

Since I don't know which plugins have been put into that directory
I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.py".
So I need to find out the *.py there and load them during startup.
I could do that with a "walk" over that directory.

See entry for __import__ at
http://www.python.org/doc/2.3/lib/built-in-funcs.html

 

[Jeff Schwab]

Dear coders...

I'm working on an application that is supposed to support "plugins".
The idea is to use the plugins as packages like this:

Plugins/
__init__.py
Plugin1.py
Plugin2.py
Plugin3.py

When the application starts up I want to have these modules loaded
dynamically. Users can put their own plugin modules into the
Plugins/ directory and the application should know about it.

Since I don't know which plugins have been put into that directory
I cannot just "import Plugin1, Plugin2, Plugin3" in the "__init__.py".
So I need to find out the *.py there and load them during startup.
I could do that with a "walk" over that directory.

Each plugin is supposed to be a class derived from a general
"Plugin" superclass. I just don't know how to 'register' every
plugin. The main application needs to know which plugin classes
there are. On IRC I was recommended walking through all objects
and finding out if the class is a subclass of "Plugin". Another
person recommended using metaclasses to automatically register
the plugin in a global list.

Since I have only little real-life Python knowledge I wonder what the
best practice for this kind of problem is.

I looked at the "supybot" IRC bot to get an idea how plugins are handled
there. Unfortunately it was still a bit over my (python) head.

I recently came up against this exact problem. My preference is to have
the plugin writer call a method to register the plugins, as this allows
him the most control. Something along these lines:

class A_plugin(Plugin):
...

class Another_plugin(Plugin):
...

register( A_plugin )
register( Another_plugin, optional_custom_registration_parameters )


I also like the Mark Pilgrim approach of having the plugins follow a
strict naming convention, then searching for them with hasattr or by
grepping each plugin module's namespace. E.g., I have a build script
written in python that I use instead of the "make" utility; for each
target, I define a method called make_<target> in my plugin module, like
this:

function = "make_%s" % target
if hasattr(module, function):
getattr(module, function)()
else:
print >>sys.stderr, 'Unknown target "%s"' % target

 

[beza1e1]

I wrote this one:
--------------------------------------
def load_plugin(self, plugin, paths):
import imp
# check if we haven't loaded it already
try:
return sys.modules[plugin]
except KeyError:
pass
# ok, the load it
fp, filename, desc = imp.find_module(plugin, paths)
try:
mod = imp.load_module(plugin, fp, filename, desc)
finally:
if fp:
fp.close()
# instantiate and put into basket
clazz = mod.main(self.config)
if "input" in clazz.types:
self.inputs.append(clazz)
if "parser" in clazz.types:
self.parser.append(clazz)
if "output" in clazz.types:
self.outputs.append(clazz)
--------------------------------------
The imp module is the key:
http://docs.python.org/lib/module-imp.html

The parameters for the load module function, are found by look through
the directory. So my plugins had a convention:
They have a class called main, which is initialized with one argument,
the config object.

That is quite a simple plugin system. You can check out the whole thing
here:
https://developer.berlios.de/projects/feedisto/

 

[Christoph Haas]

I recently came up against this exact problem. My preference is to have
the plugin writer call a method to register the plugins, as this allows
him the most control. Something along these lines:

class A_plugin(Plugin):
...

class Another_plugin(Plugin):
...

register( A_plugin )
register( Another_plugin, optional_custom_registration_parameters )

I like the idea. And "supybot" seems to do just that. What would the
"def register:" do? Maintain a global list of registered plugins?
I didn't like the idea of having global variables. Or would register
be a method of some "main" class? How would I access that?

Thanks to everybody else who posted ideas on my problem. I'm trying
all the proposals to get an idea of which approach works best.

Regards
Christoph

 

[Jeff Schwab]

I like the idea. And "supybot" seems to do just that. What would the
"def register:" do? Maintain a global list of registered plugins?
I didn't like the idea of having global variables. Or would register
be a method of some "main" class? How would I access that?

The registry clearly has to be shared between modules, so I think it's
best to make it a module-level variable. In the simple-minded code
below, I've stored it in the "plugins" module. A better choice might be
the module that defines your plugins' common base class, if they have one.


# BEGIN framework.py
if __name__ == "__main__":
import plugins
print plugins.plugin_registry
# END framework.py

# BEGIN plugins.py
# Define some plugins.

class A_plugin(object):
def __init__(self):
print "A_plugin()"

class Another_plugin(object):
def __init__(self):
print "Another_plugin()"

# Register the plugins.

import registry

plugin_registry = registry.Registry()
plugin_registry.register(A_plugin)
plugin_registry.register(Another_plugin)
# END plugins.py

# BEGIN registry.py
class Registry(object):
"Each instance may be used to store a set of objects."

def __init__(self):
"Initialize an empty registry."
self.registered_objects = set()

def register(self, o):
"Add an object to the registry."
self.registered_objects.add(o)

def __repr__(self):
return "Registry Contents:\n\t" + \
"\n\t".join(repr(o) for o in self.registered_objects)
# END registry.py