Automatically loading and initialising objects from a pluginsdirectory

[Dave Challis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm trying to write some code which:
1. Finds all modules in a plugin directory
2. Imports those modules
3. Creates an instance of each object defined in the module (each module
will contain exactly 1 object, which is a subclass of 'Plugin')

The closest I've come so far is with something like:

In plugin.py:
# taken from http://docs.python.org/lib/built-in-funcs.html
def my_import(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod

def import_plugins():
mods = []
for filename in os.listdir('/plugins'):
if filename.endswith('.py'):
name = os.path.splitext(filename)[0]
mods.append(my_import('plugins.' + name))
return mods

class Plugin(object):
pass


In plugins/exampleplugin.py:
class ExamplePlugin(Plugin):
def __init__(self):
pass


Calling import_plugins() then gives me a list containing references to
modules.

How can I loop through that list and create an instance of whatever
object was defined within the module? (In this case I'd want to
construct an instance of ExamplePlugin)

Thanks in advance,
Dave


- --
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(
Dave Challis ) ><>
(e-mail address removed)_______________________(___________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIhb/Uv26GZvAVVFERAsGaAJ9KwtFI9yXdk2gBGxy0/bjCd5318wCgsiV9
m14BZSvxqZ1EP0OvaXBZoaw=
=TYlD
-----END PGP SIGNATURE-----

 

[Diez B. Roggisch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm trying to write some code which:
1. Finds all modules in a plugin directory
2. Imports those modules
3. Creates an instance of each object defined in the module (each module
will contain exactly 1 object, which is a subclass of 'Plugin')

The closest I've come so far is with something like:

In plugin.py:
# taken from http://docs.python.org/lib/built-in-funcs.html
def my_import(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod

def import_plugins():
mods = []
for filename in os.listdir('/plugins'):
if filename.endswith('.py'):
name = os.path.splitext(filename)[0]
mods.append(my_import('plugins.' + name))
return mods

class Plugin(object):
pass


In plugins/exampleplugin.py:
class ExamplePlugin(Plugin):
def __init__(self):
pass


Calling import_plugins() then gives me a list containing references to
modules.

How can I loop through that list and create an instance of whatever
object was defined within the module? (In this case I'd want to
construct an instance of ExamplePlugin)

Like this:

for name in dir(plugin):
thing = getattr(plugin, name)
try:
if issubclass(thing, Plugin):
thing()
except ValueError: # issubclass sucks
pass

Diez

 

[Dave Challis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm trying to write some code which:
1. Finds all modules in a plugin directory
2. Imports those modules
3. Creates an instance of each object defined in the module (each module
will contain exactly 1 object, which is a subclass of 'Plugin')

The closest I've come so far is with something like:

In plugin.py:
# taken from http://docs.python.org/lib/built-in-funcs.html
def my_import(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod

def import_plugins():
mods = []
for filename in os.listdir('/plugins'):
if filename.endswith('.py'):
name = os.path.splitext(filename)[0]
mods.append(my_import('plugins.' + name))
return mods

class Plugin(object):
pass


In plugins/exampleplugin.py:
class ExamplePlugin(Plugin):
def __init__(self):
pass


Calling import_plugins() then gives me a list containing references to
modules.

How can I loop through that list and create an instance of whatever
object was defined within the module? (In this case I'd want to
construct an instance of ExamplePlugin)

Like this:

for name in dir(plugin):
thing = getattr(plugin, name)
try:
if issubclass(thing, Plugin):
thing()
except ValueError: # issubclass sucks
pass

Diez

Thanks for that, it helped as a starting point. I had some trouble with
using issubclass though (issubclass(Plugin, Plugin) returns true), which
was complicating things.

I modified your code to the following instead (which may well have it's
own pitfalls I'm not aware of!):

for name in dir(plugin):
thing = getattr(plugin, name)
if hasattr(thing, '__bases__') and \
getattr(thing, '__bases__')[0] == Plugin:
thing()

Cheers,
Dave


- --
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

><> o ) <><

. ( (
) Dave Challis <>< ) ) ><>
_([email protected]___________________________(____(__________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIhge8v26GZvAVVFERAh86AJ91DVYPPZ/1x3rjoezFL1P5FXuHNwCZAXdr
DgjzmTNV1/H8vcQ/2Hax3js=
=ZAtM
-----END PGP SIGNATURE-----

 

[Fredrik Lundh]

Thanks for that, it helped as a starting point. I had some trouble with
using issubclass though (issubclass(Plugin, Plugin) returns true), which
was complicating things.

I modified your code to the following instead (which may well have it's
own pitfalls I'm not aware of!):

for name in dir(plugin):
thing = getattr(plugin, name)
if hasattr(thing, '__bases__') and \
getattr(thing, '__bases__')[0] == Plugin:
thing()

so now you're no longer supporting mixins and multiple-level
inheritance? why not just tweak Diez' example a little:

for name in dir(plugin):

thing = getattr(plugin, name)

# make sure this is a plugin
try:
if thing is Plugin:
continue
if not issubclass(thing, Plugin):
continue
except ValueError: # issubclass sucks
continue # not a class

thing() # probably needs error handling around this

(I also moved the thing call out of the thing validation part, to allow
you to distinguish between ValueErrors caused by issubclass and errors
caused by things)

(btw, another approach would be to use a metaclass to make Plugin
classes register themselves on import)

</F>

 

[Dave Challis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

so now you're no longer supporting mixins and multiple-level
inheritance? why not just tweak Diez' example a little:

for name in dir(plugin):

thing = getattr(plugin, name)

# make sure this is a plugin
try:
if thing is Plugin:
continue
if not issubclass(thing, Plugin):
continue
except ValueError: # issubclass sucks
continue # not a class

thing() # probably needs error handling around this

(I also moved the thing call out of the thing validation part, to allow
you to distinguish between ValueErrors caused by issubclass and errors
caused by things)

(btw, another approach would be to use a metaclass to make Plugin
classes register themselves on import)

</F>

Thanks, that works much better. I've only been using python a couple of
months, so still getting to grips with how it treats classes/subclasses.

I'll have a look into metaclasses too, haven't stumbled upon those yet
at all.

Cheers,
Dave

- --
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
) <><
<>< (
Dave Challis )
(e-mail address removed)_______(___________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIibiHv26GZvAVVFERAgcmAJ4tmC7jp6TTb3Dx2Lw+rKSmJkcSLQCfYFsA
e0ZLOf8lQXhqHcz/Me8ok0E=
=Qcg7
-----END PGP SIGNATURE-----

 

[Fredrik Lundh]

I'll have a look into metaclasses too, haven't stumbled upon those yet
at all.

It's a potentially brain-exploding topic, though, so if the above
solution works for you, you might want to leave it at that ;-)

But very briefly, a metaclass is a something that's responsible for
creating a class, much like an ordinary class is responsible for
creating an object. When Python executes the following statement,

class Spam:
attrib = 1
def func(self):
pass
# <-- end of class statement

it will create a new scope for the class content, execute the class
body, and then, when it reaches the end, call the "metaclass" to create
the actual class object. The metaclass is given the requested name
("Spam" in this case), any base classes, and a dictionary containing
everything from the class scope ("attrib" and "func", in this case).
The thing that's returned is assigned to the "Spam" variable.

The default metaclass ("type") just creates an ordinary class object,
but if you replace that with your own metaclas, you can completely
override that behaviour, or just extend it (e.g. by registering the
subclasses in a common registry). Like, say, this:

registry = [] # list of subclasses

class Plugin(object):
class __metaclass__(type):
def __init__(cls, name, bases, dict):
type.__init__(name, bases, dict)
registry.append((name, cls))

class SpamPlugin(Plugin):
pass

class BaconPlugin(Plugin):
pass

for name, cls in registry:
if cls is not Plugin:
print name, cls

Here, the presence of an inner __metaclass__ class (which is a subclass
of "type") causes Python's class machinery to use that class instead of
"type" when creating class objects for Plugin or any subclass thereof.
The extra code in the __init__ method just all plugins to a list.

For more on this, see e.g.

http://www.python.org/download/releases/2.2/descrintro/#metaclasses

</F>

 

[kpd]

It's a potentially brain-exploding topic,

-that you made very understandable. Thanks for posting that
explanation and example.