How to load new class definitions at runtime?

[Carlos Ribeiro]

I'm looking for ways to load new class definitions at runtime (I'm not
talking about object instances here, so a persistent object database
isn't what I am looking for [1]). I'm aware of a few options, but I
would like to discuss the possible side effects for a potentially big,
long running application server.

I am developing a business application based on workflow concepts. The
model is quite simple: a business process is modelled as a sequence of
tasks connected by actions. Actions can be re-used in different
contexts. The result is a very complex graph. Processes, tasks and
actions are implemented as classes. I opted to declare each action as
a different class, because each action has different Python code
attached to it. So simple object instances would not cut it. To allow
the system to be customized without taking the application server
offline, there must be a solution for these new class definitions to
be loaded at runtime.

My current solution is simple. Class definitions have to be executed,
and for it to work, the class name and the source code (or the
compiled bytecode) is stored in a database. A class factory function
retrieves the correct class definition (based on the name), and simply
executes the code.

Now, talking about executing arbitrary code retrieved from a database
isn't exactly a safe proposition. But I can't see any other way to
make it work. I'm now looking for more ideas on how do other people
solved the same problem in other projects. Pointers are welcome.
Thanks in advance,

---
[1] ... or at least I assume so.
--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[Alex Martelli]

attached to it. So simple object instances would not cut it. To allow
the system to be customized without taking the application server
offline, there must be a solution for these new class definitions to
be loaded at runtime.

If you need them to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet. My vote for most useful custom metaclasses ever...

My current solution is simple. Class definitions have to be executed,
and for it to work, the class name and the source code (or the
compiled bytecode) is stored in a database. A class factory function
retrieves the correct class definition (based on the name), and simply
executes the code.

You can pickle the bytecode etc (again, cookbook recipes show how) but
that has no real advantage wrt storing the source or bytecode. Just
make sure, for performance, that if you store the source the bytecode
also gets stashed away when first made. Unless for some weird reason
you need your code repository to support multiple releases of Python at
the same time (eek), you might as well store the bytecode at the time
the source is stored -- upgrading release of Python in the future _will_
require a DB conversion and some downtime for the switch.

Now, talking about executing arbitrary code retrieved from a database
isn't exactly a safe proposition. But I can't see any other way to
make it work. I'm now looking for more ideas on how do other people
solved the same problem in other projects. Pointers are welcome.

If you want to allow arbitrary class definitions to happen at runtime,
"a safe proposition" is an oxymoron -- obviously you have to trust
*entirely* the repository of the code (and pickles would be just about
the same thing). Digital signatures to ensure the repository has not
been tampered with would also need to be stored somewhere entirely
trustable, so you might as well declare that the trust must be towards
the repository, in all practical cases I can think of.


From past experience with this sort of thing, my practical suggestion
is: make sure you have a development/debug mode in which the classes are
first looked for on ordinary modules in the filesystem, then gotten from
the DB only if the ordinary modules don't have them. When you're
developing and debugging, you'll heavily use this feature, believe me,
and also Michael Hudson's recipe above-mentioned, since tweaks and mods
to the classes will come at a hefty pace.

It's a harder deployment decision whether the normal system operation
should also support the filesystem, or just the database. The latter
makes development less convenient, but distributed processing much
easiery to manage, updates are transactional and atomic, etc. It
depends on how much customization-during-operation you'll need, vs the
other deployment issues.


Alex

 

[Carlos Ribeiro]

If you need them to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet. My vote for most useful custom metaclasses ever...

Thanks for the pointer. Michael Hudson's recipe solves one huge part
of the problem; I was not even trying to solve it yet, but it helps a
lot

[snip]

It's a harder deployment decision whether the normal system operation
should also support the filesystem, or just the database.

I'm thinking about two alternatives right now, not mutually exclusive.

1) I'm trying to understand how do packages work, and what hooks are
available. I guess I could make it work using packages to re-read the
source code whenever a "from <package> import <module>" statement is
executed; Michael Hudson's recipe could also be automatically invoked
for the redefined classes.

2) Another suggestion is to use a source-code control system as the
code repository. For example: using the Python Subversion interface, I
could checkout the code for the class. It's safer, in a sense, than
database access, because of the way it tracks changes and logs
accesses. But it's also more complex, and possibly slower, than a file
read, or even a database read (depending on the db engine, of course).

My choice now is a simple-minded 'always re-read the source code'
approach, just to make it work and test some concepts. But the problem
seems interesting enough to deserve further investigation. I'm sure a
"dynamic package" version, with automatic redefinition using the
Michaels Hudson's recipe, would make for a good recipe itself.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (e-mail address removed)
mail: (e-mail address removed)

 

[SM]

If you need [classes] to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet. My vote for most useful custom metaclasses ever...

That recipe is here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

Is it correct that this metaclass solution does not handle changes to
__init__, including new attributes that are bound there? I.e., it
basically only deals with modification or addition or deletion of
methods; if you need to add new attributes or have any other new code
in __init__, you'll have to make new instances for all the old
instances, or deal with it some other way. In which case would you be
better off just using that other way from the start?

 

[Madhusudhanan Chandrasekaran]

Hi,
I am a newbie here. I want to know how to redirect os.system("ls -af")
to a string variable without it barfing the answer of ls -af on the
screen.

Thanks in advance,
Madhusudhanan.C

 

[Alex Martelli]

(e-mail address removed) (Alex Martelli) wrote in message

If you need [classes] to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet. My vote for most useful custom metaclasses ever...

That recipe is here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

Is it correct that this metaclass solution does not handle changes to
__init__, including new attributes that are bound there? I.e., it

It doesn't rerun __init__ nor any other method on existing instances.

basically only deals with modification or addition or deletion of
methods; if you need to add new attributes or have any other new code
in __init__, you'll have to make new instances for all the old
instances, or deal with it some other way. In which case would you be
better off just using that other way from the start?

It's pretty trivial to enhance the recipe to run a per-instance or
per-subclass __make_old_into_new__ method, if that method exists, on all
existing old instances -- if you need to adjust the old instances'
states (==attributes), that strikes me as the best architecture.


Alex

 

[Harry George]

Hi,
I am a newbie here. I want to know how to redirect os.system("ls -af")
to a string variable without it barfing the answer of ls -af on the
screen.

Thanks in advance,
Madhusudhanan.C

1. If all you want is the list of files:
myfiles=os.listdir('.')

2. If you want exactly the ls commands output, simplest is:
import commands
result=commands.getoutput("ls -af")

3. If you want more control, you may need forking and piping.
Let me know if you need examples.

 

[Michael Hudson]

(e-mail address removed) (Alex Martelli) wrote in message

If you need [classes] to be _updatable_ at runtime, look at a recipe by
Michael Hudson in the cookbook -- I've updated it for the printed
version 2nd ed (since we do have __subclasses__ now, yay!) but that's
not done yet. My vote for most useful custom metaclasses ever...

Wow, thanks

It doesn't rerun __init__ nor any other method on existing instances.

Well... it runs change_class on existing instances, but see below...

It's pretty trivial to enhance the recipe to run a per-instance or
per-subclass __make_old_into_new__ method, if that method exists, on all
existing old instances -- if you need to adjust the old instances'
states (==attributes), that strikes me as the best architecture.

I'd sort of thought about this when I wrote the recipe initially --
it's why there is a change_class method -- but now realize that it
doesn't really work, because when you reload the module, it runs the
*old* change_class when it would (obviously) be much more useful to
somehow run a new method.

How about this diff (from what's on the recipe page now):

*** autoreloader.py~ 2004-10-27 17:50:03.000000000 +0100
--- autoreloader.py 2004-11-26 11:36:45.000000000 +0000
***************
*** 27,33 ****
if d.has_key(name):
old_class = d[name]
for instance in old_class.__instances__():
! instance.change_class(new_class)
new_class.__instance_refs__.append(
weakref.ref(instance))
# this section only works in 2.3
--- 27,34 ----
if d.has_key(name):
old_class = d[name]
for instance in old_class.__instances__():
! instance.__class__ = new_class
! instance.update_self()
new_class.__instance_refs__.append(
weakref.ref(instance))
# this section only works in 2.3
***************
*** 44,51 ****

class AutoReloader:
__metaclass__ = MetaAutoReloader
! def change_class(self, new_class):
! self.__class__ = new_class

class Bar(AutoReloader):
pass
--- 45,52 ----

class AutoReloader:
__metaclass__ = MetaAutoReloader
! def update_self(self):
! pass

class Bar(AutoReloader):
pass
***************
*** 67,69 ****
--- 68,94 ----
b2.meth(2)
# new Baz() instances now play too:
Baz().meth(3)
+
+ class C(AutoReloader):
+ def __init__(self, a):
+ self.a = 1
+ def get_attribute(self):
+ return self.a
+
+ c = C(1)
+
+ print c.get_attribute()
+
+ class C(AutoReloader):
+ def __init__(self, b):
+ self.b = b
+ def get_attribute(self):
+ return self.b
+
+ def update_self(self):
+ self.b = self.a
+ del self.a
+
+ print c.get_attribute()
+
+

Probably a bit more useful, though it has the disadvantage that it
will crash and burn if you forget to remove the update_self() method
before reloading again (that's probably inevitable, though). Maybe
you could introduce revision numbers on the instances and the classes
or something, though this might be getting a little complex.

Cheers,
mwh