Automatic reloading, metaclasses, and pickle

[andrewfelch]

Hello all,

I'm using the metaclass trick for automatic reloading of class member
functions, found at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

My problem is that if I
1) pickle an object that inherits from "AutoReloader"
2) unpickle the object
3) modify one of the pickled' object's derived class methods
4) reload the module holding the class

.... then the changes don't affect the unpickled object. If I unpickle
the object again, of course the changes take effect.

My friend that loves Smalltalk is laughing at me. I thought I had the
upperhand when I discovered the metaclasses but now I am not sure what
to do. I really don't want to have to unpickle again, I'm processing
video and it can take a long time.

By the way, I used to avoid all of these problems by never making
classes, and always building complex structures of lists,
dictionaries, and tuples with global functions. It's going to take me
a while to kick those horrible habits (during my transition, I'm
deriving from list, dict, etc. hehe), perhaps a link to the metaclass
trick is in order in the tutorial's comments on reload?

Any help that avoids having to unpickle again is appreciated!

Thanks,
Andrew Felch

 

[Ziga Seilnacht]

Hello all,

I'm using the metaclass trick for automatic reloading of class member
functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

My problem is that if I
1) pickle an object that inherits from "AutoReloader"
2) unpickle the object
3) modify one of the pickled' object's derived class methods
4) reload the module holding the class

... then the changes don't affect the unpickled object. If I unpickle
the object again, of course the changes take effect.

My friend that loves Smalltalk is laughing at me. I thought I had the
upperhand when I discovered the metaclasses but now I am not sure what
to do. I really don't want to have to unpickle again, I'm processing
video and it can take a long time.

By the way, I used to avoid all of these problems by never making
classes, and always building complex structures of lists,
dictionaries, and tuples with global functions. It's going to take me
a while to kick those horrible habits (during my transition, I'm
deriving from list, dict, etc. hehe), perhaps a link to the metaclass
trick is in order in the tutorial's comments on reload?

Any help that avoids having to unpickle again is appreciated!

Thanks,
Andrew Felch

This happens because unpickling doesn't recreate your object by
calling its type. MetaInstanceTracker registers an instance only
when it is created by calling a class.

You can solve this by moving the instance registration to
AutoReloader.__new__ and using pickle protocol version 2, but the
best solution is to avoid both pickle (old pickles break if you
change your code) and autoreloading (it's meant to be used in
interactive console and entertaining ircbots, not in normal code).

Ziga

 

[andrewfelch]

This happens because unpickling doesn't recreate your object by
calling its type. MetaInstanceTracker registers an instance only
when it is created by calling a class.

You can solve this by moving the instance registration to
AutoReloader.__new__ and using pickle protocol version 2, but the
best solution is to avoid both pickle (old pickles break if you
change your code) and autoreloading (it's meant to be used in
interactive console and entertaining ircbots, not in normal code).

Ziga- Hide quoted text -

- Show quoted text -

Thanks Ziga. I use pickle protocol 2 and binary file types with the
command: "cPickle.dump(obj, file, 2)"

I did your suggestion, i commented out the "__call__" function of
MetaInstanceTracker and copied the text to the __new__ function of
AutoReloader (code appended). I got a crazy recursive error message
(also appended below). In my code, I am creating a new instance,
rather than using the pickled object (it needs to work in both modes).

Thanks very much for helping me get through this. With my development
approach, finding a solution to this problem is really important to
me.

....
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
File "C:\Python25\lib\site-packages\tdbu.py", line 67, in __new__
instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
RuntimeError: maximum recursion depth exceeded

<code>
import weakref, inspect

class MetaInstanceTracker(type):
def __new__(cls, name, bases, ns):
t = super(MetaInstanceTracker, cls).__new__(cls, name, bases,
ns)
t.__instance_refs__ = []
return t
def __instances__(self):
instances = [(r, r()) for r in self.__instance_refs__]
instances = filter(lambda (x,y): y is not None, instances)
self.__instance_refs__ = [r for (r, o) in instances]
return [o for (r, o) in instances]
## def __call__(self, *args, **kw):
## instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
## self.__instance_refs__.append(weakref.ref(instance))
## return instance

class InstanceTracker:
__metaclass__ = MetaInstanceTracker

class MetaAutoReloader(MetaInstanceTracker):
def __new__(cls, name, bases, ns):
new_class = super(MetaAutoReloader, cls).__new__(
cls, name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if d.has_key(name):
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(new_class)
new_class.__instance_refs__.append(
weakref.ref(instance))
# this section only works in 2.3
for subcls in old_class.__subclasses__():
newbases = ()
for base in subcls.__bases__:
if base is old_class:
newbases += (new_class,)
else:
newbases += (base,)
subcls.__bases__ = newbases
break
return new_class

class AutoReloader:
__metaclass__ = MetaAutoReloader
def change_class(self, new_class):
self.__class__ = new_class
def __new__(self, *args, **kw):
instance = super(MetaInstanceTracker, self).__call__(*args,
**kw)
self.__instance_refs__.append(weakref.ref(instance))
return instance

Thanks,
Andrew Felch

 

[Ziga Seilnacht]

Thanks Ziga. I use pickle protocol 2 and binary file types with the
command: "cPickle.dump(obj, file, 2)"

I did your suggestion, i commented out the "__call__" function of
MetaInstanceTracker and copied the text to the __new__ function of
AutoReloader (code appended). I got a crazy recursive error message
(also appended below). In my code, I am creating a new instance,
rather than using the pickled object (it needs to work in both modes).

Thanks very much for helping me get through this. With my development
approach, finding a solution to this problem is really important to
me.

Here is a version that should work. It should work with all protocols,
see InstanceTracker.__reduce_ex__. Note that all subclasses of
InstanceTracker and AutoReloader should be cautious when overriding
the
__new__ method. They must call their base class' __new__ method,
preferably by using super(), or the tracking won't work.


# adapted from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

import weakref, inspect


class MetaInstanceTracker(type):

def __init__(cls, name, bases, ns):
super(MetaInstanceTracker, cls).__init__(name, bases, ns)
cls.__instance_refs__ = []

def __instances__(cls):
instances = []
validrefs = []
for ref in cls.__instance_refs__:
instance = ref()
if instance is not None:
instances.append(instance)
validrefs.append(ref)
cls.__instance_refs__ = validrefs
return instances


class InstanceTracker(object):

__metaclass__ = MetaInstanceTracker

def __new__(*args, **kwargs):
cls = args[0]
self = super(InstanceTracker, cls).__new__(*args, **kwargs)
cls.__instance_refs__.append(weakref.ref(self))
return self

def __reduce_ex__(self, proto):
return super(InstanceTracker, self).__reduce_ex__(2)


class MetaAutoReloader(MetaInstanceTracker):

def __init__(cls, name, bases, ns):
super(MetaAutoReloader, cls).__init__(name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if name in d:
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(cls)

cls.__instance_refs__.append(weakref.ref(instance))
for subcls in old_class.__subclasses__():
newbases = []
for base in subcls.__bases__:
if base is old_class:
newbases.append(cls)
else:
newbases.append(base)
subcls.__bases__ = tuple(newbases)
break


class AutoReloader(InstanceTracker):

__metaclass__ = MetaAutoReloader

def change_class(self, new_class):
self.__class__ = new_class


Ziga

 

[andrewfelch]

Thanks Ziga. I use pickle protocol 2 and binary file types with the
command: "cPickle.dump(obj, file, 2)"

I did your suggestion, i commented out the "__call__" function of
MetaInstanceTracker and copied the text to the __new__ function of
AutoReloader (code appended). I got a crazy recursive error message
(also appended below). In my code, I am creating a new instance,
rather than using the pickled object (it needs to work in both modes).

Thanks very much for helping me get through this. With my development
approach, finding a solution to this problem is really important to
me.

Here is a version that should work. It should work with all protocols,
see InstanceTracker.__reduce_ex__. Note that all subclasses of
InstanceTracker and AutoReloader should be cautious when overriding
the
__new__ method. They must call their base class' __new__ method,
preferably by using super(), or the tracking won't work.

# adapted fromhttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164

import weakref, inspect

class MetaInstanceTracker(type):

def __init__(cls, name, bases, ns):
super(MetaInstanceTracker, cls).__init__(name, bases, ns)
cls.__instance_refs__ = []

def __instances__(cls):
instances = []
validrefs = []
for ref in cls.__instance_refs__:
instance = ref()
if instance is not None:
instances.append(instance)
validrefs.append(ref)
cls.__instance_refs__ = validrefs
return instances

class InstanceTracker(object):

__metaclass__ = MetaInstanceTracker

def __new__(*args, **kwargs):
cls = args[0]
self = super(InstanceTracker, cls).__new__(*args, **kwargs)
cls.__instance_refs__.append(weakref.ref(self))
return self

def __reduce_ex__(self, proto):
return super(InstanceTracker, self).__reduce_ex__(2)

class MetaAutoReloader(MetaInstanceTracker):

def __init__(cls, name, bases, ns):
super(MetaAutoReloader, cls).__init__(name, bases, ns)
f = inspect.currentframe().f_back
for d in [f.f_locals, f.f_globals]:
if name in d:
old_class = d[name]
for instance in old_class.__instances__():
instance.change_class(cls)

cls.__instance_refs__.append(weakref.ref(instance))
for subcls in old_class.__subclasses__():
newbases = []
for base in subcls.__bases__:
if base is old_class:
newbases.append(cls)
else:
newbases.append(base)
subcls.__bases__ = tuple(newbases)
break

class AutoReloader(InstanceTracker):

__metaclass__ = MetaAutoReloader

def change_class(self, new_class):
self.__class__ = new_class

Ziga

I pasted the code into mine and replaced the old. It seems not to
work for either unpickled objects or new objects. I add methods to a
class that inherits from AutoReloader and reload the module, but the
new methods are not callable on the old objects. Man! It seems we're
so close, it will be huge if this ends up working. This stuff is so
over my head, I wish I could help in some additional way.

-Andrew

 

[Ziga Seilnacht]

I pasted the code into mine and replaced the old. It seems not to
work for either unpickled objects or new objects. I add methods to a
class that inherits from AutoReloader and reload the module, but the
new methods are not callable on the old objects. Man! It seems we're
so close, it will be huge if this ends up working. This stuff is so
over my head, I wish I could help in some additional way.

-Andrew

Did you copy and paste the entire module? I fiddled almost all parts
of
the original code. I did some base testing and it worked for me. Could
you post the traceback? Note that Google Groups messed up indentation;
in MetaAutoReloader.__init__, the line starting with
cls.__instance_refs__ should be at the same level as previous line.

Did you restart Python? InstanceTracker, MetaInstanceTracker and
MetaAutoReloader are not auto reloaded .

Ziga

 

[andrewfelch]

Did you copy and paste the entire module? I fiddled almost all parts
of
the original code. I did some base testing and it worked for me. Could
you post the traceback? Note that Google Groups messed up indentation;
in MetaAutoReloader.__init__, the line starting with
cls.__instance_refs__ should be at the same level as previous line.

Did you restart Python? InstanceTracker, MetaInstanceTracker and
MetaAutoReloader are not auto reloaded .

Ziga

Thanks for checking. I think I narrowed the problem down to
inheritance. I inherit from list or some other container first:

class PointList( list, AutoReloader ):
def PrintHi1(self):
print "Hi2"

class MyPrintingClass( AutoReloader ):
def PrintHi2(self):
print "Hi2v2"

Automatic reloading works for MyPrintingClass but not for PointList.
Any ideas?

-Andrew

 

[Ziga Seilnacht]

Thanks for checking. I think I narrowed the problem down to
inheritance. I inherit from list or some other container first:

class PointList( list, AutoReloader ):
def PrintHi1(self):
print "Hi2"

class MyPrintingClass( AutoReloader ):
def PrintHi2(self):
print "Hi2v2"

Automatic reloading works for MyPrintingClass but not for PointList.
Any ideas?

-Andrew

Ah yes, this is the problem of list.__new__ not calling the next
class in MRO. Try to switch the bases, so that AutoReloader's
__new__ method will be called first.

Ziga

 

[andrewfelch]

Ah yes, this is the problem of list.__new__ not calling the next
class in MRO. Try to switch the bases, so that AutoReloader's
__new__ method will be called first.

Ziga

That did it! Thanks so much!

This really seems to extend Python to be a much better rapid
prototyping langauge. It certainly allows me to now use classes where
I avoided them like the plague before. Perhaps I'll be able to make
programs > 1500 lines now!

-Andrew

 

[andrewfelch]

Ah yes, this is the problem of list.__new__ not calling the next
class in MRO. Try to switch the bases, so that AutoReloader's
__new__ method will be called first.

Ziga- Hide quoted text -

- Show quoted text -

So there is another bug crippling Python (see
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=628925&group_id=5470
). Pickle and cPickle get confused after reloading, and won't let you
dump the objects to file using protocol 2. The bug occurred for me
when an instance of an AutoReloader class had a member not derived
from AutoReloader (error text: "it's not the same object" ). The
workaround is to use text mode, which I assume means not using
protocol 2. After that, you can reload them in text mode, and then
dump them using protocol 2. This is not so bad since you only do it
when you were going to dump objects to file anyway, so it really only
takes 3x as long on those occassions. The metaclass adaptation is
still incredibly useful because it avoids having to do pickle dump-
loads when you wouldn't otherwise have had to (approaching an infinite
speedup in those cases, thanks Ziga! .

Another caveat is that pickle dumping doesn't work if there is a
pointer to a global function (that changes) that is a member of an
instance of a class derived from AutoLoader. The error message has
subtext: "it's not the same object ". That doesn't seem so bad to me,
since you don't have to do that type of thing (function pointers) so
much now that you can use classes!

The best solution would be to make pickle protocol 2 have the same
behavior as the text protocol (i.e. actually work in this case).

Regarding the function pointers, anybody (Ziga know why it is that
pickle chokes so hard on this? Is there some adaptation of the
metaclass or a less beautiful hack that could fix it?

Thanks to those readers following me down the rabbit hole. Ziga's
metaclass adaptation has saved Python in my mind, and these nits seem
to be worth discussing.

- Andrew Felch