Infinite recursion in __reduce__ when calling original base classreduce, why?

[Irmen de Jong]

Hi,

I'm having a rather obscure problem with my custom __reduce__ function. I can't use
__getstate__ to customize the pickling of my class because I want to change the actual
type that is put into the pickle stream. So I started experimenting with __reduce__, but
am running into some trouble.

I've pasted my test code below. It works fine if 'substitute' is True, but as soon as it
is set to False, it is supposed to call the original __reduce__ method of the base
class. However, that seems to crash because of infinite recursion on Jython and
IronPython and I don't know why. It works fine in CPython and Pypy.

I wonder if my understanding of __reduce__ is wrong, or that I've hit a bug in
IronPython and Jython? Do I need to do something with __reduce_ex__ as well?

Any help is very much appreciated.

Irmen de Jong



# ironpython / jython __reduce__ recursion problem test program

import pickle

class Substitute(object):
def __init__(self, name):
self.name=name
def getname(self):
return self.name

class TestClass(object):
def __init__(self, name):
self.name=name
self.substitute=True
def getname(self):
return self.name
def __reduce__(self):
if self.substitute:
return Substitute, ("SUBSTITUTED:"+self.name,)
else:
# call the original __reduce__ from the base class
return super(TestClass, self).__reduce__() # crashes on ironpython/jython

obj=TestClass("janet")

s=pickle.dumps(obj)
d=pickle.loads(s)
print(d)
print(d.getname())

# now disable the substitution and try again
obj.substitute=False
s=pickle.dumps(obj)
d=pickle.loads(s)
print(d)
print(d.getname())

 

[Chris Torek]

I've pasted my test code below. It works fine if 'substitute' is True,
but as soon as it is set to False, it is supposed to call the original
__reduce__ method of the base class. However, that seems to crash
because of infinite recursion on Jython and IronPython and I don't
know why. It works fine in CPython and Pypy.

In this particular case (no fancy inheritance going on), the base
__reduce__ method would be object.__reduce__. Perhaps in those
implementations, object.__reduce__ goes back to TestClass.__reduce__,
rather than being appropriately magic.

I wonder if my understanding of __reduce__ is wrong, or that I've
hit a bug in IronPython and Jython? Do I need to do something with
__reduce_ex__ as well?

You should not *need* to; __reduce_ex__ is just there so that you
can do something different for different versions of the pickle
protocol (I believe).

Nonetheless, there is something at least slightly suspicious here:

import pickle

class Substitute(object):
def __init__(self, name):
self.name=name
def getname(self):
return self.name

class TestClass(object):
def __init__(self, name):
self.name=name
self.substitute=True
def getname(self):
return self.name
def __reduce__(self):
if self.substitute:
return Substitute, ("SUBSTITUTED:"+self.name,)
else:
# call the original __reduce__ from the base class
return super(TestClass, self).__reduce__() # crashes on
ironpython/jython

[snip]

In general, the way __reduce__ is written in other class implementations
(as distributed with Python2.5 at least) boils down to the very
simple:

def __reduce__(self):
return self.__class__, (arg, um, ents)

For instance, consider a class with a piece that looks like this:

def __init__(self, name, value):
self.name = name
self.value = value
self.giant_cached_state = None

def make_parrot_move(self):
if self.giant_cached_state is None:
self._do_lots_of_computation()
return self._quickstuff_using_cache()

Here, the Full Internal State is fairly long but the part that
needs to be saved (or, for copy operations, copied -- but you can
override this with __copy__ and __deepcopy__ members, if copying
the cached state is a good idea) is quite short. Pickled instances
need only save the name and value, not any of the computed cached
stuff (if present). So:

def __reduce__(self):
return self.__class__, (name, value)

If you define this (and no __copy__ and no __deepcopy__), the
pickler will save the name and value and call __init__ with the
name and value arguments. The copy.copy and copy.deepcopy operations
will also call __init__ with these arguments (unless you add
__copy__(self) and __deepcopy__(self) functions).

So, it seems like in this case, you would want:

def __reduce__(self):
if self.substitute:
return Substitute, ("SUBSTITUTED:"+self.name,)
else:
return self.__class__, (self.name,)

or if you want to be paranoid and only do a Substitute if
self.__class__ is your own class:

if type(self) == TestClass and self.substitute:
return Substitute, ("SUBSTITUTED:"+self.name,)
else:
return self.__class__, (self.name,)

In CPython, if I import your code (saved in foo.py):

object.__reduce__(x)

[/QUOTE]
(<function _reconstructor at 0x70bf0>, (<class 'foo.TestClass'>, <type 'object'>, None), {'name': 'janet', 'substitute': False})

which means that CPython's object.__reduce__() uses a "smart" fallback
reconstructor. Presumably IronPython and Jython lack this.

 

[Irmen de Jong]

Nonetheless, there is something at least slightly suspicious here:

[... snip explanations...]

Many thanks Chris, for the extensive reply. There's some useful knowledge in it.

My idea to call the base class reduce as the default fallback causes the problems:
return return super(TestClass, self).__reduce__()
If, following your suggestion, I replace that with:
return self.__class__, (self.name,)
it works fine.

By the way, in the meantime I've played around with copyreg.pickle, and that code worked
in all Python implementations I've tried it in. This code feels better too, because it
is possible to simply use
return self.__reduce__()
as a fallback in it, because we're not touching the object's own __reduce__ in any way.

Anyway thanks again

Irmen