Exception classes don't follow pickle protocol, problems unpickling

[Irmen de Jong]

Hi,

I am puzzled why Python's exception classes don't seem to follow the pickle protocol.
To be more specific: an instance of a user defined exception, subclassed from Exception,
cannot be pickled/unpickled correctly in the expected way.

The pickle protocol says that:
__getinitargs__ is used if you want __init__ to be called for old-style classes
__getnewargs__ is used if you want to pass params to __new__ for new-style classes
__getstate__ is used to determine what to pickle instead of the objects __dict__

None of these are used when pickling Exception objects!
I've pasted some test code at the end of this message that creates a normal object and
an object derived from Exception, and pickles them both. Then it unpickles them.
The output is:

creating MyObj
myobj init: bla
creating MyEx
myex init: foutje
pickle myobj
myobj getnewargs # <-- great, newargs is used because of new style class
myobj getstate # getstate as well
pickle myex
unpickling MyObj # <-- huh?? where are getnewargs/getinitargs/getstate?
unpickled MyObj: [MyObj 'bla']
unpickling MyEx
Traceback (most recent call last):
File "ex.py", line 49, in <module>
o=pickle.loads(p_myex)
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.MyEx'>, ())


So there are 2 problems: the pickle protocol isn't used when exception objects (or
instances of classes derived from Exception) are pickled, and during unpickling, it then
crashes because it calls __init__ with the wrong amount of parameters. (why is it
bothering with __init__ anyway? Aren't exceptions new style classes?)


This started happening in Python 2.5, Python 2.4 works without error.

What is causing this? How can I best solve this error?
(using pickle instead of cPickle and changing the protocol number doesn't help).

--irmen


~~~~code follows~~~~

import cPickle as pickle

class MyObj(object):
def __init__(self, mymessage):
print "myobj init:",mymessage
self.mymessage=mymessage
def __str__(self):
return "[MyObj '%s']" % self.mymessage
def __getinitargs__(self):
print "myobj getinitargs"
return (self.mymessage,)
def __getnewargs__(self):
print "myobj getnewargs"
return (self.mymessage,)
def __getstate__(self):
print "myobj getstate"
return {"mymessage":self.mymessage}

class MyEx(Exception):
def __init__(self, mymessage):
print "myex init:",mymessage
self.mymessage=mymessage
def __str__(self):
return "[MyEx '%s']" % self.mymessage
def __getinitargs__(self):
print "myex getinitargs"
return (self.mymessage,)
def __getnewargs__(self):
print "myex getnewargs"
return (self.mymessage,)
def __getstate__(self):
print "myex getstate"
return {"mymessage":self.mymessage}

print "creating MyObj"
myobj=MyObj("bla")
print "creating MyEx"
myex=MyEx("foutje")
print "pickle myobj"
p_myobj=pickle.dumps(myobj,protocol=pickle.HIGHEST_PROTOCOL)
print "pickle myex"
p_myex=pickle.dumps(myex,protocol=pickle.HIGHEST_PROTOCOL)

print "unpickling MyObj"
o=pickle.loads(p_myobj)
print "unpickled MyObj:",o

print "unpickling MyEx"
o=pickle.loads(p_myex)
print "unpickled MyEx:",o

 

[Peter Otten]

I am puzzled why Python's exception classes don't seem to follow the
pickle protocol. To be more specific: an instance of a user defined
exception, subclassed from Exception, cannot be pickled/unpickled
correctly in the expected way.

The pickle protocol says that:
__getinitargs__ is used if you want __init__ to be called for old-style
classes __getnewargs__ is used if you want to pass params to __new__ for
new-style classes __getstate__ is used to determine what to pickle instead
of the objects __dict__

None of these are used when pickling Exception objects!
I've pasted some test code at the end of this message that creates a
normal object and an object derived from Exception, and pickles them both.
Then it unpickles them. The output is:

So there are 2 problems: the pickle protocol isn't used when exception
objects (or instances of classes derived from Exception) are pickled, and
during unpickling, it then
crashes because it calls __init__ with the wrong amount of parameters.
(why is it bothering with __init__ anyway? Aren't exceptions new style
classes?)

The __reduce__() method is called when you pickle an object. It returns an
argument tuple and a factory. For exceptions that factory is the class which
is why __init__() is called when the object is unpickled.

This started happening in Python 2.5, Python 2.4 works without error.

What is causing this?

I think Exceptions need special treatment because they have state that is
not stored in the instance __dict__.

How can I best solve this error?

You could override __reduce__() to call __getstate__(). I don't see the need
for __getnewargs__() because exceptions aren't immutable.

Peter

 

[Irmen de Jong]

The __reduce__() method is called when you pickle an object. It returns an
argument tuple and a factory. For exceptions that factory is the class which
is why __init__() is called when the object is unpickled.

I didn't realize exceptions are treated as "objects pickle know nothing
about" (or 'extension types'). They weren't in Python 2.4 at least

So I was expecting them to follow the pickle protocol for normal Python
types, and didn't look past __getnewargs__ and __getinitargs__...

I think Exceptions need special treatment because they have state that is
not stored in the instance __dict__.


You could override __reduce__() to call __getstate__(). I don't see the need
for __getnewargs__() because exceptions aren't immutable.

Peter

Thanks, that was enlightening.
I never had to deal with __reduce__ before

--irmen

 

[kerby2000]

Hi Guys,

I have exactly the same problem with unpickling exceptions.
Could you please give me some code which shows how to use __reduce__().
I never used it before.