Exception classes don't follow pickle protocol, problems unpickling

Discussion in 'Python' started by Irmen de Jong, Dec 6, 2009.

  1. Hi,

    I am puzzled why Python's exception classes don't seem to follow the pickle protocol.
    To be more specific: an instance of a user defined exception, subclassed from Exception,
    cannot be pickled/unpickled correctly in the expected way.

    The pickle protocol says that:
    __getinitargs__ is used if you want __init__ to be called for old-style classes
    __getnewargs__ is used if you want to pass params to __new__ for new-style classes
    __getstate__ is used to determine what to pickle instead of the objects __dict__

    None of these are used when pickling Exception objects!
    I've pasted some test code at the end of this message that creates a normal object and
    an object derived from Exception, and pickles them both. Then it unpickles them.
    The output is:

    creating MyObj
    myobj init: bla
    creating MyEx
    myex init: foutje
    pickle myobj
    myobj getnewargs # <-- great, newargs is used because of new style class
    myobj getstate # getstate as well
    pickle myex
    unpickling MyObj # <-- huh?? where are getnewargs/getinitargs/getstate?
    unpickled MyObj: [MyObj 'bla']
    unpickling MyEx
    Traceback (most recent call last):
    File "ex.py", line 49, in <module>
    o=pickle.loads(p_myex)
    TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.MyEx'>, ())


    So there are 2 problems: the pickle protocol isn't used when exception objects (or
    instances of classes derived from Exception) are pickled, and during unpickling, it then
    crashes because it calls __init__ with the wrong amount of parameters. (why is it
    bothering with __init__ anyway? Aren't exceptions new style classes?)


    This started happening in Python 2.5, Python 2.4 works without error.

    What is causing this? How can I best solve this error?
    (using pickle instead of cPickle and changing the protocol number doesn't help).

    --irmen


    ~~~~code follows~~~~

    import cPickle as pickle

    class MyObj(object):
    def __init__(self, mymessage):
    print "myobj init:",mymessage
    self.mymessage=mymessage
    def __str__(self):
    return "[MyObj '%s']" % self.mymessage
    def __getinitargs__(self):
    print "myobj getinitargs"
    return (self.mymessage,)
    def __getnewargs__(self):
    print "myobj getnewargs"
    return (self.mymessage,)
    def __getstate__(self):
    print "myobj getstate"
    return {"mymessage":self.mymessage}

    class MyEx(Exception):
    def __init__(self, mymessage):
    print "myex init:",mymessage
    self.mymessage=mymessage
    def __str__(self):
    return "[MyEx '%s']" % self.mymessage
    def __getinitargs__(self):
    print "myex getinitargs"
    return (self.mymessage,)
    def __getnewargs__(self):
    print "myex getnewargs"
    return (self.mymessage,)
    def __getstate__(self):
    print "myex getstate"
    return {"mymessage":self.mymessage}

    print "creating MyObj"
    myobj=MyObj("bla")
    print "creating MyEx"
    myex=MyEx("foutje")
    print "pickle myobj"
    p_myobj=pickle.dumps(myobj,protocol=pickle.HIGHEST_PROTOCOL)
    print "pickle myex"
    p_myex=pickle.dumps(myex,protocol=pickle.HIGHEST_PROTOCOL)

    print "unpickling MyObj"
    o=pickle.loads(p_myobj)
    print "unpickled MyObj:",o

    print "unpickling MyEx"
    o=pickle.loads(p_myex)
    print "unpickled MyEx:",o
     
    Irmen de Jong, Dec 6, 2009
    #1
    1. Advertising

  2. Irmen de Jong

    Peter Otten Guest

    Irmen de Jong wrote:

    > I am puzzled why Python's exception classes don't seem to follow the
    > pickle protocol. To be more specific: an instance of a user defined
    > exception, subclassed from Exception, cannot be pickled/unpickled
    > correctly in the expected way.
    >
    > The pickle protocol says that:
    > __getinitargs__ is used if you want __init__ to be called for old-style
    > classes __getnewargs__ is used if you want to pass params to __new__ for
    > new-style classes __getstate__ is used to determine what to pickle instead
    > of the objects __dict__
    >
    > None of these are used when pickling Exception objects!
    > I've pasted some test code at the end of this message that creates a
    > normal object and an object derived from Exception, and pickles them both.
    > Then it unpickles them. The output is:



    > So there are 2 problems: the pickle protocol isn't used when exception
    > objects (or instances of classes derived from Exception) are pickled, and
    > during unpickling, it then
    > crashes because it calls __init__ with the wrong amount of parameters.
    > (why is it bothering with __init__ anyway? Aren't exceptions new style
    > classes?)


    The __reduce__() method is called when you pickle an object. It returns an
    argument tuple and a factory. For exceptions that factory is the class which
    is why __init__() is called when the object is unpickled.

    > This started happening in Python 2.5, Python 2.4 works without error.
    >
    > What is causing this?


    I think Exceptions need special treatment because they have state that is
    not stored in the instance __dict__.

    > How can I best solve this error?


    You could override __reduce__() to call __getstate__(). I don't see the need
    for __getnewargs__() because exceptions aren't immutable.

    Peter
     
    Peter Otten, Dec 7, 2009
    #2
    1. Advertising

  3. On 7-12-2009 10:12, Peter Otten wrote:

    >> So there are 2 problems: the pickle protocol isn't used when exception
    >> objects (or instances of classes derived from Exception) are pickled, and
    >> during unpickling, it then
    >> crashes because it calls __init__ with the wrong amount of parameters.
    >> (why is it bothering with __init__ anyway? Aren't exceptions new style
    >> classes?)

    >
    > The __reduce__() method is called when you pickle an object. It returns an
    > argument tuple and a factory. For exceptions that factory is the class which
    > is why __init__() is called when the object is unpickled.


    I didn't realize exceptions are treated as "objects pickle know nothing
    about" (or 'extension types'). They weren't in Python 2.4 at least :)

    So I was expecting them to follow the pickle protocol for normal Python
    types, and didn't look past __getnewargs__ and __getinitargs__...


    >> This started happening in Python 2.5, Python 2.4 works without error.
    >>
    >> What is causing this?

    >
    > I think Exceptions need special treatment because they have state that is
    > not stored in the instance __dict__.
    >
    >> How can I best solve this error?

    >
    > You could override __reduce__() to call __getstate__(). I don't see the need
    > for __getnewargs__() because exceptions aren't immutable.
    >
    > Peter


    Thanks, that was enlightening.
    I never had to deal with __reduce__ before :)

    --irmen
     
    Irmen de Jong, Dec 8, 2009
    #3
  4. Irmen de Jong

    kerby2000

    Joined:
    Jan 26, 2011
    Messages:
    1
    Hi Guys,

    I have exactly the same problem with unpickling exceptions.
    Could you please give me some code which shows how to use __reduce__().
    I never used it before.
     
    kerby2000, Jan 26, 2011
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Erwin S. Andreasen
    Replies:
    3
    Views:
    340
    Erwin S. Andreasen
    Feb 24, 2006
  2. Boris Borcic
    Replies:
    2
    Views:
    342
    Tim Peters
    May 19, 2006
  3. -dresden.de
    Replies:
    2
    Views:
    524
    Peter Otten
    Mar 12, 2008
  4. Fredrik Tolf
    Replies:
    0
    Views:
    113
    Fredrik Tolf
    Aug 21, 2013
  5. Prasad, Ramit
    Replies:
    0
    Views:
    87
    Prasad, Ramit
    Aug 21, 2013
Loading...

Share This Page