why can't I pickle a class containing this dispatch dictionary?

Discussion in 'Python' started by jkn, Apr 2, 2012.

  1. jkn

    jkn Guest

    Hi All
    I'm clearly not understanding the 'can't pickle instancemethod
    objects' error; can someone help me to understand, & maybe suggest a
    workaround, (apart from the obvious if ... elif...).

    I'm running Python 2.6 on an embedded system.

    == testpickle.py ==
    import pickle

    class Test(object):
    def __init__(self):
    self.myDict = {
    1: self.tag1,
    2: self.tag2
    }
    def dispatch(self, v):
    try:
    self.myDict[v]()
    except KeyError:
    print "No corresponding dictionary entry!"
    #
    def tag1(self):
    print "one"
    def tag2(self):
    print "two"


    t = Test()
    t.dispatch(1)
    t.dispatch(2)
    t.dispatch(0)

    fd = open("pickle.out", "w")
    pickle.dump(t, fd)
    fd.close()
    # EOF

    $ python testpickle.py
    one
    two
    No corresponding dictionary entry!
    Traceback (most recent call last):
    File "ptest.py", line 29, in <module>
    pickle.dump(t, fd)
    File "/usr/lib/python2.6/pickle.py", line 1362, in dump
    Pickler(file, protocol).dump(obj)
    File "/usr/lib/python2.6/pickle.py", line 224, in dump
    self.save(obj)
    File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
    File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
    File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
    File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
    File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
    File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    save(v)
    File "/usr/lib/python2.6/pickle.py", line 306, in save
    rv = reduce(self.proto)
    File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle instancemethod objects
    $


    Thanks
    J^n
     
    jkn, Apr 2, 2012
    #1
    1. Advertising

  2. Pickle cannot pickle a reference to an instance method. So the
    problem is that self.myDict has values which are references to
    instance methods.

    Without questioning what this is trying to do or why (I assume it's a
    proof of concept), here is a way to make it picklable:
    http://pastebin.com/1zqE52mD

    Michael

    On Mon, Apr 2, 2012 at 7:17 AM, jkn <9.co.uk> wrote:
    > Hi All
    >    I'm clearly not understanding the 'can't pickle instancemethod
    > objects' error; can someone help me to understand, & maybe suggest a
    > workaround, (apart from the obvious if ... elif...).
    >
    > I'm running Python 2.6 on an embedded system.
    >
    > == testpickle.py ==
    > import pickle
    >
    > class Test(object):
    >    def __init__(self):
    >        self.myDict = {
    >            1: self.tag1,
    >            2: self.tag2
    >            }
    >    def dispatch(self, v):
    >        try:
    >            self.myDict[v]()
    >        except KeyError:
    >            print "No corresponding dictionary entry!"
    >        #
    >    def tag1(self):
    >        print "one"
    >    def tag2(self):
    >        print "two"
    >
    >
    > t = Test()
    > t.dispatch(1)
    > t.dispatch(2)
    > t.dispatch(0)
    >
    > fd = open("pickle.out", "w")
    > pickle.dump(t, fd)
    > fd.close()
    > # EOF
    >
    > $ python testpickle.py
    > one
    > two
    > No corresponding dictionary entry!
    > Traceback (most recent call last):
    >  File "ptest.py", line 29, in <module>
    >    pickle.dump(t, fd)
    >  File "/usr/lib/python2.6/pickle.py", line 1362, in dump
    >    Pickler(file, protocol).dump(obj)
    >  File "/usr/lib/python2.6/pickle.py", line 224, in dump
    >    self.save(obj)
    >  File "/usr/lib/python2.6/pickle.py", line 331, in save
    >    self.save_reduce(obj=obj, *rv)
    >  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    >    save(state)
    >  File "/usr/lib/python2.6/pickle.py", line 286, in save
    >    f(self, obj) # Call unbound method with explicit self
    >  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    >    self._batch_setitems(obj.iteritems())
    >  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    >    save(v)
    >  File "/usr/lib/python2.6/pickle.py", line 286, in save
    >    f(self, obj) # Call unbound method with explicit self
    >  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    >    self._batch_setitems(obj.iteritems())
    >  File "/usr/lib/python2.6/pickle.py", line 663, in _batch_setitems
    >    save(v)
    >  File "/usr/lib/python2.6/pickle.py", line 306, in save
    >    rv = reduce(self.proto)
    >  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    >    raise TypeError, "can't pickle %s objects" % base.__name__
    > TypeError: can't pickle instancemethod objects
    > $
    >
    >
    >    Thanks
    >    J^n
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
     
    Michael Hrivnak, Apr 2, 2012
    #2
    1. Advertising

  3. jkn

    Peter Otten Guest

    jkn wrote:

    > I'm clearly not understanding the 'can't pickle instancemethod
    > objects' error; can someone help me to understand,


    I think classes implemented in C need some extra work to make them
    picklable, and that hasn't been done for instance methods.

    > & maybe suggest a
    > workaround, (apart from the obvious if ... elif...).


    You can implement pickling yourself:

    import copy_reg
    import types

    def pickle_instancemethod(m):
    return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
    m.im_class)

    def unpickle_instancemethod(name, im_self, im_class):
    im_func = getattr(im_class, name)
    return im_func.__get__(im_self, im_class)

    copy_reg.pickle(types.MethodType, pickle_instancemethod)


    > I'm running Python 2.6 on an embedded system.
    >
    > == testpickle.py ==
    > import pickle
    >
    > class Test(object):
    > def __init__(self):
    > self.myDict = {
    > 1: self.tag1,
    > 2: self.tag2
    > }
    > def dispatch(self, v):
    > try:
    > self.myDict[v]()
    > except KeyError:
    > print "No corresponding dictionary entry!"
    > #
    > def tag1(self):
    > print "one"
    > def tag2(self):
    > print "two"
    >
    >
    > t = Test()
    > t.dispatch(1)
    > t.dispatch(2)
    > t.dispatch(0)
    >
    > fd = open("pickle.out", "w")
    > pickle.dump(t, fd)
    > fd.close()
    > # EOF
    >
    > $ python testpickle.py
    > one
    > two
    > No corresponding dictionary entry!


    > TypeError: can't pickle instancemethod objects
    > $
     
    Peter Otten, Apr 3, 2012
    #3
  4. jkn

    jkn Guest

    Hi Peter

    On Apr 3, 8:54 am, Peter Otten <> wrote:
    > jkn wrote:
    > >     I'm clearly not understanding the 'can't pickle instancemethod
    > > objects' error; can someone help me to understand,

    >
    > I think classes implemented in C need some extra work to make them
    > picklable, and that hasn't been done for instance methods.


    by 'classes implemented in C', doyou mean new-style classes', or what,
    please?


    >
    > > & maybe suggest a
    > > workaround, (apart from the obvious if ... elif...).

    >
    > You can implement pickling yourself:
    >
    > [...]


    Hmm - interesting, thanks. I'm more trying to understand the issue at
    the moment, but it's always nice to learn...

    Thanks
    J^n
     
    jkn, Apr 3, 2012
    #4
  5. Peter Ottenæ–¼ 2012å¹´4月3日星期二UTC+8下åˆ3時54分50秒寫é“:
    > jkn wrote:
    >
    > > I'm clearly not understanding the 'can't pickle instancemethod
    > > objects' error; can someone help me to understand,

    >
    > I think classes implemented in C need some extra work to make them
    > picklable, and that hasn't been done for instance methods.
    >
    > > & maybe suggest a
    > > workaround, (apart from the obvious if ... elif...).

    >
    > You can implement pickling yourself:
    >
    > import copy_reg
    > import types
    >
    > def pickle_instancemethod(m):
    > return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
    > m.im_class)
    >
    > def unpickle_instancemethod(name, im_self, im_class):
    > im_func = getattr(im_class, name)
    > return im_func.__get__(im_self, im_class)
    >
    > copy_reg.pickle(types.MethodType, pickle_instancemethod)
    >
    >
    > > I'm running Python 2.6 on an embedded system.
    > >
    > > == testpickle.py ==
    > > import pickle
    > >
    > > class Test(object):
    > > def __init__(self):
    > > self.myDict = {
    > > 1: self.tag1,
    > > 2: self.tag2
    > > }
    > > def dispatch(self, v):
    > > try:
    > > self.myDict[v]()
    > > except KeyError:
    > > print "No corresponding dictionary entry!"
    > > #
    > > def tag1(self):
    > > print "one"
    > > def tag2(self):
    > > print "two"
    > >
    > >
    > > t = Test()
    > > t.dispatch(1)
    > > t.dispatch(2)
    > > t.dispatch(0)
    > >
    > > fd = open("pickle.out", "w")
    > > pickle.dump(t, fd)
    > > fd.close()
    > > # EOF
    > >
    > > $ python testpickle.py
    > > one
    > > two
    > > No corresponding dictionary entry!

    >
    > > TypeError: can't pickle instancemethod objects
    > > $


    Save your python files as a package in .pyd or .py and use exec to get what you want. Of course you can use the data compression package to perform
    serialization operations, but that will increase start up time in loading your objects.
     
    88888 Dihedral, Apr 3, 2012
    #5
  6. Peter Ottenæ–¼ 2012å¹´4月3日星期二UTC+8下åˆ3時54分50秒寫é“:
    > jkn wrote:
    >
    > > I'm clearly not understanding the 'can't pickle instancemethod
    > > objects' error; can someone help me to understand,

    >
    > I think classes implemented in C need some extra work to make them
    > picklable, and that hasn't been done for instance methods.
    >
    > > & maybe suggest a
    > > workaround, (apart from the obvious if ... elif...).

    >
    > You can implement pickling yourself:
    >
    > import copy_reg
    > import types
    >
    > def pickle_instancemethod(m):
    > return unpickle_instancemethod, (m.im_func.__name__, m.im_self,
    > m.im_class)
    >
    > def unpickle_instancemethod(name, im_self, im_class):
    > im_func = getattr(im_class, name)
    > return im_func.__get__(im_self, im_class)
    >
    > copy_reg.pickle(types.MethodType, pickle_instancemethod)
    >
    >
    > > I'm running Python 2.6 on an embedded system.
    > >
    > > == testpickle.py ==
    > > import pickle
    > >
    > > class Test(object):
    > > def __init__(self):
    > > self.myDict = {
    > > 1: self.tag1,
    > > 2: self.tag2
    > > }
    > > def dispatch(self, v):
    > > try:
    > > self.myDict[v]()
    > > except KeyError:
    > > print "No corresponding dictionary entry!"
    > > #
    > > def tag1(self):
    > > print "one"
    > > def tag2(self):
    > > print "two"
    > >
    > >
    > > t = Test()
    > > t.dispatch(1)
    > > t.dispatch(2)
    > > t.dispatch(0)
    > >
    > > fd = open("pickle.out", "w")
    > > pickle.dump(t, fd)
    > > fd.close()
    > > # EOF
    > >
    > > $ python testpickle.py
    > > one
    > > two
    > > No corresponding dictionary entry!

    >
    > > TypeError: can't pickle instancemethod objects
    > > $


    Save your python files as a package in .pyd or .py and use exec to get what you want. Of course you can use the data compression package to perform
    serialization operations, but that will increase start up time in loading your objects.
     
    88888 Dihedral, Apr 3, 2012
    #6
  7. jkn

    Peter Otten Guest

    jkn wrote:

    > Hi Peter
    >
    > On Apr 3, 8:54 am, Peter Otten <> wrote:
    >> jkn wrote:
    >> > I'm clearly not understanding the 'can't pickle instancemethod
    >> > objects' error; can someone help me to understand,

    >>
    >> I think classes implemented in C need some extra work to make them
    >> picklable, and that hasn't been done for instance methods.

    >
    > by 'classes implemented in C', doyou mean new-style classes', or what,
    > please?


    Given

    >>> class A(object):

    .... def __init__(self, name):
    .... self.name = name
    .... def hello(self):
    .... print "Hello,", self.name
    ....
    >>> a = A("Peter")
    >>> hello = a.hello
    >>> hello()

    Hello, Peter

    the object bound to the name 'hello' is an instance of the 'instancemethod'
    type:

    >>> type(hello)

    <type 'instancemethod'>

    That type is implemented in C, see

    http://hg.python.org/cpython/file/9599f091faa6/Objects/classobject.c

    and doesn't support the pickle protocol while a similar class,
    functools.partial which is also written in C, see

    http://hg.python.org/cpython/file/9599f091faa6/Modules/_functoolsmodule.c

    does:

    >>> from functools import partial
    >>> import pickle
    >>> def hello(obj):

    .... print "Hi,", obj.name
    ....
    >>> hello2 = partial(hello, a)
    >>> hello2()

    Hi, Peter
    >>> s = pickle.dumps(hello2)
    >>> pickle.loads(s)()

    Hi, Peter
     
    Peter Otten, Apr 3, 2012
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    393
  2. Mr. SweatyFinger
    Replies:
    2
    Views:
    2,074
    Smokey Grindel
    Dec 2, 2006
  3. lazy
    Replies:
    2
    Views:
    350
    Alex Martelli
    Jun 15, 2007
  4. Michele Simionato
    Replies:
    2
    Views:
    1,911
    Michele Simionato
    May 23, 2008
  5. Nagu
    Replies:
    1
    Views:
    308
    Martin v. Löwis
    Jul 7, 2008
Loading...

Share This Page