Why can't you pickle instancemethods?

Discussion in 'Python' started by Chris, Oct 20, 2006.

  1. Chris

    Chris Guest

    Why can pickle serialize references to functions, but not methods?

    Pickling a function serializes the function name, but pickling a
    staticmethod, classmethod, or instancemethod generates an error. In
    these cases, pickle knows the instance or class, and the method, so
    what's the problem? Pickle doesn't serialize code objects, so why can't
    it serialize the name as it does for functions? Is this one of those
    features that's feasible, but not useful, so no one's ever gotten
    around to implementing it?

    Regards,
    Chris

    >>> import pickle
    >>>
    >>> def somefunc():

    .... return 1
    ....
    >>> class Functions(object):

    .... @staticmethod
    .... def somefunc():
    .... return 1
    ....
    >>> class Foo(object):

    .... pass
    ....
    >>> f = Foo()
    >>> f.value = somefunc
    >>> print pickle.dumps(f)

    ccopy_reg
    _reconstructor
    p0
    (c__main__
    Foo
    p1
    c__builtin__
    object
    p2
    Ntp3
    Rp4
    (dp5
    S'value'
    p6
    c__main__
    somefunc
    p7
    sb.
    >>>
    >>> f.value = Functions.somefunc
    >>> print pickle.dumps(f)

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "C:\Program Files\Python24\lib\pickle.py", line 1386, in dumps
    Pickler(file, protocol, bin).dump(obj)
    File "C:\Program Files\Python24\lib\pickle.py", line 231, in dump
    self.save(obj)
    File "C:\Program Files\Python24\lib\pickle.py", line 338, in save
    self.save_reduce(obj=obj, *rv)
    File "C:\Program Files\Python24\lib\pickle.py", line 433, in
    save_reduce
    save(state)
    File "C:\Program Files\Python24\lib\pickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Program Files\Python24\lib\pickle.py", line 663, in
    save_dict
    self._batch_setitems(obj.iteritems())
    File "C:\Program Files\Python24\lib\pickle.py", line 677, in
    _batch_setitems
    save(v)
    File "C:\Program Files\Python24\lib\pickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Program Files\Python24\lib\pickle.py", line 765, in
    save_global
    raise PicklingError(
    pickle.PicklingError: Can't pickle <function somefunc at 0x009EC5F0>:
    it's not the same object as __
    main__.somefunc
     
    Chris, Oct 20, 2006
    #1
    1. Advertising

  2. Chris

    Guest

    Chris wrote:
    > Why can pickle serialize references to functions, but not methods?
    >
    > Pickling a function serializes the function name, but pickling a
    > staticmethod, classmethod, or instancemethod generates an error. In
    > these cases, pickle knows the instance or class, and the method, so
    > what's the problem? Pickle doesn't serialize code objects, so why can't
    > it serialize the name as it does for functions? Is this one of those
    > features that's feasible, but not useful, so no one's ever gotten
    > around to implementing it?


    I have often wondered this myself. I'm convinced that it would in fact
    be useful -- more than once I've written a program that has lots of
    objects with function pointers, and where it was inconvenient that the
    method pointers could not be pickled. One compromise that I have used
    before is to write a class such as:

    class InstanceMethodSet(object):
    def __init__(self,methods):
    self.methods = set(methods)
    def __getstate__(self):
    return [(method.im_self, method.im_func.func_name)
    for method in self.method]
    def __setstate__(self,state):
    self.methods = set(getattr(obj,name) for obj,name in state)

    Obviously, this particular example is crude and not terribly robust,
    but it seems to do the job -- it effectively lets you pickle a set of
    instance method pointers. I don't know of any reason why instance
    methods (or class or static methods) couldn't be pickled directly,
    unless perhaps there exists some kind of pathological corner case that
    would create Badness?

    -Matt
     
    , Oct 20, 2006
    #2
    1. Advertising

  3. At Friday 20/10/2006 18:33, Chris wrote:

    >Why can pickle serialize references to functions, but not methods?


    Because all references must be globally accessible.

    >Pickling a function serializes the function name, but pickling a
    >staticmethod, classmethod, or instancemethod generates an error. In
    >these cases, pickle knows the instance or class, and the method, so
    >what's the problem? Pickle doesn't serialize code objects, so why can't
    >it serialize the name as it does for functions? Is this one of those
    >features that's feasible, but not useful, so no one's ever gotten
    >around to implementing it?


    Well, it's not so common to take a method from one class and glue it
    into an *instance* of another class...
    (Note that if you copy&paste a method from one *class* into another
    *class*, pickle works OK, as long as in the unpickling environment
    you rebuild your classes the same way)

    For static methods there is no way you can retrieve a globally usable
    name (like 'Functions.somefunc' in your example) - at least I don't
    know how to do it, they appear to have lost any reference to the
    defining class.
    For class methods you can build such reference using im_self and
    im_func.func_name
    For instance methods use im_class and im_func.func_name
    Then define your own __getstate__ and __setstate__.


    --
    Gabriel Genellina
    Softlab SRL

    __________________________________________________
    Correo Yahoo!
    Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
    ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar
     
    Gabriel Genellina, Oct 20, 2006
    #3
  4. Chris

    Chris Guest

    wrote:
    > Chris wrote:
    > > Why can pickle serialize references to functions, but not methods?
    > >
    > > Pickling a function serializes the function name, but pickling a
    > > staticmethod, classmethod, or instancemethod generates an error. In
    > > these cases, pickle knows the instance or class, and the method, so
    > > what's the problem? Pickle doesn't serialize code objects, so why can't
    > > it serialize the name as it does for functions? Is this one of those
    > > features that's feasible, but not useful, so no one's ever gotten
    > > around to implementing it?

    >
    > I have often wondered this myself. I'm convinced that it would in fact
    > be useful -- more than once I've written a program that has lots of
    > objects with function pointers, and where it was inconvenient that the
    > method pointers could not be pickled. One compromise that I have used
    > before is to write a class such as:
    >
    > class InstanceMethodSet(object):
    > def __init__(self,methods):
    > self.methods = set(methods)
    > def __getstate__(self):
    > return [(method.im_self, method.im_func.func_name)
    > for method in self.method]
    > def __setstate__(self,state):
    > self.methods = set(getattr(obj,name) for obj,name in state)
    >
    > Obviously, this particular example is crude and not terribly robust,
    > but it seems to do the job -- it effectively lets you pickle a set of
    > instance method pointers. I don't know of any reason why instance
    > methods (or class or static methods) couldn't be pickled directly,
    > unless perhaps there exists some kind of pathological corner case that
    > would create Badness?
    >
    > -Matt


    Thanks, that's quite clever. Although I think you'd still have to
    explicitly specify im_self for staticmethods. I could imagine a similar
    callable proxy that simply removes the direct method reference from the
    equation:

    class MethodProxy(object):
    def __init__(self, obj, method):
    self.obj = obj
    if isinstance(method, basestring):
    self.methodName = method
    else:
    assert callable(method)
    self.methodName = method.func_name
    def __call__(self, *args, **kwargs):
    return getattr(self.obj, self.methodName)(*args, **kwargs)

    picklableMethod = MethodProxy(someObj, someObj.method)
     
    Chris, Oct 21, 2006
    #4
  5. Chris wrote:
    > Why can pickle serialize references to functions, but not methods?


    Here's the recipe I use::

    def _pickle_method(method):
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    return _unpickle_method, (func_name, obj, cls)

    def _unpickle_method(func_name, obj, cls):
    for cls in cls.mro():
    try:
    func = cls.__dict__[func_name]
    except KeyError:
    pass
    else:
    break
    return func.__get__(obj, cls)

    import copy_reg
    import types
    copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

    There may be some special cases where this fails, but I haven't run into
    them yet.

    STeVe
     
    Steven Bethard, Oct 21, 2006
    #5
  6. Chris

    Guest

    Steven Bethard wrote:
    > Here's the recipe I use::
    >
    > [...]
    >
    > There may be some special cases where this fails, but I haven't run into
    > them yet.


    Wow, that's a really nice recipe; I didn't even know about the copy_reg
    module. I'll have to start using that.

    I did notice one failure mode, however--it doesn't work with methods
    named __foo because im_func.__name__ contains the *unmangled* version
    of the function name, so when you try to unpickle the method, the try
    statement never succeeds and you get an UnboundLocalError on func.

    The good news is that I think it can be fixed by mangling the name
    manually in _pickle_method(), like so:

    def _pickle_method(method):
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    if func_name.startswith('__') and not func_name.endswith('__'):
    cls_name = cls.__name__.lstrip('_')
    if cls_name: func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)
     
    , Nov 11, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mr. SweatyFinger
    Replies:
    2
    Views:
    2,270
    Smokey Grindel
    Dec 2, 2006
  2. Frank Niessink
    Replies:
    0
    Views:
    292
    Frank Niessink
    Dec 15, 2006
  3. Frank Niessink
    Replies:
    3
    Views:
    272
    Carl Banks
    Dec 17, 2006
  4. Michele Simionato
    Replies:
    2
    Views:
    1,938
    Michele Simionato
    May 23, 2008
  5. Kedar Mhaswade
    Replies:
    2
    Views:
    87
    Josh Cheek
    Dec 9, 2010
Loading...

Share This Page