pickling subclasses of dict/list

Discussion in 'Python' started by Edward Loper, Jul 3, 2004.

  1. Edward Loper

    Edward Loper Guest

    I'm having trouble pickling subclasses of dict when they contain cycles.
    In particular:

    >>> import pickle
    >>> class D(dict): pass
    >>> d = D()
    >>> d[1] = d # add a cycle.
    >>> print d

    {1: {...}}
    >>> pickle.dump(d, open('d.pickle', 'w'))

    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "/tmp/python-rSB6hN", line 6, in ?
    pickle.dump(d, open('d.pickle', 'w'))
    File "/usr/lib/python2.3/pickle.py", line 1382, in dump
    Pickler(file, protocol, bin).dump(obj)
    [...]
    File "/usr/lib/python2.3/pickle.py", line 414, in save_reduce
    save(func)
    RuntimeError: maximum recursion depth exceeded

    Note that pickling dict's that contain cycles works just fine, as does
    pickling classes that contain cycles in their instance variables. E.g.:

    >>> d = {}
    >>> d[1] = d
    >>> print d
    >>> pickle.dump(d, open('d.pickle', 'w'))
    >>> print pickle.load(open('d.pickle'))

    {1: {...}}

    I tried several things with __getstate__, __reduce__, etc., but couldn't
    get this to work. Is there some magic that I'm missing? What's the
    best way to get around this? (And should I file this as a bug in
    pickle? Or am I just not seeing the right way to do it?)

    -Edward
     
    Edward Loper, Jul 3, 2004
    #1
    1. Advertising

  2. Edward Loper

    Greg Chapman Guest

    On Sat, 03 Jul 2004 14:37:28 -0400, Edward Loper
    <> wrote:

    >I'm having trouble pickling subclasses of dict when they contain cycles.
    >
    >Note that pickling dict's that contain cycles works just fine, as does
    >pickling classes that contain cycles in their instance variables. E.g.:
    >
    > >>> d = {}
    > >>> d[1] = d
    > >>> print d
    > >>> pickle.dump(d, open('d.pickle', 'w'))
    > >>> print pickle.load(open('d.pickle'))

    > {1: {...}}
    >
    >I tried several things with __getstate__, __reduce__, etc., but couldn't
    >get this to work. Is there some magic that I'm missing? What's the
    >best way to get around this? (And should I file this as a bug in
    >pickle? Or am I just not seeing the right way to do it?)
    >


    I think the answer to this problem is to use the extended __reduce__ API
    documented here:

    http://www.python.org/peps/pep-0307.html

    Your subclass's __reduce__ can return a dictitems (or listitems) iterator, the
    items of which will get pickled after the subclass instance is memoized, thus
    avoiding the infinite recursion. The following works for your simple example:

    >>> class D(dict):

    .... def __reduce__(self):
    .... return (D, (), None, None, self.iteritems())
    ....
    >>> d = D()
    >>> d[1] = d
    >>> pickle.dumps(d)

    'c__main__\nD\np0\n(tRp1\nI1\ng1\ns.'
    >>> s = _
    >>> dd = pickle.loads(s)
    >>> dd.items()

    [(1, {1: {...}})]

    ---
    Greg Chapman
     
    Greg Chapman, Jul 4, 2004
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Thomas Guettler

    Pickling object inherited from dict

    Thomas Guettler, Oct 27, 2003, in forum: Python
    Replies:
    1
    Views:
    297
    Thomas Guettler
    Oct 28, 2003
  2. Thomas Guettler

    Pickling Objects inherited from dict (Bug?)

    Thomas Guettler, Nov 5, 2003, in forum: Python
    Replies:
    1
    Views:
    300
    Michael Hudson
    Nov 5, 2003
  3. Dan Perl
    Replies:
    0
    Views:
    360
    Dan Perl
    Nov 3, 2004
  4. dackz
    Replies:
    0
    Views:
    489
    dackz
    Feb 6, 2007
  5. Gaetan de Menten

    Pickling an extension type subclasses problems

    Gaetan de Menten, Feb 5, 2010, in forum: Python
    Replies:
    0
    Views:
    231
    Gaetan de Menten
    Feb 5, 2010
Loading...

Share This Page