Re: Is there anything that pickle + copy_reg cannot serialize?

Discussion in 'Python' started by Jean-Paul Calderone, Dec 8, 2005.

  1. On Thu, 08 Dec 2005 22:42:32 +0800, Maurice LING <> wrote:
    >Hi,
    >
    >I need to look into serialization for python objects, including codes,
    >recursive types etc etc. Currently, I have no idea exactly what needs to
    >be serialized, so my scope is to be as wide as possible.
    >
    >I understand that marshal is extended by pickle to serialize class
    >instances, shared elements, and recursive data structures
    >(http://www.effbot.org/librarybook/pickle.htm) but cannot handle code
    >types. pickle can be used together with copy_reg and marshal to
    >serialize code types as well
    >(http://www.effbot.org/librarybook/copy-reg.htm).
    >
    >So my question will be, are there anything that pickle/copy_reg/marshal
    >combination cannot serialize? If so, what are the workarounds?


    Since copy_reg lets you specify arbitrary code to serialize arbitrary
    objects, you shouldn't run into any single object that you cannot
    serialize to a pickle.

    However, both pickle implementations are recursive, so you will be
    limited by the amount of memory you can allocate for your stack. By
    default, this will limit you to something like object graphs 333 edges
    deep or so (if I'm counting stack frames correctly). Note that this
    does not mean you cannot serialize more than 333 objects at a time,
    merely that if it takes 333 or more steps to go from the first object
    to any other object in the graph (using the traversal order pickle
    uses), the pickling will fail. You can raise this limit, to a point,
    with sys.setrecursionlimit().

    Jean-Paul
     
    Jean-Paul Calderone, Dec 8, 2005
    #1
    1. Advertising

  2. Jean-Paul Calderone

    Maurice LING Guest


    > Since copy_reg lets you specify arbitrary code to serialize arbitrary
    > objects, you shouldn't run into any single object that you cannot
    > serialize to a pickle.


    In http://www.effbot.org/librarybook/pickle.htm, it specifically
    mentions that code objects cannot be pickled and require the use of
    copy_reg, as follows:

    import copy_reg
    import pickle, marshal, types

    #
    # register a pickle handler for code objects

    def code_unpickler(data):
    return marshal.loads(data)

    def code_pickler(code):
    return code_unpickler, (marshal.dumps(code),)

    copy_reg.pickle(types.CodeType, code_pickler, code_unpickler)

    #
    # try it out

    CODE = """
    print "suppose he's got a pointed stick"
    """

    code = compile(CODE, "<string>", "exec")

    exec code
    exec pickle.loads(pickle.dumps(code))


    I cannot understand 2 things, which I seek assistance for:
    1. Is code object the only thing can cannot be pickled (less facing
    recursion limits)?
    2. In the above example, how copy_reg works with pickle?

    Thanks and Cheers
    Maurice

    >
    > However, both pickle implementations are recursive, so you will be
    > limited by the amount of memory you can allocate for your stack. By
    > default, this will limit you to something like object graphs 333 edges
    > deep or so (if I'm counting stack frames correctly). Note that this
    > does not mean you cannot serialize more than 333 objects at a time,
    > merely that if it takes 333 or more steps to go from the first object to
    > any other object in the graph (using the traversal order pickle uses),
    > the pickling will fail. You can raise this limit, to a point, with
    > sys.setrecursionlimit().
    >
    > Jean-Paul
     
    Maurice LING, Dec 8, 2005
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Gordz
    Replies:
    3
    Views:
    3,939
    John Oakes
    Jun 7, 2004
  2. a pickle's pickle

    , Aug 2, 2005, in forum: Python
    Replies:
    4
    Views:
    387
  3. Maurice LING
    Replies:
    0
    Views:
    281
    Maurice LING
    Dec 8, 2005
  4. Jean-Paul Calderone
    Replies:
    2
    Views:
    541
    Fredrik Lundh
    Dec 9, 2005
  5. Michele Simionato
    Replies:
    2
    Views:
    1,889
    Michele Simionato
    May 23, 2008
Loading...

Share This Page