Pickling classes (not class instances)

Discussion in 'Python' started by Nicolas M. Thiéry, Jan 10, 2009.

  1. Dear python developers,

    Purpose of this e-mail:
    -----------------------

    How to customize how a class (not an instance of a class!!!) is
    pickled?


    Example:
    ==============================================================================
    class metaclass(type):
    def __new__(mcs, name, bases, dict):
    print " running metaclass.__new__"
    return type.__new__(mcs, name, bases, dict)
    #
    def __reduce_class__(self):
    print "reducing a class"
    # do the real work

    c = metaclass("foo", (object,), dict())

    import copy_reg
    copy_reg.pickle(metaclass, metaclass.__reduce_class__)

    pickle.dumps(c)
    ---------------------------------------------------------------------------
    PicklingError Traceback (most recent call
    last)
    ....
    PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
    __main__.foo
    ==============================================================================


    Context:
    --------

    I am working on the Sage project (www.sagemath.org), and more
    precisely on the category infrastructure. The code is building lots of
    classes on the fly by composing preexisting classes by inheritance
    (for the curious, see the doc of the class Category in
    http://sage.math.washington.edu:2144/file/1567cea09170/categories-nt.patch)..

    Since those classes are built on the fly, they cannot be pickled with
    the default mechanism of name lookup. A proper pickling would be to
    rebuild the class anew. Nothing problematic, except for the question
    above.


    Discussion:
    -----------

    It sounds like copy_reg would be the natural way to go (as in the
    example above). However, its documentation suggests that it explicitly
    is not intended for pickling classes, e.g. first paragraph of:

    http://docs.python.org/library/copy_reg.html#module-copy_reg

    is:

    The copy_reg module provides support for the pickle and cPickle
    modules. The copy module is likely to use this in the future as
    well. It provides configuration information about object
    constructors which are not classes. Such constructors may be factory
    functions or class instances.

    And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
    (and similarly in cpickle), the copy-reg dispatch is explicit bypassed
    for metaclasses:


    # Check for a class with a custom metaclass; treat as regular
    class
    try:
    issc = issubclass(t, TypeType)
    except TypeError: # t is not a class (old Boost; see SF
    #502085)
    issc = 0
    if issc:
    self.save_global(obj)
    return

    # Check copy_reg.dispatch_table
    reduce = dispatch_table.get(t)

    Which causes the failure above.


    Is there a specific reason for this restriction?

    Would it be thinkable to move up the copy reg dispatch before the
    metaclass treatment in pickle and cPickle? I did it locally, and it
    fixed my problem.

    If not, what would be the right way to achieve this?

    Many thanks in advance!

    Best regards,
    Nicolas
    --
    Nicolas M. Thiéry "Isil" <>
    http://Nicolas.Thiery.name/
     
    Nicolas M. Thiéry, Jan 10, 2009
    #1
    1. Advertising

  2. Dear python developers,

    I got no answer to my previous post on this thread "Pickling classes
    (not class instances)".
    This issue is a show stopper for our project. Any suggestion for where
    to ask?

    Thanks in advance!

    Best regards,
    Nicolas

    > Purpose of this e-mail:
    > -----------------------
    >
    > How to customize how a class (not an instance of a class!!!) is
    > pickled?
    >
    > Example:
    > ==============================================================================
    > class metaclass(type):
    > def __new__(mcs, name, bases, dict):
    > print " running metaclass.__new__"
    > return type.__new__(mcs, name, bases, dict)
    > #
    > def __reduce_class__(self):
    > print "reducing a class"
    > # do the real work
    >
    > c = metaclass("foo", (object,), dict())
    >
    > import copy_reg
    > copy_reg.pickle(metaclass, metaclass.__reduce_class__)
    >
    > pickle.dumps(c)
    > ---------------------------------------------------------------------------
    > PicklingError Traceback (most recent call
    > last)
    > ...
    > PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
    > __main__.foo
    > ==============================================================================
    >
    > Context:
    > --------
    >
    > I am working on the Sage project (www.sagemath.org), and more
    > precisely on the category infrastructure. The code is building lots of
    > classes on the fly by composing preexisting classes by inheritance
    > (for the curious, see the doc of the class Category inhttp://sage.math.washington.edu:2144/file/1567cea09170/categories-nt....).
    >
    > Since those classes are built on the fly, they cannot be pickled with
    > the default mechanism of name lookup. A proper pickling would be to
    > rebuild the class anew. Nothing problematic, except for the question
    > above.
    >
    > Discussion:
    > -----------
    >
    > It sounds like copy_reg would be the natural way to go (as in the
    > example above). However, its documentation suggests that it explicitly
    > is not intended for pickling classes, e.g. first paragraph of:
    >
    > http://docs.python.org/library/copy_reg.html#module-copy_reg
    >
    > is:
    >
    > The copy_reg module provides support for the pickle and cPickle
    > modules. The copy module is likely to use this in the future as
    > well. It provides configuration information about object
    > constructors which are not classes. Such constructors may be factory
    > functions or class instances.
    >
    > And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
    > (and similarly in cpickle), the copy-reg dispatch is explicit bypassed
    > for metaclasses:
    >
    > # Check for a class with a custom metaclass; treat as regular
    > class
    > try:
    > issc = issubclass(t, TypeType)
    > except TypeError: # t is not a class (old Boost; see SF
    > #502085)
    > issc = 0
    > if issc:
    > self.save_global(obj)
    > return
    >
    > # Check copy_reg.dispatch_table
    > reduce = dispatch_table.get(t)
    >
    > Which causes the failure above.
    >
    > Is there a specific reason for this restriction?
    >
    > Would it be thinkable to move up the copy reg dispatch before the
    > metaclass treatment in pickle and cPickle? I did it locally, and it
    > fixed my problem.
    >
    > If not, what would be the right way to achieve this?
    >
    > Many thanks in advance!
    >
    > Best regards,
    > Nicolas
    > --
    > Nicolas M. Thiéry "Isil" <>http://Nicolas.Thiery.name/
     
    Nicolas M. Thiéry, Feb 14, 2009
    #2
    1. Advertising

  3. Nicolas M. Thiéry

    Aaron Brady Guest

    On Feb 14, 2:19 am, Nicolas M. Thiéry <>
    wrote:
    >          Dear python developers,
    >
    > I got no answer to my previous post on this thread "Pickling classes
    > (not class instances)".
    > This issue is a show stopper for our project. Any suggestion for where
    > to ask?

    snip to
    http://groups.google.com/group/comp...2afc04aa39c/4d9304aa0cc791c2#4d9304aa0cc791c2
    > > How to customize how a class (not an instance of a class!!!) is
    > > pickled?

    snip
    > > Is there a specific reason for this restriction?

    >
    > > Would it be thinkable to move up the copy reg dispatch before the
    > > metaclass treatment in pickle and cPickle? I did it locally, and it
    > > fixed my problem.

    >
    > > If not, what would be the right way to achieve this?


    Hi Nicolas,

    I couldn't get yours to work. I also tried defining '__reduce__' and
    '__getstate__' in the metaclass; they didn't work either.

    Can you just subclass Pickler, and override the 'save' method?

    > > Is there a specific reason for this restriction?


    It sounds very odd. I'm not a core dev, so I can't say.

    I also tried defining '__subclasshook__' to convince 'metaclass' it is
    not a subclass of 'type'. It didn't work, but 'register' was not
    working the way I wanted.

    Perhaps you could define your own 'custom_type' type, which looks and
    acts just like type, but just doesn't pass the 'subclass' test. It
    may be as simple as just cutting-and-pasting the TypeObject definition
    from the core.
     
    Aaron Brady, Feb 14, 2009
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John Wohlbier
    Replies:
    2
    Views:
    368
    Josiah Carlson
    Feb 22, 2004
  2. Erik Max Francis
    Replies:
    1
    Views:
    394
    Erik Max Francis
    Aug 9, 2005
  3. Alex
    Replies:
    1
    Views:
    324
    Alex Martelli
    Oct 29, 2005
  4. Replies:
    2
    Views:
    299
  5. lars van gemerden
    Replies:
    10
    Views:
    611
    lars van gemerden
    Jan 3, 2012
Loading...

Share This Page