N
Nicolas M. Thiéry
Dear python developers,
Purpose of this e-mail:
-----------------------
How to customize how a class (not an instance of a class!!!) is
pickled?
Example:
==============================================================================
class metaclass(type):
def __new__(mcs, name, bases, dict):
print " running metaclass.__new__"
return type.__new__(mcs, name, bases, dict)
#
def __reduce_class__(self):
print "reducing a class"
# do the real work
c = metaclass("foo", (object,), dict())
import copy_reg
copy_reg.pickle(metaclass, metaclass.__reduce_class__)
pickle.dumps(c)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call
last)
....
PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
__main__.foo
==============================================================================
Context:
--------
I am working on the Sage project (www.sagemath.org), and more
precisely on the category infrastructure. The code is building lots of
classes on the fly by composing preexisting classes by inheritance
(for the curious, see the doc of the class Category in
http://sage.math.washington.edu:2144/file/1567cea09170/categories-nt.patch)..
Since those classes are built on the fly, they cannot be pickled with
the default mechanism of name lookup. A proper pickling would be to
rebuild the class anew. Nothing problematic, except for the question
above.
Discussion:
-----------
It sounds like copy_reg would be the natural way to go (as in the
example above). However, its documentation suggests that it explicitly
is not intended for pickling classes, e.g. first paragraph of:
http://docs.python.org/library/copy_reg.html#module-copy_reg
is:
The copy_reg module provides support for the pickle and cPickle
modules. The copy module is likely to use this in the future as
well. It provides configuration information about object
constructors which are not classes. Such constructors may be factory
functions or class instances.
And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
(and similarly in cpickle), the copy-reg dispatch is explicit bypassed
for metaclasses:
# Check for a class with a custom metaclass; treat as regular
class
try:
issc = issubclass(t, TypeType)
except TypeError: # t is not a class (old Boost; see SF
#502085)
issc = 0
if issc:
self.save_global(obj)
return
# Check copy_reg.dispatch_table
reduce = dispatch_table.get(t)
Which causes the failure above.
Is there a specific reason for this restriction?
Would it be thinkable to move up the copy reg dispatch before the
metaclass treatment in pickle and cPickle? I did it locally, and it
fixed my problem.
If not, what would be the right way to achieve this?
Many thanks in advance!
Best regards,
Nicolas
Purpose of this e-mail:
-----------------------
How to customize how a class (not an instance of a class!!!) is
pickled?
Example:
==============================================================================
class metaclass(type):
def __new__(mcs, name, bases, dict):
print " running metaclass.__new__"
return type.__new__(mcs, name, bases, dict)
#
def __reduce_class__(self):
print "reducing a class"
# do the real work
c = metaclass("foo", (object,), dict())
import copy_reg
copy_reg.pickle(metaclass, metaclass.__reduce_class__)
pickle.dumps(c)
---------------------------------------------------------------------------
PicklingError Traceback (most recent call
last)
....
PicklingError: Can't pickle <class '__main__.foo'>: it's not found as
__main__.foo
==============================================================================
Context:
--------
I am working on the Sage project (www.sagemath.org), and more
precisely on the category infrastructure. The code is building lots of
classes on the fly by composing preexisting classes by inheritance
(for the curious, see the doc of the class Category in
http://sage.math.washington.edu:2144/file/1567cea09170/categories-nt.patch)..
Since those classes are built on the fly, they cannot be pickled with
the default mechanism of name lookup. A proper pickling would be to
rebuild the class anew. Nothing problematic, except for the question
above.
Discussion:
-----------
It sounds like copy_reg would be the natural way to go (as in the
example above). However, its documentation suggests that it explicitly
is not intended for pickling classes, e.g. first paragraph of:
http://docs.python.org/library/copy_reg.html#module-copy_reg
is:
The copy_reg module provides support for the pickle and cPickle
modules. The copy module is likely to use this in the future as
well. It provides configuration information about object
constructors which are not classes. Such constructors may be factory
functions or class instances.
And indeed, looking up at the code of pickle (l. 289-299 of pickle.py)
(and similarly in cpickle), the copy-reg dispatch is explicit bypassed
for metaclasses:
# Check for a class with a custom metaclass; treat as regular
class
try:
issc = issubclass(t, TypeType)
except TypeError: # t is not a class (old Boost; see SF
#502085)
issc = 0
if issc:
self.save_global(obj)
return
# Check copy_reg.dispatch_table
reduce = dispatch_table.get(t)
Which causes the failure above.
Is there a specific reason for this restriction?
Would it be thinkable to move up the copy reg dispatch before the
metaclass treatment in pickle and cPickle? I did it locally, and it
fixed my problem.
If not, what would be the right way to achieve this?
Many thanks in advance!
Best regards,
Nicolas