pickling instances of metaclass generated classes

Discussion in 'Python' started by lars van gemerden, Dec 29, 2011.

  1. Hello,

    Can someone help me with the following:

    I am using metaclasses to make classes and these classes to make
    instances. Now I want to use multiprocessing, which needs to pickle
    these instances.

    Pickle cannot find the class definitions of the instances. I am trying
    to add a line to the __new__ of the metaclass to add the new class
    under the right name in the right module/place, so pickle can find
    it.

    Is this the right approach? Can anyone explain to me where/how to add
    these classes for pickle to find and maybe why?

    Thanks in advance,

    Lars
     
    lars van gemerden, Dec 29, 2011
    #1
    1. Advertising

  2. lars van gemerden

    Robert Kern Guest

    On 12/29/11 9:55 AM, lars van gemerden wrote:
    > Hello,
    >
    > Can someone help me with the following:
    >
    > I am using metaclasses to make classes and these classes to make
    > instances. Now I want to use multiprocessing, which needs to pickle
    > these instances.
    >
    > Pickle cannot find the class definitions of the instances. I am trying
    > to add a line to the __new__ of the metaclass to add the new class
    > under the right name in the right module/place, so pickle can find
    > it.
    >
    > Is this the right approach? Can anyone explain to me where/how to add
    > these classes for pickle to find and maybe why?


    Can you post some code (preferably pared down to a minimal example that fails)?
    I'm not really clear on what you are doing. I would expect that a class defined
    by a class statement would usually work fine unless if the metaclass is doing
    something particularly weird to it.

    In any case, you can probably just explicitly register a reduction function for
    each type using copy_reg.pickle():

    http://docs.python.org/library/copy_reg

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
     
    Robert Kern, Dec 29, 2011
    #2
    1. Advertising

  3. lars van gemerden

    Ian Kelly Guest

    On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <> wrote:
    > Hello,
    >
    > Can someone help me with the following:
    >
    > I am using metaclasses to make classes and these classes to make
    > instances. Now I want to use multiprocessing, which needs to pickle
    > these instances.
    >
    > Pickle cannot find the class definitions of the instances. I am trying
    > to add a line to the __new__ of the metaclass to add the new class
    > under the right name in the right module/place, so pickle can find
    > it.
    >
    > Is this the right approach? Can anyone explain to me where/how to add
    > these classes for pickle to find and maybe why?


    It sounds like you're trying to do something like this?

    >>> class MetaClass(type):

    .... pass
    ....
    >>> instance = MetaClass('<Anonymous>', (object,), {})()
    >>> instance

    <__main__.<Anonymous> object at 0x00CC00F0>
    >>> import pickle
    >>> pickle.dumps(instance)

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "c:\python27\lib\pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
    File "c:\python27\lib\pickle.py", line 224, in dump
    self.save(obj)
    File "c:\python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
    File "c:\python27\lib\pickle.py", line 401, in save_reduce
    save(args)
    File "c:\python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    File "c:\python27\lib\pickle.py", line 562, in save_tuple
    save(element)
    File "c:\python27\lib\pickle.py", line 295, in save
    self.save_global(obj)
    File "c:\python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
    pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    it's not found as __main__.<Anonymous>


    Yeah, pickle's not going to work with anonymous classes. As you
    suggest, you could dynamically add the classes to the module namespace
    so that pickle.dumps will find them, but bear in mind that they will
    also have to exist when calling pickle.loads, so you will need to be
    able to reconstruct the same anonymous classes before unpickling later
    on.

    Cheers,
    Ian
     
    Ian Kelly, Dec 29, 2011
    #3
  4. On Dec 29, 8:55 pm, Ian Kelly <> wrote:
    > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <>wrote:
    >
    > > Hello,

    >
    > > Can someone help me with the following:

    >
    > > I am using metaclasses to make classes and these classes to make
    > > instances. Now I want to use multiprocessing, which needs to pickle
    > > these instances.

    >
    > > Pickle cannot find the class definitions of the instances. I am trying
    > > to add a line to the __new__ of the metaclass to add the new class
    > > under the right name in the right module/place, so pickle can find
    > > it.

    >
    > > Is this the right approach? Can anyone explain to me where/how to add
    > > these classes for pickle to find and maybe why?

    >
    > It sounds like you're trying to do something like this?
    >
    > >>> class MetaClass(type):

    >
    > ...     pass
    > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
    > >>> instance

    >
    > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
    > >>> pickle.dumps(instance)

    >
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "c:\python27\lib\pickle.py", line 1374, in dumps
    >     Pickler(file, protocol).dump(obj)
    >   File "c:\python27\lib\pickle.py", line 224, in dump
    >     self.save(obj)
    >   File "c:\python27\lib\pickle.py", line 331, in save
    >     self.save_reduce(obj=obj, *rv)
    >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
    >     save(args)
    >   File "c:\python27\lib\pickle.py", line 286, in save
    >     f(self, obj) # Call unbound method with explicit self
    >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
    >     save(element)
    >   File "c:\python27\lib\pickle.py", line 295, in save
    >     self.save_global(obj)
    >   File "c:\python27\lib\pickle.py", line 748, in save_global
    >     (obj, module, name))
    > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    > it's not found as __main__.<Anonymous>
    >
    > Yeah, pickle's not going to work with anonymous classes.  As you
    > suggest, you could dynamically add the classes to the module namespace
    > so that pickle.dumps will find them, but bear in mind that they will
    > also have to exist when calling pickle.loads, so you will need to be
    > able to reconstruct the same anonymous classes before unpickling later
    > on.
    >
    > Cheers,
    > Ian


    Thank you Ian for the minimal example. This is almost the case i was
    trying to discribe, however the classes will be named at runtime and
    the class definitions will be persisted in a database.

    Can you help me with how to add the classes to the correct namespace?
    The application spans multiple modules (or compared to the example,
    the metaclass definition will be in another module then one where the
    class and instance will be generated).

    Cheers, Lars
     
    lars van gemerden, Dec 30, 2011
    #4
  5. On Dec 29, 8:55 pm, Ian Kelly <> wrote:
    > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <>wrote:
    >
    > > Hello,

    >
    > > Can someone help me with the following:

    >
    > > I am using metaclasses to make classes and these classes to make
    > > instances. Now I want to use multiprocessing, which needs to pickle
    > > these instances.

    >
    > > Pickle cannot find the class definitions of the instances. I am trying
    > > to add a line to the __new__ of the metaclass to add the new class
    > > under the right name in the right module/place, so pickle can find
    > > it.

    >
    > > Is this the right approach? Can anyone explain to me where/how to add
    > > these classes for pickle to find and maybe why?

    >
    > It sounds like you're trying to do something like this?
    >
    > >>> class MetaClass(type):

    >
    > ...     pass
    > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
    > >>> instance

    >
    > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
    > >>> pickle.dumps(instance)

    >
    > Traceback (most recent call last):
    >   File "<stdin>", line 1, in <module>
    >   File "c:\python27\lib\pickle.py", line 1374, in dumps
    >     Pickler(file, protocol).dump(obj)
    >   File "c:\python27\lib\pickle.py", line 224, in dump
    >     self.save(obj)
    >   File "c:\python27\lib\pickle.py", line 331, in save
    >     self.save_reduce(obj=obj, *rv)
    >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
    >     save(args)
    >   File "c:\python27\lib\pickle.py", line 286, in save
    >     f(self, obj) # Call unbound method with explicit self
    >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
    >     save(element)
    >   File "c:\python27\lib\pickle.py", line 295, in save
    >     self.save_global(obj)
    >   File "c:\python27\lib\pickle.py", line 748, in save_global
    >     (obj, module, name))
    > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    > it's not found as __main__.<Anonymous>
    >
    > Yeah, pickle's not going to work with anonymous classes.  As you
    > suggest, you could dynamically add the classes to the module namespace
    > so that pickle.dumps will find them, but bear in mind that they will
    > also have to exist when calling pickle.loads, so you will need to be
    > able to reconstruct the same anonymous classes before unpickling later
    > on.
    >
    > Cheers,
    > Ian


    Ian also wrote:

    '''
    Actually, I was wrong, you probably don't need to do that. I suggest
    going with Robert Kern's suggestion to either register the class with
    the copy_reg module, or (perhaps better since it won't leak
    registrations) implement a __reduce__ method on the class. For
    example, this seems to work:

    >>> def reconstructor(*metaclass_args):

    .... cls = MetaClass.build_class(*metaclass_args)
    .... self = cls.__new__(cls)
    .... return self
    ....
    >>> class MetaClass(type):

    .... @classmethod
    .... def build_class(mcs, arg1, arg2, arg3):
    .... # Do something useful with the args...
    .... class _AnonymousClass(object):
    .... __metaclass__ = mcs
    .... def __reduce__(self):
    .... return (reconstructor, ('foo', 'bar', 'baz'),
    self.__dict__)
    .... return _AnonymousClass
    ....
    >>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
    >>> instance

    <__main__._AnonymousClass object at 0x011DB410>
    >>> instance.banana = 42
    >>> import pickle
    >>> s = pickle.dumps(instance)
    >>> s

    "c__main__\nreconstructor
    \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI42\nsb."
    >>> inst2 = pickle.loads(s)
    >>> inst2

    <__main__._AnonymousClass object at 0x011DBE90>
    >>> inst2.banana

    42
    >>> inst2.__class__ is instance.__class__

    False

    Cheers,
    Ian

    '''
     
    lars van gemerden, Dec 30, 2011
    #5
  6. lars van gemerden

    Peter Otten Guest

    lars van gemerden wrote:

    > On Dec 29, 8:55 pm, Ian Kelly <> wrote:
    >> On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <>
    >> wrote:
    >>
    >> > Hello,

    >>
    >> > Can someone help me with the following:

    >>
    >> > I am using metaclasses to make classes and these classes to make
    >> > instances. Now I want to use multiprocessing, which needs to pickle
    >> > these instances.

    >>
    >> > Pickle cannot find the class definitions of the instances. I am trying
    >> > to add a line to the __new__ of the metaclass to add the new class
    >> > under the right name in the right module/place, so pickle can find
    >> > it.

    >>
    >> > Is this the right approach? Can anyone explain to me where/how to add
    >> > these classes for pickle to find and maybe why?

    >>
    >> It sounds like you're trying to do something like this?
    >>
    >> >>> class MetaClass(type):

    >>
    >> ... pass
    >> ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
    >> >>> instance

    >>
    >> <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
    >> >>> pickle.dumps(instance)

    >>
    >> Traceback (most recent call last):
    >> File "<stdin>", line 1, in <module>
    >> File "c:\python27\lib\pickle.py", line 1374, in dumps
    >> Pickler(file, protocol).dump(obj)
    >> File "c:\python27\lib\pickle.py", line 224, in dump
    >> self.save(obj)
    >> File "c:\python27\lib\pickle.py", line 331, in save
    >> self.save_reduce(obj=obj, *rv)
    >> File "c:\python27\lib\pickle.py", line 401, in save_reduce
    >> save(args)
    >> File "c:\python27\lib\pickle.py", line 286, in save
    >> f(self, obj) # Call unbound method with explicit self
    >> File "c:\python27\lib\pickle.py", line 562, in save_tuple
    >> save(element)
    >> File "c:\python27\lib\pickle.py", line 295, in save
    >> self.save_global(obj)
    >> File "c:\python27\lib\pickle.py", line 748, in save_global
    >> (obj, module, name))
    >> pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    >> it's not found as __main__.<Anonymous>
    >>
    >> Yeah, pickle's not going to work with anonymous classes. As you
    >> suggest, you could dynamically add the classes to the module namespace
    >> so that pickle.dumps will find them, but bear in mind that they will
    >> also have to exist when calling pickle.loads, so you will need to be
    >> able to reconstruct the same anonymous classes before unpickling later
    >> on.
    >>
    >> Cheers,
    >> Ian

    >
    > Thank you Ian for the minimal example. This is almost the case i was
    > trying to discribe, however the classes will be named at runtime and
    > the class definitions will be persisted in a database.
    >
    > Can you help me with how to add the classes to the correct namespace?
    > The application spans multiple modules (or compared to the example,
    > the metaclass definition will be in another module then one where the
    > class and instance will be generated).


    If the metaclass is global in whatever module you don't need to bother about
    that. The problem is that you cannot access the classes (metaclass
    instances) under a dotted name. One workaround is a pseudo-module that does
    whatever is needed to recreate the class:

    import pickle
    import sys

    class MetaClass(type):
    pass

    class M(object):
    def __init__(self, module):
    self.__module = module
    def __getattr__(self, name):
    print "creating class", name
    class_ = MetaClass(name, (), {"__module__": self.__module})
    setattr(self, name, class_)
    return class_

    sys.modules["m"] = M("m")
    import m
    c = m.x
    s = pickle.dumps(c)
    print repr(s)
    d = pickle.loads(s)

    assert c is d

    sys.modules["m"] = M("m")
    e = pickle.loads(s)

    assert c is not e

    The official way is probably what Robert mentioned, via the copy_reg module,
    but I didn't get it to work.
     
    Peter Otten, Dec 30, 2011
    #6
  7. On Dec 30, 12:16 pm, lars van gemerden <> wrote:
    > On Dec 29, 8:55 pm, Ian Kelly <> wrote:
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <> wrote:

    >
    > > > Hello,

    >
    > > > Can someone help me with the following:

    >
    > > > I am using metaclasses to make classes and these classes to make
    > > > instances. Now I want to use multiprocessing, which needs to pickle
    > > > these instances.

    >
    > > > Pickle cannot find the class definitions of the instances. I am trying
    > > > to add a line to the __new__ of the metaclass to add the new class
    > > > under the right name in the right module/place, so pickle can find
    > > > it.

    >
    > > > Is this the right approach? Can anyone explain to me where/how to add
    > > > these classes for pickle to find and maybe why?

    >
    > > It sounds like you're trying to do something like this?

    >
    > > >>> class MetaClass(type):

    >
    > > ...     pass
    > > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
    > > >>> instance

    >
    > > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
    > > >>> pickle.dumps(instance)

    >
    > > Traceback (most recent call last):
    > >   File "<stdin>", line 1, in <module>
    > >   File "c:\python27\lib\pickle.py", line 1374, in dumps
    > >     Pickler(file, protocol).dump(obj)
    > >   File "c:\python27\lib\pickle.py", line 224, in dump
    > >     self.save(obj)
    > >   File "c:\python27\lib\pickle.py", line 331, in save
    > >     self.save_reduce(obj=obj, *rv)
    > >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
    > >     save(args)
    > >   File "c:\python27\lib\pickle.py", line 286, in save
    > >     f(self, obj) # Call unbound method with explicit self
    > >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
    > >     save(element)
    > >   File "c:\python27\lib\pickle.py", line 295, in save
    > >     self.save_global(obj)
    > >   File "c:\python27\lib\pickle.py", line 748, in save_global
    > >     (obj, module, name))
    > > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    > > it's not found as __main__.<Anonymous>

    >
    > > Yeah, pickle's not going to work with anonymous classes.  As you
    > > suggest, you could dynamically add the classes to the module namespace
    > > so that pickle.dumps will find them, but bear in mind that they will
    > > also have to exist when calling pickle.loads, so you will need to be
    > > able to reconstruct the same anonymous classes before unpickling later
    > > on.

    >
    > > Cheers,
    > > Ian

    > Ian also wrote:
    >
    > '''
    > Actually, I was wrong, you probably don't need to do that.  I suggest
    > going with Robert Kern's suggestion to either register the class with
    > the copy_reg module, or (perhaps better since it won't leak
    > registrations) implement a __reduce__ method on the class.  For
    > example, this seems to work:
    >
    > >>> def reconstructor(*metaclass_args):

    >
    > ...     cls = MetaClass.build_class(*metaclass_args)
    > ...     self = cls.__new__(cls)
    > ...     return self
    > ...>>> class MetaClass(type):
    >
    > ...     @classmethod
    > ...     def build_class(mcs, arg1, arg2, arg3):
    > ...         # Do something useful with the args...
    > ...         class _AnonymousClass(object):
    > ...             __metaclass__ = mcs
    > ...             def __reduce__(self):
    > ...                 return (reconstructor, ('foo', 'bar','baz'),
    > self.__dict__)
    > ...         return _AnonymousClass
    > ...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
    > >>> instance

    >
    > <__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42
    > >>> import pickle
    > >>> s = pickle.dumps(instance)
    > >>> s

    >
    > "c__main__\nreconstructor
    > \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)
    > >>> inst2

    >
    > <__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
    > 42
    > >>> inst2.__class__ is instance.__class__

    >
    > False
    >
    > Cheers,
    > Ian
    >
    > '''


    Interesting, though I cannot say I completely understand this solution
    (looked up __reduce__, but still). I am trying to adapt this example
    to a situation where the metaclass generated classes are named at
    runtime (not anonymous), but cannot figure it out.

    Cheers, Lars
     
    lars van gemerden, Dec 30, 2011
    #7
  8. On Dec 30, 4:56 pm, lars van gemerden <> wrote:
    > On Dec 30, 12:16 pm, lars van gemerden <> wrote:
    >
    >
    >
    >
    >
    >
    >
    >
    >
    > > On Dec 29, 8:55 pm, Ian Kelly <> wrote:

    >
    > > > On Thu, Dec 29, 2011 at 2:55 AM, lars van gemerden <> wrote:

    >
    > > > > Hello,

    >
    > > > > Can someone help me with the following:

    >
    > > > > I am using metaclasses to make classes and these classes to make
    > > > > instances. Now I want to use multiprocessing, which needs to pickle
    > > > > these instances.

    >
    > > > > Pickle cannot find the class definitions of the instances. I am trying
    > > > > to add a line to the __new__ of the metaclass to add the new class
    > > > > under the right name in the right module/place, so pickle can find
    > > > > it.

    >
    > > > > Is this the right approach? Can anyone explain to me where/how to add
    > > > > these classes for pickle to find and maybe why?

    >
    > > > It sounds like you're trying to do something like this?

    >
    > > > >>> class MetaClass(type):

    >
    > > > ...     pass
    > > > ...>>> instance = MetaClass('<Anonymous>', (object,), {})()
    > > > >>> instance

    >
    > > > <__main__.<Anonymous> object at 0x00CC00F0>>>> import pickle
    > > > >>> pickle.dumps(instance)

    >
    > > > Traceback (most recent call last):
    > > >   File "<stdin>", line 1, in <module>
    > > >   File "c:\python27\lib\pickle.py", line 1374, in dumps
    > > >     Pickler(file, protocol).dump(obj)
    > > >   File "c:\python27\lib\pickle.py", line 224, in dump
    > > >     self.save(obj)
    > > >   File "c:\python27\lib\pickle.py", line 331, in save
    > > >     self.save_reduce(obj=obj, *rv)
    > > >   File "c:\python27\lib\pickle.py", line 401, in save_reduce
    > > >     save(args)
    > > >   File "c:\python27\lib\pickle.py", line 286, in save
    > > >     f(self, obj) # Call unbound method with explicit self
    > > >   File "c:\python27\lib\pickle.py", line 562, in save_tuple
    > > >     save(element)
    > > >   File "c:\python27\lib\pickle.py", line 295, in save
    > > >     self.save_global(obj)
    > > >   File "c:\python27\lib\pickle.py", line 748, in save_global
    > > >     (obj, module, name))
    > > > pickle.PicklingError: Can't pickle <class '__main__.<Anonymous>'>:
    > > > it's not found as __main__.<Anonymous>

    >
    > > > Yeah, pickle's not going to work with anonymous classes.  As you
    > > > suggest, you could dynamically add the classes to the module namespace
    > > > so that pickle.dumps will find them, but bear in mind that they will
    > > > also have to exist when calling pickle.loads, so you will need to be
    > > > able to reconstruct the same anonymous classes before unpickling later
    > > > on.

    >
    > > > Cheers,
    > > > Ian

    > > Ian also wrote:

    >
    > > '''
    > > Actually, I was wrong, you probably don't need to do that.  I suggest
    > > going with Robert Kern's suggestion to either register the class with
    > > the copy_reg module, or (perhaps better since it won't leak
    > > registrations) implement a __reduce__ method on the class.  For
    > > example, this seems to work:

    >
    > > >>> def reconstructor(*metaclass_args):

    >
    > > ...     cls = MetaClass.build_class(*metaclass_args)
    > > ...     self = cls.__new__(cls)
    > > ...     return self
    > > ...>>> class MetaClass(type):

    >
    > > ...     @classmethod
    > > ...     def build_class(mcs, arg1, arg2, arg3):
    > > ...         # Do something useful with the args...
    > > ...         class _AnonymousClass(object):
    > > ...             __metaclass__ = mcs
    > > ...             def __reduce__(self):
    > > ...                 return (reconstructor, ('foo', 'bar', 'baz'),
    > > self.__dict__)
    > > ...         return _AnonymousClass
    > > ...>>> instance = MetaClass.build_class('foo', 'bar', 'baz')()
    > > >>> instance

    >
    > > <__main__._AnonymousClass object at 0x011DB410>>>> instance.banana = 42
    > > >>> import pickle
    > > >>> s = pickle.dumps(instance)
    > > >>> s

    >
    > > "c__main__\nreconstructor
    > > \np0\n(S'foo'\np1\nS'bar'\np2\nS'baz'\np3\ntp4\nRp5\n(dp6\nS'banana'\np7\nI 42\nsb.">>> inst2 = pickle.loads(s)
    > > >>> inst2

    >
    > > <__main__._AnonymousClass object at 0x011DBE90>>>> inst2.banana
    > > 42
    > > >>> inst2.__class__ is instance.__class__

    >
    > > False

    >
    > > Cheers,
    > > Ian

    >
    > > '''

    >
    > Interesting, though I cannot say I completely understand this solution
    > (looked up __reduce__, but still). I am trying to adapt this example
    > to a situation where the metaclass generated classes are named at
    > runtime (not anonymous), but cannot figure it out.
    >
    > Cheers, Lars


    Found a way to name the classes:

    def reconstructor(*metaclass_args):
    cls = MetaClass2.build_class(*metaclass_args)
    self = cls.__new__(cls)
    return self

    class MetaClass(type):
    @classmethod
    def build_class(mcs, name, arg1, arg2, arg3):
    return mcs(name, (object,), {"__reduce__": lambda e:
    (reconstructor2, (name, arg1, arg2, arg3), e.__dict__)})

    I still wonder whether it might be easier to add the class to the
    namespace. Can anyone help me with that?

    Regards, Lars
     
    lars van gemerden, Dec 30, 2011
    #8
  9. lars van gemerden

    Ian Kelly Guest

    On Fri, Dec 30, 2011 at 9:51 AM, lars van gemerden <> wrote:
    > I still wonder whether it might be easier to add the class to the
    > namespace. Can anyone help me with that?


    from mypackage import mymodule

    setattr(mymodule, myclass.__name__, myclass)
     
    Ian Kelly, Dec 30, 2011
    #9
  10. lars van gemerden

    Peter Otten Guest

    lars van gemerden wrote:

    >> import pickle
    >> import sys
    >>
    >> class MetaClass(type):
    >> pass
    >>
    >> class M(object):
    >> def __init__(self, module):
    >> self.__module = module
    >> def __getattr__(self, name):
    >> print "creating class", name
    >> class_ = MetaClass(name, (), {"__module__": self.__module})
    >> setattr(self, name, class_)
    >> return class_
    >>
    >> sys.modules["m"] = M("m")
    >> import m
    >> c = m.x
    >> s = pickle.dumps(c)
    >> print repr(s)
    >> d = pickle.loads(s)
    >>
    >> assert c is d
    >>
    >> sys.modules["m"] = M("m")
    >> e = pickle.loads(s)
    >>
    >> assert c is not e
    >>
    >> The official way is probably what Robert mentioned, via the copy_reg
    >> module, but I didn't get it to work.

    >
    > I will look further into this. does "sys.modules["m"] = M("m")" create
    > a new module?


    Assigning to sys.modules[modulename] can put arbitrary objects into the
    module cache, in this case an M instance. To drive the point home:

    >>> import sys
    >>> sys.modules["x"] = 42
    >>> import x
    >>> x

    42
    >>> sys.modules["x"] = "spam"
    >>> import x
    >>> x

    'spam'

    > Cheers, Lars
    >
    > PS: I get an error when posting this to the usenet group


    Sorry, that seems to happen when I post via gmane and don't manually clear
    the follow-up that my newsreader helpfully (knode) inserts. I've not yet
    found a permanent fix, but if that was the problem you should be able to
    answer this post.
     
    Peter Otten, Jan 1, 2012
    #10
  11. On Dec 29 2011, 10:55 am, lars van gemerden <>
    wrote:
    > Hello,
    >
    > Can someone help me with the following:
    >
    > I am using metaclasses to make classes and these classes to make
    > instances. Now I want to use multiprocessing, which needs to pickle
    > these instances.
    >
    > Pickle cannot find the class definitions of the instances. I am trying
    > to add a line to the __new__ of the metaclass to add the new class
    > under the right name in the right module/place, so pickle can find
    > it.
    >
    > Is this the right approach? Can anyone explain to me where/how to add
    > these classes for pickle to find and maybe why?
    >
    > Thanks in advance,
    >
    > Lars


    Ok,

    After reading all posts (thanks a lot), I am considering to use the
    following base metaclass for all metaclasses that must lead to
    pickleable instances (not pickleable classes):


    import sys

    class Meta(type):
    def __new__(mcls, name, bases, attrs):
    cls = type.__new__(mcls, name, bases, attrs)
    setattr(sys.modules[__name__], name, cls)
    return cls


    if __name__ == '__main__':
    instance = Meta("Klass", (str,),{})("apple")
    s = pickle.dumps(instance)
    delattr(sys.modules[__name__], "Klass")
    Meta("Klass", (str,),{})
    inst = pickle.loads(s)
    print instance
    print inst
    print type(instance) is type(inst)

    Can anyone see any drawbacks to this approach? I've also tested the
    case where the Meta metaclass is defined in another module.

    Cheers, Lars
     
    lars van gemerden, Jan 3, 2012
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ironfroggy
    Replies:
    16
    Views:
    440
    Michele Simionato
    Jun 3, 2005
  2. Erik Max Francis
    Replies:
    1
    Views:
    395
    Erik Max Francis
    Aug 9, 2005
  3. George Sakkis

    Pickling dynamically generated classes

    George Sakkis, Jan 26, 2008, in forum: Python
    Replies:
    0
    Views:
    288
    George Sakkis
    Jan 26, 2008
  4. Nicolas M. Thiéry

    Pickling classes (not class instances)

    Nicolas M. Thiéry, Jan 10, 2009, in forum: Python
    Replies:
    2
    Views:
    359
    Aaron Brady
    Feb 14, 2009
  5. Steven D'Aprano

    Metaclass of a metaclass

    Steven D'Aprano, Jun 5, 2012, in forum: Python
    Replies:
    1
    Views:
    307
Loading...

Share This Page