can't use multiprocessing with class factory?

Discussion in 'Python' started by Alan, Jan 28, 2011.

  1. Alan

    Alan Guest

    Can the below example be fixed to work?
    Thanks,
    Alan Isaac

    import multiprocessing as mp

    class Test(object):
    pass

    def class_factory(x):
    class ConcreteTest(Test):
    _x = x
    return ConcreteTest

    def f(cls):
    print cls._x

    if __name__ == '__main__':
    pool = mp.Pool(2)
    pool.map(f, [class_factory(i) for i in range(4)])
    Alan, Jan 28, 2011
    #1
    1. Advertising

  2. Alan

    Robert Kern Guest

    On 1/28/11 1:02 PM, Alan wrote:
    > Can the below example be fixed to work?
    > Thanks,
    > Alan Isaac
    >
    > import multiprocessing as mp
    >
    > class Test(object):
    > pass
    >
    > def class_factory(x):
    > class ConcreteTest(Test):
    > _x = x
    > return ConcreteTest
    >
    > def f(cls):
    > print cls._x
    >
    > if __name__ == '__main__':
    > pool = mp.Pool(2)
    > pool.map(f, [class_factory(i) for i in range(4)])


    Send the (pickleable) factory and the arguments used to construct the instance,
    not the unpickleable instance itself.

    def g(factory, i):
    cls = factory(i)
    print cls._x

    if __name__ == '__main__':
    pool = mp.Pool(2)
    pool.map(g, zip([class_factory] * 4, range(4)))

    By the way, when asking for help like this, show us what your code did and
    describe what results you want. It can often be hard to figure out exactly what
    you mean by "work".

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Jan 28, 2011
    #2
    1. Advertising

  3. Alan

    Daniel Urban Guest

    On Fri, Jan 28, 2011 at 20:02, Alan <> wrote:
    > Can the below example be fixed to work?
    > Thanks,
    > Alan Isaac
    >
    > import multiprocessing as mp
    >
    > class Test(object):
    >    pass
    >
    > def class_factory(x):
    >    class ConcreteTest(Test):
    >        _x = x
    >    return ConcreteTest
    >
    > def f(cls):
    >    print cls._x
    >
    > if __name__ == '__main__':
    >    pool = mp.Pool(2)
    >    pool.map(f, [class_factory(i) for i in range(4)])


    Only classes defined on the top level of a module are picklable (see
    http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
    ). The collections.namedtuple class factory function works around this
    limitation by setting the __module__ attribute of the created class,
    but I'm not sure if this solution can be used in this case.


    Daniel
    Daniel Urban, Jan 28, 2011
    #3
  4. Alan

    Hidura Guest

    What is the output?

    2011/1/28, Alan <>:
    > Can the below example be fixed to work?
    > Thanks,
    > Alan Isaac
    >
    > import multiprocessing as mp
    >
    > class Test(object):
    > pass
    >
    > def class_factory(x):
    > class ConcreteTest(Test):
    > _x = x
    > return ConcreteTest
    >
    > def f(cls):
    > print cls._x
    >
    > if __name__ == '__main__':
    > pool = mp.Pool(2)
    > pool.map(f, [class_factory(i) for i in range(4)])
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >


    --
    Enviado desde mi dispositivo móvil

    Diego I. Hidalgo D.
    Hidura, Jan 28, 2011
    #4
  5. Alan

    Robert Kern Guest

    On 1/28/11 1:25 PM, Daniel Urban wrote:

    > Only classes defined on the top level of a module are picklable (see
    > http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
    > ). The collections.namedtuple class factory function works around this
    > limitation by setting the __module__ attribute of the created class,
    > but I'm not sure if this solution can be used in this case.


    namedtuple's trick only works when you assign the created class to a name at the
    module level. E.g.

    MyFancyTuple = collections.namedtuple(...)

    The trick won't work for "anonymous" classes like the above use case.

    --
    Robert Kern

    "I have come to believe that the whole world is an enigma, a harmless enigma
    that is made terrible by our own mad attempt to interpret it as though it had
    an underlying truth."
    -- Umberto Eco
    Robert Kern, Jan 28, 2011
    #5
  6. Alan

    Alan Guest

    On Jan 28, 2:23 pm, Robert Kern <> wrote:
    > Send the (pickleable) factory and the arguments used to construct the instance,
    > not the unpickleable instance itself.
    >
    > def g(factory, i):
    > cls = factory(i)
    > print cls._x
    >
    > if __name__ == '__main__':
    > pool = mp.Pool(2)
    > pool.map(g, zip([class_factory] * 4, range(4)))




    If I change that to g((factory,i)) it does work.

    Thanks!

    Alan
    Alan, Jan 28, 2011
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Medi Montaseri
    Replies:
    17
    Views:
    852
    Medi Montaseri
    Sep 3, 2003
  2. Johan Bergman
    Replies:
    6
    Views:
    419
    Johan Bergman
    Jan 20, 2004
  3. Simon Elliott
    Replies:
    0
    Views:
    375
    Simon Elliott
    Jan 11, 2005
  4. C#
    Replies:
    4
    Views:
    400
  5. Alan
    Replies:
    4
    Views:
    249
    Bill Felton
    Jan 27, 2011
Loading...

Share This Page