can't use multiprocessing with class factory?

A

Alan

Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
pass

def class_factory(x):
class ConcreteTest(Test):
_x = x
return ConcreteTest

def f(cls):
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(f, [class_factory(i) for i in range(4)])
 
R

Robert Kern

Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
pass

def class_factory(x):
class ConcreteTest(Test):
_x = x
return ConcreteTest

def f(cls):
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(f, [class_factory(i) for i in range(4)])

Send the (pickleable) factory and the arguments used to construct the instance,
not the unpickleable instance itself.

def g(factory, i):
cls = factory(i)
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(g, zip([class_factory] * 4, range(4)))

By the way, when asking for help like this, show us what your code did and
describe what results you want. It can often be hard to figure out exactly what
you mean by "work".

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
D

Daniel Urban

Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
   pass

def class_factory(x):
   class ConcreteTest(Test):
       _x = x
   return ConcreteTest

def f(cls):
   print cls._x

if __name__ == '__main__':
   pool = mp.Pool(2)
   pool.map(f, [class_factory(i) for i in range(4)])

Only classes defined on the top level of a module are picklable (see
http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
). The collections.namedtuple class factory function works around this
limitation by setting the __module__ attribute of the created class,
but I'm not sure if this solution can be used in this case.


Daniel
 
H

Hidura

What is the output?

2011/1/28 said:
Can the below example be fixed to work?
Thanks,
Alan Isaac

import multiprocessing as mp

class Test(object):
pass

def class_factory(x):
class ConcreteTest(Test):
_x = x
return ConcreteTest

def f(cls):
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(f, [class_factory(i) for i in range(4)])
 
R

Robert Kern

Only classes defined on the top level of a module are picklable (see
http://docs.python.org/dev/py3k/library/pickle#what-can-be-pickled-and-unpickled
). The collections.namedtuple class factory function works around this
limitation by setting the __module__ attribute of the created class,
but I'm not sure if this solution can be used in this case.

namedtuple's trick only works when you assign the created class to a name at the
module level. E.g.

MyFancyTuple = collections.namedtuple(...)

The trick won't work for "anonymous" classes like the above use case.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
A

Alan

Send the (pickleable) factory and the arguments used to construct the instance,
not the unpickleable instance itself.

def g(factory, i):
cls = factory(i)
print cls._x

if __name__ == '__main__':
pool = mp.Pool(2)
pool.map(g, zip([class_factory] * 4, range(4)))



If I change that to g((factory,i)) it does work.

Thanks!

Alan
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,480
Members
44,900
Latest member
Nell636132

Latest Threads

Top