multiprocessing question/error

[Eduardo Lenz]

Hi,

I was using the former processing package with python 2.5 with no problems.
After switching to python 2.6.1 I am having some problems with the same code.
The problem seems to be related to the fact that I am using Pool.map
with a bounded method, since it is inside a class. To clarify a little bit,
let me show some parts of the code ....

class Pygen3(self)....
....
....
....
def calcula(self,indiv):
....
....
....

def evaluate(self):
....
....
indiv = range(mult*self.popsize,(mult+1)*self.popsize)
pool = Pool(processes=nproc)
results = pool.map(self.calcula,indiv)
...
...

the error is the following

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed


Thanks for your help.

--

Eduardo Lenz Cardoso
Dr. Eng.
Associate Professor

State University of Santa Catarina
Department of Mechanical Engineering
89223-100 - Joinville-SC - Brasil

Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
E-mail: (e-mail address removed)
---------------------------------------------

 

[Aaron Brady]

Hi,

I was using the former processing package with python 2.5 with no problems.
After switching to python 2.6.1 I am having some problems with the same code.
The problem seems to be related to the fact that I am using Pool.map
with a bounded method, since it is inside a class. To clarify a little bit,
let me show some parts of the code ....
Hello,

class Pygen3(self)....

I infer this was a typo.

    ....
    ....
    ....
    def calcula(self,indiv):
        ....
        ....
        ....

   def evaluate(self):
     ....
     ....
    indiv = range(mult*self.popsize,(mult+1)*self.popsize)
   pool = Pool(processes=nproc)
   results = pool.map(self.calcula,indiv)

I infer the indentation drifted.

   ...
   ...

the error is the following

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed

Thanks for your help.

Checking on two typos above. I have this working:

from multiprocessing import Process

class C(object):
def m1( self ):
p1 = Process(target=self.m2, args=('bob1',))
p2 = Process(target=self.m2, args=('bob2',))
p1.start()
p2.start()
p1.join()
p2.join()
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

But I get the same error as you with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= range( 10 )
pool= Pool( 5 )
results = pool.map(self.m2,arg)
print results
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

I build a workaround with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= zip( [ self ]* 10, range( 10 ) )
pool= Pool( 5 )
results = pool.map(C_m2,arg)
print results
def m2( self, arg ):
print 'hello', arg
return arg* 2
def C_m2( ar, **kwar ):
return C.m2( *ar, **kwar )

if __name__ == '__main__':
c= C()
c.m1()

Note that you have to explicitly include 'self' as a first parameter
when packing the arguments, and then explicitly unpack it in the
helper function. Also, wrapping 'm2' with '@staticmethod' also fails
with this error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\programs\python26\lib\threading.py", line 522, in
__bootstrap_inner
self.run()
File "c:\programs\python26\lib\threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

If this works for you, it's possible to make the helper function more
dynamic, by accepting a class name and an attribute name as string
arguments, then retrieving them in the subprocesses.

 

[Gabriel Genellina]

Another question: is it possible to
change a bound method to a function on the fly ? I was wondering if
something
like class.__dict__["function_name"] could do the job.

Exactly.
Or, if you already have the method, use its im_func attribute to obtain
the original function. BTW, im_self is the "self" argument.

 

[Eduardo Lenz]

Hi,

I was using the former processing package with python 2.5 with no
problems. After switching to python 2.6.1 I am having some problems with
the same code. The problem seems to be related to the fact that I am
using Pool.map with a bounded method, since it is inside a class. To
clarify a little bit, let me show some parts of the code ....
Hello,

class Pygen3(self)....

I infer this was a typo.

    ....
    ....
    ....
    def calcula(self,indiv):
        ....
        ....
        ....

   def evaluate(self):
     ....
     ....
    indiv = range(mult*self.popsize,(mult+1)*self.popsize)
   pool = Pool(processes=nproc)
   results = pool.map(self.calcula,indiv)

I infer the indentation drifted.

   ...
   ...

the error is the following

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 225, in
_handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed

Thanks for your help.

Checking on two typos above. I have this working:

from multiprocessing import Process

class C(object):
def m1( self ):
p1 = Process(target=self.m2, args=('bob1',))
p2 = Process(target=self.m2, args=('bob2',))
p1.start()
p2.start()
p1.join()
p2.join()
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

But I get the same error as you with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= range( 10 )
pool= Pool( 5 )
results = pool.map(self.m2,arg)
print results
def m2( self, name ):
print 'hello', name

if __name__ == '__main__':
c= C()
c.m1()

I build a workaround with this:

from multiprocessing import Process, Pool

class C(object):
def m1( self ):
arg= zip( [ self ]* 10, range( 10 ) )
pool= Pool( 5 )
results = pool.map(C_m2,arg)
print results
def m2( self, arg ):
print 'hello', arg
return arg* 2
def C_m2( ar, **kwar ):
return C.m2( *ar, **kwar )

if __name__ == '__main__':
c= C()
c.m1()

Note that you have to explicitly include 'self' as a first parameter
when packing the arguments, and then explicitly unpack it in the
helper function. Also, wrapping 'm2' with '@staticmethod' also fails
with this error:

Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\programs\python26\lib\threading.py", line 522, in
__bootstrap_inner
self.run()
File "c:\programs\python26\lib\threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File "c:\programs\python26\lib\multiprocessing\pool.py", line 225,
in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup
__builtin__.function failed

If this works for you, it's possible to make the helper function more
dynamic, by accepting a class name and an attribute name as string
arguments, then retrieving them in the subprocesses.

Thanks for your help ! The only question remaninig is why it iused to work
with the former processing package. Another question: is it possible to
change a bound method to a function on the fly ? I was wondering if something
like class.__dict__["function_name"] could do the job. Actually, I was
wondering with something like a casting, but a guess it is not possible.

Thanks again,
Eduardo.
--

Eduardo Lenz Cardoso
Dr. Eng.
Associate Professor

State University of Santa Catarina
Department of Mechanical Engineering
89223-100 - Joinville-SC - Brasil

Tel: +55 47 4009-7971 - Fax: +55 47 4009-7940
E-mail: (e-mail address removed)
---------------------------------------------