Deadlock problem using multiprocessing

[À¶É«»ùÒò]

This is my first touch on the multiprocessing module, and I admit not
having a deep understanding of parallel programming, forgive me if
there's any obvious error. This is my test code:

# deadlock.py

import multiprocessing

class MPTask:
def __init__(self):
self._tseq= range(10) # task sequence
self._pool= multiprocessing.Pool(2) # process pool
def _exe(self, num):
return num**2
def run(self):
result= self._pool.map_async(self._exe, self._tseq)
return result.get()

result= MPTask().run()
print(result)

And seemingly it creates a deadlock, I have to manually kill the
processes in the system monitor, yet it would be OK if the _exe()
function was defined outside the MPTask class:

# no_deadlock.py
import multiprocessing

def _exe(num):
return num**2

class MPTask:
def __init__(self):
self._tseq= range(10) # task sequence
self._pool= multiprocessing.Pool(2) # process pool
def run(self):
result= self._pool.map_async(_exe, self._tseq)
return result.get()

result= MPTask().run()
print(result)

This would give the correct answer without any deadlock. My questions:

1. Why is this, where did the deadlock come from?

2. Besides, is there any material I can refer to about how to avoid
deadlocks when using multiple processes in the program?

Thanks!

 

[Kushal Kumaran]

This is my first touch on the multiprocessing module, and I admit not
having a deep understanding of parallel programming, forgive me if
there's any obvious error. This is my test code:

# deadlock.py

import multiprocessing

class MPTask:
       def __init__(self):
               self._tseq= range(10)   # task sequence
               self._pool= multiprocessing.Pool(2)     # process pool
       def _exe(self, num):
               return num**2
       def run(self):
               result= self._pool.map_async(self._exe, self._tseq)
               return result.get()

result= MPTask().run()
print(result)

And seemingly it creates a deadlock, I have to manually kill the
processes in the system monitor, yet it would be OK if the _exe()
function was defined outside the MPTask class:

# no_deadlock.py
import multiprocessing

def _exe(num):
       return num**2

class MPTask:
       def __init__(self):
               self._tseq= range(10)   # task sequence
               self._pool= multiprocessing.Pool(2)     # process pool
       def run(self):
               result= self._pool.map_async(_exe, self._tseq)
               return result.get()

result= MPTask().run()
print(result)

This would give the correct answer without any deadlock. My questions:

1. Why is this, where did the deadlock come from?

2. Besides, is there any material I can refer to about how to avoid
deadlocks when using multiple processes in the program?

I get this exception when I run the first program:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
self.run()
File "/usr/lib/python3.1/threading.py", line 469, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_tasks
put(task)
File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
builtins.method failed

There is no deadlock. You are hitting this problem:
http://stackoverflow.com/questions/...d-when-using-pythons-multiprocessing-pool-map

 

[Jacky Liu]

I get this exception when I run the first program:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.1/threading.py", line 516, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.1/threading.py", line 469, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.1/multiprocessing/pool.py", line 231, in _handle_tasks
    put(task)
  File "/usr/lib/python3.1/pickle.py", line 1349, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup
builtins.method failed

There is no deadlock.  You are hitting this problem:http://stackoverflow.com/questions/1816958/cant-pickle-type-instancem...

You're right. I just realized that I was running the program through
my Vim plug-in which would use the subprocess module to open the
process, wait for its termination and get the output. It seems that
multi-process program would act quite differently such that my Vim
plug-in had been halted without getting any error information, so I
was to falsely take it as a deadlock.

The thread in stackoverflow you referenced is great, now I'm trying to
fit the solution in my own program, thanks a lot!