multiprocessing: child process race to answer

[smhall05]

I am using a basic multiprocessing snippet I found:

#-----------------------------------------------------
from multiprocessing import Pool

def f(x):
return x*x

if __name__ == '__main__':
pool = Pool(processes=4) # start 4 worker processes
result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
print result.get(timeout=1)
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
#---------------------------------------------------------

I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.

I have tried:

sys.exit(0) #this causes the program to hang
pool.close()
pool.terminate

These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you

 

[MRAB]

I am using a basic multiprocessing snippet I found:

#-----------------------------------------------------
from multiprocessing import Pool

def f(x):
return x*x

if __name__ == '__main__':
pool = Pool(processes=4) # start 4 worker processes
result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
print result.get(timeout=1)
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
#---------------------------------------------------------

I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.

I have tried:

sys.exit(0) #this causes the program to hang
pool.close()
pool.terminate

Did you actually mean "pool.terminate", or is that a typo for
"pool.terminate()"?

 

[smhall05]

I am using a basic multiprocessing snippet I found:


from multiprocessing import Pool

def f(x):

return x*x

if __name__ == '__main__':

pool = Pool(processes=4) # start 4 worker processes

result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously

print result.get(timeout=1)

print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"

#---------------------------------------------------------

I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.
I have tried:
sys.exit(0) #this causes the program to hang

pool.close()

pool.terminate

Did you actually mean "pool.terminate", or is that a typo for

"pool.terminate()"?

These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you

I am not sure to be honest, however it turns out that I can't use pool.terminate() because pool is defined in main and not accessible under my def in which I check for the correct answer.

 

[Sherard Hall]

Thank you for the response. Processing time is very important so I suspect
having to write to disk will take more time than letting the other
processes complete without finding the answer. So I did some profiling one
process finds the answer in about 250ms, but since I can't stop the other
processes, it takes about 800ms before I can use the answer. Do you
recommend a global variable flag? Any other suggestions?

On 02/11/2013 02:35, smhall05 wrote:

I am using a basic multiprocessing snippet I found:

#-----------------------------------------------------
from multiprocessing import Pool

def f(x):
return x*x

if __name__ == '__main__':
pool = Pool(processes=4) # start 4 worker processes
result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
print result.get(timeout=1)
print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
#---------------------------------------------------------

I am using this code to have each process go off and solve the same

problem, just with different inputs to the problem. I need to be able to
kill all processes once 1 of n processes has come up with the solution.
There will only be one answer.What else can I try? I am not able to share the exact code at this time. I
can provide more detail if I am unclear. Thank you

I am not sure to be honest, however it turns out that I can't use

pool.terminate() because pool is defined in main and not accessible under
my def in which I check for the correct answer.
So, the simplest solution to that situation is to have whichever
subprocess that finds the correct answer set a flag which the calling
process can check. Depending on your OS, that flag can be anything from
setting a lock to something as simple as creating a file which the calling
process periodically wakes up and looks for, maybe just a file in which the
subprocess has written the answer.

Bill

 

[cappleman]

I am using a basic multiprocessing snippet I found:



#-----------------------------------------------------

from multiprocessing import Pool



def f(x):

return x*x



if __name__ == '__main__':

pool = Pool(processes=4) # start 4 worker processes

result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously

print result.get(timeout=1)

print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"

#---------------------------------------------------------



I am using this code to have each process go off and solve the same problem, just with different inputs to the problem. I need to be able to kill all processes once 1 of n processes has come up with the solution. There will only be one answer.



I have tried:



sys.exit(0) #this causes the program to hang

pool.close()

pool.terminate



These still allow further processing before the program terminates. What else can I try? I am not able to share the exact code at this time. I can provide more detail if I am unclear. Thank you

You could take a look at the Mutiprocessing module capabilities for exchanging objects between processes:

http://docs.python.org/3.3/library/multiprocessing.html#exchanging-objects-between-processes