multiprocessing: child process share changes to global variable


A

Alan

In the code below, child processes see changes to global variables caused by other child processes. That is, pool.map is producing a list where the value of ``lst`` is being (non-deterministically) shared across child processes. Standard advice is that, "Processes have independent memory space" and"each process works with its own copy" of global variables, so I thought this was not supposed to happen. In the end, the globals are not changed, which matches my expectation. Still, clearly I have misunderstood what seemedto be standard advice. Can someone clarify for me?

Expected result from pool.map: [[0],[1],[2]]
Usual result: [[0],[0,1],[0,1,2]]
Occasional result: [[0],[1],[0,2]]

Thanks,
Alan Isaac


#--------------- temp.py -------------------------
#run at Python 2.7 command prompt
import time
import multiprocessing as mp
lst = []
lstlst = []

def alist(x):
lst.append(x)
lstlst.append(lst)
print "a"
return lst

if __name__=='__main__':
pool = mp.Pool(3)
print pool.map(alist,range(3)) #UNEXPECTED RESULTS
print "b"
time.sleep(0.1)

print "c"
print lst
print lstlst
 
Ad

Advertisements

J

Jason Friedman

#--------------- temp.py -------------------------
#run at Python 2.7 command prompt
import time
import multiprocessing as mp
lst = []
lstlst = []

def alist(x):
lst.append(x)
lstlst.append(lst)
print "a"
return lst

if __name__=='__main__':
pool = mp.Pool(3)
print pool.map(alist,range(3)) #UNEXPECTED RESULTS
print "b"
time.sleep(0.1)

print "c"
print lst
print lstlst
--

I get several different results:

$ for i in {1..1000}; do
python2.7 temp.py | md5sum >> result
done

$ wc -l result
1000 result

$ sort result | uniq | wc -l
9
 
Ad

Advertisements

A

Alan

I received a suggestion off list that my results indicated that the jobs were not being distributed across the pool workers. I used mp.current_process().name to confirm this suggestion.

Alan Isaac
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top