multiprocessing.sharedctypes and built-in locks

[Ahmad Syukri bin Abdollah]

I'm trying this on Python 3.0.1
Consider the following code:
"""
import multiprocessing as mp

def jambu(b,i,gl):
for n in range(100000):
with gl:
b+=2
with gl[3-i]:
b[3-i]-=1

def main():
b = mp.RawArray('i',4)
gl = []
proc = []
for i in range(len(b)):
gl.append(mp.Lock())
proc.append(mp.Process(target=jambu,args=(b,i,gl)))
for p in proc:
p.start()
for p in proc:
p.join()
print(b[:])
print(sum(b[:]))
main()
"""
(Yes, I'm aware that I didn't pass the lock array as shared variable,
but since they're not reassigned, it should be okay) The above code
should produce an output like this:
[100000, 100000, 100000, 100000]
400000
Now, in the documentation for multiprocessing module, it says that
multiprocessing.Array will automatically create a lock to ensure
process-safe synchronization. So I suppose the above code can be
simplified as follows:
"""
import multiprocessing as mp

def jambu(b,i):
for n in range(100000):
b+=2
b[3-i]-=1

def main():
b = mp.Array('i',4)
proc = []
for i in range(len(b)):
gl.append(mp.Lock())
proc.append(mp.Process(target=jambu,args=(b,i)))
for p in proc:
p.start()
for p in proc:
p.join()
print(b[:])
print(sum(b[:]))
main()
"""
The output of this second code isn't consistent with the first one,
implying multiprocessing.Array (or even multiprocessing.Value; I've
also tried with this one) isn't as atomic as I understand it should
be. So what is the actual meaning of having "a Lock object which will
be used to synchronize access to the value"? Is it only for getting,
and not for assigning values?

Regards,
Ahmad Syukri

 

[Aaron Brady]

I'm trying this on Python 3.0.1
Consider the following code:
"""
import multiprocessing as mp

def jambu(b,i,gl):
    for n in range(100000):
       with gl:
            b+=2
       with gl[3-i]:
            b[3-i]-=1

def main():
    b = mp.RawArray('i',4)
    gl = []
    proc = []
    for i in range(len(b)):
        gl.append(mp.Lock())
        proc.append(mp.Process(target=jambu,args=(b,i,gl)))
    for p in proc:
        p.start()
    for p in proc:
        p.join()
    print(b[:])
    print(sum(b[:]))
main()
"""
(Yes, I'm aware that I didn't pass the lock array as shared variable,
but since they're not reassigned, it should be okay) The above code
should produce an output like this:
  [100000, 100000, 100000, 100000]
  400000
Now, in the documentation for multiprocessing module, it says that
multiprocessing.Array will automatically create a lock to ensure
process-safe synchronization. So I suppose the above code can be
simplified as follows:
"""
import multiprocessing as mp

def jambu(b,i):
    for n in range(100000):
        b+=2
        b[3-i]-=1

def main():
    b = mp.Array('i',4)
    proc = []
    for i in range(len(b)):
        gl.append(mp.Lock())
        proc.append(mp.Process(target=jambu,args=(b,i)))
    for p in proc:
        p.start()
    for p in proc:
        p.join()
    print(b[:])
    print(sum(b[:]))
main()
"""
The output of this second code isn't consistent with the first one,
implying multiprocessing.Array (or even multiprocessing.Value; I've
also tried with this one) isn't as atomic as I understand it should
be. So what is the actual meaning of having "a Lock object which will
be used to synchronize access to the value"? Is it only for getting,
and not for assigning values?

Regards,
Ahmad Syukri

Your code hung on my machine. The call to 'main()' should be in an
'if __name__' block:

if __name__== '__main__':
main()

Is it possible you are just seeing the effects of the non-atomic
'__iadd__' operation? That is, the value is read, added, and written
at different times, between which other processes might have
intervened.

 

[Ahmad Syukri b]

Your code hung on my machine.  The call to 'main()' should be in an
'if __name__' block:

if __name__== '__main__':
    main()

Is it possible you are just seeing the effects of the non-atomic
'__iadd__' operation?  That is, the value is read, added, and written
at different times, between which other processes might have
intervened.

I've forgotten to remove ' gl.append(mp.Lock()) ' from the second
code.

Anyways if what you said is true, then why doesn't the documentation
say so? I also realized that the created lock's release and acquire
methods are also imbued to the created shared object, and this is also
undocumented.

 

[Aaron Brady]

I've forgotten to remove ' gl.append(mp.Lock()) ' from the second
code.

Anyways if what you said is true, then why doesn't the documentation
say so?

I couldn't find it either. It's not one of the guarantees that Python
makes. Operations aren't atomic by default, such as unless stated
otherwise. Though it does seem counterintuitive to have to write:

with a_lock:
a+= 1

I also realized that the created lock's release and acquire
methods are also imbued to the created shared object, and this is also
undocumented.

Sorry, I don't follow you. By the way, your first example didn't
produce the expected result on my machine-- even with the 'if
__name__' statement.

 

[Ahmad Syukri b]

It's not one of the guarantees that Python
makes.  Operations aren't atomic by default, such as unless stated
otherwise.  

Well, in the documentation for RawArray:
"Note that setting and getting an element is potentially non-atomic –
use Array() instead to make sure that access is automatically
synchronized using a lock."
So I assumed operations for Array() would be atomic

Sorry, I don't follow you.

Sorry if that wasn't clear. By this, I meant:

b = multiprocessing.Array('i',5)
if b.acquire():
b[0] += 1
b.release()

though you don't get to use the with statement with this one. Also
only one RLock is created per array, whilst my code needed a lock for
each array member.

By the way, your first example didn't
produce the expected result on my machine-- even with the 'if
__name__' statement.

Strange, I just copied and pasted the first example on a new blank
file and ran python3 with it again, it comes out find and dandy
(totalling 400000). I also tried the second example with the gl.append
line deleted, and it also runs (of course, with the wrong results)