multiprocessing.sharedctypes and built-in locks

Discussion in 'Python' started by Ahmad Syukri bin Abdollah, Mar 14, 2009.

  1. I'm trying this on Python 3.0.1
    Consider the following code:
    """
    import multiprocessing as mp

    def jambu(b,i,gl):
    for n in range(100000):
    with gl:
    b+=2
    with gl[3-i]:
    b[3-i]-=1

    def main():
    b = mp.RawArray('i',4)
    gl = []
    proc = []
    for i in range(len(b)):
    gl.append(mp.Lock())
    proc.append(mp.Process(target=jambu,args=(b,i,gl)))
    for p in proc:
    p.start()
    for p in proc:
    p.join()
    print(b[:])
    print(sum(b[:]))
    main()
    """
    (Yes, I'm aware that I didn't pass the lock array as shared variable,
    but since they're not reassigned, it should be okay) The above code
    should produce an output like this:
    [100000, 100000, 100000, 100000]
    400000
    Now, in the documentation for multiprocessing module, it says that
    multiprocessing.Array will automatically create a lock to ensure
    process-safe synchronization. So I suppose the above code can be
    simplified as follows:
    """
    import multiprocessing as mp

    def jambu(b,i):
    for n in range(100000):
    b+=2
    b[3-i]-=1

    def main():
    b = mp.Array('i',4)
    proc = []
    for i in range(len(b)):
    gl.append(mp.Lock())
    proc.append(mp.Process(target=jambu,args=(b,i)))
    for p in proc:
    p.start()
    for p in proc:
    p.join()
    print(b[:])
    print(sum(b[:]))
    main()
    """
    The output of this second code isn't consistent with the first one,
    implying multiprocessing.Array (or even multiprocessing.Value; I've
    also tried with this one) isn't as atomic as I understand it should
    be. So what is the actual meaning of having "a Lock object which will
    be used to synchronize access to the value"? Is it only for getting,
    and not for assigning values?

    Regards,
    Ahmad Syukri
     
    Ahmad Syukri bin Abdollah, Mar 14, 2009
    #1
    1. Advertisements

  2. Ahmad Syukri bin Abdollah

    Aaron Brady Guest



    Your code hung on my machine. The call to 'main()' should be in an
    'if __name__' block:

    if __name__== '__main__':
    main()

    Is it possible you are just seeing the effects of the non-atomic
    '__iadd__' operation? That is, the value is read, added, and written
    at different times, between which other processes might have
    intervened.
     
    Aaron Brady, Mar 14, 2009
    #2
    1. Advertisements

  3. I've forgotten to remove ' gl.append(mp.Lock()) ' from the second
    code.

    Anyways if what you said is true, then why doesn't the documentation
    say so? I also realized that the created lock's release and acquire
    methods are also imbued to the created shared object, and this is also
    undocumented.
     
    Ahmad Syukri b, Mar 16, 2009
    #3
  4. Ahmad Syukri bin Abdollah

    Aaron Brady Guest

    I couldn't find it either. It's not one of the guarantees that Python
    makes. Operations aren't atomic by default, such as unless stated
    otherwise. Though it does seem counterintuitive to have to write:

    with a_lock:
    a+= 1
    Sorry, I don't follow you. By the way, your first example didn't
    produce the expected result on my machine-- even with the 'if
    __name__' statement.
     
    Aaron Brady, Mar 16, 2009
    #4
  5. Well, in the documentation for RawArray:
    "Note that setting and getting an element is potentially non-atomic –
    use Array() instead to make sure that access is automatically
    synchronized using a lock."
    So I assumed operations for Array() would be atomic
    Sorry if that wasn't clear. By this, I meant:

    b = multiprocessing.Array('i',5)
    if b.acquire():
    b[0] += 1
    b.release()

    though you don't get to use the with statement with this one. Also
    only one RLock is created per array, whilst my code needed a lock for
    each array member.
    Strange, I just copied and pasted the first example on a new blank
    file and ran python3 with it again, it comes out find and dandy
    (totalling 400000). I also tried the second example with the gl.append
    line deleted, and it also runs (of course, with the wrong results)
     
    Ahmad Syukri b, Mar 16, 2009
    #5
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.