Read-Write Lock vs primitive Lock()

Discussion in 'Python' started by k3xji, Dec 29, 2008.

  1. k3xji

    k3xji Guest

    Hi,

    I am trying to see on which situations does the Read-Write Lock
    performs better on primitive Lock() itself. Below is the code I am
    using to test the performance:
    import threading
    import locks
    import time

    class mylock(object):
    def __init__(self):
    self.__notreading = threading.Event()
    self.__notwriting = threading.Event()
    self.__notreading.set()
    self.__notwriting.set()
    def acquire_read(self):
    self.__notreading.clear()
    self.__notwriting.wait()
    def acquire_write(self):
    self.__notreading.wait()
    self.__notwriting.clear()
    def release_read(self):
    self.__notreading.set()
    def release_write(self):
    self.__notwriting.set()

    GLOBAL_VAR = 1
    #GLOBAL_LOCK = locks.ReadWriteLock()
    GLOBAL_LOCK = threading.Lock()
    #GLOBAL_LOCK = mylock()
    GLOBAL_LOOP_COUNT = 100000
    GLOBAL_READER_COUNT = 1000
    GLOBAL_WRITER_COUNT = 1



    class wthread(threading.Thread):
    def run(self):
    try:
    #GLOBAL_LOCK.acquireWrite()
    #GLOBAL_LOCK.acquire_write()
    GLOBAL_LOCK.acquire()
    for i in range(GLOBAL_LOOP_COUNT):
    GLOBAL_VAR = 4
    finally:
    #GLOBAL_LOCK.release_write()
    GLOBAL_LOCK.release()

    class rthread(threading.Thread):
    def run(self):
    try:
    #GLOBAL_LOCK.acquireRead()
    #GLOBAL_LOCK.acquire_read()
    GLOBAL_LOCK.acquire()
    for i in range(GLOBAL_LOOP_COUNT):
    GLOBAL_VAR = 3
    finally:
    #GLOBAL_LOCK.release_read()
    GLOBAL_LOCK.release()

    # module executed?
    if __name__ == "__main__":
    starttime = time.clock()
    threads = []
    for i in range(GLOBAL_READER_COUNT):
    rt = rthread()
    threads.append(rt)
    for i in range(GLOBAL_WRITER_COUNT):
    wt = wthread()
    threads.append(wt)

    for thread in threads:
    thread.start()
    for thread in threads:
    thread.join()
    print "All operations took " + str(time.clock() - starttime) + "
    msecs"


    What I am doing is: I am creating multiple readers and try to do
    something. I had assumed that with using primitive Lock() on the above
    situation, it will create a bottleneck on the rthreads. But the
    numbers indicate that there are no difference at all. I had
    implemented my own READ-WRIET lock as can be seen above mylock and
    also used the one here: code.activestate.com/recipes/502283/.

    Both have the same numbers:
    above test with primitive Lock:
    C:\Python25\mytest>python test_rw.py
    All operations took 14.4584082614 msecs
    above test with mylock:
    C:\Python25\mytest>python test_rw.py
    All operations took 14.5185156214 msecs
    abive test with the one in recipe:
    C:\Python25\mytest>python test_rw.py
    All operations took 14.4641975447 msecs

    So, I am confused in which situations Read-Write lock scales better?

    Thanks,
    k3xji, Dec 29, 2008
    #1
    1. Advertising

  2. En Mon, 29 Dec 2008 05:56:10 -0200, k3xji <> escribió:

    > I am trying to see on which situations does the Read-Write Lock
    > performs better on primitive Lock() itself. Below is the code I am
    > using to test the performance:
    > import threading
    > import locks
    > import time
    >
    > class mylock(object):


    (I'm not convinced your lock is correct)

    > GLOBAL_VAR = 1
    > #GLOBAL_LOCK = locks.ReadWriteLock()
    > GLOBAL_LOCK = threading.Lock()
    > #GLOBAL_LOCK = mylock()
    > GLOBAL_LOOP_COUNT = 100000
    > GLOBAL_READER_COUNT = 1000
    > GLOBAL_WRITER_COUNT = 1


    Only one writer? If this is always the case, you don't need a lock at all.

    > class wthread(threading.Thread):
    > def run(self):
    > try:
    > #GLOBAL_LOCK.acquireWrite()
    > #GLOBAL_LOCK.acquire_write()
    > GLOBAL_LOCK.acquire()
    > for i in range(GLOBAL_LOOP_COUNT):
    > GLOBAL_VAR = 4
    > finally:
    > #GLOBAL_LOCK.release_write()
    > GLOBAL_LOCK.release()


    Note that the thread acquires the lock ONCE, repeats several thousand
    times an assignment to a *local* variable called GLOBAL_VAR (!), finally
    releases the lock and exits. As every thread does the same, they just run
    one after another, they never have a significant overlap.

    Also, you should acquire the lock *before* the try block (you have to
    ensure that, *after* acquiring the lock, it is always released; such
    requisite does not apply *before* acquiring the lock)

    I'd test again with something like this:

    class wthread(threading.Thread):
    def run(self):
    global GLOBAL_VAR
    for i in xrange(GLOBAL_LOOP_COUNT):
    GLOBAL_LOCK.acquire()
    try:
    GLOBAL_VAR += 1
    finally:
    GLOBAL_LOCK.release()

    > class rthread(threading.Thread):
    > def run(self):
    > try:
    > #GLOBAL_LOCK.acquireRead()
    > #GLOBAL_LOCK.acquire_read()
    > GLOBAL_LOCK.acquire()
    > for i in range(GLOBAL_LOOP_COUNT):
    > GLOBAL_VAR = 3
    > finally:
    > #GLOBAL_LOCK.release_read()
    > GLOBAL_LOCK.release()


    Hmmm, it's a reader but attempts to modify the value?
    You don't have to protect a read operation on a global variable - so a
    lock isn't required here.

    > What I am doing is: I am creating multiple readers and try to do
    > something. I had assumed that with using primitive Lock() on the above
    > situation, it will create a bottleneck on the rthreads. But the
    > numbers indicate that there are no difference at all. I had
    > implemented my own READ-WRIET lock as can be seen above mylock and
    > also used the one here: code.activestate.com/recipes/502283/.


    I hope you now understand why you got the same numbers always.

    --
    Gabriel Genellina
    Gabriel Genellina, Dec 29, 2008
    #2
    1. Advertising

  3. k3xji

    k3xji Guest

    On 29 Aralýk, 11:52, "Gabriel Genellina" <>
    wrote:
    > En Mon, 29 Dec 2008 05:56:10 -0200, k3xji <> escribió:
    >
    > > I  am trying to see on which situations does the Read-Write Lock
    > > performs better on primitive Lock() itself. Below is the code I am
    > > using to test the performance:
    > > import threading
    > > import locks
    > > import time

    >
    > > class mylock(object):

    >
    > (I'm not convinced your lock is correct)


    No problem.:)

    > > GLOBAL_VAR = 1
    > > #GLOBAL_LOCK = locks.ReadWriteLock()
    > > GLOBAL_LOCK = threading.Lock()
    > > #GLOBAL_LOCK = mylock()
    > > GLOBAL_LOOP_COUNT = 100000
    > > GLOBAL_READER_COUNT = 1000
    > > GLOBAL_WRITER_COUNT = 1

    >
    > Only one writer? If this is always the case, you don't need a lock at all..


    No just used for testing. It does not matter, what I want to see is
    that I want to create a botleneck on readers.

    > > class wthread(threading.Thread):
    > >     def run(self):
    > >             try:
    > >                 #GLOBAL_LOCK.acquireWrite()
    > >                 #GLOBAL_LOCK.acquire_write()
    > >                 GLOBAL_LOCK.acquire()
    > >                 for i in range(GLOBAL_LOOP_COUNT):
    > >                     GLOBAL_VAR = 4
    > >             finally:
    > >                 #GLOBAL_LOCK.release_write()
    > >                 GLOBAL_LOCK.release()

    >
    > Note that the thread acquires the lock ONCE, repeats several thousand
    > times an assignment to a *local* variable called GLOBAL_VAR (!), finally
    > releases the lock and exits. As every thread does the same, they just run
    > one after another, they never have a significant overlap.


    If I put the for loop outside, in that case, readers will not overlap
    at all, and you would be amazed by the numbers for that test. They
    indicate primitive lock is faster than read-write lock, as it requires
    the lock, executes only one bytecode operation and releases the lock.
    So, in order to create a bottlenneck on readers, we need to somehow do
    not release the lock immediately.

    > Also, you should acquire the lock *before* the try block (you have to
    > ensure that, *after* acquiring the lock, it is always released; such
    > requisite does not apply *before* acquiring the lock)


    Yeah, you are right but it is irrelevant.

    > I'd test again with something like this:
    >
    > class wthread(threading.Thread):
    >       def run(self):
    >           global GLOBAL_VAR
    >           for i in xrange(GLOBAL_LOOP_COUNT):
    >               GLOBAL_LOCK.acquire()
    >               try:
    >                   GLOBAL_VAR += 1
    >               finally:
    >                   GLOBAL_LOCK.release()


    With that, primitive locks perform 10 times better than Read-Write
    lock. See above.


    > > class rthread(threading.Thread):
    > >     def run(self):
    > >             try:
    > >                 #GLOBAL_LOCK.acquireRead()
    > >                 #GLOBAL_LOCK.acquire_read()
    > >                 GLOBAL_LOCK.acquire()
    > >                 for i in range(GLOBAL_LOOP_COUNT):
    > >                     GLOBAL_VAR = 3
    > >             finally:
    > >                 #GLOBAL_LOCK.release_read()
    > >                 GLOBAL_LOCK.release()

    >
    > Hmmm, it's a reader but attempts to modify the value?
    > You don't have to protect a read operation on a global variable - so a
    > lock isn't required here.


    This is just for testing. Suppose that I am actually reading the
    value. I don't understand why a lock is not required? Are you saying
    lock is not necesary beacuse GLOBAL_VALUE is an immutable object, if
    then, suppose it is not. This is just a test. Suppose GLOBAl_VAR is a
    list and we are calling append() on it which is nt an atomic
    operation.

    > > What I am doing is: I am creating multiple readers and try to do
    > > something. I had assumed that with using primitive Lock() on the above
    > > situation, it will create a bottleneck on the rthreads. But the
    > > numbers indicate that there are no difference at all. I had
    > > implemented my own READ-WRIET lock as can be seen above mylock and
    > > also used the one here: code.activestate.com/recipes/502283/.

    >
    > I hope you now understand why you got the same numbers always.


    Unfortunately, I do not understand anyhing.

    Thanks.
    k3xji, Dec 29, 2008
    #3
  4. k3xji

    Aaron Brady Guest

    On Dec 29, 4:17 am, k3xji <> wrote:
    > On 29 Aralýk, 11:52, "Gabriel Genellina" <>
    > wrote:
    >
    > > En Mon, 29 Dec 2008 05:56:10 -0200, k3xji <> escribió:

    >

    snip
    > > > class wthread(threading.Thread):
    > > >     def run(self):
    > > >             try:
    > > >                 #GLOBAL_LOCK.acquireWrite()
    > > >                 #GLOBAL_LOCK.acquire_write()
    > > >                 GLOBAL_LOCK.acquire()
    > > >                 for i in range(GLOBAL_LOOP_COUNT):
    > > >                     GLOBAL_VAR = 4
    > > >             finally:
    > > >                 #GLOBAL_LOCK.release_write()
    > > >                 GLOBAL_LOCK.release()

    >
    > > Note that the thread acquires the lock ONCE, repeats several thousand
    > > times an assignment to a *local* variable called GLOBAL_VAR (!)

    snip
    > > class wthread(threading.Thread):
    > >       def run(self):
    > >           global GLOBAL_VAR
    > >           for i in xrange(GLOBAL_LOOP_COUNT):
    > >               GLOBAL_LOCK.acquire()
    > >               try:
    > >                   GLOBAL_VAR += 1
    > >               finally:
    > >                   GLOBAL_LOCK.release()

    >
    > With that, primitive locks perform 10 times better than Read-Write
    > lock. See above.

    snip

    Gabriel's point (one of them) was that 'GLOBAL_VAR' isn't global in
    your example. Your 'wthread' objects aren't sharing anything. He
    added the 'global GLOBAL_VAR' statement, which is important
    Aaron Brady, Dec 29, 2008
    #4
  5. En Mon, 29 Dec 2008 08:17:42 -0200, k3xji <> escribió:

    > On 29 Aralık, 11:52, "Gabriel Genellina" <>
    > wrote:
    >> En Mon, 29 Dec 2008 05:56:10 -0200, k3xji <> escribió:
    >>
    >> > I  am trying to see on which situations does the Read-Write Lock
    >> > performs better on primitive Lock() itself. Below is the code I am
    >> > using to test the performance:
    >> > import threading
    >> > import locks
    >> > import time

    >>
    >> > class mylock(object):

    >>
    >> (I'm not convinced your lock is correct)

    >
    > No problem.:)
    >
    >> > GLOBAL_VAR = 1
    >> > #GLOBAL_LOCK = locks.ReadWriteLock()
    >> > GLOBAL_LOCK = threading.Lock()
    >> > #GLOBAL_LOCK = mylock()
    >> > GLOBAL_LOOP_COUNT = 100000
    >> > GLOBAL_READER_COUNT = 1000
    >> > GLOBAL_WRITER_COUNT = 1

    >>
    >> Only one writer? If this is always the case, you don't need a lock at
    >> all.

    >
    > No just used for testing. It does not matter, what I want to see is
    > that I want to create a botleneck on readers.


    Is this a theoretical question? What do your readers/writers actually do?
    Just reading a global variable does not require any locking. If it is
    being modified, you either get the previous value, or the new value.

    >> Note that the thread acquires the lock ONCE, repeats several thousand
    >> times an assignment to a *local* variable called GLOBAL_VAR (!), finally
    >> releases the lock and exits. As every thread does the same, they just
    >> run
    >> one after another, they never have a significant overlap.

    >
    > If I put the for loop outside, in that case, readers will not overlap
    > at all, and you would be amazed by the numbers for that test. They
    > indicate primitive lock is faster than read-write lock, as it requires
    > the lock, executes only one bytecode operation and releases the lock.
    > So, in order to create a bottlenneck on readers, we need to somehow do
    > not release the lock immediately.


    Again, what is your specific use case? why do you think readers will see a
    bottleneck?

    > With that, primitive locks perform 10 times better than Read-Write
    > lock. See above.


    So what? You don't like such results? The read-write lock in the recipe
    you posted is rather complex, and isn't obvious whether it is correct or
    not, and the author doesn't prove it nor provide a reference. I'd stick to
    the standard Lock object unless I have specific needs.
    Note that threading.Lock objects are implemented in C and don't have
    significant overhead (other than the wait itself); even threading.RLock
    objects are a lot slower. So depending on what you really have to do, the
    overhead of using a class like ReadWriteLock may be worse than using a
    standard Lock.

    >> Hmmm, it's a reader but attempts to modify the value?
    >> You don't have to protect a read operation on a global variable - so a
    >> lock isn't required here.

    >
    > This is just for testing. Suppose that I am actually reading the
    > value. I don't understand why a lock is not required? Are you saying
    > lock is not necesary beacuse GLOBAL_VALUE is an immutable object, if
    > then, suppose it is not. This is just a test. Suppose GLOBAl_VAR is a
    > list and we are calling append() on it which is nt an atomic
    > operation.


    It doesn't matter whether it is mutable or immutable. Remember that, in
    Python, a global variable is just a *name* that refers to an object; you
    either get the old object referred, or the new one. If it is a list and it
    is being modified "in place", you always get the same list -- its contents
    may differ from one instant to another so iterating over it isn't safe.
    That's why I insist on knowing your use case: you may me doing things more
    complicated than they should.

    Attached there is a modified test using a shared list. A group of writers
    append elements; another group of writers pop elements. Readers
    continuously check the list sanity. The list starts empty, should be empty
    at the end, and should be equal to range(len(global_list)) at any time.
    Run the code as-is, with all locks enabled, and it should work fine.
    Disable the locking in each place, and things start going wrong.

    >> > What I am doing is: I am creating multiple readers and try to do
    >> > something. I had assumed that with using primitive Lock() on the above
    >> > situation, it will create a bottleneck on the rthreads. But the
    >> > numbers indicate that there are no difference at all. I had
    >> > implemented my own READ-WRIET lock as can be seen above mylock and
    >> > also used the one here: code.activestate.com/recipes/502283/.

    >>
    >> I hope you now understand why you got the same numbers always.

    >
    > Unfortunately, I do not understand anyhing.


    Each thread in your original code were like this:

    begin thread; wait for lock; execute something locally; release lock; exit
    thread;

    There was no shared state between threads. Even if you fix them to share
    some variable, they were a single, sequential, piece of code (the inner
    loop is irrelevant here). Once a thread gets the lock, it just runs to its
    end. No one waits more than once for the lock. So all of them run
    sequentially, in arbitrary order, but just one after the other. They never
    overlap.

    --
    Gabriel Genellina
    Gabriel Genellina, Dec 30, 2008
    #5
  6. k3xji

    k3xji Guest


    > >> Note that the thread acquires the lock ONCE, repeats several thousand
    > >> times an assignment to a *local* variable called GLOBAL_VAR (!), finally
    > >> releases the lock and exits. As every thread does the same, they just  
    > >> run
    > >> one after another, they never have a significant overlap.


    Ok. Forget about the GLOBAL_VAR. See the below code:

    import threading
    import locks
    import time

    GLOBAL_VAR = []
    #GLOBAL_LOCK = locks.ReadWriteLock()
    GLOBAL_LOCK = threading.Lock()
    #GLOBAL_LOCK = mylock()
    GLOBAL_LOOP_COUNT = 10000000
    GLOBAL_READER_COUNT = 100
    GLOBAL_WRITER_COUNT = 1
    global GLOBAL_LEN


    class wthread(threading.Thread):
    def run(self):
    try:
    GLOBAL_LOCK.acquireWrite()
    #GLOBAL_LOCK.acquire_write()
    #GLOBAL_LOCK.acquire()
    for i in range(GLOBAL_LOOP_COUNT):
    pass
    finally:
    #GLOBAL_LOCK.release_write()
    GLOBAL_LOCK.release()

    class rthread(threading.Thread):
    def run(self):
    try:
    GLOBAL_LOCK.acquireRead()
    #GLOBAL_LOCK.acquire_read()
    print 'ENTER:'+str(threading.currentThread())
    #GLOBAL_LOCK.acquire()
    print 'ACQUIRE:'+str(threading.currentThread())
    for i in range(GLOBAL_LOOP_COUNT):
    pass
    finally:
    #GLOBAL_LOCK.release_read()
    GLOBAL_LOCK.release()
    print 'RELEASE:'+str(threading.currentThread())

    # module executed?
    if __name__ == "__main__":
    starttime = time.clock()
    threads = []
    for i in range(GLOBAL_READER_COUNT):
    rt = rthread()
    threads.append(rt)
    for i in range(GLOBAL_WRITER_COUNT):
    wt = wthread()
    threads.append(wt)

    for thread in threads:
    thread.start()
    for thread in threads:
    thread.join()
    print "All operations took " + str(time.clock() - starttime) + "
    msecs"

    As GLOBAL_LOOP_COUNT is 10000000, now this is making a bottleneck on
    the readers. I had assumed that as everythread is given only 100
    bytecodes to execute, that it will be enough to have a 10000 value for
    this number to let other rthread start() and try to acquire the lock.
    But, Python undocumentedly, does not grab the GIL easily if a Lock is
    held. This is strange for me.

    > > If I put the for loop outside, in that case, readers will not overlap
    > > at all, and you would be amazed by the numbers for that test. They
    > > indicate primitive lock is faster than read-write lock, as it requires
    > > the lock, executes only one bytecode operation and releases the lock.
    > > So, in order to create a bottlenneck on readers, we need to somehow do
    > > not release the lock immediately.

    >
    > Again, what is your specific use case? why do you think readers will see a  
    > bottleneck?
    >
    > > With that, primitive locks perform 10 times better than Read-Write
    > > lock. See above.

    >
    > So what? You don't like such results? The read-write lock in the recipe  
    > you posted is rather complex, and isn't obvious whether it is correct or  
    > not, and the author doesn't prove it nor provide a reference. I'd stick to  
    > the standard Lock object unless I have specific needs.
    > Note that threading.Lock objects are implemented in C and don't have  
    > significant overhead (other than the wait itself); even threading.RLock  
    > objects are a lot slower. So depending on what you really have to do, the  
    > overhead of using a class like ReadWriteLock may be worse than using a  
    > standard Lock.


    That may be the point. That is why I am trying to test this. When will
    ReadWrite() lock permforms better over the primitive lock. By the way,
    for the reference, the code in the recipe is used in CherryPy and
    based on the Tanenbaum's book Operating Systems Design.

    > >> Hmmm, it's a reader but attempts to modify the value?
    > >> You don't have to protect a read operation on a global variable - so a
    > >> lock isn't required here.

    >
    > > This is just for testing. Suppose that I am actually reading the
    > > value. I don't understand why a lock is not required? Are you saying
    > > lock is not necesary beacuse GLOBAL_VALUE is an immutable object, if
    > > then, suppose it is not. This is just a test. Suppose GLOBAl_VAR is a
    > > list and we are calling append() on it which is nt an atomic
    > > operation.

    >
    > It doesn't matter whether it is mutable or immutable. Remember that, in  
    > Python, a global variable is just a *name* that refers to an object; you  
    > either get the old object referred, or the new one. If it is a list and it  
    > is being modified "in place", you always get the same list -- its contents  
    > may differ from one instant to another so iterating over it isn't safe.
    > That's why I insist on knowing your use case: you may me doing things more  
    > complicated than they should.


    I understand your point completely, but let's change anything inside
    the loop. Just assume that we a thread that is supposed to read
    something, and a thread that is supposed to write. If I create
    thousands of readers, and if the operation is enormously huge
    calculation(as Python does not grab GIL easily inside a Lock), then
    this will create a bottlencek on readers. But, with ReadWrite Lock
    this problem *SHOULD* be fixed and in my tests I see it is fixed, when
    I increase the time spent in the loop in huge amounts.


    >
    > Each thread in your original code were like this:
    >
    > begin thread; wait for lock; execute something locally; release lock; exit  
    > thread;
    >
    > There was no shared state between threads. Even if you fix them to share  
    > some variable, they were a single, sequential, piece of code (the inner  
    > loop is irrelevant here). Once a thread gets the lock, it just runs to its  
    > end. No one waits more than once for the lock. So all of them run  
    > sequentially, in arbitrary order, but just one after the other. They never  
    > overlap.


    You are correct about this. But this is an undocumented thing. I had
    assumed that Python will release GIL for other threads after 100
    bytecodes are executed by default. However, as I stated, when a Lock()
    is held this is changing. I think this is because to avoid simple MT
    problems for new programmers. Not sure. I will read threads.c for
    information.

    Thanks.
    k3xji, Dec 30, 2008
    #6
  7. k3xji

    k3xji Guest


    > >> Note that the thread acquires the lock ONCE, repeats several thousand
    > >> times an assignment to a *local* variable called GLOBAL_VAR (!), finally
    > >> releases the lock and exits. As every thread does the same, they just  
    > >> run
    > >> one after another, they never have a significant overlap.


    Ok. Forget about the GLOBAL_VAR. See the below code:

    import threading
    import locks
    import time

    GLOBAL_VAR = []
    #GLOBAL_LOCK = locks.ReadWriteLock()
    GLOBAL_LOCK = threading.Lock()
    #GLOBAL_LOCK = mylock()
    GLOBAL_LOOP_COUNT = 10000000
    GLOBAL_READER_COUNT = 100
    GLOBAL_WRITER_COUNT = 1
    global GLOBAL_LEN


    class wthread(threading.Thread):
    def run(self):
    try:
    GLOBAL_LOCK.acquireWrite()
    #GLOBAL_LOCK.acquire_write()
    #GLOBAL_LOCK.acquire()
    for i in range(GLOBAL_LOOP_COUNT):
    pass
    finally:
    #GLOBAL_LOCK.release_write()
    GLOBAL_LOCK.release()

    class rthread(threading.Thread):
    def run(self):
    try:
    GLOBAL_LOCK.acquireRead()
    #GLOBAL_LOCK.acquire_read()
    print 'ENTER:'+str(threading.currentThread())
    #GLOBAL_LOCK.acquire()
    print 'ACQUIRE:'+str(threading.currentThread())
    for i in range(GLOBAL_LOOP_COUNT):
    pass
    finally:
    #GLOBAL_LOCK.release_read()
    GLOBAL_LOCK.release()
    print 'RELEASE:'+str(threading.currentThread())

    # module executed?
    if __name__ == "__main__":
    starttime = time.clock()
    threads = []
    for i in range(GLOBAL_READER_COUNT):
    rt = rthread()
    threads.append(rt)
    for i in range(GLOBAL_WRITER_COUNT):
    wt = wthread()
    threads.append(wt)

    for thread in threads:
    thread.start()
    for thread in threads:
    thread.join()
    print "All operations took " + str(time.clock() - starttime) + "
    msecs"

    As GLOBAL_LOOP_COUNT is 10000000, now this is making a bottleneck on
    the readers. I had assumed that as everythread is given only 100
    bytecodes to execute, that it will be enough to have a 10000 value for
    this number to let other rthread start() and try to acquire the lock.
    But, Python undocumentedly, does not grab the GIL easily if a Lock is
    held. This is strange for me.

    > > If I put the for loop outside, in that case, readers will not overlap
    > > at all, and you would be amazed by the numbers for that test. They
    > > indicate primitive lock is faster than read-write lock, as it requires
    > > the lock, executes only one bytecode operation and releases the lock.
    > > So, in order to create a bottlenneck on readers, we need to somehow do
    > > not release the lock immediately.

    >
    > Again, what is your specific use case? why do you think readers will see a  
    > bottleneck?
    >
    > > With that, primitive locks perform 10 times better than Read-Write
    > > lock. See above.

    >
    > So what? You don't like such results? The read-write lock in the recipe  
    > you posted is rather complex, and isn't obvious whether it is correct or  
    > not, and the author doesn't prove it nor provide a reference. I'd stick to  
    > the standard Lock object unless I have specific needs.
    > Note that threading.Lock objects are implemented in C and don't have  
    > significant overhead (other than the wait itself); even threading.RLock  
    > objects are a lot slower. So depending on what you really have to do, the  
    > overhead of using a class like ReadWriteLock may be worse than using a  
    > standard Lock.


    That may be the point. That is why I am trying to test this. When will
    ReadWrite() lock permforms better over the primitive lock. By the way,
    for the reference, the code in the recipe is used in CherryPy and
    based on the Tanenbaum's book Operating Systems Design.

    > >> Hmmm, it's a reader but attempts to modify the value?
    > >> You don't have to protect a read operation on a global variable - so a
    > >> lock isn't required here.

    >
    > > This is just for testing. Suppose that I am actually reading the
    > > value. I don't understand why a lock is not required? Are you saying
    > > lock is not necesary beacuse GLOBAL_VALUE is an immutable object, if
    > > then, suppose it is not. This is just a test. Suppose GLOBAl_VAR is a
    > > list and we are calling append() on it which is nt an atomic
    > > operation.

    >
    > It doesn't matter whether it is mutable or immutable. Remember that, in  
    > Python, a global variable is just a *name* that refers to an object; you  
    > either get the old object referred, or the new one. If it is a list and it  
    > is being modified "in place", you always get the same list -- its contents  
    > may differ from one instant to another so iterating over it isn't safe.
    > That's why I insist on knowing your use case: you may me doing things more  
    > complicated than they should.


    I understand your point completely, but let's change anything inside
    the loop. Just assume that we a thread that is supposed to read
    something, and a thread that is supposed to write. If I create
    thousands of readers, and if the operation is enormously huge
    calculation(as Python does not grab GIL easily inside a Lock), then
    this will create a bottlencek on readers. But, with ReadWrite Lock
    this problem *SHOULD* be fixed and in my tests I see it is fixed, when
    I increase the time spent in the loop in huge amounts.


    >
    > Each thread in your original code were like this:
    >
    > begin thread; wait for lock; execute something locally; release lock; exit  
    > thread;
    >
    > There was no shared state between threads. Even if you fix them to share  
    > some variable, they were a single, sequential, piece of code (the inner  
    > loop is irrelevant here). Once a thread gets the lock, it just runs to its  
    > end. No one waits more than once for the lock. So all of them run  
    > sequentially, in arbitrary order, but just one after the other. They never  
    > overlap.


    You are correct about this. But this is an undocumented thing. I had
    assumed that Python will release GIL for other threads after 100
    bytecodes are executed by default. However, as I stated, when a Lock()
    is held this is changing. I think this is because to avoid simple MT
    problems for new programmers. Not sure. I will read threads.c for
    information.

    Thanks.
    k3xji, Dec 30, 2008
    #7
  8. En Tue, 30 Dec 2008 06:16:23 -0200, k3xji <> escribió:

    > As GLOBAL_LOOP_COUNT is 10000000, now this is making a bottleneck on
    > the readers. I had assumed that as everythread is given only 100
    > bytecodes to execute, that it will be enough to have a 10000 value for
    > this number to let other rthread start() and try to acquire the lock.
    > But, Python undocumentedly, does not grab the GIL easily if a Lock is
    > held. This is strange for me.


    This has nothing to do with the GIL, nor undocumented behavior. If all
    threads are blocked waiting for the same lock, yielding the CPU has no
    effect, because nobody can progress until the lock is released.

    I'll ask you again: what is your use case? what do you want to do? what is
    a "reader" in your problem? what is a "writer"? what is the resource you
    want to protect using a lock?
    Different problems have different behaviors; for example, for
    reading/writing files, your model is basically wrong.

    > That may be the point. That is why I am trying to test this. When will
    > ReadWrite() lock permforms better over the primitive lock. By the way,
    > for the reference, the code in the recipe is used in CherryPy and
    > based on the Tanenbaum's book Operating Systems Design.


    Ah, good to know at least it has some background. Anyway, isn't a
    guarantee of correctness. By exampe, Tanenbaum's algorithm for the dining
    philosophers problem were (is?) flawed (according to W. Stallings).

    > I understand your point completely, but let's change anything inside
    > the loop. Just assume that we a thread that is supposed to read
    > something, and a thread that is supposed to write. If I create
    > thousands of readers, and if the operation is enormously huge
    > calculation(as Python does not grab GIL easily inside a Lock), then
    > this will create a bottlencek on readers. But, with ReadWrite Lock
    > this problem *SHOULD* be fixed and in my tests I see it is fixed, when
    > I increase the time spent in the loop in huge amounts.


    But why would you protect the whole long calculation with a lock? And why
    do you say "Python does not grab GIL easily inside a Lock", whatever that
    means?
    If you hold a scarce resource for a long time, this is likely to cause a
    bottleneck, and the language (be it Python or other) is mostly irrelevant
    here.

    > I had
    > assumed that Python will release GIL for other threads after 100
    > bytecodes are executed by default.


    Yes. You can alter how often this occurs, using sys.setcheckinterval (the
    code I posted does that, just to see how results vary)

    > However, as I stated, when a Lock()
    > is held this is changing. I think this is because to avoid simple MT
    > problems for new programmers. Not sure. I will read threads.c for
    > information.


    No, it's always the same thing, locks don't affect this behavior. But as I
    said above, if no other thread can progress because all of them are
    waiting to acquire the same lock, you gain nothing by releasing the GIL.

    --
    Gabriel Genellina
    Gabriel Genellina, Dec 30, 2008
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    7
    Views:
    590
    Victor Bazarov
    May 9, 2005
  2. Fuzzyman
    Replies:
    3
    Views:
    476
    Andrew MacIntyre
    Dec 5, 2003
  3. Robert Brewer
    Replies:
    0
    Views:
    478
    Robert Brewer
    Dec 5, 2003
  4. Daniel Pitts
    Replies:
    7
    Views:
    462
  5. Geoff Bache

    Read-write lock for Python

    Geoff Bache, Apr 28, 2011, in forum: Python
    Replies:
    6
    Views:
    1,817
    Geoff Bache
    Apr 29, 2011
Loading...

Share This Page