Multithreading and Queue

[Jeffrey Barish]

Several methods in Queue.Queue have warnings in their doc strings that they
are not reliable (e.g., qsize). I note that the code in all these methods
is bracketed with lock acquire/release. These locks are intended to
protect the enclosed code from collisions with other threads. I am
wondering whether I understand correctly that the reason these methods are
still not reliable is that from the point where a thread calls qsize (for
example) to the point in Queue where a thread acquires a lock there is a
bunch of code, none of which is protected by a lock, (and moreover there is
another bunch of code between the point where a thread releases a lock and
then actually returns to the calling program) and so despite the locks in
Queue it is still possible for values to change before a thread acts on
them.

 

[robert]

Several methods in Queue.Queue have warnings in their doc strings that they
are not reliable (e.g., qsize). I note that the code in all these methods
is bracketed with lock acquire/release. These locks are intended to
protect the enclosed code from collisions with other threads. I am
wondering whether I understand correctly that the reason these methods are
still not reliable is that from the point where a thread calls qsize (for
example) to the point in Queue where a thread acquires a lock there is a
bunch of code, none of which is protected by a lock, (and moreover there is
another bunch of code between the point where a thread releases a lock and
then actually returns to the calling program) and so despite the locks in
Queue it is still possible for values to change before a thread acts on
them.

That warnings are quite meaningless: If you get a certain q size, it
might be changed the next moment/OP anyway.

simply len() is the same as that

def qsize(self):
"""Return the approximate size of the queue (not reliable!)."""
self.mutex.acquire()
n = self._qsize()
self.mutex.release()
return n
...
def _qsize(self):
return len(self.queue)


All that time-consuming locking in Queue is quite unnecessary. As
assumed in many other locations in the std lib, list.append() / .pop() /
len() are atomic. Thus doing in a single location a IndexError-catch on
a .pop() race, and all care is done.

(the only additional guarantee through that lock's ( at .append() time)
is that the q size is never one over the exact "maximum" - but thats
pedantic in a threaded multi-producer mess.)

The whole CallQueue in
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/491281 for
example doesn't use a single lock - though its an inter-thread
communication hot spot.

-robert