multiprocessing: queue.get() blocks even if queue.qsize() != 0

[redbaron]

I run into problem with queue from multiprocessing. Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
raises Emtpy error.

I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?

while qresult.qsize():
result = qresult.get() #this code blocks!
doWithResult(result)

 

[MRAB]

I run into problem with queue from multiprocessing. Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
raises Emtpy error.

I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?

while qresult.qsize():
    result = qresult.get()  #this code blocks!
    doWithResult(result)

From Python v2.5 onwards queues also have a task_done() method. Try:

while qresult.qsize():
result = qresult.get() #this code blocks!
doWithResult(result)
qresult.task_done()

 

[Antoon Pardon]

I run into problem with queue from multiprocessing. Even if I
queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
raises Emtpy error.

I'm unable to cut my big part to small test case, because smaller test
case similair to my real app by design is works. In what conditions is
it possible?

while qresult.qsize():
result = qresult.get() #this code blocks!
doWithResult(result)

If you have more than one consumer the above code can block.
The two consumers both see that there is an item present
in the queue. One removes the item and the second blocks.

 

[Paul Rubin]

while qresult.qsize():
result = qresult.get() #this code blocks!
doWithResult(result)

That is unreliable for the reason Antoon explained, and as is
documented in the manual for the Queue module. Write instead
something like (untested):

while True:
try:
result = qresult.get_nowait()
except Empty:
break
doWithResult(result)