multiprocessing: queue.get() blocks even if queue.qsize() != 0

Discussion in 'Python' started by redbaron, Oct 15, 2008.

  1. redbaron

    redbaron Guest

    I run into problem with queue from multiprocessing. Even if I
    queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
    raises Emtpy error.

    I'm unable to cut my big part to small test case, because smaller test
    case similair to my real app by design is works. In what conditions is
    it possible?

    while qresult.qsize():
    result = qresult.get() #this code blocks!
    doWithResult(result)
     
    redbaron, Oct 15, 2008
    #1
    1. Advertising

  2. redbaron

    MRAB Guest

    On Oct 15, 2:05 pm, redbaron <> wrote:
    > I run into problem with queue from multiprocessing. Even if I
    > queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
    > raises Emtpy error.
    >
    > I'm unable to cut my big part to small test case, because smaller test
    > case similair to my real app by design is works. In what conditions is
    > it possible?
    >
    > while qresult.qsize():
    >     result = qresult.get()  #this code blocks!
    >     doWithResult(result)


    From Python v2.5 onwards queues also have a task_done() method. Try:

    while qresult.qsize():
    result = qresult.get() #this code blocks!
    doWithResult(result)
    qresult.task_done()
     
    MRAB, Oct 15, 2008
    #2
    1. Advertising

  3. On 2008-10-15, redbaron <> wrote:
    > I run into problem with queue from multiprocessing. Even if I
    > queue.qsize() != 0 queue.get() still blocks and queue.get_nowait()
    > raises Emtpy error.
    >
    > I'm unable to cut my big part to small test case, because smaller test
    > case similair to my real app by design is works. In what conditions is
    > it possible?
    >
    > while qresult.qsize():
    > result = qresult.get() #this code blocks!
    > doWithResult(result)


    If you have more than one consumer the above code can block.
    The two consumers both see that there is an item present
    in the queue. One removes the item and the second blocks.

    --
    Antoon Pardon
     
    Antoon Pardon, Oct 20, 2008
    #3
  4. redbaron

    Paul Rubin Guest

    redbaron <> writes:
    > while qresult.qsize():
    > result = qresult.get() #this code blocks!
    > doWithResult(result)


    That is unreliable for the reason Antoon explained, and as is
    documented in the manual for the Queue module. Write instead
    something like (untested):

    while True:
    try:
    result = qresult.get_nowait()
    except Empty:
    break
    doWithResult(result)
     
    Paul Rubin, Oct 21, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. James R. Saker Jr.

    Queue qsize = unreliable?

    James R. Saker Jr., Aug 6, 2004, in forum: Python
    Replies:
    4
    Views:
    461
    Jacob Hallen
    Aug 10, 2004
  2. Mike C. Fletcher

    Re: Queue qsize = unreliable?

    Mike C. Fletcher, Aug 6, 2004, in forum: Python
    Replies:
    2
    Views:
    258
    Mike C. Fletcher
    Aug 6, 2004
  3. Tim Peters

    Re: Queue qsize = unreliable?

    Tim Peters, Aug 6, 2004, in forum: Python
    Replies:
    2
    Views:
    522
    Dan Sommers
    Aug 7, 2004
  4. Kris
    Replies:
    0
    Views:
    532
  5. matt
    Replies:
    1
    Views:
    296
    George Ogata
    Aug 6, 2004
Loading...

Share This Page