Get multiprocessing.Queue to do priorities

Discussion in 'Python' started by uuid, May 9, 2009.

  1. uuid

    uuid Guest

    Hello,
    I was wondering whether there was a way to make multiprocessing.Queue
    behave in a priority queue-like fashion. Subclassing with heappush and
    heappop for put and get doesn't work the old way (multiprocessing.Queue
    seems to use different data structures than Queue.Queue?)

    Could one create a heapq within the producer as a proxy, and then feed
    a proper queue from that? Does anyone have an idea on how to deal with
    the queue item flow control? (Since all of the sorting has to happen
    within the heapq, it should only pass items to the real Queue if it's
    empty?)

    Thanks in advance!
    uuid, May 9, 2009
    #1
    1. Advertising

  2. uuid

    uuid Guest

    I just read up and it seems that no matter the approach, it's futile to
    use multiprocessing.Queue, since there is a bug that prevents true
    FIFO. (http://bugs.python.org/issue4999)

    Any recommendation on an alternate way to build a priority queue to use
    with a "one producer, many consumers" type multiprocessing setup would
    be welcomed!

    :|

    On 2009-05-09 18:42:34 +0200, uuid <> said:

    > Hello,
    > I was wondering whether there was a way to make multiprocessing.Queue
    > behave in a priority queue-like fashion. Subclassing with heappush and
    > heappop for put and get doesn't work the old way (multiprocessing.Queue
    > seems to use different data structures than Queue.Queue?)
    >
    > Could one create a heapq within the producer as a proxy, and then feed
    > a proper queue from that? Does anyone have an idea on how to deal with
    > the queue item flow control? (Since all of the sorting has to happen
    > within the heapq, it should only pass items to the real Queue if it's
    > empty?)
    >
    > Thanks in advance!
    uuid, May 9, 2009
    #2
    1. Advertising

  3. uuid

    uuid Guest

    The Queue module, apparently, is thread safe, but *not* process safe.
    If you try to use an ordinary Queue, it appears inaccessible to the
    worker process. (Which, after all, is quite logical, since methods for
    moving items between the threads of the same process are quite
    different from inter-process communication.) It appears that creating a
    manager that holds a shared queue might be an option
    (http://stackoverflow.com/questions/342556/python-2-6-multiprocessing-queue-compatible-with-threads).

    Just

    for illustration: This shows that Queue.Queue doesn't work with processes:

    ------------------------
    def worker(queue):
    while True:
    item = queue.get()
    print item
    queue.task_done()

    queue_queue = Queue.Queue()

    worker_thread = multiprocessing.Process(target=worker, args=(queue_queue,))
    worker_thread.start()
    for i in range(10):
    queue_queue.put(str(i))
    time.sleep(10)
    while True:
    try:
    print 'still on queue: ' + queue_queue.get(False)
    except Queue.Empty:
    break
    worker_thread.join()
    ------------------------

    This yields:

    still on queue: 0
    still on queue: 1
    still on queue: 2
    still on queue: 3
    still on queue: 4
    still on queue: 5
    still on queue: 6
    still on queue: 7
    still on queue: 8
    still on queue: 9

    So no queue item ever arrives at the worker process.



    On 2009-05-09 22:00:36 +0200, Scott David Daniels <> said:

    > 2.6 has a PriorityQueue in the Queue module.
    > If you aren't using 2.6, you could copy the code for your own version.
    uuid, May 9, 2009
    #3
  4. uuid

    Jesse Noller Guest

    On Sat, May 9, 2009 at 6:11 PM, uuid <> wrote:
    > The Queue module, apparently, is thread safe, but *not* process safe. If you
    > try to use an ordinary Queue, it appears inaccessible to the worker process.
    > (Which, after all, is quite logical, since methods for moving items between
    > the threads of the same process are quite different from inter-process
    > communication.) It appears that creating a manager that holds a shared queue
    > might be an option
    > (http://stackoverflow.com/questions/342556/python-2-6-multiprocessing-queue-compatible-with-threads).


    Using a manager, or submitting a patch which adds priority queue to
    the multiprocessing.queue module is the correct solution for this.

    You can file an enhancement in the tracker, and assign/add me to it,
    but without a patch it may take me a bit (wicked busy right now).

    jesse
    Jesse Noller, May 10, 2009
    #4
  5. uuid

    uuid Guest

    Dear Jesse,
    thanks for the hint.

    I see you are already assigned to the FIFO bug
    (http://bugs.python.org/issue4999), so I won't burden you even more.
    Clearly, a reliable FIFO behavior of multiprocessing.Queue helps more
    than a priority queue, since it can be used to build one, so that
    should really be the first thing to fix.

    In the meantime, I think I'll whip up a hack that uses sort of a
    bucket- strategy: fill up a prioritized heapq, and then, in regular
    intervals, unload its contents into a size-limited multiprocessing
    queue.

    I'll post this as soon as it works.

    -u

    On 2009-05-10 15:35:03 +0200, Jesse Noller <> said:

    > Using a manager, or submitting a patch which adds priority queue to
    > the multiprocessing.queue module is the correct solution for this.
    >
    > You can file an enhancement in the tracker, and assign/add me to it,
    > but without a patch it may take me a bit (wicked busy right now).
    >
    > jesse
    uuid, May 10, 2009
    #5
  6. uuid

    Guest

    Was this issue ever resolved? What is the current best practice for those wishing to use a priority queue with multiprocessing?

    On Sunday, May 10, 2009 6:35:03 AM UTC-7, Jesse Noller wrote:
    > On Sat, May 9, 2009 at 6:11 PM, uuid <> wrote:
    > > The Queue module, apparently, is thread safe, but *not* process safe. If you
    > > try to use an ordinary Queue, it appears inaccessible to the worker process.
    > > (Which, after all, is quite logical, since methods for moving items between
    > > the threads of the same process are quite different from inter-process
    > > communication.) It appears that creating a manager that holds a shared queue
    > > might be an option
    > > (http://stackoverflow.com/questions/342556/python-2-6-multiprocessing-queue-compatible-with-threads).

    >
    > Using a manager, or submitting a patch which adds priority queue to
    > the multiprocessing.queue module is the correct solution for this.
    >
    > You can file an enhancement in the tracker, and assign/add me to it,
    > but without a patch it may take me a bit (wicked busy right now).
    >
    > jesse
    , Apr 1, 2013
    #6
  7. uuid

    Guest

    Was this issue ever resolved? What is the current best practice for those wishing to use a priority queue with multiprocessing?

    On Sunday, May 10, 2009 6:35:03 AM UTC-7, Jesse Noller wrote:
    > On Sat, May 9, 2009 at 6:11 PM, uuid <> wrote:
    > > The Queue module, apparently, is thread safe, but *not* process safe. If you
    > > try to use an ordinary Queue, it appears inaccessible to the worker process.
    > > (Which, after all, is quite logical, since methods for moving items between
    > > the threads of the same process are quite different from inter-process
    > > communication.) It appears that creating a manager that holds a shared queue
    > > might be an option
    > > (http://stackoverflow.com/questions/342556/python-2-6-multiprocessing-queue-compatible-with-threads).

    >
    > Using a manager, or submitting a patch which adds priority queue to
    > the multiprocessing.queue module is the correct solution for this.
    >
    > You can file an enhancement in the tracker, and assign/add me to it,
    > but without a patch it may take me a bit (wicked busy right now).
    >
    > jesse
    , Apr 1, 2013
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    666
    Russell Warren
    Jun 27, 2006
  2. redbaron
    Replies:
    1
    Views:
    343
    redbaron
    Oct 15, 2008
  3. redbaron
    Replies:
    3
    Views:
    484
    Paul Rubin
    Oct 21, 2008
  4. Travis  Miller
    Replies:
    3
    Views:
    382
    Gabriel Genellina
    Mar 20, 2009
  5. Kris
    Replies:
    0
    Views:
    464
Loading...

Share This Page