multiprocessing: Correct usage of pool & queue?

Discussion in 'Python' started by Allen Fowler, Sep 4, 2009.

  1. Allen Fowler

    Allen Fowler Guest

    Hello,

    I have a list of tasks/items that I want handed off to
    threads/processes to complete. (I would like to stick with process if
    I could, since there is some CPU work here. )

    Each task involves some calculations and a call to a remote server over urllib2/HTTP.

    The
    time to complete each task varies from 1 to 20 seconds depending on a
    number of factors including variable delay on the remote server.

    I would like to:

    1) Have a maximum of 20 "in-flight" tasks. (thus worker processes?)

    2)
    Not overload the external server that each task is calling. No more
    than "3 new tasks" per second. More "waiting" tasks may be OK, i need
    to test it.

    3) Certain tasks in my list must be processed in
    the correct order. (I guess the asignment logic must somehow tag those
    to by done by the same worker?)


    Do any of you have suggestions? Can someone point me in the direction of sample code for this?

    Thank you,
    :)
    Allen Fowler, Sep 4, 2009
    #1
    1. Advertising

  2. Allen Fowler

    Aahz Guest

    In article <>,
    Allen Fowler <> wrote:
    >
    >1) Have a maximum of 20 "in-flight" tasks. (thus worker processes?)


    Good bet.

    >3) Certain tasks in my list must be processed in the correct order. (I
    >guess the asignment logic must somehow tag those to by done by the same
    >worker?)


    The simpler way to do this would be to bundle these tasks into a single
    queue object that contains a list of tasks. Each worker iterates over
    the list of tasks that it receives, which could be a single task.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "To me vi is Zen. To use vi is to practice zen. Every command is a
    koan. Profound to the user, unintelligible to the uninitiated. You
    discover truth everytime you use it."
    Aahz, Sep 10, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Russell Warren

    Is Queue.Queue.queue.clear() thread-safe?

    Russell Warren, Jun 22, 2006, in forum: Python
    Replies:
    4
    Views:
    678
    Russell Warren
    Jun 27, 2006
  2. redbaron
    Replies:
    3
    Views:
    494
    Paul Rubin
    Oct 21, 2008
  3. masher
    Replies:
    4
    Views:
    870
    ryles
    Jul 3, 2009
  4. John Ladasky
    Replies:
    7
    Views:
    883
    John Ladasky
    Sep 17, 2011
  5. Kris
    Replies:
    0
    Views:
    478
Loading...

Share This Page