Pool module -- does map pre-assign tasks to processes?

Discussion in 'Python' started by Luca, Sep 28, 2009.

  1. Luca

    Luca Guest

    I would like to use the Pool module, but my tasks take sometimes
    unpredictably different time to complete. The simplest way to write
    the code would be to put all task descriptions in an array, then call

    p = Pool(8)
    p.map(f, a)

    But I don't want to preassign process 0 with elements a[0], a[8], a
    [16], process 1 with elements a[1], a[9], a[17], and so forth.
    Rather, I would like all tasks to be put into a queue, and I would
    like the processes to each time grab the next task to be done, and do
    it. This would ensure fairly equal loading.

    My question is: does the map() method of Pool pre-assign which task
    gets done by each process, or is this done at runtime, on a get-first-
    task-to-be-done basis?

    Many thanks,

    Luca
    Luca, Sep 28, 2009
    #1
    1. Advertising

  2. Luca

    Aahz Guest

    In article <>,
    Luca <> wrote:
    >I would like to use the Pool module, but my tasks take sometimes
    >unpredictably different time to complete. The simplest way to write
    >the code would be to put all task descriptions in an array, then call
    >
    >p = Pool(8)
    >p.map(f, a)
    >
    >But I don't want to preassign process 0 with elements a[0], a[8], a
    >[16], process 1 with elements a[1], a[9], a[17], and so forth.
    >Rather, I would like all tasks to be put into a queue, and I would
    >like the processes to each time grab the next task to be done, and do
    >it. This would ensure fairly equal loading.
    >
    >My question is: does the map() method of Pool pre-assign which task
    >gets done by each process, or is this done at runtime, on a get-first-
    >task-to-be-done basis?


    You can probably get what you want by setting chunksize to 1.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "....Normal is what cuts off your sixth finger and your tail..." --Siobhan
    Aahz, Oct 5, 2009
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. DT
    Replies:
    0
    Views:
    437
  2. Jordan Tan
    Replies:
    0
    Views:
    493
    Jordan Tan
    May 10, 2006
  3. Rick Lawson
    Replies:
    8
    Views:
    783
    Graham Dumpleton
    Jul 17, 2009
  4. John Ladasky
    Replies:
    7
    Views:
    869
    John Ladasky
    Sep 17, 2011
  5. Abdul-rahman Advany

    Managing a fork pool to handle tasks

    Abdul-rahman Advany, Apr 29, 2008, in forum: Ruby
    Replies:
    6
    Views:
    150
    John Carter
    Apr 30, 2008
Loading...

Share This Page