Scheduling used in the multiprocessing.Pool.map() function

Discussion in 'Python' started by rpg, Oct 31, 2009.

  1. rpg

    rpg Guest

    Hi all,

    I have been using the map() function in the multiprocessing module to
    parallelize my tasks on a dual core CPU. My tasks are embarrassingly
    parallel, shared nothing tasks. In one of my runs, I found that the
    this function interleaves execution of two processes over a single
    list.

    So far so good. But the problem is that the last remnant job is
    executed serially. I mean that it seems that the job scheduling is
    essentially static, and the last piece does not execute in parallel.

    Why can't there be a task-stealing scheduler in multiprocessing? Each
    of my individual function call in map takes over half hour (Each
    function call internally calls out to c++ code). This could be a very
    useful addition to multiprocessing's utility.

    Thanks.
    rpg, Oct 31, 2009
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    2,080
    Aaron Brady
    May 1, 2009
  2. Replies:
    1
    Views:
    333
    Jesse Noller
    Apr 30, 2009
  3. masher
    Replies:
    4
    Views:
    878
    ryles
    Jul 3, 2009
  4. Rick Lawson
    Replies:
    8
    Views:
    800
    Graham Dumpleton
    Jul 17, 2009
  5. Allen Fowler
    Replies:
    1
    Views:
    1,206
Loading...

Share This Page