multiprocessing as batch system

Discussion in 'Python' started by Neal Becker, Jan 21, 2010.

  1. Neal Becker

    Neal Becker Guest

    I'm using multiprocessing as a poor man's batch system. It is working OK,
    except that it doesn't seem to do load balancing quite right.

    I have a 8-cpu machine. If I start, using multiprocessing pool, calling map
    with more than 8 elements, it will start 8 processes. It seems, though,
    that when some of them finish, it does not start others right away. The
    load average may be a lot less than 8 but still additional processes are not
    active.

    Is there a way to use multiprocessing to achieve the effect I want? (I
    really want a simple batch system, but without complex setup)
     
    Neal Becker, Jan 21, 2010
    #1
    1. Advertisements

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Amit N
    Replies:
    4
    Views:
    713
    Paddy
    Sep 13, 2007
  2. sturlamolden

    multiprocessing module (PEP 371)

    sturlamolden, Jun 4, 2008, in forum: Python
    Replies:
    6
    Views:
    576
    John Nagle
    Jun 8, 2008
  3. Max Ivanov

    multiprocessing eats memory

    Max Ivanov, Sep 25, 2008, in forum: Python
    Replies:
    6
    Views:
    486
    redbaron
    Sep 27, 2008
  4. Aaron \Castironpi\ Brady

    2.6 multiprocessing and pdb

    Aaron \Castironpi\ Brady, Oct 2, 2008, in forum: Python
    Replies:
    1
    Views:
    863
    Gabriel Genellina
    Oct 3, 2008
  5. nhwarriors

    Using multiprocessing

    nhwarriors, Oct 10, 2008, in forum: Python
    Replies:
    4
    Views:
    435
    Aaron \Castironpi\ Brady
    Oct 11, 2008
  6. redbaron
    Replies:
    1
    Views:
    410
    redbaron
    Oct 15, 2008
  7. redbaron
    Replies:
    3
    Views:
    718
    Paul Rubin
    Oct 21, 2008
  8. Replies:
    9
    Views:
    1,226
Loading...