multiprocessing as batch system

Discussion in 'Python' started by Neal Becker, Jan 21, 2010.

  1. Neal Becker

    Neal Becker Guest

    I'm using multiprocessing as a poor man's batch system. It is working OK,
    except that it doesn't seem to do load balancing quite right.

    I have a 8-cpu machine. If I start, using multiprocessing pool, calling map
    with more than 8 elements, it will start 8 processes. It seems, though,
    that when some of them finish, it does not start others right away. The
    load average may be a lot less than 8 but still additional processes are not
    active.

    Is there a way to use multiprocessing to achieve the effect I want? (I
    really want a simple batch system, but without complex setup)
     
    Neal Becker, Jan 21, 2010
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tam

    batch file

    Tam, Nov 11, 2003, in forum: ASP .Net
    Replies:
    4
    Views:
    5,421
    cemal eroglu
    Dec 25, 2003
  2. Dave
    Replies:
    0
    Views:
    919
  3. brad
    Replies:
    2
    Views:
    3,968
    Joe Smith
    Jan 19, 2004
  4. Mike John

    How to run a batch file.

    Mike John, Jul 23, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    419
    Saravana
    Jul 23, 2003
  5. Mike John
    Replies:
    2
    Views:
    371
    Ray Dixon [MVP]
    Jul 23, 2003
Loading...

Share This Page