Tips for load balancing multiple Python apps on dual/quad coreprocessors?

Discussion in 'Python' started by Malcolm Greene, Apr 7, 2008.

  1. I'm looking for tips on how to load balance running multiple Python
    applications in multi-CPU environments. My understanding is that Python
    applications and their threads are limited to a specific CPU.

    Background: I have a Python utility that processes email messages. I
    suspect there's a lot of idle time while this utility waits on a remote
    email server. I would like to run as many simultaneous copies of this
    utility as possible without overwhelming the server these utilities are
    running on. My thought is that I should write a dispatcher that monitors
    CPU load and launches/cancels multiple instances of my utility with
    specific job queues to process.

    Is there a cross-platform way to monitor CPU load?

    Is there a pre-existing Python module I should be looking at for
    building (subclassing) my dispatcher?

    Thanks!
    Malcolm
    Malcolm Greene, Apr 7, 2008
    #1
    1. Advertising

  2. Malcolm Greene

    Paul Rubin Guest

    Re: Tips for load balancing multiple Python apps on dual/quad core processors?

    "Malcolm Greene" <> writes:
    > Is there a cross-platform way to monitor CPU load?


    Cross-platform: not that I know of.

    Linux: /proc/loadav (load average),
    /proc/cpuinfo (to figure out number of cpu's).

    You want the load average and # of cpu's to be about equal,
    i.e. all cpu's should be kept busy but not overloaded.
    Paul Rubin, Apr 7, 2008
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Kamilche

    Quad Perspective Transformation

    Kamilche, Feb 1, 2007, in forum: Python
    Replies:
    1
    Views:
    928
    Gabriel Genellina
    Feb 1, 2007
  2. Sanny
    Replies:
    29
    Views:
    987
    Twisted
    Jul 25, 2007
  3. Sanny
    Replies:
    2
    Views:
    1,009
    Roedy Green
    Jan 10, 2008
  4. Sanny
    Replies:
    4
    Views:
    430
    Mark Space
    Feb 3, 2008
  5. Replies:
    1
    Views:
    3,112
    Arne Vajhøj
    Jul 31, 2012
Loading...

Share This Page