Tips for load balancing multiple Python apps on dual/quad coreprocessors?

[Malcolm Greene]

I'm looking for tips on how to load balance running multiple Python
applications in multi-CPU environments. My understanding is that Python
applications and their threads are limited to a specific CPU.

Background: I have a Python utility that processes email messages. I
suspect there's a lot of idle time while this utility waits on a remote
email server. I would like to run as many simultaneous copies of this
utility as possible without overwhelming the server these utilities are
running on. My thought is that I should write a dispatcher that monitors
CPU load and launches/cancels multiple instances of my utility with
specific job queues to process.

Is there a cross-platform way to monitor CPU load?

Is there a pre-existing Python module I should be looking at for
building (subclassing) my dispatcher?

Thanks!
Malcolm

 

[Paul Rubin]

Is there a cross-platform way to monitor CPU load?

Cross-platform: not that I know of.

Linux: /proc/loadav (load average),
/proc/cpuinfo (to figure out number of cpu's).

You want the load average and # of cpu's to be about equal,
i.e. all cpu's should be kept busy but not overloaded.