Cross platform way of finding number of processors on a machine?

[John]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

 

[Nicholas Bastin]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)

For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.

For Win32, the cheap and dirty way is to read the NUMBER_OF_PROCESSORS
environment variable. 99% of the time, this will be correct, so it
might be sufficient for your purposes. There is a system call
available on windows that will net you the true number of virtual
cpus, but I don't know what it is.

I don't have a cygwin install laying around, but my guess is there's a
sysctl option there too.

One note: *all* of these methods will tell you the virtual number of
CPUs in a machine, not the physical number. There's almost no reason
why you care about the distinction between these two numbers, but if
you do, you'll have to go to great lengths to probe the actual
hardware on each platform. (And pre-WinXP, Windows doesn't actually
know the difference - all processors are presumed to be physical).

Also, Darwin and Linux will easily allow you to get the speed of the
processors, but on x86 these numbers are not the maximums due to C1E
and EIST (x86 processors from all vendors are capable of changing
speeds depending on load).

 

[Tim Golden]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:

http://msdn2.microsoft.com/en-us/library/aa394373.aspx

but, as Nicholas noted:

"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.

TJG

 

[Kay Schluehr]

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:

http://msdn2.microsoft.com/en-us/library/aa394373.aspx

but, as Nicholas noted:

"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""

since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.

TJG

Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:

import os
os.environ['NUMBER_OF_PROCESSORS']

2

However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?

Kay

 

[Tim Golden]

[Tim Golden]

Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:

import os
os.environ['NUMBER_OF_PROCESSORS']

2

However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?

Ahem. I was referring (and not very clearly, now I look
back at my post) to a particular property within the
WMI class, not to the general concept of counting
processors.

TJG

 

[Nick Craig-Wood]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.

For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)

For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.

On any unix/posix system (OSX and linux should both qualify) you can use

(From my Core 2 Duo laptop running linux)

 

[Lawrence Oluyede]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

From processing <http://cheeseshop.python.org/pypi/processing/0.34> :

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass

if num >= 1:
return num
else:
raise NotImplementedError

 

[Paul Boddie]

From processing <http://cheeseshop.python.org/pypi/processing/0.34> :
[...]

num = os.sysconf('SC_NPROCESSORS_ONLN')

It's interesting what new (or obscure) standard library functions (and
system functions) can be discovered through these kinds of
discussions. However, this one seems to report the number of "virtual
CPUs" in situations like mine where the CPU supports hyperthreading,
as Nicholas Bastin says. One way to get the real number of CPUs (on
Linux-based systems) is to parse /proc/cpuinfo, and to count the
number of distinct "physical id" values.

From the experiments I've done with pprocess [1], pretending that a

hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.

Paul

[1] http://www.python.org/pypi/pprocess

 

[Lawrence Oluyede]

From the experiments I've done with pprocess [1], pretending that a
hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.

We came the same conclusion at work. One of my colleagues has a machine
with HyperThreading and he noticed no real performance increase with the
processing library altough on my machine (which is a dual Core 2 Duo)
the performance boost was definitely there

The good side of this is hyperthreading with parallel processes is like
using a mono-CPU machine so we don't have to take into account it like
if it was an entirely different kind of architecture.

 

[Nick Craig-Wood]

Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?

From processing <http://cheeseshop.python.org/pypi/processing/0.34> :

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass

if num >= 1:
return num
else:
raise NotImplementedError

There is a bug in that code...

NotImplementedError will never be raised because num won't have been
set. It will raise "UnboundLocalError: local variable 'num'
referenced before assignment" instead

 

[Nicholas Bastin]

A few reiterated notes inline, from my previous message.

def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])

The user can do bad things to this environment variable, but it's
probably ok most of the time. (Hey, they change it, they pay the
consequences).

else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')

This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.

 

[Lawrence Oluyede]

This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.

I'm not the author of the library. I think you should address these bug
reports elsewhere...