python threads on multi-CPU machines

T

Thomas Womack

If I have a dual-processor hyperthreaded machine (so with four CPU
contexts), will a python program distribute threads over all four
logical processors?

I ask because I'm fairly sure that this *does* happen using the
threading extensions in MFC, and fairly sure that it *doesn't* when
using Java, so I don't see that the result is obvious for python.

Tom
 
D

Duncan Booth

If I have a dual-processor hyperthreaded machine (so with four CPU
contexts), will a python program distribute threads over all four
logical processors?

I ask because I'm fairly sure that this *does* happen using the
threading extensions in MFC, and fairly sure that it *doesn't* when
using Java, so I don't see that the result is obvious for python.

The C implementation of Python uses a global interpreter lock that only
allows one thread to interpret bytecode at a time, so while the threads may
be distributed across multiple processors you will get little or no speedup
over a single processor. (If your threads spend most of their time in a
non-Python extension, they may be able to get some benefit from multiple
processors).

The only way to take advantage of multiple processors with Python is to run
at least one separate process for each processor. For example, I believe
Zope will take advantage of multiple processor systems if you run it in ZEO
client/server mode with several Zope client processes.

I believe that Jython simply does whatever the Java implementation does.
 
A

Andrew Dalke

Thomas Womack:
If I have a dual-processor hyperthreaded machine (so with four CPU
contexts), will a python program distribute threads over all four
logical processors?

Yes, unless you are CPU bound in the Python interpreter.
(CPU bound in an extension is fine, and I/O bound, as when
doing network is also fine.)

For a longer answer, see back threads concerning the global
interpreter lock and "free-threading".

Andrew
(e-mail address removed)
 
K

Kevin Cazabon

However, C extensions can release the
GIL, and almost all I/O code in Python does that, so I/O-heavy programs
will make good use of the SMP.

FYI, I'm also working on adding that functionality to the PIL library,
seeing as imaging operations can be fairly expensive too. It makes
these processes more "friendly" to other threads (and Tk) even on
single-CPU boxes, and allows taking advantage of multiple CPUs where
available.

I'll be posting links on the PIL mailing list when it's complete (just
needs exhaustive testing now... it'll be a while), and Fred has agreed
to merge it into the core distribution afterwards.

Kevin.
 
A

Aahz

Aahz:

FYI, I'm also working on adding that functionality to the PIL library,
seeing as imaging operations can be fairly expensive too. It makes
these processes more "friendly" to other threads (and Tk) even on
single-CPU boxes, and allows taking advantage of multiple CPUs where
available.

Cool! When you're done, I'd appreciate it if you could write up a short
summary of the problems you ran into and post it here. I'm interested
in pushing computational threading in Python, but I haven't written
enough C code to have any idea how it differs from other kinds of
threading issues.
 
L

Lothar Scholz

Thomas Womack said:
If I have a dual-processor hyperthreaded machine (so with four CPU
contexts), will a python program distribute threads over all four
logical processors?

Okay others told you already about the GIL.
I ask because I'm fairly sure that this *does* happen using the
threading extensions in MFC, and fairly sure that it *doesn't* when
using Java,

Of couse java *does* this. It's the number one reason for SUN to
develop and maintain Java: selling a reason to buy some of there
overpriced multiprocessor machines. Maybe you don't know anything
about Java.
 
P

Paul McGuire

Just one general caveat, based on some multi-thread code I worked on about a
year ago: if you want to support your users running your multithread code on
multi-CPU machines, you *must, must, must* stress test on multi-CPU
machines!

We had a very nasty intermittent bug (collision with threads in Tcl
interpreter) that I could *not* reproduce on my single-processor development
box, but which showed up in the field on the customer's quad-processor
machine.
Finally, after about a day and a half, I moved over to our dual processor
test machine, and the bug showed up after about 20 minutes of stress
testing. It turned out that the 3rd party thread library we were using was
built to be compatible with the Win98 (!) thread API - once I rebuilt to run
using the WinNT thread API, the window was closed, and the app ran great on
the multiCPU servers.

I'm sorry if this is off-topic, and I am fairly new in this newsgroup, but I
know we were very eager to push multiCPU machines on our customers, and we
did not realize what risk we were taking on.

Sincerely,
-- Paul McGuire
Austin, TX
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,901
Latest member
Noble71S45

Latest Threads

Top