Threading in python

[Carl J. Van Arsdall]

Hi everyone, I'm trying to use the threading module with python 2.2 and
I have some questions regarding python's threading.

1. Who schedules which threads run and when? Is this something left up
to the operating system or does python provide a mechanism for this?

2. I've read that python threads don't like to allow other threads to
run except in certain situations, these would be situations where there
is sleep or I/O happening, is this true? If so, what are the cases in
which a python thread would not give up the processor?

3. Is there a way to which thread is running? I mean something a bit
more robust than a print statement inside the thread, I want to be able
to see when a context switch occurs, is this possible? The reason for
this is I will have threads deadlocked waiting for I/O and I am
interested to see how often context switching occurs.

Thanks in advance,

-carl

--

Carl J. Van Arsdall
(e-mail address removed)
Build and Release
MontaVista Software

 

[Aahz]

I have some questions regarding python's threading.

These answers assume you're using CPython; Jython and IronPython have
different answers.

1. Who schedules which threads run and when? Is this something left up
to the operating system or does python provide a mechanism for this?

Strictly OS -- Python provides no control.

2. I've read that python threads don't like to allow other threads to
run except in certain situations, these would be situations where there
is sleep or I/O happening, is this true? If so, what are the cases in
which a python thread would not give up the processor?

There is only a single thread of Python code running at any time.
Calling out to external libraries (e.g. C code) can release the Global
Interpreter Lock. Python's standard I/O routines do this automatically
for you, but you're free to write your own code that does this (e.g.
mxODBC).

3. Is there a way to which thread is running? I mean something a bit
more robust than a print statement inside the thread, I want to be able
to see when a context switch occurs, is this possible? The reason for
this is I will have threads deadlocked waiting for I/O and I am
interested to see how often context switching occurs.

You'd have to write some kind of logging. You can use the logging
module.

 

[Carl J. Van Arsdall]

These answers assume you're using CPython; Jython and IronPython have
different answers.

This confuses me. What is CPython versus Jython and IronPython? I'm
using compiled source from python.org, would this be CPython?

Strictly OS -- Python provides no control.

I did some research yesterday, and I'd like some clarification. So the
OS handles which thread runs and when, however if one of the python
processes currently holds the global interpreter lock, when the OS
switches to a python thread that does not have this lock, this thread
will do nothing. Does this sound right? Ultimately python does
control what thread runs by controlling the global interpreter lock
although its the underlying OS that handles all the context switching etc.

Because of this global interpreter lock does this mean its impossible to
get speed up with threading on multiple processor systems? I would
think so because only one python thread can execute at any one time. Is
there a way to get around this? This isn't something I need to do, I'm
just curious at this point.



-c


--

Carl J. Van Arsdall
(e-mail address removed)
Build and Release
MontaVista Software

 

[Lawrence Oluyede]

This confuses me. What is CPython versus Jython and IronPython? I'm
using compiled source from python.org, would this be CPython?

CPython is the official distribution, the one you find on python.org
It's C and Python based.

Jython is the Python implementation targetting the JVM

IronPython is the Python implementation targetting the .NET CLR

 

[Aahz]

[BTW, please follow standard Usenet convention and attribute the quotes;
I've added them back in for you]

Aahz:

I did some research yesterday, and I'd like some clarification. So the
OS handles which thread runs and when, however if one of the python
processes currently holds the global interpreter lock, when the OS
switches to a python thread that does not have this lock, this thread
will do nothing. Does this sound right? Ultimately python does
control what thread runs by controlling the global interpreter lock
although its the underlying OS that handles all the context switching
etc.

No. Python simply uses a standard OS thread lock. When a Python thread
gives up the GIL, the OS decides which thread acquires the lock -- it
could even be the same thread that released the lock.

Because of this global interpreter lock does this mean its impossible to
get speed up with threading on multiple processor systems? I would
think so because only one python thread can execute at any one time. Is
there a way to get around this? This isn't something I need to do, I'm
just curious at this point.

You either need to run multiple processes or run code that mostly calls
into C libraries that release the GIL. For example, a threaded spider
scales nicely on SMP.

 

[Peter Hansen]

Yes. Nearly as well as a single-threaded spider

I'm confused about how a single-threaded spider -- even using Twisted!
-- could "scale well" on a multiprocessor machine (that's what SMP
means, right?) without using multiple processes. I believe that's the
scenario to which Aahz was referring.

-Peter