SMP, GIL and Threads

[catsup]

Hi,

I have an app written under version Python 2.3.5. The problem I'm
having is that it hangs on one of its threads. The thread that hangs
does updates to a standard dictionary shared with another thread that
only reads this dictionary. This app works beautifully on a single
processor boxes in my testing environment, but this problem quickly
occurs when the software runs on a dual cpu SMP blade server with
hyperthreading turned off, Windows 2003 server.

I do not bother using application level locks because I figure the GIL
is going to do the job for me, and because with one thread doing
updates and the other only reading, things remain consistent logically
for the app. The app will not have a problem if the dictionary changes
just before it does a read.

I have searched this group on this subject and seen one warning against
sharing objects between threads. I don't recall every writing a
threaded app that didn't share data between threads in some way. I've
also seen a recomendation in this list against using threads at all
with Python. I'm hoping that is an extreme view and not general wisdom
here. Python has never failed me when analysis indicated that it would
be the correct tool for the job.

Thank you for your time and attention.

Randy

 

[Steve Holden]

Hi,

I have an app written under version Python 2.3.5. The problem I'm
having is that it hangs on one of its threads. The thread that hangs
does updates to a standard dictionary shared with another thread that
only reads this dictionary. This app works beautifully on a single
processor boxes in my testing environment, but this problem quickly
occurs when the software runs on a dual cpu SMP blade server with
hyperthreading turned off, Windows 2003 server.

I don't see why you should get problems on SMP hardware, since the
threads are all part of the same process and should therefore (I'd have
thought) be tagged with the same processor affinity. Hence the GIL
should manage contention successfully.

I do not bother using application level locks because I figure the GIL
is going to do the job for me, and because with one thread doing
updates and the other only reading, things remain consistent logically
for the app. The app will not have a problem if the dictionary changes
just before it does a read.

I believe dictionary access is an atomic operation wrt thread switches
anyway, so I'm (again) not sure why a thread is hanging.

I have searched this group on this subject and seen one warning against
sharing objects between threads. I don't recall every writing a
threaded app that didn't share data between threads in some way. I've
also seen a recomendation in this list against using threads at all
with Python. I'm hoping that is an extreme view and not general wisdom
here. Python has never failed me when analysis indicated that it would
be the correct tool for the job.

Threads most often use Queue.Queue to communicate, precisely because its
operations are guaranteed thread-safe. Could we see some code, or do we
have to guess what the problem might be ;-) (I appreciate the code may
be too large or proprietary, but in that case could you build a test to
demonstrate the problem?).

I wouldn't worry about the "don't use threads" camp. Although there is a
lot to be said for non-blocking asynchronous operations managed by a
single thread this is by no means the only way to do things.

regards
Steve

 

[Thomas Heller]

I don't see why you should get problems on SMP hardware, since the
threads are all part of the same process and should therefore (I'd
have thought) be tagged with the same processor affinity. Hence the
GIL should manage contention successfully.

I believe dictionary access is an atomic operation wrt thread
switches anyway, so I'm (again) not sure why a thread is hanging.

Well, there's at least the 'do not mutate a dictionary (or list) while
iterating over it' advice. Having one thread writing and another
reading makes it IMO impossible to avoid the conflict.

Thomas

 

[Donn Cave]

I don't see why you should get problems on SMP hardware, since the
threads are all part of the same process and should therefore (I'd have
thought) be tagged with the same processor affinity. Hence the GIL
should manage contention successfully.

Could you explain your thinking there? I don't know that much
about SMP. Would have thought that affinity might make a difference
with on-processor cache and so forth, but would be transparent to
applications. Are you thinking that the processor affinity would
essentially serialize execution, so SMP hardware doesn't matter
because your threads won't execute concurrently anyway?

Threads most often use Queue.Queue to communicate, precisely because its
operations are guaranteed thread-safe.

(Just thought that might bear repetition.)

Donn Cave, (e-mail address removed)

 

[Greg Copeland]

In situations like this, you need to guard the resource with a mutex.
In Python, things like insertions are atomic but iterations are not.
Thusly, if you wrap it with a mutex, things can be made safe. I saw,
"can be", because you then have to ensure you always use the mutex to
satify your concurrent access requirements.

Greg

 

[catsup]

Yes. Iterating over a list that you concurrently update will
definately cause problems. That is not the type of "read" I am doing
in the application. My read is one key/value translation. Below is an
example of the operations I am performing to help clarify.

The update thread, running once every fifteen minutes, gathers updates
for the dictionary and applies them:

def run_thread(self):
while( not self.terminate ):
sleep(900)
for agentRec in self.agentListUpdates:
agentInfo = agentRec[1]
if agentRec[0] == 'a':
self.agentByIVRSeq[agentInfo.IVRSeq] = agentInfo #->
thread hangs here

The accessing thread takes command requests off a queue, every
half-second, placed there by an altogether different thread, and does a
lookup on this same dictionary before performing this command:

def run_thread(self):
while( not self.terminate ):
cmd = self.cmdQueue.get(False)
agentInfo = self.agentByIVRSeq[cmd[0]]
self.performCmd(cmd,agentInfo)
sleep(.5)

 

[Peter Hansen]

The accessing thread takes command requests off a queue, every
half-second, placed there by an altogether different thread, and does a
lookup on this same dictionary before performing this command:

def run_thread(self):
while( not self.terminate ):
cmd = self.cmdQueue.get(False)
agentInfo = self.agentByIVRSeq[cmd[0]]
self.performCmd(cmd,agentInfo)
sleep(.5)

You can guarantee that there will always be exactly one item available
in the Queue for this thread to pull out? Or is this a heavily edited
example, with the required "try/except Queue.Empty" handling removed for
tender minds?

(Note the "exactly one", too, since having either fewer or more will be
a problem for the above code.)

-Peter

 

[catsup]

Yes. The test for Empty was edited out. There is a great deal going
on in this application, most not germane to the problem. I should
perhaps let all of you be the judge of that. Hopefully, this will be
enough to help generate ideas.

Thanks,
Randy

 

[Steve Holden]

Could you explain your thinking there? I don't know that much
about SMP. Would have thought that affinity might make a difference
with on-processor cache and so forth, but would be transparent to
applications. Are you thinking that the processor affinity would
essentially serialize execution, so SMP hardware doesn't matter
because your threads won't execute concurrently anyway?

I was just trying to underline that the separate threads won't run
concurrently, I suppose, and choosing a bad way to do it (since the
affinity need not be set and the process can freely migrate between
processors).

(Just thought that might bear repetition.)

Yes!

regards
Steve

 

[Aahz]

I have an app written under version Python 2.3.5. The problem I'm
having is that it hangs on one of its threads. The thread that hangs
does updates to a standard dictionary shared with another thread that
only reads this dictionary. This app works beautifully on a single
processor boxes in my testing environment, but this problem quickly
occurs when the software runs on a dual cpu SMP blade server with
hyperthreading turned off, Windows 2003 server.

Hrm. This is either a Python bug or a bug in your application; I'd bet
on the latter, but the former is possible. (That is, there have been
bugs in previous versions of Python that were at least theoretically
stimulatable on single-CPU machines but in practice only showed up with
SMP machines.)

To find out which this is, you need to provide a reasonably small chunk
of code that demonstrates the problem. Until you prove otherwise, we'll
have to assume that it's a bug in your code, because many other people
are running threaded applications just fine. You might check the dev
logs and see if any thread bugs were fixed for 2.4 (or just try using
2.4 to see whether that helps).

I do not bother using application level locks because I figure the GIL
is going to do the job for me, and because with one thread doing
updates and the other only reading, things remain consistent logically
for the app. The app will not have a problem if the dictionary changes
just before it does a read.

This is usually safe.

I have searched this group on this subject and seen one warning against
sharing objects between threads. I don't recall every writing a
threaded app that didn't share data between threads in some way. I've
also seen a recomendation in this list against using threads at all
with Python. I'm hoping that is an extreme view and not general wisdom
here. Python has never failed me when analysis indicated that it would
be the correct tool for the job.

What the warning actually means is that the only shared object between
threads should be a Queue; you pass other objects between threads using
the Queue, so that only one thread at a time uses the non-Queue objects.

 

[catsup]

It was actually quite a minor adjustment in the application to follow
the oft repeated advice here on this subject to share only the Queue
object between threads. Making the update of the dictionary just
another queued command request caused both the dictionary read and
write to be performed by the same thread. This seems to have
eliminated the problem.

Thank you all for your thoughtful comments.

Randy

 

[Aahz]

It was actually quite a minor adjustment in the application to follow
the oft repeated advice here on this subject to share only the
Queue object between threads. Making the update of the dictionary
just another queued command request caused both the dictionary read
and write to be performed by the same thread. This seems to have
eliminated the problem.

Great!