Protocol for thread communication

[Michael Torrie]

Does anyone have any recommended ideas/ways of implementing a proper
control and status protocol for communicating with threads? I have a
program that spawns a few worker threads, and I'd like a good, clean way
of communicating the status of these threads back to the main thread.
Each thread (wrapped in a very simple class) has only a few states, and
progress levels in those states. And sometimes they can error out,
although if the main thread knew about it, it could ask the thread to
retry (start over). How would any of you do this? A callback method
that the thread can call (synchronizing one-way variables isn't a
problem)? A queue? How would the main thread check these things?
Currently the main thread is polling some simple status variables. This
works, and polling will likely continue to be the simplest and easiest
way, but simple status variables are very limited. Are there any
pythonic patterns people have developed for this.

thanks.

Michael

 

[Samuel]

Does anyone have any recommended ideas/ways of implementing a proper
control and status protocol for communicating with threads? I have a
program that spawns a few worker threads, and I'd like a good, clean way
of communicating the status of these threads back to the main thread.
Each thread (wrapped in a very simple class) has only a few states, and
progress levels in those states. And sometimes they can error out,
although if the main thread knew about it, it could ask the thread to
retry (start over). How would any of you do this? A callback method
that the thread can call (synchronizing one-way variables isn't a
problem)? A queue? How would the main thread check these things?

I implemented a generic library that could easily support these things:

http://code.google.com/p/exscript/source/browse/trunk/lib/WorkQueue/

Basically, it's an asynchronous queue into which the jobs are added. The
"MainLoop" class manages jobs (like, making sure that n jobs are running
at the same time, and providing signals when jobs are done). One way to
add a status to each job is by adding a status attribute to the Job
class, in addition, a Job may emit a signal (using a simple signal/event
mechanism such as this one:

http://code.google.com/p/exscript/source/browse/trunk/lib/Exscript/
Trackable.py

) whenever a status changes, and have the MainLoop class fetch that.

-Samuel

 

[bockman]

Does anyone have any recommended ideas/ways of implementing a proper
control and status protocol for communicating with threads?  I have a
program that spawns a few worker threads, and I'd like a good, clean way
of communicating the status of these threads back to the main thread.
Each thread (wrapped in a very simple class) has only a few states, and
progress levels in those states.  And sometimes they can error out,
although if the main thread knew about it, it could ask the thread to
retry (start over).  How would any of you do this?  A callback method
that the thread can call (synchronizing one-way variables isn't a
problem)?  A queue?  How would the main thread check these things?
Currently the main thread is polling some simple status variables.  This
works, and polling will likely continue to be the simplest and easiest
way, but simple status variables are very limited.  Are there any
pythonic patterns people have developed for this.

thanks.

Michael

I've found that Queue.Queue objects are the easiest way to communicate
between threads in Python.
So, I'd suggets that you attach a Queue to each thread: the main
thread will use its queue to receive
the status messages from the other threads; the other threads will use
their queues to receive the retry command (or any other command that
may be needed) from the main thread.

Ciao

 

[castironpi]

Does anyone have any recommended ideas/ways of implementing a proper
control and status protocol for communicating with threads?  I have a
program that spawns a few worker threads, and I'd like a good, clean way
of communicating the status of these threads back to the main thread.
Each thread (wrapped in a very simple class) has only a few states, and
progress levels in those states.  And sometimes they can error out,
although if the main thread knew about it, it could ask the thread to
retry (start over).  How would any of you do this?  A callback method
that the thread can call (synchronizing one-way variables isn't a
problem)?  A queue?  How would the main thread check these things?
Currently the main thread is polling some simple status variables.  This
works, and polling will likely continue to be the simplest and easiest
way, but simple status variables are very limited.  Are there any
pythonic patterns people have developed for this.

thanks.

Michael

It depends on the messages. In the general form, 'do X', there are a
lot of specifics relevant.

- Do X at time T

- for relative time T
- for absolute time T
- for conditional time T

While the wording is vague, it illustrates a contrast.

- Do X at time T

- relative T -- T is a time offset into the future
- absolute time T -- T is an absolute time, independent of the rest of
the program
- conditional time T -- under certain conditions, such as, right after
X' finishes, right before X' starts, if Y happens during X

In the big picture, the choice of X is irrelevant. However, in a
particular PL, in a particular data, encapsulation, or abstraction
model, some combinations of ( T, X ) may be impossible, or even long-
winded, ugly, or wordy.

For the idealists, an pure imperative model may fit best, so let X =
'set a bit', and conflict never arises. However such one restricts
the universal turing machine-- can only write one value, which may or
may not forbid solution of some problems.

But without loss of generality, for X in { set B, reset B }, you have
a conflict when for OPS= { (T1, X1), (T1, X2) }, X1= set B, and X2=
reset B. Then there exists an ( X, T ) such that the semantics of the
program are undefined.

 

[castironpi]

Does anyone have any recommended ideas/ways of implementing a proper
There is the PyThreadState_SetAsyncExc API. "To prevent naive misuse,
you must write your own C extension to call this."