How do i : Python Threads + KeyboardInterrupt exception

[Brendon Costa]

Hi all,

I have a small python project i am working on. Basically i always have
two threads. A "Read" thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a "Proc" thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.

What i want to know is: "What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?"

Some sample code is shown below. This code works as is, except in the
case where the "Proc" thread wants to initiate the exit of the
application.

For example:
* running the code below and pressing Ctrl + C works fine as the Read
thread is initiating the shutdown.
* running the code below and entering:
pquit
some other data
<Ctrl + D>

will cause oytput:

Processing: pquit
Proc: Initiating quit

and then it HANGS waiting for the Read thread to exit.

Some questions i have that are:
* KeyboardInterrupt exception seems to only be recieved by the main
thread. Is this ALWAYS the case across all UNIX + windows platforms
(not so worried about others)?
* Can i somehow get the Proc thread to force the Read thread to
generate a KeyboardInterrupt or somehow exit its blocking "for line in
fin:" call?


Thanks,
Brendon


--- SNIP ---
# Two or more threads
#
# proc : Is a processing thread that basically reads events from a
event queue and processes them
# read : Is a thread reading in a loop from stdin and generating
events for "proc"
# * : Other additional threads that may asynchronously add events to
the queue to be processed

import Queue
import threading
import sys

def Read(queue, fin, fout):
ret = (1, 'Normal Exit')
try:
for line in fin:
queue.put((0, line))
#raise Exception("Blah")
#raise "Blah"
except KeyboardInterrupt: ret = (1, 'KeyboardInterrupt')
except Exception, e: ret = (1, 'ERROR: ' + str(e))
except: ret = (1, 'UNKNOWN-ERROR')

# Notify Proc thread that we are exiting.
queue.put(ret)
print >>fout, 'Read: Initiating quit'


def Proc(queue, fout, ignore):
quit = False
while not quit:
(evt_type, evt_data) = queue.get()

if evt_type == 0:
print >>fout, 'Processing: ' + str(evt_data)
if evt_data.startswith('pquit'):
print >>fout, 'Proc: Initiating quit'
quit = True

elif evt_type == 1:
print >>fout, 'Quit: ' + str(evt_data)
quit = True

class MyThread(threading.Thread):
def __init__(self, func, queue, file1, file2, *args, **kwds):
threading.Thread.__init__(self, *args, **kwds)
self.func = func
self.queue = queue
self.file1 = file1
self.file2 = file2
self.start()

def run(self):
return self.func(self.queue, self.file1, self.file2)


if __name__ == '__main__':
queue = Queue.Queue()

# Read thread is the main thread and seems to get the
KeyboardInterrupt exception.
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)

# Read thread is NOT the main thread and never seems to get the
KeyboardInterrupt exception.
# This doesnt work for that reason.
#t = MyThread(Read, queue, sys.stdin, sys.stderr)
#Proc(queue, sys.stderr, None)


# @@@Brendon How do we notify the Read thread that they should
exit?
# If the Read thread initiated the quit then all is fine.
# If the Proc thread initiated the quit then i need to get the
Read
# thread to exit too somehow. But it is currently blocking in a
read
# on an input file.
print >>sys.stderr, 'Joining thread.'
t.join()

 

[Gabriel Genellina]

I have a small python project i am working on. Basically i always have
two threads. A "Read" thread that sits in a loop reading a line at a
time from some input (Usually stdin) and then generating events to be
processed and a "Proc" thread that processes incoming events from a
queue. There will be additional threads as well that asynchronously
insert events into the queue to be processed, but they are not a part
of this so i have omitted them.

What i want to know is: "What is the standard/best way of implementing
such a pattern that works in the presence of errors and particularly
with the KeyboardInterrupt exception?"

I don't know the "standard" way, but perhaps you can get some ideas from
this recent thread:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/82636f1bdd1d2d83/

Some sample code is shown below. This code works as is, except in the
case where the "Proc" thread wants to initiate the exit of the
application.

You might try using the PyThreadState_SetAsyncExc function (from the
Python C API) to inject a KeyboardInterrupt exception into the Read thread
- but I don't know if it will work at all, the execution might be blocked
waiting for an I/O call to complete, and never return to Python code...

 

[Brendon Costa]

I don't know the "standard" way, but perhaps you can get some ideas from

this recent thread:http://groups.google.com/group/comp.lang.python/browse_thread/thread/...

I had a quick read through that thread. I think i will need some more
time to think about what they are saying in there though. They seem to
talking about killing a python thread kind of similar to the C
functions TerminateThread() in windows or pthread_cancel() on UNIX
which are not suitable for my purpose.

You might try using the PyThreadState_SetAsyncExc function (from the
Python C API) to inject a KeyboardInterrupt exception into the Read thread
- but I don't know if it will work at all, the execution might be blocked
waiting for an I/O call to complete, and never return to Python code...

Unfortunately that is the problem. It is blocking in a IO system call
and i want to force it to exit that with an error, hopefully causing a
Python exception. I looked at what you mentioned and it is described a
bit here too: http://sebulba.wikispaces.com/recipe+thread2

I really need a python mechanism for interrupting blocking system
calls. have dealt with this sort of thing before in C/C++ on windows
and UNIX. For UNIX it is really a matter of sending a signal to the
process (SIGINTR), the main thread which is the only one in Python to
accept signals (others as i understand are masked) will get the signal
and and return with an EINTR breaking out of the blocking read
hopefully a python exception of Interrupted IO or KeyboardInterrupt.
Note that this only works for the one thread , but that is all i need.

For windows, it is possible to do a similar thing using:
GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, 0) with which behaves a bit
like a UNIX signal and i assume is what causes the KeyboardInterrupt
in the first place.

The problem i have is how do I achieve this in python?

 

[Brendon Costa]

I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.

Is there some other method rather than a blocking "for line in fin:"
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?

Sorry for all the posts. I am giving updates as i look further into
the problem.

 

[Brendon Costa]

I tested this a bit more. My windows example was incorrect. It should
have used CTRL_C_EVENT. But even then, the problem is that the process
will also close the console window from which it was called because of
the 0. Also this requires that the process actually have a console and
is not a GUI application.

Is there some other method rather than a blocking "for line in fin:"
that i can use for reading from a file like device that plays nicely
with other threads asynchronously waking it up?

Sorry for all the posts. I am giving updates as i look further into
the problem.

 

[MRAB]

        There is no "safe" means of forcing a Python thread to exit... Heck,
I believe the MSDN is full of warnings that killing a thread at the
Windows OS level is fraught with danger.

        Unfortunately, Windows doesn't allow for use of select() on anything
other than sockets -- otherwise you could code a wait that included a
"die" object and "write" to that object to break the blocking wait.

        If your blocking call has no time-out variant, you may be stuck...

If only the main thread can receive KeyboardInterrupt, is there any
reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.

You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.

 

[Brendon Costa]

If only the main thread can receive KeyboardInterrupt, is there any

reason why you couldn't move the functionality of the Read thread into
the main thread? It looks like it's not doing any work, just waiting
for the Proc thread to finish.

You could start the Proc thread, do the current Read thread
functionality until the interrupt occurs, put the apporpriate message
in the queue, and then wait for the Proc thread to finish.

It is already doing that. You will notice that the Proc() function is
called by a threading.Thread instance so Proc() is running in a
thread, but the Read() function is being called by the main thread
right after this. It DOES work with the Ctrl + C, but i can find no
way at all of closing down the script from within the Proc() thread.

The relevant bit of code is:
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)

In the end, the problem is that i am trying to multiplex IO and other
operations. In UNIX i would use select with the input file descriptor
and an anonymous pipe for the extra commands to achieve this without
any threads, but this script needs to run primarily on a windows box
and i would like to use it on UNIX too. I thought i could use threads
to achieve the IO Multiplexing in python, but it seems not or at least
not simply.

How do people usually manage IO multiplexing (not just using sockets)
cross platform in python?

I only need to multiplex two sources really:
* Input file or stdin
* A input event queue
This will have messages injected from various locations: timers,
the processing thread itself, and possibly from a single GUI thread at
a later point in time.

Though i can foresee that at some point in the future i may also need
to multiplex those two above and some sockets (For a server with a few
clients).

I was thinking of looking at some asynchronous IO libraries for python
on Windows + UNIX, any suggestions (Must include more than just
sockets)?

 

[Rhamphoryncus]

It is already doing that. You will notice that the Proc() function is
called by a threading.Thread instance so Proc() is running in a
thread, but the Read() function is being called by the main thread
right after this. It DOES work with the Ctrl + C, but i can find no
way at all of closing down the script from within the Proc() thread.

The relevant bit of code is:
t = MyThread(Proc, queue, sys.stderr, None)
Read(queue, sys.stdin, sys.stderr)

In the end, the problem is that i am trying to multiplex IO and other
operations. In UNIX i would use select with the input file descriptor
and an anonymous pipe for the extra commands to achieve this without
any threads, but this script needs to run primarily on a windows box
and i would like to use it on UNIX too. I thought i could use threads
to achieve the IO Multiplexing in python, but it seems not or at least
not simply.

How do people usually manage IO multiplexing (not just using sockets)
cross platform in python?

I only need to multiplex two sources really:
* Input file or stdin
* A input event queue
   This will have messages injected from various locations: timers,
the processing thread itself, and possibly from a single GUI thread at
a later point in time.

Though i can foresee that at some point in the future i may also need
to multiplex those two above and some sockets (For a server with a few
clients).

I was thinking of looking at some asynchronous IO libraries for python
on Windows + UNIX, any suggestions (Must include more than just
sockets)?

They either use an event-driven library.. or they use a timeout of
around 1 second. 1 second will definitely waste power on laptops (and
desktops), but it *works*.

python-safethread has this fixed - any lowlevel trickery needed is
done for you - but it's not ported to windows yet.