Killing threads, and os.system()

[Laurent Claessens]

Hello all


I've a program that launches a lot of threads and each of them
launches a os.system("my_command").

My program also keeps a list of the launched threads, so I can make
"for" loops on the threads.


My aim is to kill everything with ctrl-C (KeyboardInterrupt).

Of course I tried to do

try:
[...]
except KeyboardInterrupt :
for task in task_list :
task.stop()
#task_list is the list of threads to be killed


It does not work.

How can I produce an "emergency" stop of all processes, including the
externals programs that were called by os.system() ?

My aim is of course to write an ultimate log file containing the status
of the program when KeyboardInterupt was raised. (if not, the unix
command "kill" does the job )


Thanks for any help
have a good day
Laurent

 

[Dennis Lee Bieber]

My aim is to kill everything with ctrl-C (KeyboardInterrupt).

Of course I tried to do

try:
[...]
except KeyboardInterrupt :
for task in task_list :
task.stop()
#task_list is the list of threads to be killed

When did .stop() become a method of threads? Or is it a method
created by you which is supposed to set some local data in the thread to
signal is should exit?

It does not work.

For years, the recommended coding style requires the thread to wait
on an event/condition or poll a local/global flag value; the thread is
then responsible for cleanly exiting /itself/.

Of course, if that thread is stuck waiting for a call to os.system()
to complete, then it can not do anything...

os.system() is a rather limited, restrictive, call -- best used for
quick one-of operations. If running Python 2.6+, I'd recommend
converting from os.system() to subprocess.Popen(). .Popen() objects now
have .terminate() and .kill() methods.

Heck, if the only reason you have been using threads is to permit
multiple parallel os.system() calls, you can probably drop the threads
entirely, and use a collection of subprocess.Popen() objects directly.
You will/should code something to handle stdin/stdout/stderr of those
objects -- unfortunately lowest-common-denominator IPC system does not
make it easy to do interactive operations ({muse: who do we have to kill
to persuade OS designers to incorporate something like the Amiga ARexx

How can I produce an "emergency" stop of all processes, including the
externals programs that were called by os.system() ?

As hinted above: Python 2.6 or newer, subprocess.Popen(), using
..terminate(), .kill(), or maybe .send_signal()

 

[Laurent Claessens]

Le 31/01/2012 17:04, Dennis Lee Bieber a écrit :

Of course, if that thread is stuck waiting for a call to os.system()
to complete, then it can not do anything...

os.system() is a rather limited, restrictive, call -- best used for
quick one-of operations. If running Python 2.6+, I'd recommend
converting from os.system() to subprocess.Popen(). .Popen() objects now
have .terminate() and .kill() methods.

Ok, I'll try that Popen. Indeed I think that my threads are stuck
waiting os.system to complete.

Thanks
Laurent

 

[Laurent Claessens]

Le 31/01/2012 17:04, Dennis Lee Bieber a écrit :

Of course, if that thread is stuck waiting for a call to os.system()
to complete, then it can not do anything...

os.system() is a rather limited, restrictive, call -- best used for
quick one-of operations. If running Python 2.6+, I'd recommend
converting from os.system() to subprocess.Popen(). .Popen() objects now
have .terminate() and .kill() methods.

Ok, I'll try that Popen. Indeed I think that my threads are stuck
waiting os.system to complete.

Thanks
Laurent

 

[Dennis Lee Bieber]

Ok, I'll try that Popen. Indeed I think that my threads are stuck
waiting os.system to complete.

Since the return value of os.system() is the /completion status/ of
the spawned command, that would seem to be a given.

 

[John Nagle]

({muse: who do we have to kill
to persuade OS designers to incorporate something like the Amiga ARexx
"rexxport" system<G>}).

QNX, which is a real-time microkernel which looks like POSIX to
applications. actually got interprocess communication right. It
has to; everything in QNX is done by interprocess communication,
including all I/O. File systems and drivers are ordinary programs.
The kernel just handles message passing, CPU dispatching, and timers.
QNX's message passing looks more like a subroutine call than an
I/O operation, and this has important implications for efficient CPU
dispatching.

Any QNX system call that can block is really a message pass. Message
passes can be given a timeout, and they can be canceled from another
thread. The "system call" then returns with an error status. This
provides a way to keep threads from getting "stuck" in a system call.

(Unfortunately, QNX, which survived as a separate company for decades,
sold out to Harmon (car audio) a few years ago. They had no clue
what to do with an OS. They sold it to Research In Motion, the
Blackberry company, which is in the process of tanking.)

Python's thread model is unusually dumb. You can't send signals
to other threads, you can't force an exception in another thread, and
I won't even get into the appalling mess around the Global Interpreter
Lock. This has forced the use of subprocesses where, in other
languages, you'd use threads. Of course, you load a new copy of the
interpreter in each thread, so this bloats memory usage.

John Nagle

 

[Steven D'Aprano]

I won't even get into the appalling mess around the Global Interpreter
Lock.

You know full well that IronPython and Jython don't have a GIL. If the
GIL was as harmful as you repeatedly tell us, why haven't you, and
everyone else, migrated to IronPython and Jython?

Oh yeah, maybe it's because CPython, even with the GIL, is significantly
faster than the two implementations without a GIL. Go figure.

But never mind the facts, spreading the FUD about Python is much more
fun. Hey, I hear that Python 3 is tanking too, and millions of Python
developers are rushing, rushing I say, to port their code to Go. Or was
it Ruby? Possibly Lua. Maybe VBScript.

This has forced the use of subprocesses where, in other
languages, you'd use threads.

Only if by "forced" you mean "not forced at all".

http://docs.python.org/library/multiprocessing.html
http://www.stackless.com/
http://pypi.python.org/pypi/greenlet/
http://twistedmatrix.com/trac/
http://www.gevent.org/
http://code.google.com/p/cogen/
http://www.kamaelia.org/Home
http://code.google.com/p/syncless/
http://opensource.hyves.org/concurrence/
http://www.tornadoweb.org/
http://docs.python.org/library/asyncore.html
http://pyro.sourceforge.net/
http://wiki.python.org/moin/ParallelProcessing

 

[Dennis Lee Bieber]

QNX, which is a real-time microkernel which looks like POSIX to
applications. actually got interprocess communication right. It
has to; everything in QNX is done by interprocess communication,
including all I/O. File systems and drivers are ordinary programs.
The kernel just handles message passing, CPU dispatching, and timers.
QNX's message passing looks more like a subroutine call than an
I/O operation, and this has important implications for efficient CPU
dispatching.

Sounds a lot like the Amiga kernal. Though, since message passing
was just a matter of adding/removing entries from a linked list, it
prevented easy porting to protected virtual memory spaces.

I/O was user code passing an I/O request to a file-handler, which
then determined which device driver was to receive the next lower level
operation... Eventual these messages passed back in the other direction
with the result data.

Rexxports were just regular Amiga message ports with some ARexx
overhead... From an ARexx script, sending to another program that
implemented a Rexxport was as simple as

address <targetport> command_and_data_parameters

The library included statements for ARexx scripts to create their
own ports, and to receive packets via them.

Unfortunately, the limitations of most OS's have led to
implementations like the Regina Rexx -- where the ADDRESS command is
basically a synonym for Python's subprocess.Popen() and the "targetport"
controls whether the command is executed directly or if a shell
processor is invoked to handle it.

 

[Paul Rubin]

QNX's message passing looks more like a subroutine call than an
I/O operation,

How do they enforce process isolation, or do they decide they don't need to?