Killing threads, and os.system()

Discussion in 'Python' started by Laurent Claessens, Jan 31, 2012.

  1. Hello all


    I've a program that launches a lot of threads and each of them
    launches a os.system("my_command").

    My program also keeps a list of the launched threads, so I can make
    "for" loops on the threads.


    My aim is to kill everything with ctrl-C (KeyboardInterrupt).

    Of course I tried to do

    try:
    [...]
    except KeyboardInterrupt :
    for task in task_list :
    task.stop()
    #task_list is the list of threads to be killed


    It does not work.

    How can I produce an "emergency" stop of all processes, including the
    externals programs that were called by os.system() ?

    My aim is of course to write an ultimate log file containing the status
    of the program when KeyboardInterupt was raised. (if not, the unix
    command "kill" does the job ;) )


    Thanks for any help
    have a good day
    Laurent
    Laurent Claessens, Jan 31, 2012
    #1
    1. Advertising

  2. On Tue, 31 Jan 2012 08:45:53 +0100, Laurent Claessens
    <> wrote:

    >
    >My aim is to kill everything with ctrl-C (KeyboardInterrupt).
    >
    >Of course I tried to do
    >
    >try:
    > [...]
    >except KeyboardInterrupt :
    > for task in task_list :
    > task.stop()
    > #task_list is the list of threads to be killed
    >

    When did .stop() become a method of threads? Or is it a method
    created by you which is supposed to set some local data in the thread to
    signal is should exit?

    >
    >It does not work.
    >

    For years, the recommended coding style requires the thread to wait
    on an event/condition or poll a local/global flag value; the thread is
    then responsible for cleanly exiting /itself/.

    Of course, if that thread is stuck waiting for a call to os.system()
    to complete, then it can not do anything...

    os.system() is a rather limited, restrictive, call -- best used for
    quick one-of operations. If running Python 2.6+, I'd recommend
    converting from os.system() to subprocess.Popen(). .Popen() objects now
    have .terminate() and .kill() methods.

    Heck, if the only reason you have been using threads is to permit
    multiple parallel os.system() calls, you can probably drop the threads
    entirely, and use a collection of subprocess.Popen() objects directly.
    You will/should code something to handle stdin/stdout/stderr of those
    objects -- unfortunately lowest-common-denominator IPC system does not
    make it easy to do interactive operations ({muse: who do we have to kill
    to persuade OS designers to incorporate something like the Amiga ARexx
    "rexxport" system <G>}).

    >How can I produce an "emergency" stop of all processes, including the
    >externals programs that were called by os.system() ?


    As hinted above: Python 2.6 or newer, subprocess.Popen(), using
    ..terminate(), .kill(), or maybe .send_signal()
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Jan 31, 2012
    #2
    1. Advertising

  3. Le 31/01/2012 17:04, Dennis Lee Bieber a écrit :

    > Of course, if that thread is stuck waiting for a call to os.system()
    > to complete, then it can not do anything...
    >
    > os.system() is a rather limited, restrictive, call -- best used for
    > quick one-of operations. If running Python 2.6+, I'd recommend
    > converting from os.system() to subprocess.Popen(). .Popen() objects now
    > have .terminate() and .kill() methods.


    Ok, I'll try that Popen. Indeed I think that my threads are stuck
    waiting os.system to complete.

    Thanks
    Laurent
    Laurent Claessens, Jan 31, 2012
    #3
  4. Le 31/01/2012 17:04, Dennis Lee Bieber a écrit :

    > Of course, if that thread is stuck waiting for a call to os.system()
    > to complete, then it can not do anything...
    >
    > os.system() is a rather limited, restrictive, call -- best used for
    > quick one-of operations. If running Python 2.6+, I'd recommend
    > converting from os.system() to subprocess.Popen(). .Popen() objects now
    > have .terminate() and .kill() methods.


    Ok, I'll try that Popen. Indeed I think that my threads are stuck
    waiting os.system to complete.

    Thanks
    Laurent
    Laurent Claessens, Jan 31, 2012
    #4
  5. On Tue, 31 Jan 2012 18:18:28 +0100, Laurent Claessens
    <> wrote:

    >
    >Ok, I'll try that Popen. Indeed I think that my threads are stuck
    >waiting os.system to complete.
    >

    Since the return value of os.system() is the /completion status/ of
    the spawned command, that would seem to be a given.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Jan 31, 2012
    #5
  6. Laurent Claessens

    John Nagle Guest

    On 1/31/2012 8:04 AM, Dennis Lee Bieber wrote:
    > ({muse: who do we have to kill
    > to persuade OS designers to incorporate something like the Amiga ARexx
    > "rexxport" system<G>}).


    QNX, which is a real-time microkernel which looks like POSIX to
    applications. actually got interprocess communication right. It
    has to; everything in QNX is done by interprocess communication,
    including all I/O. File systems and drivers are ordinary programs.
    The kernel just handles message passing, CPU dispatching, and timers.
    QNX's message passing looks more like a subroutine call than an
    I/O operation, and this has important implications for efficient CPU
    dispatching.

    Any QNX system call that can block is really a message pass. Message
    passes can be given a timeout, and they can be canceled from another
    thread. The "system call" then returns with an error status. This
    provides a way to keep threads from getting "stuck" in a system call.

    (Unfortunately, QNX, which survived as a separate company for decades,
    sold out to Harmon (car audio) a few years ago. They had no clue
    what to do with an OS. They sold it to Research In Motion, the
    Blackberry company, which is in the process of tanking.)

    Python's thread model is unusually dumb. You can't send signals
    to other threads, you can't force an exception in another thread, and
    I won't even get into the appalling mess around the Global Interpreter
    Lock. This has forced the use of subprocesses where, in other
    languages, you'd use threads. Of course, you load a new copy of the
    interpreter in each thread, so this bloats memory usage.

    John Nagle
    John Nagle, Feb 3, 2012
    #6
  7. On Fri, 03 Feb 2012 00:14:33 -0800, John Nagle wrote:

    > I won't even get into the appalling mess around the Global Interpreter
    > Lock.


    You know full well that IronPython and Jython don't have a GIL. If the
    GIL was as harmful as you repeatedly tell us, why haven't you, and
    everyone else, migrated to IronPython and Jython?

    Oh yeah, maybe it's because CPython, even with the GIL, is significantly
    faster than the two implementations without a GIL. Go figure.

    But never mind the facts, spreading the FUD about Python is much more
    fun. Hey, I hear that Python 3 is tanking too, and millions of Python
    developers are rushing, rushing I say, to port their code to Go. Or was
    it Ruby? Possibly Lua. Maybe VBScript.


    > This has forced the use of subprocesses where, in other
    > languages, you'd use threads.


    Only if by "forced" you mean "not forced at all".

    http://docs.python.org/library/multiprocessing.html
    http://www.stackless.com/
    http://pypi.python.org/pypi/greenlet/
    http://twistedmatrix.com/trac/
    http://www.gevent.org/
    http://code.google.com/p/cogen/
    http://www.kamaelia.org/Home
    http://code.google.com/p/syncless/
    http://opensource.hyves.org/concurrence/
    http://www.tornadoweb.org/
    http://docs.python.org/library/asyncore.html
    http://pyro.sourceforge.net/
    http://wiki.python.org/moin/ParallelProcessing


    --
    Steven
    Steven D'Aprano, Feb 3, 2012
    #7
  8. On Fri, 03 Feb 2012 00:14:33 -0800, John Nagle <>
    wrote:

    >On 1/31/2012 8:04 AM, Dennis Lee Bieber wrote:
    >> ({muse: who do we have to kill
    >> to persuade OS designers to incorporate something like the Amiga ARexx
    >> "rexxport" system<G>}).

    >
    > QNX, which is a real-time microkernel which looks like POSIX to
    >applications. actually got interprocess communication right. It
    >has to; everything in QNX is done by interprocess communication,
    >including all I/O. File systems and drivers are ordinary programs.
    >The kernel just handles message passing, CPU dispatching, and timers.
    >QNX's message passing looks more like a subroutine call than an
    >I/O operation, and this has important implications for efficient CPU
    >dispatching.
    >

    Sounds a lot like the Amiga kernal. Though, since message passing
    was just a matter of adding/removing entries from a linked list, it
    prevented easy porting to protected virtual memory spaces.

    I/O was user code passing an I/O request to a file-handler, which
    then determined which device driver was to receive the next lower level
    operation... Eventual these messages passed back in the other direction
    with the result data.

    Rexxports were just regular Amiga message ports with some ARexx
    overhead... From an ARexx script, sending to another program that
    implemented a Rexxport was as simple as

    address <targetport> command_and_data_parameters

    The library included statements for ARexx scripts to create their
    own ports, and to receive packets via them.

    Unfortunately, the limitations of most OS's have led to
    implementations like the Regina Rexx -- where the ADDRESS command is
    basically a synonym for Python's subprocess.Popen() and the "targetport"
    controls whether the command is executed directly or if a shell
    processor is invoked to handle it.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Feb 3, 2012
    #8
  9. Laurent Claessens

    Paul Rubin Guest

    John Nagle <> writes:
    > QNX's message passing looks more like a subroutine call than an
    > I/O operation,


    How do they enforce process isolation, or do they decide they don't need to?
    Paul Rubin, Feb 3, 2012
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. cppaddict
    Replies:
    20
    Views:
    806
    Chris Uppal
    Aug 11, 2004
  2. Don Garrett

    Killing threads during development.

    Don Garrett, Jun 5, 2005, in forum: Python
    Replies:
    0
    Views:
    291
    Don Garrett
    Jun 5, 2005
  3. jimzat

    Killing subservient threads

    jimzat, Feb 20, 2009, in forum: Python
    Replies:
    6
    Views:
    316
  4. Killing threads

    , Apr 5, 2009, in forum: Python
    Replies:
    13
    Views:
    526
  5. Chris Angelico
    Replies:
    2
    Views:
    268
    Antoon Pardon
    Sep 21, 2011
Loading...

Share This Page