KeyboardInterrupt and threading

Discussion in 'Python' started by Ivan Nestlerode, Jan 2, 2004.

  1. Hello comp.lang.python,

    I am attempting to write a threaded program right now in Python
    (version 2.3 on Debian unstable) and the current behavior of
    KeyboardInterrupt is causing me grief. From the documentation of the
    thread module in Python 2.3.3:

    "Threads interact strangely with interrupts: the KeyboardInterrupt
    exception will be received by an arbitrary thread. (When the signal
    module is available, interrupts always go to the main thread.)"

    So the current behavior is that some "arbitrarily chosen" thread
    receives the KeyboardInterrupt exception when the user hits Ctrl-C. My
    question is whether there is some way to always make the main thread
    receive the KeyboardInterrupt. The snippet at the end of the
    documentation is vague/confusing. It sounds like KeyboardInterrupt
    would always be delivered to the main thread, but I have tested this
    and it is definitely not behaving that way. Maybe it just means that
    the Unix signal is received by the main thread's signal handler (and
    at that point an arbitrary thread is picked for KeyboardInterrupt). I
    don't know.

    The reason I cannot use the current "arbitrary thread"
    KeyboardInterrupt delivery model:

    The current model seems to deliver the KeyboardInterrupt to the
    busiest thread (i.e. some sub-thread doing lots of computation, as
    opposed to my main thread that is sitting in a sleep() loop). This
    causes problems because the busy thread that I spawn runs library code
    that was not written by me. That library code may contain lines like
    "except:" that mask the KeyboardInterrupt so that the user experience
    is that the app seems to ignore Ctrl-C.

    If I could guarantee delivery of KeyboardInterrupt to the main thread,
    I could control things much more easily/consistently. With the current
    semantics code reuse becomes very difficult (I cannot use any code
    that is "sloppy" with its exception catching or Ctrl-C will be
    ignored).

    Any ideas (besides rewriting the sloppy library code) would be much
    appreciated.

    Thanks,
    -Ivan
     
    Ivan Nestlerode, Jan 2, 2004
    #1
    1. Advertising

  2. Ivan Nestlerode

    Paul McGuire Guest

    "Ivan Nestlerode" <> wrote in message
    news:...
    > Hello comp.lang.python,
    >
    > I am attempting to write a threaded program right now in Python
    > (version 2.3 on Debian unstable) and the current behavior of
    > KeyboardInterrupt is causing me grief. From the documentation of the
    > thread module in Python 2.3.3:
    >
    > "Threads interact strangely with interrupts: the KeyboardInterrupt
    > exception will be received by an arbitrary thread. (When the signal
    > module is available, interrupts always go to the main thread.)"
    >

    Is the signal module not available to you?
    <snip>
    > Any ideas (besides rewriting the sloppy library code) would be much
    > appreciated.
    >
    > Thanks,
    > -Ivan


    Assuming you have access to the signal module:
    1. Create a boolean that you can periodically test for in your thread code,
    to see if you should keep processing. This may be a global singleton, a
    global boolean, or something, but it needs to be visible to both the main
    thread and the threaded code. You probably have some compute intensive
    loops such as:
    while( prime_factors < 1e15 ):
    or
    while( optimization != converged ):
    or even just
    while( True ):

    Change these to reference the "keep processing" flag.
    while( keep_processing and prime_factors < 1e15 ):
    or
    while( keep_processing and optimization != converged ):
    or even just
    while( keep_processing ): # no need to 'and' with True

    2. Create a signal handler to trap for SIGINT. In the signal handler, set
    the "keep-processing" bit to False:

    import signal

    def discontinue_processing(signl, frme):
    global keep_processing
    keep_processing = False
    return 0

    signal.signal( signal.SIGINT, discontinue_processing )


    Now your main thread will get the ^C interrupt and clear the
    "keep_processing" flag, which will eventually get picked up by your worker
    thread the next time it tests for it.

    HTH,
    -- Paul
     
    Paul McGuire, Jan 3, 2004
    #2
    1. Advertising

  3. (Ivan Nestlerode) writes:

    > Hello comp.lang.python,
    >
    > I am attempting to write a threaded program right now in Python
    > (version 2.3 on Debian unstable) and the current behavior of
    > KeyboardInterrupt is causing me grief. From the documentation of the
    > thread module in Python 2.3.3:
    >
    > "Threads interact strangely with interrupts: the KeyboardInterrupt
    > exception will be received by an arbitrary thread. (When the signal
    > module is available, interrupts always go to the main thread.)"


    On debian, the signal module certainly should be available, and the
    KeyboardInterrupt exception should go to the main thread. Or at
    least, that's what I thought. You're definitely seeing it being
    delivered to an arbitrary thread?

    Cheers,
    mwh

    --
    The Programmer's Quick Guide To Python (Time Machine version):
    You try to shoot yourself in the foot, only to realize that
    there's no need, since Guido thoughtfully shot you in the foot
    years ago. -- Nick Mathewson, comp.lang.python
     
    Michael Hudson, Jan 5, 2004
    #3
  4. Michael Hudson <> wrote in message news:<>...
    > On debian, the signal module certainly should be available, and the
    > KeyboardInterrupt exception should go to the main thread. Or at
    > least, that's what I thought. You're definitely seeing it being
    > delivered to an arbitrary thread?
    >
    > Cheers,
    > mwh


    After a closer look, I am not seeing KeyboardInterrupt being delivered
    to non-main threads. What I am seeing is that Ctrl-C does not work at
    all when non-main threads are CPU intensive. I had previously jumped
    to the conclusion that because Ctrl-C didn't work and because the
    non-main thread was sloppy ("except:"), that Ctrl-C wasn't working
    because KeyboardInterrupt was going to the sub-thread. That was an
    incorrect conclusion.

    So I guess the entire problem can be restated as:
    How do I make Ctrl-C work properly when the non-main thread is busier
    than the main thread?

    The short program at the end of this post demonstrates the problem
    with Ctrl-C and busy non-main threads. The idea behind this script is
    that the main thread launches a busy sub-thread. Ctrl-C is supposed to
    be caught in the main thread, stopping both threads, and ending the
    program (with some printouts). As written, the program does not
    respond to Ctrl-C even if I hold it down continuously. If you
    uncomment the print statement in the main thread, all of the sudden it
    starts to work. Why?

    I think it has something to do with the main thread needing some
    larger amount of CPU to detect the signal and the KeyboardInterrupt. I
    think this is a bug though (I shouldn't have to put hacks into the
    main thread to get this to work).

    Any thoughts?

    Thanks,
    -Ivan


    #!/usr/bin/python

    from threading import Event, Thread
    from time import sleep

    class CpuHogThread(Thread):
    def __init__(self):
    Thread.__init__(self)
    self.counter = 0
    self.stopEvent = Event()

    def join(self):
    self.stopEvent.set()
    Thread.join(self)

    def run(self):
    try:
    while not self.stopEvent.isSet():
    self.counter += 1
    except KeyboardInterrupt:
    print 'CPU hog subthread caught KeyboardInterrupt'

    if __name__ == '__main__':
    t = CpuHogThread()
    t.start()
    print 'CPU hog subthread spawned'
    try:
    while True:
    sleep(2)
    # without the next print, Ctrl-C is completely ignored
    #print 'still sleeping'
    except KeyboardInterrupt:
    print 'main thread caught KeyboardInterrupt'
    t.join()
    print 'CPU hog subthread joined at counter %d' % t.counter
     
    Ivan Nestlerode, Jan 7, 2004
    #4
  5. "Paul McGuire" <> wrote in message news:<XhEJb.40848$>...
    > Is the signal module not available to you?
    > <snip>
    > Assuming you have access to the signal module:
    > 1. Create a boolean that you can periodically test for in your thread code,
    > to see if you should keep processing. This may be a global singleton, a
    > global boolean, or something, but it needs to be visible to both the main
    > thread and the threaded code. You probably have some compute intensive
    > loops such as:
    > while( prime_factors < 1e15 ):
    > or
    > while( optimization != converged ):
    > or even just
    > while( True ):
    >
    > Change these to reference the "keep processing" flag.
    > while( keep_processing and prime_factors < 1e15 ):
    > or
    > while( keep_processing and optimization != converged ):
    > or even just
    > while( keep_processing ): # no need to 'and' with True
    >
    > 2. Create a signal handler to trap for SIGINT. In the signal handler, set
    > the "keep-processing" bit to False:
    >
    > <snip>
    >
    > Now your main thread will get the ^C interrupt and clear the
    > "keep_processing" flag, which will eventually get picked up by your worker
    > thread the next time it tests for it.
    >
    > HTH,
    > -- Paul


    I do have the signal module (Python 2.3.2 on Linux), but
    unfortunately, even registering a signal handler doesn't fix the
    problem where busy non-main threads prevent Ctrl-C from working.

    Here is another code snippet using a signal handler (this code is very
    similar to the code from my second post). If the main thread's loop
    has the print statement, everything works fine. If it is commented out
    (as written here), Ctrl-C is ignored completely and the program just
    keeps running. Why won't this work? Is this a bug?

    Thanks,
    -Ivan

    #!/usr/bin/python

    from signal import signal, SIGINT
    from threading import Event, Thread
    from time import sleep

    class CpuHogThread(Thread):
    def __init__(self):
    Thread.__init__(self)
    self.counter = 0
    self.stopEvent = Event()

    def join(self):
    self.stopEvent.set()
    Thread.join(self)

    def run(self):
    try:
    while not self.stopEvent.isSet():
    self.counter += 1
    except KeyboardInterrupt:
    print 'CPU hog subthread caught KeyboardInterrupt'

    def handler(signal, frme):
    stopEvent.set()
    print 'main thread SIGINT handler'
    t.join()
    print 'CPU hog subthread joined at counter %d' % t.counter
    return 0

    if __name__ == '__main__':
    stopEvent = Event()
    signal(SIGINT, handler)
    t = CpuHogThread()
    t.start()

    print 'CPU hog subthread spawned'
    try:
    while not stopEvent.isSet():
    sleep(2)
    # without the next print, Ctrl-C is completely ignored
    #print 'still sleeping'
    except KeyboardInterrupt:
    print 'main thread caught KeyboardInterrupt'
    t.join()
    print 'CPU hog subthread joined at counter %d' % t.counter
     
    Ivan Nestlerode, Jan 7, 2004
    #5
  6. Ivan Nestlerode

    Jeff Epler Guest

    Running your attached program, it works just fine whereever I try it,
    except on a system so old it only has Python 1.5 installed. Even that
    worked after a few seconds of editing.

    Here's the behavior I saw on Fedora Core 1:
    $ python ivan.py
    CPU hog subthread spawned
    main thread caught KeyboardInterrupt
    CPU hog subthread joined at counter 76429

    This is the behavior I saw on redhat 9:
    $ python /tmp/ivan.py
    CPU hog subthread spawned
    main thread caught KeyboardInterrupt
    CPU hog subthread joined at counter 89897

    Here's the behavior I saw on Red Hat 7.2:
    CPU hog subthread spawned
    main thread caught KeyboardInterrupt
    CPU hog subthread joined at counter 31764

    Here's the behavior I saw on Windows XP:
    C:\...> python ivan.py
    CPU hog subthread spawned
    main thread caught KeyboardInterrupt
    CPU hog subthread joined at counter 665928

    Here's the behavior I saw on Red Hat 6.2:
    $ python1.5 ivan.py
    File "ivan.py", line 19
    self.counter += 1
    ^
    SyntaxError: invalid syntax
    $ vi ivan.py
    $ python1.5 ivan.py
    CPU hog subthread spawned
    main thread caught KeyboardInterrupt
    CPU hog subthread joined at counter 238983

    Jeff
     
    Jeff Epler, Jan 7, 2004
    #6
  7. (Ivan Nestlerode) writes:

    > Michael Hudson <> wrote in message news:<>...
    > > On debian, the signal module certainly should be available, and the
    > > KeyboardInterrupt exception should go to the main thread. Or at
    > > least, that's what I thought. You're definitely seeing it being
    > > delivered to an arbitrary thread?
    > >
    > > Cheers,
    > > mwh

    >
    > After a closer look, I am not seeing KeyboardInterrupt being delivered
    > to non-main threads. What I am seeing is that Ctrl-C does not work at
    > all when non-main threads are CPU intensive.


    Oh...

    > I had previously jumped to the conclusion that because Ctrl-C didn't
    > work and because the non-main thread was sloppy ("except:"), that
    > Ctrl-C wasn't working because KeyboardInterrupt was going to the
    > sub-thread. That was an incorrect conclusion.


    Well, in a way, good, but perhaps not for you...

    > So I guess the entire problem can be restated as: How do I make
    > Ctrl-C work properly when the non-main thread is busier than the
    > main thread?


    This is hard for me to answer -- it works fine for me. What platform
    are you on? ISTR Debian, but not versions of anything.

    I'm somewhat inclined to start blaming libc at this point...

    As you may well be aware, the combination of threads and signals is a
    bit of a minefield. Python runs all threads other than the main
    thread with all signals masked. My understanding is that when a signal
    is delivered to a process it's delivered to an arbitrary thread that
    has that signal unmasked -- in Python's case, this should be the main
    thread. It kind of sounds like in your case the signal is being
    queued up on a non-main thread, but as the signal is never unblocked,
    it never gets handled.

    [...]
    > I think this is a bug though (I shouldn't have to put hacks into the
    > main thread to get this to work).


    Agreed, but I'm not sure they're in Python.

    > Any thoughts?


    Upgrade glibc?

    Cheers,
    mwh

    --
    ARTHUR: Why are there three of you?
    LINTILLAS: Why is there only one of you?
    ARTHUR: Er... Could I have notice of that question?
    -- The Hitch-Hikers Guide to the Galaxy, Episode 11
     
    Michael Hudson, Jan 7, 2004
    #7
  8. Michael Hudson <> wrote in message news:<>...
    > This is hard for me to answer -- it works fine for me. What platform
    > are you on? ISTR Debian, but not versions of anything.
    >
    > I'm somewhat inclined to start blaming libc at this point...
    >
    > As you may well be aware, the combination of threads and signals is a
    > bit of a minefield. Python runs all threads other than the main
    > thread with all signals masked. My understanding is that when a signal
    > is delivered to a process it's delivered to an arbitrary thread that
    > has that signal unmasked -- in Python's case, this should be the main
    > thread. It kind of sounds like in your case the signal is being
    > queued up on a non-main thread, but as the signal is never unblocked,
    > it never gets handled.
    >
    > [...]
    > > I think this is a bug though (I shouldn't have to put hacks into the
    > > main thread to get this to work).

    >
    > Agreed, but I'm not sure they're in Python.
    >
    > > Any thoughts?

    >
    > Upgrade glibc?
    >
    > Cheers,
    > mwh


    I've filed the bug with Debian since I think it is something wrong
    with their Python or their glibc. The bug is #226547.

    FYI, here are the exact details of the kernel/packages that don't seem
    to work:

    $ uname -a
    Linux debian 2.4.23 #1 Sun Nov 30 21:11:49 EST 2003 i686 GNU/Linux

    packages and versions:
    ii libbz2-1.0 1.0.2-1
    ii libc6 2.3.2.ds1-10
    ii libdb4.1 4.1.25-10
    ii libncurses5 5.3.20030719-4
    ii libreadline4 4.3-8
    ii libssl0.9.7 0.9.7c-5
    ii python 2.3.3-4
    ii python2.3 2.3.3-4
    ii zlib1g 1:1.2.1-3


    -Ivan
     
    Ivan Nestlerode, Jan 8, 2004
    #8
  9. Ivan Nestlerode

    Anand Pillai Guest

    There is an easier way than using signals.

    Add a keyboard interrupt handler for your main function.
    And in the handler code, write a clean up function which
    will be able to then manage threads.

    This is the method I have used in my HarvestMan program.
    I am copying this code here.

    ------------(copied code)----------------------------------------
    try:
    self.start_project()
    except (KeyboardInterrupt, EOFError):
    # Localise links
    if not self._cfg.ignorekbinterrupt:
    # dont allow to write cache, since it
    # screws up existing cache.
    GetObject('datamanager').conditional_cache_set()
    self.clean_up()
    ---------------------------------------------------------------------

    The clean_up method takes care of cleaning the thread.
    There you can add code to pass on the interrupt to the
    main thread or perform your own clean up actions.

    One way to do this is to use the currentThread() method
    of threading module which can tell you whether the thread
    is the main or a worker.

    One drawback with this method is that you might need to
    set your threads as 'daemon' threads to make sure that
    the threads dont hang your python program. This is because
    the entire python program exits when the only threads left
    are daemon threads.

    -Anand

    (Ivan Nestlerode) wrote in message news:<>...
    > Michael Hudson <> wrote in message news:<>...
    > > This is hard for me to answer -- it works fine for me. What platform
    > > are you on? ISTR Debian, but not versions of anything.
    > >
    > > I'm somewhat inclined to start blaming libc at this point...
    > >
    > > As you may well be aware, the combination of threads and signals is a
    > > bit of a minefield. Python runs all threads other than the main
    > > thread with all signals masked. My understanding is that when a signal
    > > is delivered to a process it's delivered to an arbitrary thread that
    > > has that signal unmasked -- in Python's case, this should be the main
    > > thread. It kind of sounds like in your case the signal is being
    > > queued up on a non-main thread, but as the signal is never unblocked,
    > > it never gets handled.
    > >
    > > [...]
    > > > I think this is a bug though (I shouldn't have to put hacks into the
    > > > main thread to get this to work).

    > >
    > > Agreed, but I'm not sure they're in Python.
    > >
    > > > Any thoughts?

    > >
    > > Upgrade glibc?
    > >
    > > Cheers,
    > > mwh

    >
    > I've filed the bug with Debian since I think it is something wrong
    > with their Python or their glibc. The bug is #226547.
    >
    > FYI, here are the exact details of the kernel/packages that don't seem
    > to work:
    >
    > $ uname -a
    > Linux debian 2.4.23 #1 Sun Nov 30 21:11:49 EST 2003 i686 GNU/Linux
    >
    > packages and versions:
    > ii libbz2-1.0 1.0.2-1
    > ii libc6 2.3.2.ds1-10
    > ii libdb4.1 4.1.25-10
    > ii libncurses5 5.3.20030719-4
    > ii libreadline4 4.3-8
    > ii libssl0.9.7 0.9.7c-5
    > ii python 2.3.3-4
    > ii python2.3 2.3.3-4
    > ii zlib1g 1:1.2.1-3
    >
    >
    > -Ivan
     
    Anand Pillai, Jan 8, 2004
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. PantherSE
    Replies:
    0
    Views:
    365
    PantherSE
    May 16, 2005
  2. darren kirby
    Replies:
    1
    Views:
    532
    Diez B. Roggisch
    Nov 27, 2005
  3. Replies:
    3
    Views:
    813
    Donn Cave
    Nov 28, 2005
  4. Strax-Haber, Matthew (LARC-D320)

    FW: [Tutor] Multi-Threading and KeyboardInterrupt

    Strax-Haber, Matthew (LARC-D320), Jun 11, 2009, in forum: Python
    Replies:
    0
    Views:
    973
    Strax-Haber, Matthew (LARC-D320)
    Jun 11, 2009
  5. Dennis Lee Bieber
    Replies:
    13
    Views:
    563
    Mike Kazantsev
    Jun 16, 2009
Loading...

Share This Page