Terminating a Python program that uses multi-process, multi-threading

Discussion in 'Python' started by akineko, Jan 28, 2009.

  1. akineko

    akineko Guest

    Hello Python experts,

    I have a program that uses three processes (invoked by
    multiprocessing) and several threads.
    The program is terminated when ^C is typed (KeyboardInterrupt).
    The main process takes the KeyboardInterrupt Exception and it orderly
    shutdown the program.

    It works fine in normal situation.

    However, KeyboardInterrupt is not accepted when, for example, the
    program is stuck somewhere due to error in network. I understand that
    the KeyboardInterrupt won't be processed until the program moves out
    from an atomic step in a Python program.

    Probably this is partly my fault as I use some blocking statements
    (without timeout) as they should not block the program in normal
    situation.

    As the program freezes up, I cannot do anything except killing three
    processes using kill command.
    So, I cannot tell which statement is actually blocking when this
    occurs (and there are many suspects).

    Is there any good way to deal with this kind of problem?
    Killing three processes when it hangs is not a fun thing to do.

    Any suggestions will be greatly appreciated.

    Best regards,
    Aki Niimura
     
    akineko, Jan 28, 2009
    #1
    1. Advertising

  2. akineko

    MRAB Guest

    Re: Terminating a Python program that uses multi-process,multi-threading

    akineko wrote:
    > Hello Python experts,
    >
    > I have a program that uses three processes (invoked by
    > multiprocessing) and several threads.
    > The program is terminated when ^C is typed (KeyboardInterrupt).
    > The main process takes the KeyboardInterrupt Exception and it orderly
    > shutdown the program.
    >
    > It works fine in normal situation.
    >
    > However, KeyboardInterrupt is not accepted when, for example, the
    > program is stuck somewhere due to error in network. I understand that
    > the KeyboardInterrupt won't be processed until the program moves out
    > from an atomic step in a Python program.
    >
    > Probably this is partly my fault as I use some blocking statements
    > (without timeout) as they should not block the program in normal
    > situation.
    >
    > As the program freezes up, I cannot do anything except killing three
    > processes using kill command.
    > So, I cannot tell which statement is actually blocking when this
    > occurs (and there are many suspects).
    >
    > Is there any good way to deal with this kind of problem?
    > Killing three processes when it hangs is not a fun thing to do.
    >
    > Any suggestions will be greatly appreciated.
    >

    The scripts could write to a log around where they could block (write to
    the log just before and just after), so that you can then see where they
    blocked.

    A hacky way of killing the processes would be for the processes to write
    their process IDs to the log and then you can write another script which
    will parse the log to get the process IDs and then kill the processes
    with those IDs. (Left as an exercise for the reader. :))
     
    MRAB, Jan 28, 2009
    #2
    1. Advertising

  3. akineko

    akineko Guest

    Re: Terminating a Python program that uses multi-process,multi-threading

    On Jan 28, 1:11 pm, MRAB <> wrote:
    > A hacky way of killing the processes would be for the processes to write
    > their process IDs to the log and then you can write another script which
    > will parse the log to get the process IDs and then kill the processes
    > with those IDs. (Left as an exercise for the reader.


    Thank you MRAB.
    Your suggestion is very reasonable. I think I will do both.
    Try to reduce the chance of freeze-up and create a script to clean-up.

    Aki-
     
    akineko, Jan 28, 2009
    #3
  4. akineko

    Jesse Noller Guest

    Re: Terminating a Python program that uses multi-process,multi-threading

    On Wed, Jan 28, 2009 at 3:46 PM, akineko <> wrote:
    > Hello Python experts,
    >
    > I have a program that uses three processes (invoked by
    > multiprocessing) and several threads.
    > The program is terminated when ^C is typed (KeyboardInterrupt).
    > The main process takes the KeyboardInterrupt Exception and it orderly
    > shutdown the program.
    >
    > It works fine in normal situation.
    >
    > However, KeyboardInterrupt is not accepted when, for example, the
    > program is stuck somewhere due to error in network. I understand that
    > the KeyboardInterrupt won't be processed until the program moves out
    > from an atomic step in a Python program.
    >
    > Probably this is partly my fault as I use some blocking statements
    > (without timeout) as they should not block the program in normal
    > situation.
    >
    > As the program freezes up, I cannot do anything except killing three
    > processes using kill command.
    > So, I cannot tell which statement is actually blocking when this
    > occurs (and there are many suspects).
    >
    > Is there any good way to deal with this kind of problem?
    > Killing three processes when it hangs is not a fun thing to do.
    >
    > Any suggestions will be greatly appreciated.
    >
    > Best regards,
    > Aki Niimura
    >
    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >


    See also:
    http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/

    jesse
     
    Jesse Noller, Jan 29, 2009
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Georg Andersson

    terminating applet-process

    Georg Andersson, Jun 8, 2004, in forum: Java
    Replies:
    7
    Views:
    2,522
    Andrew Thompson
    Jun 10, 2004
  2. Wayne Marrison

    Terminating a process tree

    Wayne Marrison, Nov 8, 2004, in forum: Java
    Replies:
    9
    Views:
    3,466
    John C. Bollinger
    Nov 9, 2004
  3. Earl Eiland

    terminating an inactive process

    Earl Eiland, Apr 2, 2005, in forum: Python
    Replies:
    1
    Views:
    384
    fred.dixon
    Apr 2, 2005
  4. Trent Mick

    Re: terminating an inactive process

    Trent Mick, Apr 4, 2005, in forum: Python
    Replies:
    1
    Views:
    339
    fred.dixon
    Apr 5, 2005
  5. jfcg
    Replies:
    0
    Views:
    399
Loading...

Share This Page