Terminating a Python program that uses multi-process, multi-threading

[akineko]

Hello Python experts,

I have a program that uses three processes (invoked by
multiprocessing) and several threads.
The program is terminated when ^C is typed (KeyboardInterrupt).
The main process takes the KeyboardInterrupt Exception and it orderly
shutdown the program.

It works fine in normal situation.

However, KeyboardInterrupt is not accepted when, for example, the
program is stuck somewhere due to error in network. I understand that
the KeyboardInterrupt won't be processed until the program moves out
from an atomic step in a Python program.

Probably this is partly my fault as I use some blocking statements
(without timeout) as they should not block the program in normal
situation.

As the program freezes up, I cannot do anything except killing three
processes using kill command.
So, I cannot tell which statement is actually blocking when this
occurs (and there are many suspects).

Is there any good way to deal with this kind of problem?
Killing three processes when it hangs is not a fun thing to do.

Any suggestions will be greatly appreciated.

Best regards,
Aki Niimura

 

[MRAB]

Hello Python experts,

I have a program that uses three processes (invoked by
multiprocessing) and several threads.
The program is terminated when ^C is typed (KeyboardInterrupt).
The main process takes the KeyboardInterrupt Exception and it orderly
shutdown the program.

It works fine in normal situation.

However, KeyboardInterrupt is not accepted when, for example, the
program is stuck somewhere due to error in network. I understand that
the KeyboardInterrupt won't be processed until the program moves out
from an atomic step in a Python program.

Probably this is partly my fault as I use some blocking statements
(without timeout) as they should not block the program in normal
situation.

As the program freezes up, I cannot do anything except killing three
processes using kill command.
So, I cannot tell which statement is actually blocking when this
occurs (and there are many suspects).

Is there any good way to deal with this kind of problem?
Killing three processes when it hangs is not a fun thing to do.

Any suggestions will be greatly appreciated.

The scripts could write to a log around where they could block (write to
the log just before and just after), so that you can then see where they
blocked.

A hacky way of killing the processes would be for the processes to write
their process IDs to the log and then you can write another script which
will parse the log to get the process IDs and then kill the processes
with those IDs. (Left as an exercise for the reader. )

 

[akineko]

A hacky way of killing the processes would be for the processes to write
their process IDs to the log and then you can write another script which
will parse the log to get the process IDs and then kill the processes
with those IDs. (Left as an exercise for the reader.

Thank you MRAB.
Your suggestion is very reasonable. I think I will do both.
Try to reduce the chance of freeze-up and create a script to clean-up.

Aki-

 

[Jesse Noller]

Hello Python experts,

I have a program that uses three processes (invoked by
multiprocessing) and several threads.
The program is terminated when ^C is typed (KeyboardInterrupt).
The main process takes the KeyboardInterrupt Exception and it orderly
shutdown the program.

It works fine in normal situation.

However, KeyboardInterrupt is not accepted when, for example, the
program is stuck somewhere due to error in network. I understand that
the KeyboardInterrupt won't be processed until the program moves out
from an atomic step in a Python program.

Probably this is partly my fault as I use some blocking statements
(without timeout) as they should not block the program in normal
situation.

As the program freezes up, I cannot do anything except killing three
processes using kill command.
So, I cannot tell which statement is actually blocking when this
occurs (and there are many suspects).

Is there any good way to deal with this kind of problem?
Killing three processes when it hangs is not a fun thing to do.

Any suggestions will be greatly appreciated.

Best regards,
Aki Niimura

See also:
http://jessenoller.com/2009/01/08/multiprocessingpool-and-keyboardinterrupt/

jesse