Dumping the state of a deadlocked process

[andre.naess]

Hi all

I'm currently having some issues with a process getting deadlocked. The
problem is that the only way I can seem to find information about where
it deadlocks is by making a wild guess, insert a pdb.set_trace() before
this point, and then step until it locks up, hoping that I've guessed
right.

The frustrating part is that most of the time my guesses are wrong.

It would be really nice if I could send the python process some signal
which would cause it to print the current stacktrace and exit
immediately. That way I would quickly be able to pinpoint where in the
code the deadlock happens. Java has a somewhat similar feature where
you can send a running VM process a SIGQUIT, to which it will respond
by dumping all current threads and lots of other information on stdout.

Is this possible somehow?

 

[MrJean1]

Did you try using the signal module? If not, a basic example is here
<http://docs.python.org/lib/node546.html> which may need to be
extended.

/Jean Brouwers

 

[Dennis Lee Bieber]

Hi all

I'm currently having some issues with a process getting deadlocked. The
problem is that the only way I can seem to find information about where
it deadlocks is by making a wild guess, insert a pdb.set_trace() before
this point, and then step until it locks up, hoping that I've guessed
right.

The frustrating part is that most of the time my guesses are wrong.

I presume the process is using threads? If it is truly deadlocked,
then you must have some mutual calls to lock objects somewhere... It
would seem that rather than just randomly inserting debug statements you
should surround each call to a lock with statements.

print "Locking xyz"
xyz.acquire() #or whatever the syntax is
print "Locked xyz"



print "Releasing xyz"
xyz.release()
print "Released xyz"


You'd need something like that around any potentially blocking
operation -- queue operations, subprocess operations, socket
operations... Rather than print statements you may wish to implement it
via the logging module.

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Hendrik van Rooyen]

Hi all

I'm currently having some issues with a process getting deadlocked. The
problem is that the only way I can seem to find information about where
it deadlocks is by making a wild guess, insert a pdb.set_trace() before
this point, and then step until it locks up, hoping that I've guessed
right.

The frustrating part is that most of the time my guesses are wrong.

Welcome to the wonderful world of crash and burn....

It would be really nice if I could send the python process some signal
which would cause it to print the current stacktrace and exit
immediately. That way I would quickly be able to pinpoint where in the
code the deadlock happens. Java has a somewhat similar feature where
you can send a running VM process a SIGQUIT, to which it will respond
by dumping all current threads and lots of other information on stdout.

Is this possible somehow?

Have you tried to sprinkle your code with print statements of the "We get here
No: 7" kind - you can get quite a good idea of what is going on if you do, and
if there are threads running - the results are often surprisingly insightful...

- Hendrik

 

[andre.naess]

Did you try using the signal module? If not, a basic example is here
<http://docs.python.org/lib/node546.html> which may need to be
extended.

I looks useful. I gave it a try, and the only weakness it has is that
when my process locks, it locks so badly that it doesn't respond to
CTRL-C, or any other signal. But by sending it a SIGQUIT which causes
it to dump the current state, and then kill it, I get the dump I need.

This is actually not a multi-threaded app. It's an application which
uses a SQL DB. The problem I was having was that I had a cursor which
started a transaction, and then never finished. Then some other cursor
came along and tried to perform a delete table, and they both locked
up. The cursor isn't ending it's transaction, and the transaction
prevents the delete table from being executed. Luckily Postgresql
allows me to list current activity, otherwise I would have been
scratching my head still.

Using logging or print statements to debug this sort of things is
highly unsatisfactory. I think the way Java uses SIGQUIT is pretty
neat, are there any reasons why Python can't adopt something similar?

 

[Ziga Seilnacht]

Hi all

I'm currently having some issues with a process getting deadlocked. The
problem is that the only way I can seem to find information about where
it deadlocks is by making a wild guess, insert a pdb.set_trace() before
this point, and then step until it locks up, hoping that I've guessed
right.

The frustrating part is that most of the time my guesses are wrong.

It would be really nice if I could send the python process some signal
which would cause it to print the current stacktrace and exit
immediately. That way I would quickly be able to pinpoint where in the
code the deadlock happens. Java has a somewhat similar feature where
you can send a running VM process a SIGQUIT, to which it will respond
by dumping all current threads and lots of other information on stdout.

Is this possible somehow?

Check out the sys._current_frames() function, new in Python 2.5:
http://docs.python.org/lib/module-sys.html#l2h-5122

Hope this helps,
Ziga

 

[Dennis Lee Bieber]

I presume the process is using threads? If it is truly deadlocked,
then you must have some mutual calls to lock objects somewhere... It

Well, based on your recent response ... I'm all wet...
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[fumanchu]

I presume the process is using threads? If it is truly deadlocked,
then you must have some mutual calls to lock objects somewhere... It
would seem that rather than just randomly inserting debug statements you
should surround each call to a lock with statements.

print "Locking xyz"
xyz.acquire() #or whatever the syntax is
print "Locked xyz"



print "Releasing xyz"
xyz.release()
print "Released xyz"


You'd need something like that around any potentially blocking
operation -- queue operations, subprocess operations, socket
operations... Rather than print statements you may wish to implement it
via the logging module.

If you don't mind a potentially large log file, use the pyconquer
module I maintain here: http://projects.amor.org/misc/wiki/PyConquer
which uses settrace to do the logging in a much more readable and
manageable way than printlining. Try an initial run using the default
settings to narrow down the culprit, and then a run with C events
turned on if the first run wasn't enough. It should help out even if
your program is not multi-threaded, but it realy shines with threads.



Robert Brewer
System Architect
Amor Ministries
(e-mail address removed)

P.S. Oh, and ignore the times in the output for now; that's still work
in progress.

 

[MrJean1]

I looks useful. I gave it a try, and the only weakness it has is that
when my process locks, it locks so badly that it doesn't respond to
CTRL-C, or any other signal. But by sending it a SIGQUIT which causes
it to dump the current state, and then kill it, I get the dump I need.

The Ctrl-C signal SIGINT is caught by Python by default and the signal
handler
raises a KeyboardInterrupt exception. For any other signals, the
signal is caught
but the signal handler is not called until Python returns to the main
loop.

Therefore, if some extension -like Postgresql in this case- is busy or
hangs, nothing
will happen until Python regains control.

This is actually not a multi-threaded app. It's an application which
uses a SQL DB. The problem I was having was that I had a cursor which
started a transaction, and then never finished. Then some other cursor
came along and tried to perform a delete table, and they both locked
up. The cursor isn't ending it's transaction, and the transaction
prevents the delete table from being executed. Luckily Postgresql
allows me to list current activity, otherwise I would have been
scratching my head still.

Using logging or print statements to debug this sort of things is
highly unsatisfactory. I think the way Java uses SIGQUIT is pretty
neat, are there any reasons why Python can't adopt something similar?

I can not anwer that.

/Jean Brouwers