Multithreaded python program freezes

[Andreas Røsdal]

Hello,

I have some problems with a http proxy which is implemented
in Python 2.6. A few times a day, the proxy begins using 100% CPU
and doesn't work any more.

I have created a thread dump when the problem occurs here:
http://www.pvv.ntnu.no/~andrearo/thread-dump.html

This is a thread dump during normal operation:
http://www.pvv.ntnu.no/~andrearo/thread-dump-normal.html

The source code can be found here:
http://code.google.com/p/freeciv-forever/source/browse/trunk/freeciv-proxy/


The application is heavily multi-threaded, since its main function
is being a proxy converting packets containing C struct data to JSON
over HTTP.

Any advice on why this Python program appears to freeze? Is there anything
more I can do to find out why this problem occurs?


Andreas R

 

[Cameron Simpson]

| I have some problems with a http proxy which is implemented
| in Python 2.6. A few times a day, the proxy begins using 100% CPU
| and doesn't work any more.
|
| I have created a thread dump when the problem occurs here:
| http://www.pvv.ntnu.no/~andrearo/thread-dump.html
|
| This is a thread dump during normal operation:
| http://www.pvv.ntnu.no/~andrearo/thread-dump-normal.html

These two URLs don't work for me.

[...]
| Any advice on why this Python program appears to freeze? Is there anything
| more I can do to find out why this problem occurs?

Sounds like a livelock:
http://en.wikipedia.org/wiki/Deadlock#Livelock
or busy lock.

Can you find out which bits of your program are busy?

Are there places in your code that loop, polling for data, instead of
using blocking I/O. With a network proxy, a typical place that might
happen is a loop testing select() (or its poll-like friends) with a short or
zero timeout on inactive file handles.

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

No electrons were harmed in the production of this message.
- Dr. P. Gensheimer <[email protected]>