find memory leaks in running program

[Marco Hornung]

Hey,

------------------------------------------------------------------------------------------
questions
------------------------------------------------------------------------------------------
1. What are the best tools to analyze pythons memory stack, while it is running?
2. Is there a possibility to analyze the memory stack of a program with external programs? (without to change the source code - I am only interested in the object size)
3. Can I sort of "break" into the memory to see what objects consume how much memory?

------------------------------------------------------------------------------------------
my scenario
------------------------------------------------------------------------------------------
I have to debug a multithreaded server, which is written in the twisted-framework. One of the processes has some sort of memory leak - After one of our jobs is finished the main process has still over 59% of the entire memory allocated. I will probably have to recreate our scenario and equip our server with some memory sensors - but it takes 12h to reproduce the scenario and I will have to change the source code(at least I do not know of other options).
Therefore I am looking for quicker possibilities to look into what causes our memory leak.

Regards,
Marco

 

[shearichard]

Hey,

------------------------------------------------------------------------------------------
questions
------------------------------------------------------------------------------------------
1. What are the best tools to analyze pythons memory stack, while it is running?
2. Is there a possibility to analyze the memory stack of a program with external programs? (without to change the source code - I am only interested in the object size)
3. Can I sort of "break" into the memory to see what objects consume how much memory?

------------------------------------------------------------------------------------------
my scenario
------------------------------------------------------------------------------------------
I have to debug a multithreaded server, which is written in the twisted-framework. One of the processes has some sort of memory leak - After one of our jobs is finished the main process has still over 59% of the entire memory allocated. I will probably have to recreate our scenario and equip our server with some memory sensors - but  it takes 12h to reproduce the scenario and I will have to change the source code(at least I do not know of other options).
Therefore I am looking for quicker possibilities to look into what causes our memory leak.

Regards,
Marco

I haven't used this myself but sometime ago I bookmarked yappi which
may go some way towards helping you ...

http://code.google.com/p/yappi/
http://code.google.com/p/yappi/wiki/apiyappi

.... also sometime ago I was at presentation (Pycon NZ 2009) where the
twisted.manhole functionality was used to hook a remote console up to
a running twisted task and examine the interior of the target task.
Can't remember the details but someone else here may be able to help.
This touches upon the idea ...

http://stackoverflow.com/questions/1721699/is-there-any-remote-console-for-twisted-server

.... and in turn references Heapy which I'd forgotten about but may
also be useful in your circs ...

http://guppy-pe.sourceforge.net/

regards

Richard.