Debugging memory leaks

[Frank Millman]

Hi all,

I've written a program using Twisted that uses SqlAlchemy to access a
database using threads.deferToThread(...) and SqlAlchemy's
scoped_session(...). This program runs for a long time, but leaks memory
slowly to the point of needing to be restarted. I don't know that the
SqlAlchemy/threads thing is the problem, but thought I'd make you aware of
it.

Anyway, my real question is how to go about debugging memory leak problems
in Python, particularly for a long running server process written with
Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell
me enough to locate the problem. If anyone as any suggestions or pointers
it would be very much appreciated!

Thanks in advance,
Doug

You have received lots of good advice, but there is one technique that I
have found useful that has not been mentioned.

As you are probably aware, one of the main causes of a 'memory leak' in
python is an object that is supposed to be garbage collected, but hangs
around because there is still a reference pointing to it.

You cannot directly confirm that an object has been deleted, because
invoking its '__del__' method causes side-effects which can prevent it from
being deleted even if it is otherwise ok.

However, there is an indirect way of confirming it - a 'DelWatcher' class. I
got this idea from a thread on a similar subject in this forum a long time
ago. Here is how it works.

class DelWatcher:
def __init__(self, obj):
# do not store a reference to obj - that would create a circular
reference
# store some attribute that uniquely identifies the 'obj' instance
self.name = obj.name
print(self.name, 'created')
def __del__(self):
print(self.name, 'deleted')

class MyClass:
def __init__(self, ...):
[...]
self._del = DelWatcher(self)

Now you can watch the objects as they are created, and then check that they
are deleted when you expect them to be.

This can help to pinpoint where the memory leak is occurring.

HTH

Frank Millman