Detecting memory leaks on apache, mod_python

I

Ilias Lazaridis

How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
* mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

context:

http://dev.lazaridis.com/base/ticket/148
 
I

Ilias Lazaridis

How to detect memory leaks of python programms, which run in an
environment like this:

* Suse Linux 9.3
* Apache
* mod_python

The problem occoured after some updates on the infrastructure. It's
most possibly caused by trac and it's dependencies, but several
components of the OS where updated, too.

Any nice tools which play with the above constellation?

Thank's for any hints!

No tool available to detect memory leaks for python applications?
 
I

Ilias Lazaridis

No tool available to detect memory leaks for python applications?

So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python
b) that this community is not very helpful
 
G

Graham Dumpleton

So, anyone who hit's on this thread via a search will think

a) that there's really no memory leak detection for python
b) that this community is not very helpful

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

Maybe you should read:

http://www.catb.org/~esr/faqs/smart-questions.html

Two things to note in here. First, choose your forum appropriately and
secondly show some courtesy rather than making accusations against the
community if no one answers.

If you want to see perhaps how you might be viewed by the open source
community when you make such a comment, perhaps also watch:

http://video.google.com/videoplay?docid=-4216011961522818645

Now, since you think this is a Trac problem, why don't you go ask on
the Trac mailing list.

http://groups.google.com/group/trac-users?lnk=li

Even a search of that forum will most likely yield previous
discussions about growing memory use of Trac as a result of things
like Python wrappers for Subversion or certain database adapters. It
may be a case of using a different version, or in some cases
configuration of your hosting environment, if using Apache, to recycle
Apache child processes after a set number of requests so as to restore
process sizes back to a low level.

So, do some research first in the correct places and then perhaps ask
any additional questions in the correct place also.

Graham
 
D

Diez B. Roggisch

So, anyone who hit's on this thread via a search will think
c) That finally people in this forum are smart enough to detect your
flamebait & refuse to comment on it, Ilias...

Comments like (b) will not help your chances one bit in respect of
getting an answer from anyone.

Maybe you should read:

http://www.catb.org/~esr/faqs/smart-questions.html

Two things to note in here. First, choose your forum appropriately and
secondly show some courtesy rather than making accusations against the
community if no one answers.

If you want to see perhaps how you might be viewed by the open source
community when you make such a comment, perhaps also watch:

http://video.google.com/videoplay?docid=-4216011961522818645

Now, since you think this is a Trac problem, why don't you go ask on
the Trac mailing list.

http://groups.google.com/group/trac-users?lnk=li

Even a search of that forum will most likely yield previous
discussions about growing memory use of Trac as a result of things
like Python wrappers for Subversion or certain database adapters. It
may be a case of using a different version, or in some cases
configuration of your hosting environment, if using Apache, to recycle
Apache child processes after a set number of requests so as to restore
process sizes back to a low level.

So, do some research first in the correct places and then perhaps ask
any additional questions in the correct place also.

Graham, I'm not sure how long you have been around here - but googling
for Ilias in general and specificly in this group will reveal that he
often posts spiteful & inflammatory messages and has a tendency to get
cross with open source projects. It's debatable if he can be called a
troll - but he surely is a nuisance best let alone, so he doesn't gain
traction.

Diez
 
S

Steven D'Aprano

c) That finally people in this forum are smart enough to detect your
flamebait & refuse to comment on it, Ilias...

Not me. I didn't (still don't actually) recognise Ilias' name. I was
actually thinking:

(d) Python is so great that there can never be any memory leaks!

but I guess that's probably wishful thinking.

So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some* sort
of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?

Unless I've missed something, I don't believe Python does *any* of that.
It could be quite useful. The timeit module is good for measuring the
time taken for code to run, but as near as I can tell, there's no way to
optimize memory use except prematurely.
 
F

Fredrik Lundh

Steven said:
> Not me.

You're quite knew to this internet thing, aren't you? ;-)
So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some* sort
of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?

And what languages would that be? I cannot think of a single modern
language that does any of that. Including low-level stuff like C/C++.

And things like "how much memory is free in the heap" isn't even a
meaningful concept on a modern machine, thanks to the wonders of virtual
memory (especially the overcommitting kind).

For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns. Just run the program under a few different
scenarios, and see what it does. If the memory use looks suspicious,
use standard debugging techniques to locate the problematic area, and
standard benchmarking techniques to look for unexpected blowups and leaks.

For really hard problems, use the "gc" module, instrumenting memory
allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

</F>
 
S

Steven D'Aprano

You're quite knew to this internet thing, aren't you? ;-)
:-D



And what languages would that be? I cannot think of a single modern
language that does any of that. Including low-level stuff like C/C++.

I didn't actually say they were *modern* languages. E.g. THINK Pascal for
Apple Mac, circa 1990.

And things like "how much memory is free in the heap" isn't even a
meaningful concept on a modern machine, thanks to the wonders of virtual
memory (especially the overcommitting kind).

Maybe the memory model used in modern multi-tasking virtual-memory PCs
makes the concept of "free memory" obsolete. Although if so, somebody
should have a quiet word with the author of the Linux free command:

$ free
total used free shared buffers cached
Mem: 1002524 988736 13788 0 7044 98916
-/+ buffers/cache: 882776 119748
Swap: 4241080 3939736 301344

(Admittedly that's system-wide memory usage, rather than for a single
process.)


For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns. Just run the program under a few different
scenarios, and see what it does.

Are you saying that Python programs can't monitor their own memory use?

I'm happy to accept that "free memory" is not a meaningful concept for a
process in a modern system. That makes sense. But surely it is reasonable
for a process to have an idea of how much memory it has actually used.
Yes? No?

If the memory use looks suspicious,
use standard debugging techniques to locate the problematic area, and
standard benchmarking techniques to look for unexpected blowups and
leaks.

What sort of "standard debugging techniques" work in the absence of any
way (that I know of) to measure memory usage? In Python, standard
debugging techniques usually start with the print statement, but one
can't do anything like this:

# problematic area of code
last = memory()
for i in xrange(100):
x = foo()
if memory() >= last:
print "memory use increased", memory()


So what are you suggesting is standard?

For really hard problems, use the "gc" module, instrumenting memory
allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

I'm not so much concerned about the really hard problems as I am about
the really easy ones.
 
F

Fredrik Lundh

Steven said:

(hmm. why is that whenever you make some silly last-second addition to a
post, you end up making a stupid typo?)
Maybe the memory model used in modern multi-tasking virtual-memory PCs
makes the concept of "free memory" obsolete. Although if so, somebody
should have a quiet word with the author of the Linux free command:

$ free
total used free shared buffers cached
Mem: 1002524 988736 13788 0 7044 98916
-/+ buffers/cache: 882776 119748
Swap: 4241080 3939736 301344

(Admittedly that's system-wide memory usage, rather than for a single
process.)

the problem isn't that you can get a number, it's that the number you
get has no *direct* correlation to the amount of memory your process can
actually allocate -- or at least allocate without causing excessive
swapping or triggering the OOM killer or otherwise annoying all the
other processes on the machine.
Are you saying that Python programs can't monitor their own memory use?

I'm happy to accept that "free memory" is not a meaningful concept for a
process in a modern system. That makes sense. But surely it is reasonable
for a process to have an idea of how much memory it has actually used.
Yes? No?

unless you do utterly trivial things, a process allocates memory from a
lot of different resource pools (i/o buffers, virtual memory buffers,
network buffers, graphics resources, etc). there's simply no way for
the process itself to know how much memory it uses. if you want to
know, ask the operating system.
What sort of "standard debugging techniques" work in the absence of any
way (that I know of) to measure memory usage?

as I said, use standard process monitoring tools (i.e. "ps" and "top"
etc on Unix, the task manager on Windows) takes you a long way.
> In Python, standard
debugging techniques usually start with the print statement
> but one can't do anything like this:

# problematic area of code
last = memory()
for i in xrange(100):
x = foo()
if memory() >= last:
print "memory use increased", memory()

it doesn't have to be harder than this:

# problematic area of code
raw_input("before: check memory here")
for i in xrange(100):
x = foo()
raw_input("after: check memory here")

on unix, you can also do e.g. os.system("ps ux") or os.system("ps -F -p
%d" % os.getpid()) or some variation thereof.
So what are you suggesting is standard?

running individual modules/classes/functions under test harnesses, so
you can analyze their behaviour under well-known conditions.

</F>
 
I

Ilias Lazaridis

For Python, standard process monitoring tools (combined with a basic
understanding of how dynamic memory allocation works on modern
platforms) are usually sufficient to get a good view of an application's
memory usage patterns.
[...]

which "standard process monitoring tools" are those?

Aren't there any python specific tools (e.g. which list the python
modules which use the memory)?

.

http://case.lazaridis.com/wiki/PythonAudit
 
D

Dennis Lee Bieber

So... how do you measure memory usage in Python? Every programming
language I've used before (not a huge range, I'll admit) had *some* sort
of facility to measure memory usage, typically things like:

* how much memory is free in the stack?

* how much memory is free in the heap?

* how big a block does this pointer point to?

* how much memory does this record/struct/object/string use?
Well, in my 30 years with various languages -- FORTRAN (IV under
CP/V, 77 & 90 under VMS), COBOL, a smidgen of APL, a pinch of LISP
(TRS-80!), Pascal (UCSD, Alcor, DEC), Ada, C/C++ (TRSDOS, Amiga, DEC,
Windows), Python, VB6, (A)Rexx -- I've never encountered such items
supported by the language.

OS specific extensions MIGHT supply it... For example, if one can
obtain the size of the stack allocated to the process, and then the
address of the current top-of-stack, computing the free stack space is a
simple operation.

Determining free heap space is not as easy, unless it either does
free-space compression (the old Macintosh double pointer handle scheme,
where it could move the end-object to compress heap, by only updating
the pointer in the handle structure which doesn't get moved).

"how big a block"... On the Amiga, I believe that was available by
playing games -- negative indexing from the start of the malloc'd memory
would get one into a "block header" that defined the length of the
block; used when chaining the linked list of heap space upon free (if
block A ends where block B began, and both are free, combine into a
single block AB)

As for generic object size... Typically by reading the detailed
language manual, and then hand calculations.

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 
S

Steven D'Aprano

I've never encountered such items
supported by the language.

OS specific extensions MIGHT supply it...

Picky picky... but of course you are right. When I said that programming
languages I have used before had facilities to measure memory usage, I
meant that the *implementation* had those facilities rather than the
language itself.
 
I

Ilias Lazaridis

Picky picky... but of course you are right. When I said that programming
languages I have used before had facilities to measure memory usage, I
meant that the *implementation* had those facilities rather than the
language itself.


yes, that's what this thread is about.

"
I'm really just looking for a low-level python (memory) profiling
tool, and thus i'm asking in c.l.p. for experiences, mainly in context
of an apache mod_python environment.
"

But seeing the responses within this thread, it looks like there's no
such tool available.

..

http://case.lazaridis.com/wiki/PythonAudit
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,733
Messages
2,569,439
Members
44,829
Latest member
PIXThurman

Latest Threads

Top