Detecting memory leaks on apache, mod_python

Discussion in 'Python' started by Ilias Lazaridis, Dec 17, 2007.

  1. How to detect memory leaks of python programms, which run in an
    environment like this:

    * Suse Linux 9.3
    * Apache
    * mod_python

    The problem occoured after some updates on the infrastructure. It's
    most possibly caused by trac and it's dependencies, but several
    components of the OS where updated, too.

    Any nice tools which play with the above constellation?

    Thank's for any hints!

    context:

    http://dev.lazaridis.com/base/ticket/148
     
    Ilias Lazaridis, Dec 17, 2007
    #1
    1. Advertising

  2. On Dec 17, 8:41 am, Ilias Lazaridis <> wrote:
    > How to detect memory leaks of python programms, which run in an
    > environment like this:
    >
    > * Suse Linux 9.3
    > * Apache
    > * mod_python
    >
    > The problem occoured after some updates on the infrastructure. It's
    > most possibly caused by trac and it's dependencies, but several
    > components of the OS where updated, too.
    >
    > Any nice tools which play with the above constellation?
    >
    > Thank's for any hints!


    No tool available to detect memory leaks for python applications?

    > context:
    >
    > http://dev.lazaridis.com/base/ticket/148
     
    Ilias Lazaridis, Dec 19, 2007
    #2
    1. Advertising

  3. On Dec 19, 5:40 am, Ilias Lazaridis <> wrote:
    > On Dec 17, 8:41 am, Ilias Lazaridis <> wrote:
    >
    > > How to detect memory leaks of python programms, which run in an
    > > environment like this:

    >
    > > * Suse Linux 9.3
    > > * Apache
    > > * mod_python

    >
    > > The problem occoured after some updates on the infrastructure. It's
    > > most possibly caused by trac and it's dependencies, but several
    > > components of the OS where updated, too.

    >
    > > Any nice tools which play with the above constellation?

    >
    > > Thank's for any hints!

    >
    > No tool available to detect memory leaks for python applications?


    So, anyone who hit's on this thread via a search will think

    a) that there's really no memory leak detection for python
    b) that this community is not very helpful

    >
    > > context:

    >
    > >http://dev.lazaridis.com/base/ticket/148
     
    Ilias Lazaridis, Dec 21, 2007
    #3
  4. On Dec 21, 7:42 pm, Ilias Lazaridis <> wrote:
    > On Dec 19, 5:40 am, Ilias Lazaridis <> wrote:
    >
    >
    >
    > > On Dec 17, 8:41 am, Ilias Lazaridis <> wrote:

    >
    > > > How to detect memory leaks of python programms, which run in an
    > > > environment like this:

    >
    > > >  * Suse Linux 9.3
    > > >  * Apache
    > > >  *mod_python

    >
    > > > The problem occoured after some updates on the infrastructure. It's
    > > > most possibly caused by trac and it's dependencies, but several
    > > > components of the OS where updated, too.

    >
    > > > Any nice tools which play with the above constellation?

    >
    > > > Thank's for any hints!

    >
    > > No tool available to detect memory leaks for python applications?

    >
    > So, anyone who hit's on this thread via a search will think
    >
    > a) that there's really no memory leak detection for python
    > b) that this community is not very helpful


    Comments like (b) will not help your chances one bit in respect of
    getting an answer from anyone.

    Maybe you should read:

    http://www.catb.org/~esr/faqs/smart-questions.html

    Two things to note in here. First, choose your forum appropriately and
    secondly show some courtesy rather than making accusations against the
    community if no one answers.

    If you want to see perhaps how you might be viewed by the open source
    community when you make such a comment, perhaps also watch:

    http://video.google.com/videoplay?docid=-4216011961522818645

    Now, since you think this is a Trac problem, why don't you go ask on
    the Trac mailing list.

    http://groups.google.com/group/trac-users?lnk=li

    Even a search of that forum will most likely yield previous
    discussions about growing memory use of Trac as a result of things
    like Python wrappers for Subversion or certain database adapters. It
    may be a case of using a different version, or in some cases
    configuration of your hosting environment, if using Apache, to recycle
    Apache child processes after a set number of requests so as to restore
    process sizes back to a low level.

    So, do some research first in the correct places and then perhaps ask
    any additional questions in the correct place also.

    Graham
     
    Graham Dumpleton, Dec 21, 2007
    #4
  5. >> So, anyone who hit's on this thread via a search will think
    >>
    >> a) that there's really no memory leak detection for python
    >> b) that this community is not very helpful


    c) That finally people in this forum are smart enough to detect your
    flamebait & refuse to comment on it, Ilias...


    > Comments like (b) will not help your chances one bit in respect of
    > getting an answer from anyone.
    >
    > Maybe you should read:
    >
    > http://www.catb.org/~esr/faqs/smart-questions.html
    >
    > Two things to note in here. First, choose your forum appropriately and
    > secondly show some courtesy rather than making accusations against the
    > community if no one answers.
    >
    > If you want to see perhaps how you might be viewed by the open source
    > community when you make such a comment, perhaps also watch:
    >
    > http://video.google.com/videoplay?docid=-4216011961522818645
    >
    > Now, since you think this is a Trac problem, why don't you go ask on
    > the Trac mailing list.
    >
    > http://groups.google.com/group/trac-users?lnk=li
    >
    > Even a search of that forum will most likely yield previous
    > discussions about growing memory use of Trac as a result of things
    > like Python wrappers for Subversion or certain database adapters. It
    > may be a case of using a different version, or in some cases
    > configuration of your hosting environment, if using Apache, to recycle
    > Apache child processes after a set number of requests so as to restore
    > process sizes back to a low level.
    >
    > So, do some research first in the correct places and then perhaps ask
    > any additional questions in the correct place also.


    Graham, I'm not sure how long you have been around here - but googling
    for Ilias in general and specificly in this group will reveal that he
    often posts spiteful & inflammatory messages and has a tendency to get
    cross with open source projects. It's debatable if he can be called a
    troll - but he surely is a nuisance best let alone, so he doesn't gain
    traction.

    Diez
     
    Diez B. Roggisch, Dec 21, 2007
    #5
  6. On Dec 21, 12:25 pm, Graham Dumpleton <>
    wrote:
    > On Dec 21, 7:42 pm, Ilias Lazaridis <> wrote:
    > > On Dec 19, 5:40 am, Ilias Lazaridis <> wrote:
    > > > On Dec 17, 8:41 am, Ilias Lazaridis <> wrote:
    > > > > How to detect memory leaks of python programms, which run in an
    > > > > environment like this:

    >
    > > > > * Suse Linux 9.3
    > > > > * Apache
    > > > > *mod_python

    >
    > > > > The problem occoured after some updates on the infrastructure. It's
    > > > > most possibly caused by trac and it's dependencies, but several
    > > > > components of the OS where updated, too.

    >
    > > > > Any nice tools which play with the above constellation?

    >
    > > > > Thank's for any hints!

    >
    > > > No tool available to detect memory leaks for python applications?

    >
    > > So, anyone who hit's on this thread via a search will think

    >
    > > a) that there's really no memory leak detection for python
    > > b) that this community is not very helpful

    >
    > Comments like (b) will not help your chances one bit in respect of
    > getting an answer from anyone.

    [...]

    I'm really just looking for a low-level python (memory) profiling
    tool, and thus i'm asking in c.l.p. for experiences, mainly in context
    of an apache mod_python environment.

    As to Open Source etc.:

    http://case.lazaridis.com/wiki/PythonAudit
    http://case.lazaridis.com/wiki/CoreLiveEval

    Than you for your reply.

    ..
     
    Ilias Lazaridis, Dec 21, 2007
    #6
  7. On Dec 21, 3:21 pm, "Diez B. Roggisch" <> wrote:
    [...]

    Please get serious, Mr.!

    (and avoid further off-topics)

    ..
     
    Ilias Lazaridis, Dec 21, 2007
    #7
  8. On Fri, 21 Dec 2007 14:21:27 +0100, Diez B. Roggisch wrote:

    >>> So, anyone who hit's on this thread via a search will think
    >>>
    >>> a) that there's really no memory leak detection for python b) that
    >>> this community is not very helpful

    >
    > c) That finally people in this forum are smart enough to detect your
    > flamebait & refuse to comment on it, Ilias...


    Not me. I didn't (still don't actually) recognise Ilias' name. I was
    actually thinking:

    (d) Python is so great that there can never be any memory leaks!

    but I guess that's probably wishful thinking.

    So... how do you measure memory usage in Python? Every programming
    language I've used before (not a huge range, I'll admit) had *some* sort
    of facility to measure memory usage, typically things like:

    * how much memory is free in the stack?

    * how much memory is free in the heap?

    * how big a block does this pointer point to?

    * how much memory does this record/struct/object/string use?

    Unless I've missed something, I don't believe Python does *any* of that.
    It could be quite useful. The timeit module is good for measuring the
    time taken for code to run, but as near as I can tell, there's no way to
    optimize memory use except prematurely.



    --
    Steven.
     
    Steven D'Aprano, Dec 22, 2007
    #8
  9. Steven D'Aprano wrote:

    > Not me.


    You're quite knew to this internet thing, aren't you? ;-)

    > So... how do you measure memory usage in Python? Every programming
    > language I've used before (not a huge range, I'll admit) had *some* sort
    > of facility to measure memory usage, typically things like:
    >
    > * how much memory is free in the stack?
    >
    > * how much memory is free in the heap?
    >
    > * how big a block does this pointer point to?
    >
    > * how much memory does this record/struct/object/string use?


    And what languages would that be? I cannot think of a single modern
    language that does any of that. Including low-level stuff like C/C++.

    And things like "how much memory is free in the heap" isn't even a
    meaningful concept on a modern machine, thanks to the wonders of virtual
    memory (especially the overcommitting kind).

    For Python, standard process monitoring tools (combined with a basic
    understanding of how dynamic memory allocation works on modern
    platforms) are usually sufficient to get a good view of an application's
    memory usage patterns. Just run the program under a few different
    scenarios, and see what it does. If the memory use looks suspicious,
    use standard debugging techniques to locate the problematic area, and
    standard benchmarking techniques to look for unexpected blowups and leaks.

    For really hard problems, use the "gc" module, instrumenting memory
    allocation libraries (e.g. dmalloc), or C/C++-level debuggers.

    </F>
     
    Fredrik Lundh, Dec 22, 2007
    #9
  10. On Sat, 22 Dec 2007 08:09:50 +0100, Fredrik Lundh wrote:

    > Steven D'Aprano wrote:
    >
    > > Not me.

    >
    > You're quite knew to this internet thing, aren't you? ;-)


    :-D


    >> So... how do you measure memory usage in Python? Every programming
    >> language I've used before (not a huge range, I'll admit) had *some*
    >> sort of facility to measure memory usage, typically things like:
    >>
    >> * how much memory is free in the stack?
    >>
    >> * how much memory is free in the heap?
    >>
    >> * how big a block does this pointer point to?
    >>
    >> * how much memory does this record/struct/object/string use?

    >
    > And what languages would that be? I cannot think of a single modern
    > language that does any of that. Including low-level stuff like C/C++.


    I didn't actually say they were *modern* languages. E.g. THINK Pascal for
    Apple Mac, circa 1990.


    > And things like "how much memory is free in the heap" isn't even a
    > meaningful concept on a modern machine, thanks to the wonders of virtual
    > memory (especially the overcommitting kind).


    Maybe the memory model used in modern multi-tasking virtual-memory PCs
    makes the concept of "free memory" obsolete. Although if so, somebody
    should have a quiet word with the author of the Linux free command:

    $ free
    total used free shared buffers cached
    Mem: 1002524 988736 13788 0 7044 98916
    -/+ buffers/cache: 882776 119748
    Swap: 4241080 3939736 301344

    (Admittedly that's system-wide memory usage, rather than for a single
    process.)



    > For Python, standard process monitoring tools (combined with a basic
    > understanding of how dynamic memory allocation works on modern
    > platforms) are usually sufficient to get a good view of an application's
    > memory usage patterns. Just run the program under a few different
    > scenarios, and see what it does.


    Are you saying that Python programs can't monitor their own memory use?

    I'm happy to accept that "free memory" is not a meaningful concept for a
    process in a modern system. That makes sense. But surely it is reasonable
    for a process to have an idea of how much memory it has actually used.
    Yes? No?


    > If the memory use looks suspicious,
    > use standard debugging techniques to locate the problematic area, and
    > standard benchmarking techniques to look for unexpected blowups and
    > leaks.


    What sort of "standard debugging techniques" work in the absence of any
    way (that I know of) to measure memory usage? In Python, standard
    debugging techniques usually start with the print statement, but one
    can't do anything like this:

    # problematic area of code
    last = memory()
    for i in xrange(100):
    x = foo()
    if memory() >= last:
    print "memory use increased", memory()


    So what are you suggesting is standard?


    > For really hard problems, use the "gc" module, instrumenting memory
    > allocation libraries (e.g. dmalloc), or C/C++-level debuggers.


    I'm not so much concerned about the really hard problems as I am about
    the really easy ones.



    --
    Steven
     
    Steven D'Aprano, Dec 22, 2007
    #10
  11. Steven D'Aprano wrote:

    >> > Not me.

    >>
    >> You're quite knew to this internet thing, aren't you? ;-)

    >
    > :-D


    (hmm. why is that whenever you make some silly last-second addition to a
    post, you end up making a stupid typo?)

    >> And things like "how much memory is free in the heap" isn't even a
    >> meaningful concept on a modern machine, thanks to the wonders of virtual
    >> memory (especially the overcommitting kind).

    >
    > Maybe the memory model used in modern multi-tasking virtual-memory PCs
    > makes the concept of "free memory" obsolete. Although if so, somebody
    > should have a quiet word with the author of the Linux free command:
    >
    > $ free
    > total used free shared buffers cached
    > Mem: 1002524 988736 13788 0 7044 98916
    > -/+ buffers/cache: 882776 119748
    > Swap: 4241080 3939736 301344
    >
    > (Admittedly that's system-wide memory usage, rather than for a single
    > process.)


    the problem isn't that you can get a number, it's that the number you
    get has no *direct* correlation to the amount of memory your process can
    actually allocate -- or at least allocate without causing excessive
    swapping or triggering the OOM killer or otherwise annoying all the
    other processes on the machine.

    >> For Python, standard process monitoring tools (combined with a basic
    >> understanding of how dynamic memory allocation works on modern
    >> platforms) are usually sufficient to get a good view of an application's
    >> memory usage patterns. Just run the program under a few different
    >> scenarios, and see what it does.

    >
    > Are you saying that Python programs can't monitor their own memory use?
    >
    > I'm happy to accept that "free memory" is not a meaningful concept for a
    > process in a modern system. That makes sense. But surely it is reasonable
    > for a process to have an idea of how much memory it has actually used.
    > Yes? No?


    unless you do utterly trivial things, a process allocates memory from a
    lot of different resource pools (i/o buffers, virtual memory buffers,
    network buffers, graphics resources, etc). there's simply no way for
    the process itself to know how much memory it uses. if you want to
    know, ask the operating system.

    >> If the memory use looks suspicious,
    >> use standard debugging techniques to locate the problematic area, and
    >> standard benchmarking techniques to look for unexpected blowups and
    >> leaks.

    >
    > What sort of "standard debugging techniques" work in the absence of any
    > way (that I know of) to measure memory usage?


    as I said, use standard process monitoring tools (i.e. "ps" and "top"
    etc on Unix, the task manager on Windows) takes you a long way.

    > In Python, standard
    > debugging techniques usually start with the print statement
    > but one can't do anything like this:
    >
    > # problematic area of code
    > last = memory()
    > for i in xrange(100):
    > x = foo()
    > if memory() >= last:
    > print "memory use increased", memory()


    it doesn't have to be harder than this:

    # problematic area of code
    raw_input("before: check memory here")
    for i in xrange(100):
    x = foo()
    raw_input("after: check memory here")

    on unix, you can also do e.g. os.system("ps ux") or os.system("ps -F -p
    %d" % os.getpid()) or some variation thereof.

    > So what are you suggesting is standard?


    running individual modules/classes/functions under test harnesses, so
    you can analyze their behaviour under well-known conditions.

    </F>
     
    Fredrik Lundh, Dec 22, 2007
    #11
  12. On 22 Äåê, 09:09, Fredrik Lundh <> wrote:
    [...]
    > For Python, standard process monitoring tools (combined with a basic
    > understanding of how dynamic memory allocation works on modern
    > platforms) are usually sufficient to get a good view of an application's
    > memory usage patterns.

    [...]

    which "standard process monitoring tools" are those?

    Aren't there any python specific tools (e.g. which list the python
    modules which use the memory)?

    .

    http://case.lazaridis.com/wiki/PythonAudit
     
    Ilias Lazaridis, Dec 22, 2007
    #12
  13. On Sat, 22 Dec 2007 04:51:43 -0000, Steven D'Aprano
    <> declaimed the following in
    comp.lang.python:

    >
    > So... how do you measure memory usage in Python? Every programming
    > language I've used before (not a huge range, I'll admit) had *some* sort
    > of facility to measure memory usage, typically things like:
    >
    > * how much memory is free in the stack?
    >
    > * how much memory is free in the heap?
    >
    > * how big a block does this pointer point to?
    >
    > * how much memory does this record/struct/object/string use?
    >

    Well, in my 30 years with various languages -- FORTRAN (IV under
    CP/V, 77 & 90 under VMS), COBOL, a smidgen of APL, a pinch of LISP
    (TRS-80!), Pascal (UCSD, Alcor, DEC), Ada, C/C++ (TRSDOS, Amiga, DEC,
    Windows), Python, VB6, (A)Rexx -- I've never encountered such items
    supported by the language.

    OS specific extensions MIGHT supply it... For example, if one can
    obtain the size of the stack allocated to the process, and then the
    address of the current top-of-stack, computing the free stack space is a
    simple operation.

    Determining free heap space is not as easy, unless it either does
    free-space compression (the old Macintosh double pointer handle scheme,
    where it could move the end-object to compress heap, by only updating
    the pointer in the handle structure which doesn't get moved).

    "how big a block"... On the Amiga, I believe that was available by
    playing games -- negative indexing from the start of the malloc'd memory
    would get one into a "block header" that defined the length of the
    block; used when chaining the linked list of heap space upon free (if
    block A ends where block B began, and both are free, combine into a
    single block AB)

    As for generic object size... Typically by reading the detailed
    language manual, and then hand calculations.

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Dec 22, 2007
    #13
  14. On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote:

    > I've never encountered such items
    > supported by the language.
    >
    > OS specific extensions MIGHT supply it...


    Picky picky... but of course you are right. When I said that programming
    languages I have used before had facilities to measure memory usage, I
    meant that the *implementation* had those facilities rather than the
    language itself.


    --
    Steven
     
    Steven D'Aprano, Dec 23, 2007
    #14
  15. On Dec 23, 2:47 am, Steven D'Aprano <st...@REMOVE-THIS-
    cybersource.com.au> wrote:
    > On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote:
    > > I've never encountered such items
    > > supported by the language.

    >
    > > OS specific extensions MIGHT supply it...

    >
    > Picky picky... but of course you are right. When I said that programming
    > languages I have used before had facilities to measure memory usage, I
    > meant that the *implementation* had those facilities rather than the
    > language itself.



    yes, that's what this thread is about.

    "
    I'm really just looking for a low-level python (memory) profiling
    tool, and thus i'm asking in c.l.p. for experiences, mainly in context
    of an apache mod_python environment.
    "

    But seeing the responses within this thread, it looks like there's no
    such tool available.

    ..

    http://case.lazaridis.com/wiki/PythonAudit
     
    Ilias Lazaridis, Dec 23, 2007
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. ASP.Confused

    ASP.NET - Detecting memory leaks

    ASP.Confused, Jul 16, 2004, in forum: ASP .Net
    Replies:
    2
    Views:
    2,749
    Marina
    Jul 16, 2004
  2. Jive

    Detecting memory leaks

    Jive, Nov 21, 2004, in forum: Python
    Replies:
    2
    Views:
    382
    Eddie Parker
    Nov 21, 2004
  3. Diego Martins
    Replies:
    4
    Views:
    320
    Diego Martins
    Oct 9, 2006
  4. Jerome Sui

    detecting memory leaks in Perl

    Jerome Sui, Jul 13, 2003, in forum: Perl Misc
    Replies:
    0
    Views:
    133
    Jerome Sui
    Jul 13, 2003
  5. Replies:
    5
    Views:
    1,021
Loading...

Share This Page