Memory Leak

Discussion in 'Python' started by Guy, Feb 24, 2004.

  1. Guy

    Guy Guest

    Hi

    It might take me a little time to explain this but here goes.
    Firstly I'm not using the latest upto date python releases so my first
    plan is to try more upto date rels of python and win32 libs, as this
    has fixed problems for me in the past (Don't like doing this as it
    usally breaks stuff and sys admin are slow to update software,
    although they do a cracking job, just covering my back.)

    I've created a build script for my companies product, the product is
    in c and can be built using makefiles and gmake and dsps if using
    win32. The script seems to work ok, it builds by setting processes off
    and then it returns the built libs exe to a network area.

    The problems occure when I build more than one flavour of the product,
    (there are 16 possible flavours and about 50 modules which get turned
    into libs and exe's this means theres 800 processes to be set off one
    after the other or if using a clean build 1600 process and thats just
    for win32, to build one flavour it takes about 40 mins on my comp.)

    When using the build script it keeps logs of what its built and
    records all output from the process all these python objs and lists
    are destroyed by using del to try and free up memory, this never seems
    to have that much effect the memory just keeps getting used up until
    usally the script or python crashes due to the lack of mem.

    Sometimes for no reason the mem that python uses seem to purge itself
    and start a fresh, I don't know why this is or how to control it.

    I also don't think the del is working, not sure.

    The script itself is made up of my code, python, python win32 libs and
    a 3rd party module off the internet which controls the processes
    (which has been of great help)

    I just wondering if anybody can give me any ideas about whats going on
    here, or maybe some memory purge commands or memory dianosics so I can
    see which python objs are using all the mem.

    TIA
    Guy
     
    Guy, Feb 24, 2004
    #1
    1. Advertising

  2. Guy

    Peter Hansen Guest

    Guy wrote:
    >
    > It might take me a little time to explain this but here goes.


    Thanks for the details you did provide; it's more than most manage!

    > Firstly I'm not using the latest upto date python releases so my first
    > plan is to try more upto date rels of python and win32 libs, as this
    > has fixed problems for me in the past (Don't like doing this as it
    > usally breaks stuff and sys admin are slow to update software,
    > although they do a cracking job, just covering my back.)


    Definitely do that. There have been memory leaks in past versions
    (and possibly are in the latest too, but it's perhaps less likely unless
    you are using some of the newest features, which you obviously aren't).

    I'd even suggest that doing much before you try that would be a waste.

    > The problems occure when I build more than one flavour of the product,
    > (there are 16 possible flavours and about 50 modules which get turned
    > into libs and exe's this means theres 800 processes to be set off one
    > after the other or if using a clean build 1600 process and thats just
    > for win32, to build one flavour it takes about 40 mins on my comp.)
    >
    > When using the build script it keeps logs of what its built and
    > records all output from the process all these python objs and lists
    > are destroyed by using del to try and free up memory, this never seems
    > to have that much effect the memory just keeps getting used up until
    > usally the script or python crashes due to the lack of mem.


    Memory leaks are extremely difficult to debug, perhaps more difficult
    than any other problem except race conditions.

    Thankfully, real memory leaks are very rare. I don't think you've
    described enough to prove that you actually have a memory leak. What
    you are describing sounds more like you're just using up all the
    resources too quickly, which is more of a capacity (or design) problem
    than anything else.

    More importantly, you've got way too many variables involved to be
    able to troubleshoot this, if it's really a memory leak. You absolutely
    will have to simplify the setup, while keeping the problem occurring,
    in order to solve this issue. If this is a memory leak involving Python,
    you will be able to reproduce it with a program that involves nothing
    but Python, and perhaps a small external dummy program that you launch
    repeatedly in the other processes.

    You shouldn't have to bother with "del" at all, since all that does is
    release the name from the object it's bound to (or vice versa) and if
    there are no other names bound to the object, Python will free the memory
    anyway. (Note that it might not return the memory to the OS, however.
    You can't force that to happen, either.) Doing a "del" followed by
    reassigning the name to a new object is basically a waste of time.

    Consider this. From the looks of things, you are doing so much that
    is actually *outside* of the scope of Python that you might simply
    be running out of memory elsewhere.

    Also consider whether this might be due to memory fragmentation rather
    than true leaking. You might need to analyze your pattern of memory
    usage to see whether this could be the case.

    -Peter
     
    Peter Hansen, Feb 24, 2004
    #2
    1. Advertising

  3. In article <>,
    Peter Hansen <> wrote:
    .
    .
    .
    >Memory leaks are extremely difficult to debug, perhaps more difficult
    >than any other problem except race conditions.
    >
    >Thankfully, real memory leaks are very rare. I don't think you've

    .
    [LOTS more sound advice]
    .
    .
    Mostly, I just want to repeat Peter's sound counsel.

    There's a LOT to say about memory errors; in fact, I'm even
    trying to introduce the subject to C and C++ programmers <URL:
    http://informit.com/isapi/product_id~{51BA92A2-20C7-4142-8707-FA7140CE4275}/session_id~{549AF63E-92F6-4EC1-B6B7-6A9147DF259E}/content/index.asp >
    I'll pull out a few points present in Peter's description that
    I think summarize the right attitude:
    1. Look to yourself. Until you have strong evidence
    to the contrary, assume that apparent memory pro-
    blems are under your control, and are NOT an
    intrinsic fault of Python.
    2. The biggest return-on-investment you can reap is
    to upgrade your version.
    3. Next is to rationalize your own coding.
    4. Next is to inspect code for memory faults. A
    good developer can certify source, without re-
    course to external tools. It's possible, even
    practical, to learn all the basic categories
    of memory error and be alert to their appear-
    ance.
    --

    Cameron Laird <>
    Business: http://www.Phaseit.net
     
    Cameron Laird, Feb 24, 2004
    #3
  4. (Guy) writes:

    > Hi
    >
    > It might take me a little time to explain this but here goes.
    > Firstly I'm not using the latest upto date python releases so my first
    > plan is to try more upto date rels of python and win32 libs, as this
    > has fixed problems for me in the past (Don't like doing this as it
    > usally breaks stuff and sys admin are slow to update software,
    > although they do a cracking job, just covering my back.)
    >
    > I've created a build script for my companies product, the product is
    > in c and can be built using makefiles and gmake and dsps if using
    > win32. The script seems to work ok, it builds by setting processes off
    > and then it returns the built libs exe to a network area.
    >
    > The problems occure when I build more than one flavour of the product,
    > (there are 16 possible flavours and about 50 modules which get turned
    > into libs and exe's this means theres 800 processes to be set off one
    > after the other or if using a clean build 1600 process and thats just
    > for win32, to build one flavour it takes about 40 mins on my comp.)
    >
    > When using the build script it keeps logs of what its built and
    > records all output from the process all these python objs and lists
    > are destroyed by using del to try and free up memory, this never seems
    > to have that much effect the memory just keeps getting used up until
    > usally the script or python crashes due to the lack of mem.


    Um, I don't think you understand what 'del' does.

    For an oblique perspective on these issues, you might find reading the
    messages mentioned in

    http://starship.python.net/crew/mwh/hacks/pep310/

    enlightening, particularly my last one. Quoting:

    >> I guess I have a lispers approach to this: view memory as
    >> infinite and assume the implementation does enough behind its
    >> back to maintain this illusion so long as not too many objects
    >> are actually live.

    >
    > If you seperate the memory and object concepts, an object can
    > cease to exist even though memory is still allocated for it.


    Well, this begs another question that I guess underlies all of
    this: what does it mean for an object to be "alive"? Again, my
    background is functional/lisp-like languages, and I'm stealing
    this description more-or-less from Appel's "Modern Compiler
    Implementation in ML":

    An object is *live* if it will be used again by the executing
    program.

    At any given point of program execution, determining the set of
    live objects can obviously be at least as hard as the halting
    problem, so we settle for a weaker notion:

    An object is *reachable* if it can be reached by traversing the
    object graph from a given set of roots (in Python, things like
    locals and the globals of all loaded modules).

    The runtime pessimistically assumes that all reachable objects are
    live (and reachable but not live objects can be a cause of memory
    leaks in Python and other GCed languages).

    From this point of view, you never actually /destroy/ objects, you
    just forget about them, and a __del__ method is a side-effect of
    dropping the last reference to an object -- a slightly unnatural
    action to have a side effect!

    This is obviously in contrast to C++, where you /do/ destroy
    objects (or in the case of local variables, it is clear when the
    object will be destroyed as a function of program flow).

    I guess you are probably in the "reachable but not live" case of
    memory leaks.

    Instead of doing 'del alist' you might want to try 'del alist[:]' and
    see if behaviour improves.

    Cheers,
    mwh

    --
    112. Computer Science is embarrassed by the computer.
    -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html
     
    Michael Hudson, Feb 25, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?Y3liZXJzdHJpa2U=?=

    datagrid memory leak?

    =?Utf-8?B?Y3liZXJzdHJpa2U=?=, Jan 3, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    466
    =?Utf-8?B?Y3liZXJzdHJpa2U=?=
    Jan 3, 2005
  2. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    707
    Eric Sosman
    Mar 22, 2005
  3. Richard Heathfield

    Leak or no leak ??

    Richard Heathfield, Jul 10, 2006, in forum: C Programming
    Replies:
    4
    Views:
    354
    Richard Heathfield
    Jul 10, 2006
  4. cham
    Replies:
    5
    Views:
    772
  5. Mark Probert
    Replies:
    4
    Views:
    331
    Mark Probert
    Feb 9, 2005
Loading...

Share This Page