Memory Leak

[Guy]

Hi

It might take me a little time to explain this but here goes.
Firstly I'm not using the latest upto date python releases so my first
plan is to try more upto date rels of python and win32 libs, as this
has fixed problems for me in the past (Don't like doing this as it
usally breaks stuff and sys admin are slow to update software,
although they do a cracking job, just covering my back.)

I've created a build script for my companies product, the product is
in c and can be built using makefiles and gmake and dsps if using
win32. The script seems to work ok, it builds by setting processes off
and then it returns the built libs exe to a network area.

The problems occure when I build more than one flavour of the product,
(there are 16 possible flavours and about 50 modules which get turned
into libs and exe's this means theres 800 processes to be set off one
after the other or if using a clean build 1600 process and thats just
for win32, to build one flavour it takes about 40 mins on my comp.)

When using the build script it keeps logs of what its built and
records all output from the process all these python objs and lists
are destroyed by using del to try and free up memory, this never seems
to have that much effect the memory just keeps getting used up until
usally the script or python crashes due to the lack of mem.

Sometimes for no reason the mem that python uses seem to purge itself
and start a fresh, I don't know why this is or how to control it.

I also don't think the del is working, not sure.

The script itself is made up of my code, python, python win32 libs and
a 3rd party module off the internet which controls the processes
(which has been of great help)

I just wondering if anybody can give me any ideas about whats going on
here, or maybe some memory purge commands or memory dianosics so I can
see which python objs are using all the mem.

TIA
Guy

 

[Peter Hansen]

It might take me a little time to explain this but here goes.

Thanks for the details you did provide; it's more than most manage!

Firstly I'm not using the latest upto date python releases so my first
plan is to try more upto date rels of python and win32 libs, as this
has fixed problems for me in the past (Don't like doing this as it
usally breaks stuff and sys admin are slow to update software,
although they do a cracking job, just covering my back.)

Definitely do that. There have been memory leaks in past versions
(and possibly are in the latest too, but it's perhaps less likely unless
you are using some of the newest features, which you obviously aren't).

I'd even suggest that doing much before you try that would be a waste.

The problems occure when I build more than one flavour of the product,
(there are 16 possible flavours and about 50 modules which get turned
into libs and exe's this means theres 800 processes to be set off one
after the other or if using a clean build 1600 process and thats just
for win32, to build one flavour it takes about 40 mins on my comp.)

When using the build script it keeps logs of what its built and
records all output from the process all these python objs and lists
are destroyed by using del to try and free up memory, this never seems
to have that much effect the memory just keeps getting used up until
usally the script or python crashes due to the lack of mem.

Memory leaks are extremely difficult to debug, perhaps more difficult
than any other problem except race conditions.

Thankfully, real memory leaks are very rare. I don't think you've
described enough to prove that you actually have a memory leak. What
you are describing sounds more like you're just using up all the
resources too quickly, which is more of a capacity (or design) problem
than anything else.

More importantly, you've got way too many variables involved to be
able to troubleshoot this, if it's really a memory leak. You absolutely
will have to simplify the setup, while keeping the problem occurring,
in order to solve this issue. If this is a memory leak involving Python,
you will be able to reproduce it with a program that involves nothing
but Python, and perhaps a small external dummy program that you launch
repeatedly in the other processes.

You shouldn't have to bother with "del" at all, since all that does is
release the name from the object it's bound to (or vice versa) and if
there are no other names bound to the object, Python will free the memory
anyway. (Note that it might not return the memory to the OS, however.
You can't force that to happen, either.) Doing a "del" followed by
reassigning the name to a new object is basically a waste of time.

Consider this. From the looks of things, you are doing so much that
is actually *outside* of the scope of Python that you might simply
be running out of memory elsewhere.

Also consider whether this might be due to memory fragmentation rather
than true leaking. You might need to analyze your pattern of memory
usage to see whether this could be the case.

-Peter

 

[Cameron Laird]

.
.
.

Memory leaks are extremely difficult to debug, perhaps more difficult
than any other problem except race conditions.

Thankfully, real memory leaks are very rare. I don't think you've

.
[LOTS more sound advice]
.
.
Mostly, I just want to repeat Peter's sound counsel.

There's a LOT to say about memory errors; in fact, I'm even
trying to introduce the subject to C and C++ programmers <URL:
http://informit.com/isapi/product_id~{51BA92A2-20C7-4142-8707-FA7140CE4275}/session_id~{549AF63E-92F6-4EC1-B6B7-6A9147DF259E}/content/index.asp >
I'll pull out a few points present in Peter's description that
I think summarize the right attitude:
1. Look to yourself. Until you have strong evidence
to the contrary, assume that apparent memory pro-
blems are under your control, and are NOT an
intrinsic fault of Python.
2. The biggest return-on-investment you can reap is
to upgrade your version.
3. Next is to rationalize your own coding.
4. Next is to inspect code for memory faults. A
good developer can certify source, without re-
course to external tools. It's possible, even
practical, to learn all the basic categories
of memory error and be alert to their appear-
ance.

 

[Michael Hudson]

Hi

It might take me a little time to explain this but here goes.
Firstly I'm not using the latest upto date python releases so my first
plan is to try more upto date rels of python and win32 libs, as this
has fixed problems for me in the past (Don't like doing this as it
usally breaks stuff and sys admin are slow to update software,
although they do a cracking job, just covering my back.)

I've created a build script for my companies product, the product is
in c and can be built using makefiles and gmake and dsps if using
win32. The script seems to work ok, it builds by setting processes off
and then it returns the built libs exe to a network area.

The problems occure when I build more than one flavour of the product,
(there are 16 possible flavours and about 50 modules which get turned
into libs and exe's this means theres 800 processes to be set off one
after the other or if using a clean build 1600 process and thats just
for win32, to build one flavour it takes about 40 mins on my comp.)

When using the build script it keeps logs of what its built and
records all output from the process all these python objs and lists
are destroyed by using del to try and free up memory, this never seems
to have that much effect the memory just keeps getting used up until
usally the script or python crashes due to the lack of mem.

Um, I don't think you understand what 'del' does.

For an oblique perspective on these issues, you might find reading the
messages mentioned in

http://starship.python.net/crew/mwh/hacks/pep310/

enlightening, particularly my last one. Quoting:

>
> If you seperate the memory and object concepts, an object can
> cease to exist even though memory is still allocated for it.

Well, this begs another question that I guess underlies all of
this: what does it mean for an object to be "alive"? Again, my
background is functional/lisp-like languages, and I'm stealing
this description more-or-less from Appel's "Modern Compiler
Implementation in ML":

An object is *live* if it will be used again by the executing
program.

At any given point of program execution, determining the set of
live objects can obviously be at least as hard as the halting
problem, so we settle for a weaker notion:

An object is *reachable* if it can be reached by traversing the
object graph from a given set of roots (in Python, things like
locals and the globals of all loaded modules).

The runtime pessimistically assumes that all reachable objects are
live (and reachable but not live objects can be a cause of memory
leaks in Python and other GCed languages).

From this point of view, you never actually /destroy/ objects, you
just forget about them, and a __del__ method is a side-effect of
dropping the last reference to an object -- a slightly unnatural
action to have a side effect!

This is obviously in contrast to C++, where you /do/ destroy
objects (or in the case of local variables, it is clear when the
object will be destroyed as a function of program flow).

I guess you are probably in the "reachable but not live" case of
memory leaks.

Instead of doing 'del alist' you might want to try 'del alist[:]' and
see if behaviour improves.

Cheers,
mwh