Python not giving free memory back to the os get's me in real problems ...

[leuchte]

So I read quite a few things about this phenomenon in Python 2.4.x but
I can hardly believe that there is really no solution to my problem.

We use a commercial tool that has a macro functionality. These macros
are written in python. So far nothing extraordinary.

Our (python-)macro uses massively nested loops which are unfortunately
necessary. These loops perform complex calculations in this commercial
tool. To give you a quick overview how long this macros runs:

The outer loop takes 5-7 hours for one cycle. Each cycle creates one
outputfile. So we would like to perform 3-5 outer cycles en bloc.
Unfortunately one of our computers (768MB RAM) crashes after just ~10%
of the first cycle with the following error message:

http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg

while another computer (1GB RAM) crashes after ~10% of the fourth
loop. While the virtual memory on the 1gb machine was full to the
limit when it crashed the memory usage of the 768mb machine looked
this this:

http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg

The moment I close the application that launched the macro, my
ressources get freed.

So is there a way to free my memory inside my nested loops?

thanks in advance,
tim

 

[Larry Bates]

So I read quite a few things about this phenomenon in Python 2.4.x but
I can hardly believe that there is really no solution to my problem.

We use a commercial tool that has a macro functionality. These macros
are written in python. So far nothing extraordinary.

Our (python-)macro uses massively nested loops which are unfortunately
necessary. These loops perform complex calculations in this commercial
tool. To give you a quick overview how long this macros runs:

The outer loop takes 5-7 hours for one cycle. Each cycle creates one
outputfile. So we would like to perform 3-5 outer cycles en bloc.
Unfortunately one of our computers (768MB RAM) crashes after just ~10%
of the first cycle with the following error message:

http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg

while another computer (1GB RAM) crashes after ~10% of the fourth
loop. While the virtual memory on the 1gb machine was full to the
limit when it crashed the memory usage of the 768mb machine looked
this this:

http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg

The moment I close the application that launched the macro, my
ressources get freed.

So is there a way to free my memory inside my nested loops?

thanks in advance,
tim

Let's see for this I need to get out my crystal ball...

If it is a commercial application, you should contact their tech
support for a solution. The problem isn't specifically a Python
problem but rather an implementation problem with their app.

-Larry

 

[TimC]

Let's see for this I need to get out my crystal ball...

If it is a commercial application, you should contact their tech
support for a solution. The problem isn't specifically a Python
problem but rather an implementation problem with their app.

-Larry

Well one part of the problem maybe really is on the side of this
tool. Because the implementation of the macro functionality is
really crappy. But unfortunately there are no competitive-product
we could switch too, so we have to live with this poor environment.

But I don't think that the WHOLE problem lies inside this tool.
Because I read

http://mail.python.org/pipermail/python-dev/2005-January/051255.html
and
http://evanjones.ca/python-memory.html

Sadly we can't switch from Python 2.4 to 2.5 because this stupid
macro functionality only supports 2.4.x

I really know that all this is not the best setup to get happy with
but there really is no other setup available.

So I'm happy for every input all of you can give me.

greetings,
tim

 

[Chris Mellon]

Well one part of the problem maybe really is on the side of this
tool. Because the implementation of the macro functionality is
really crappy. But unfortunately there are no competitive-product
we could switch too, so we have to live with this poor environment.

But I don't think that the WHOLE problem lies inside this tool.
Because I read

http://mail.python.org/pipermail/python-dev/2005-January/051255.html
and
http://evanjones.ca/python-memory.html

Sadly we can't switch from Python 2.4 to 2.5 because this stupid
macro functionality only supports 2.4.x

I really know that all this is not the best setup to get happy with
but there really is no other setup available.

So I'm happy for every input all of you can give me.

The problem you're seeing almost certainly isn't related to the memory
arena issues referenced on the list.

"Returning memory to the OS" doesn't affect exhaustion of your virtual
address space. More likely, your nested loops are just creating more
objects than is possible to hold within a single process. That may be
because you're working with very large data sets, or because you have
some sort of ref cycle problem, or simply because you hold onto
references you no longer need. You'll need to do more work to diagnose
the exact details before you can find a fix. Contacting your vendor
support would be a really good first step.

It's possible that you have a degenerate case that triggers some of
the arena problems (like allocation 10 million integers, and then
never using them again), and thats the root of your problem. Without
knowing more about the usage pattern of your application, theres no
way we can determine that. If you can't upgrade to python 2.5, your
only solution is going to be to rework your algorithm to not do that.
Again, contacting your vendor support is going to be the first step.

 

[Grant Edwards]

"Returning memory to the OS" doesn't affect exhaustion of your virtual
address space. More likely, your nested loops are just creating more
objects than is possible to hold within a single process.

I'm a bit fuzzy on this, but I don't think there _is_ a
practical way to "return memory to the OS" in many OSes. For
example in Unix the C library uses the sbrk() call to increase
the size of the data segment when additional memory is needed
to handle soemthing like malloc() calls.

In theory, one can use brk() to reduce the size of the data
segment, but I think making the segment smaller will produce
undefined behavior if any of the standard C library's dynamic
memory routines (e.g. malloc/free) have ever been used by the
program prior to the call to brk().

 

[Steven Howe]

I'm a bit fuzzy on this, but I don't think there _is_ a
practical way to "return memory to the OS" in many OSes. For
example in Unix the C library uses the sbrk() call to increase
the size of the data segment when additional memory is needed
to handle soemthing like malloc() calls.

In theory, one can use brk() to reduce the size of the data
segment, but I think making the segment smaller will produce
undefined behavior if any of the standard C library's dynamic
memory routines (e.g. malloc/free) have ever been used by the
program prior to the call to brk().

Interesting questions. What happens when an object is 'cleaned' up by
using the 'del' command. Does the memory space stay in the python
process, get handed back to the OS, or some combination of both?
I remember 'C' on VMS at least, could be coerced into return memory on
block boundaries. 'C++' was suppose to have garbage collect, but I was
always doubtful it worked well.

sph

 

[Donald 'Paddy' McCarthy]

So I read quite a few things about this phenomenon in Python 2.4.x but
I can hardly believe that there is really no solution to my problem.

We use a commercial tool that has a macro functionality. These macros
are written in python. So far nothing extraordinary.

Our (python-)macro uses massively nested loops which are unfortunately
necessary. These loops perform complex calculations in this commercial
tool. To give you a quick overview how long this macros runs:

The outer loop takes 5-7 hours for one cycle. Each cycle creates one
outputfile. So we would like to perform 3-5 outer cycles en bloc.
Unfortunately one of our computers (768MB RAM) crashes after just ~10%
of the first cycle with the following error message:

http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg

while another computer (1GB RAM) crashes after ~10% of the fourth
loop. While the virtual memory on the 1gb machine was full to the
limit when it crashed the memory usage of the 768mb machine looked
this this:

http://img2.freeimagehosting.net/uploads/dd15127b7a.jpg

The moment I close the application that launched the macro, my
ressources get freed.

So is there a way to free my memory inside my nested loops?

thanks in advance,
tim

Could you split the program into one handling the outer loop and
calling another program, with data transfer, to handle the inner
loops?

- Paddy.

 

[Grant Edwards]

Interesting questions. What happens when an object is
'cleaned' up by using the 'del' command. Does the memory space
stay in the python process, get handed back to the OS, or some
combination of both?

Assuming the python interpreter free()s the memory, my
understanding is that on Unixes the memory is returned to the
pool used by malloc(), but is not returned to the OS since
there isn't a practical way to ensure that the memory at the
"end" of the data segment is not used.

In theory, you could walk the data structures used by
free/malloc and try to figure out if a free() should allow
brk() to be called to reduce the data segment size. That would
only happen if the free() call was freeing data that was at the
"end" of the data segment. It's possible that some standard C
libraries do that, but most of what I remember reading implies
that they don't. I don't really keep current on libc details,
so my info might be way out of date.

Asking on the gnu libc mailing list would probably provide a
more authoritative answer. I'd wager that they even know what
other non-Gnu libc implementations do.

I remember 'C' on VMS at least, could be coerced into return
memory on block boundaries.

You can call brk() in a C program to reduce the size of the
data segment, but that may result in free and malloc breakage
because you've returned (behind their back) some of the memory
they are managing.

 

[Donn Cave]

Interesting questions. What happens when an object is 'cleaned' up by
using the 'del' command. Does the memory space stay in the python
process, get handed back to the OS, or some combination of both?
I remember 'C' on VMS at least, could be coerced into return memory on
block boundaries. 'C++' was suppose to have garbage collect, but I was
always doubtful it worked well.

Note that UNIX (and VMS) use "virtual" memory. Real memory
space gets "handed back to the OS" by default -- if you don't
use it, you lose it. It isn't coercion, but it does happen on
a per page basis, so fragmentation wastes space.

If a Python program uses more space than it ought to need, then
some knowledge of Python's reference counting allocation scheme
will be useful. In particular, you need a fairly good grasp of
what a "reference" is.

Donn Cave, (e-mail address removed)

 

[TimC]

Could you split the program into one handling the outer loop and
calling another program, with data transfer, to handle the inner
loops?

- Paddy.

I'm afraid this isn't possible, because the python macro is called
and started from within our commercial tool which leads to the
cirumstance that this tool eats up more and more memory.

Due to the fact that there is no way we can start the tool via
commandline giving an argument that tells the tool it has to
start macroXY.py after startup I don't see a way to lower the
memory usage by splitting it up in two loops.

thanks for all your help so far,
tim

 

[Diez B. Roggisch]

I'm afraid this isn't possible, because the python macro is called
and started from within our commercial tool which leads to the
cirumstance that this tool eats up more and more memory.

Due to the fact that there is no way we can start the tool via
commandline giving an argument that tells the tool it has to
start macroXY.py after startup I don't see a way to lower the
memory usage by splitting it up in two loops.

Is the python macro crippled in any way - e.g. threading? Maybe
delegating the actual processing to an external python process you feed
through the macro might work.

Diez

 

[Charles Sanders]

Assuming the python interpreter free()s the memory, my
understanding is that on Unixes the memory is returned to the
pool used by malloc(), but is not returned to the OS since
there isn't a practical way to ensure that the memory at the
"end" of the data segment is not used.
http://www.dent.med.uni-muenchen.de/~wmglo/malloc-slides.html

> Large chunks are allocated via mmap(). The threshold is set
> through the environment variable MALLOC_MMAP_THRESHOLD_.
> The maximum number of chunks allocated in this way is limited
> through MALLOC_MMAP_MAX_.

I think these large chunks are returned to the OS when
freed.

> Trimming occurs when the size of the top chunk exceeds
> MALLOC_TRIM_THRESHOLD_.

I think this means that memory for small chunks
is returned to the OS if the free space at the top of the
heap exceeds the threshold.

Asking on the gnu libc mailing list would probably provide a
more authoritative answer. I'd wager that they even know what
other non-Gnu libc implementations do.

Agreed, they would be authoritative for GNU libc.

Charles

 

[Paul McGuire]

You might try looking at references between objects, especially if
there are any cyclic refs. For instance, if you have a data structure
in which child nodes have back refs to their parents, try changing
these to use weakref's. This may help the garbage collector to better
reclaim discarded objects, so that not so many will need to be held in
memory.

-- Paul

 

[TimC]

Maybe delegating the actual processing to an external python
process you feed through the macro might work.

Thanks for the idea. I'll check it out soon and will report
about possible improvements.

 

[=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=]

I'm a bit fuzzy on this, but I don't think there _is_ a

practical way to "return memory to the OS" in many OSes.

That's not true at all. Most C libraries these days manage
to return memory to the operating system.

On Win32 (which the OP is most likely to use), the
operating system offers the VirtualFree function to
release memory to the operating system. In MSVCRT,
the small block allocator will invoke VirtualFree
inside __sbh_free_block, when a block group becomes
completely free on the C level; likewise,
_heap_free_region invokes VirtualFree, so does
_heapmin_region.

In glibc 2 on Linux, public_fREe will invoke munmap
to release the memory to the operating system if the
chunk being freed was mmap'ed. glibc allocates chunks
greater than mmap_threshold with mmap, which defaults
to 128kiB - so Python's obmalloc arenas fall under
that algorithm. For small blocks, glibc releases
topmost memory through sbrk when it gets more than
M_TRIM_THRESHOLD bytes (defaults to 128kiB as well).
This algorithm isn't used when MORECORE_CANNOT_TRIM
is defined, which it would only on systems where
sbrk does not support negative arguments.

Regards,
Martin

 

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

The moment I close the application that launched the macro, my

ressources get freed.

So is there a way to free my memory inside my nested loops?

Yes. Most likely, it is your algorithm itself which is flawed
(not Python, not the application that embeds Python): You keep,
most likely, references to objects which you should release in
order to be able to reuse the memory.

Regards,
Martin

 

[Magnus Lycka]

Our (python-)macro uses massively nested loops which are unfortunately
necessary. These loops perform complex calculations in this commercial
tool. To give you a quick overview how long this macros runs:

The outer loop takes 5-7 hours for one cycle. Each cycle creates one
outputfile. So we would like to perform 3-5 outer cycles en bloc.
Unfortunately one of our computers (768MB RAM) crashes after just ~10%
of the first cycle with the following error message:

http://img2.freeimagehosting.net/uploads/7157b1dd7e.jpg

while another computer (1GB RAM) crashes after ~10% of the fourth
loop. While the virtual memory on the 1gb machine was full to the
limit when it crashed the memory usage of the 768mb machine looked
this this:

While Python won't return memory to the OS, this is not
the same as a memory leak. When your Python program is
done using some objects, the memory they needed can be
used by other Python objects in the same process. The
process size should never grow bigger than the actually
needed memory at a point in time.

The (not only) Python problem is that the virtual memory
of your process will never shrink, once it has attained
its maximum size. Since the unused parts of its virtual
memory will be paged out to disk, this is probably not
big a problem anyway.

If you have a long running process that you want to
keep slim, it might be worth factoring out the memory
intensive parts to separate processes like this in your
macros:

for x,y,z in outer_loop_sequence:
os.system('python the_real_worker.py %s %s %s' % (x,y,z))

This should contain the big memory usage to short-running
processes. It should not solve your crash problem though.
If your crashes go away if you rig it like this, something
was broken, either in your macro, or in the way the
application uses Python for macros. You won't spend less
memory because you divide the problem to two processes.

To actually solve the problem, you need to look closer at
your algorithms and data structures. Are you using effective
ways of storing data? For instance, I imagine that numarray
or whatever those things are called these days are much,
much more effective at handling a vector of numerical values
than a Python list of e.g. ints. Lists are pretty expensive.
Lists of ints are at least twice as big as efficient integer
arrays. Are you copying data when you should just share
references? Are you hanging on to data longer than you
actually need it?

I don't see why there would be a big difference if you
use Python or e.g. C or C++ to solve this kind of problem.
If you use lots of data, you need to understand the data
structures and use them effectively. With Python you are
get shorter development time and less code to maintain to
solve a certain problem. The price for that is typically
longer execution time, but disregarding small systems that
can't bear the size of the Python runtime, memory should
not be a big problem. Python isn't Java.

 

[Grant Edwards]

That's not true at all. Most C libraries these days manage
to return memory to the operating system.

[...]

Thanks for providing a real answer. Like I said in my post, my
information wasn't very current.