python 3.0 memory leaking?

[rupert.thurner]

the edgewall trac release 0.11 is blocked now since more than one
month for a memory leak nobody is able to find, see
http://groups.google.com/group/trac-dev/browse_thread/thread/116e519da54f16b
..

does python-3.0 improve something to avoid writing memory leaking
applications?

rupert.

 

[Jeroen Ruigrok van der Werven]

Hi Rupert,

the edgewall trac release 0.11 is blocked now since more than one
month for a memory leak nobody is able to find, see
http://groups.google.com/group/trac-dev/browse_thread/thread/116e519da54f16b
.

You are slightly mistaken that nobody is able to find anything. There is
currently a lot underway to fix this. The problem was not just one area, but
problems on a number of areas. Given the fact nobody uses a large enough
site with a 0.11 trunk installation these sort of things will start to show
during beta test, as it did. This is still beta period and I personally
would rather release a month later (even despite the already long release
time) than give our users a solution that eats away at their resources like
crazy. I am sure people will appreciate the latter more in the end. Already
the current fixes in trunk for Trac and Genshi stabilized a few large beta
deployment sites in terms of resource eating.

does python-3.0 improve something to avoid writing memory leaking
applications?

There are efforts underway to optimize the memory usage for 2.6 on a lot of
levels more than 2.5 already did. Check Python-dev of the last 2-3 weeks.
However, this is not interesting in the short term, since 2.4 and 2.5 (and
later on 2.6) will be versions that will be used for the coming 1-2 years
for sure. Avoidance is very difficult, especially given Python's dynamic
nature. But I'll let more knowledgeable minds talk about that.

For me, a relative newbie to Python, the entire memory allocation issue is
not transparent at all and information about it is scattered across the net.
One of the things I hope to contribute to in the coming year is to make sure
the entire area of Python memory use and proper coding is better understood.

 

[Christian Heimes]

does python-3.0 improve something to avoid writing memory leaking
applications?

No, it doesn't. Python 3.0 and maybe 2.6 are going to have some small
improvements but the improvements aren't related to memory leaking. I'm
working on the matter for a while now. I've limited some free lists and
I've a patch that gives allocated memory back to the OS earlier.

Have you followed my advice and checked all places that deal with
frames, tracebacks, exception objects and __del__ methods?

Christian

 

[Christian Heimes]

For me, a relative newbie to Python, the entire memory allocation issue is
not transparent at all and information about it is scattered across the net.
One of the things I hope to contribute to in the coming year is to make sure
the entire area of Python memory use and proper coding is better understood.

Python uses its own memory allocator for small objecst (< 257 bytes).
Larger objects are allocated directly with malloc, smaller objects end
up in arenas. The code is well documented in
http://svn.python.org/view/python/trunk/Objects/obmalloc.c?rev=56476&view=auto

Several objects keep a free list for performance reasons. Free list save
some extra mallocs and initialization of data structures. I've renamed
all free lists in Python 2.6 to "free_list".

Ints and floats are using their own block allocation algorithm. The code
predates Python's pymalloc code. I'm working on replacing the code with
pymalloc because pymalloc-ed memory is given back to the OS. The int
and float free lists keep their sizes until the Python process ends.

Christian

 

[rupert.thurner]

No, it doesn't. Python 3.0 and maybe 2.6 are going to have some small
improvements but the improvements aren't related to memory leaking. I'm
working on the matter for a while now. I've limited some free lists and
I've a patch that gives allocated memory back to the OS earlier.

Have you followed my advice and checked all places that deal with
frames, tracebacks, exception objects and __del__ methods?

Christian

many thanks christian! do you have any code examples how to handle
frames, tracebacks, exception objects and __del__ correctly?

btw, i put your hints on http://wiki.python.org/moin/FreeMemory.

rupert.