memory recycling/garbage collecting problem

[Yuanxin Xi]

I'm having some problems with the memory recycling/garbage collecting
of the following testing code:

a=[str(i) for i in xrange(10000000)]

This takes 635m/552m/2044 memory (VIRT/RES/SHR)
.... b[str(i)]=i

Then the memory usage increased to 1726m/1.6g/2048
I expect the memory usage drop to the ammount before b was
created(635m/552m/2044), but it's actually 1341m/1.2g/2048

Could anyone please explain why this happens? It seems some memory
are not freed. I'm running into problems with this as my program is
very memory cosuming and need to frequently free some object to reuse
the memory. What is the best way to free the memory of b completely
(i.e. back to the status as b was created)? I'm using Python 2.5.2

Thanks.

Yuanxin

 

[Marc 'BlackJack' Rintsch]

I'm having some problems with the memory recycling/garbage collecting of
the following testing code:

a=[str(i) for i in xrange(10000000)]

This takes 635m/552m/2044 memory (VIRT/RES/SHR)
... b[str(i)]=i

Then the memory usage increased to 1726m/1.6g/2048
I expect the memory usage drop to the ammount before b was
created(635m/552m/2044), but it's actually 1341m/1.2g/2048

Could anyone please explain why this happens? It seems some memory are
not freed.

It seems the memory is not given back to the operating system. This
doesn't mean that it is not freed by Python and can't be used again by
Python. Create the dictionary again and see if the memory usage rises
again or if it stays stable.

Ciao,
Marc 'BlackJack' Rintsch

 

[Aaron Brady]

I'm having some problems with the memory recycling/garbage collecting
of the following testing code:

a=[str(i) for i in xrange(10000000)]

This takes 635m/552m/2044 memory (VIRT/RES/SHR)

...     b[str(i)]=i

Then the memory usage increased to 1726m/1.6g/2048

I expect the memory usage drop to the ammount before b was
created(635m/552m/2044), but it's actually 1341m/1.2g/2048

snip

'gc.collect()' -- I believe, but I'm not the specialist in it.

 

[David Niergarth]

Could anyone please explain why this happens?  It seems some memory
are not freed.

There is a "bug" in versions of Python prior to 2.5 where memory
really isn't released back to the OS. Python 2.5 contains a new object
allocator that is able to return memory to the operating system that
fixes this issue. Here's an explanation:

http://evanjones.ca/python-memory-part3.html

What version of Python are you using? I have a machine running several
long-running processes, each of which occasionally spike up to 500M
memory usage, although normally they only require about 25M. Prior to
2.5, those processes never released that memory back to the OS and I
would need to periodically restart them. With 2.5, this is no longer a
problem. I don't always see memory usage drop back down immediately
but the OS does recover the memory eventually. Make sure you use 2.5
if this is an issue for you.

--David

 

[David Niergarth]

Tim Peters showed a way to demonstrate the fix in

http://mail.python.org/pipermail/python-dev/2006-March/061991.html

For simpler fun, run this silly little program, and look at memory
consumption at the prompts:

"""
x = []
for i in xrange(1000000):
x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

For example, in a release build on WinXP, VM size is about 48MB at the
"full" prompt, and drops to 3MB at the "empty" prompt. In the trunk
(without this patch), VM size falls relatively little from what it is
at the "full" prompt (the contiguous vector holding a million
PyObject* pointers is freed, but the obmalloc arenas holding a
million+1 list objects are never freed).

For more info about the patch, see Evan's slides from _last_ year's PyCon:

http://evanjones.ca/memory-allocator.pdf

I'm not sure what deleting a slice accomplishes (del x[:]); the
behavior is the same whether I do del x or del x[:]. Any ideas?

--David

 

[Steve Holden]

Tim Peters showed a way to demonstrate the fix in

http://mail.python.org/pipermail/python-dev/2006-March/061991.html

For simpler fun, run this silly little program, and look at memory
consumption at the prompts:

"""
x = []
for i in xrange(1000000):
x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

For example, in a release build on WinXP, VM size is about 48MB at the
"full" prompt, and drops to 3MB at the "empty" prompt. In the trunk
(without this patch), VM size falls relatively little from what it is
at the "full" prompt (the contiguous vector holding a million
PyObject* pointers is freed, but the obmalloc arenas holding a
million+1 list objects are never freed).

For more info about the patch, see Evan's slides from _last_ year's PyCon:

http://evanjones.ca/memory-allocator.pdf

I'm not sure what deleting a slice accomplishes (del x[:]); the
behavior is the same whether I do del x or del x[:]. Any ideas?

del x removes the name x from the current namespace, garbage collecting
the object to which it referred. del x[:] leaves x referencing a cleared
list.

regards
Steve

 

[Terry Reedy]

Tim Peters showed a way to demonstrate the fix in

http://mail.python.org/pipermail/python-dev/2006-March/061991.html

For simpler fun, run this silly little program, and look at memory
consumption at the prompts:

"""
x = []
for i in xrange(1000000):
x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

For example, in a release build on WinXP, VM size is about 48MB at the
"full" prompt, and drops to 3MB at the "empty" prompt. In the trunk
(without this patch), VM size falls relatively little from what it is
at the "full" prompt (the contiguous vector holding a million
PyObject* pointers is freed, but the obmalloc arenas holding a
million+1 list objects are never freed).

For more info about the patch, see Evan's slides from _last_ year's PyCon:

http://evanjones.ca/memory-allocator.pdf

I'm not sure what deleting a slice accomplishes (del x[:]); the
behavior is the same whether I do del x or del x[:]. Any ideas?

del x removes the name x from the current namespace, garbage collecting
the object to which it referred.

If there is another reference to the list, which there well might be in
an actual application with memory problems, then 'del x' only
disassociates the name but the object and its large contents are not gc'ed.

> del x[:] leaves x referencing a cleared list.

which is guaranteed to be cleared, regardless of other refs.

Terry

 

[Steve Holden]

David Niergarth wrote: [...]

I'm not sure what deleting a slice accomplishes (del x[:]); the
behavior is the same whether I do del x or del x[:]. Any ideas?

del x removes the name x from the current namespace, garbage collecting
the object to which it referred.

If there is another reference to the list, which there well might be in
an actual application with memory problems, then 'del x' only
disassociates the name but the object and its large contents are not gc'ed.

del x[:] leaves x referencing a cleared list.

which is guaranteed to be cleared, regardless of other refs.

Nice catch, Terry! You correctly spotted I was assuming that d
represented the only reference to the list.

regards
Steve