Stephen said:
[Note that actually it was Hermann Maier that wrote the following
but as quoted, it may look like it was I that wrote it.]
Python Memory Validator.
Run your program to completion.
Switch to the hotspots tab.
Search for your function.
All memory used in that function will be shown in the tree (with the
effective callstack) underneath that function node in the tree.
I can't try it out easily since it doesn't seem to support Linux. Also
looking at the home page I couldn't find any description of what it
was actually doing. The info links didn't work, they showed blank
pages with my browser (Netscape 3.01.)
So if you would like to explain here, it would be helpful.
Especially, it isn't clear to me what this means (requoting):
All memory used in that function
Since Python has a reference-counting allocator, and various
kinds of objects use specialized allocation pools, it seems
unclear and implementation defined what is meant with
'all memory used'.
I could think of some different things this could mean.
1. The total memory allocated, regardless of whether it is released
again. If it is to mean really all the memory, it should include
memory allocated by special allocators such as the allocator for
ints. In this case, the following example function
def f():
x = 0
while x < 1000000:
x += 1
would have allocated (at least) 1000000 objects,
one for each iteration since int's are immutable
and one is allocated each time x is assigned to
(although each one is also deallocated.)
So according to the meaning (1.), the
'total memory used' would be 1000000 * the size of ints.
[ Though with some alternative interpreter implementation
it might be an entirely different value such as 0 bytes since
it may not need to allocate every integer in the heap. Would
'allocating' on the stack count? In a CPU register?]
Is something like this what would be reported by your system?
2. The second alternative meaning would be the memory allocated in
the function, MINUS the memory deallocated in the function.
Is this what is reported by your system?
If the case is according to the 2nd alternative, 'memory used in' f
would
be 0 bytes since all memory allocated is deallocated, and for a
function like the following
def g():
return range(1000)
it would be the size allocated for the list of length 1000 + any int
objects allocated to fill the list. Given the above function g, its
memory usage could be tested using heapy and the following function.
def test(g):
from guppy import hpy
hp=hpy()
hp.setrelheap()
hp.setrelheap() # Has to be done twice after the first import (XXX)
x=g()
print hp.heap()
Partition of a set of 902 objects. Total size = 15364 bytes.
Index Count % Size % Cumulative % Kind (class / dict of
class)
0 900 100 10800 70 10800 70 int
1 1 0 4128 27 14928 97 list
2 1 0 436 3 15364 100 types.FrameType
[ Why it shows an object of type FrameType is something of a mystery,
I'm not sure of the best way to get rid of it.]
I guess I could add a function somewhat like test(g) to Heapy,
depending on what common use cases seem to require.
On the other hand, test for memory usage according to alternative (1)
would be harder to add, and it is somewhat undefined what it
means. And it is perhaps not so useful as the 2nd alternative?
Sverker
Project home page:
http://guppy-pe.sourceforge.net