Memory usage problem of twisted server

[Victor Lin]

Hi,

I encountered an increasing memory usage problem of my twisted server.
I have posted a question on stackoverflow:
http://stackoverflow.com/questions/...f-increasing-memory-usage-of-a-twisted-server

I have read the article "Improving Python's Memory Allocator" (
http://evanjones.ca/memoryallocator/ ) and Python Memory Management
( http://evanjones.ca/python-memory.html ). And I now know little
about how Python manages memory. I am wondering, is that the
increasing memory usage problem of my audio broadcasting caused by the
how python manage memory?

In my server, there are lots of audio data chunks, they are not all in
fixed size. For example, the size of audio stream might looks like
this:

....
chunk size 822
chunk size 878
chunk size 1690
chunk size 1659
chunk size 1643
chunk size 1952
chunk size 373
chunk size 763
chunk size 1535
chunk size 1665
....

All of those chunks are str object. The size of chunks depend on mp3
encoder. You can see most the size of those chunks is bigger than 256
bytes, so that Python uses malloc to allocate spaces for those chunks.
In my mind, I saw those chunks occupy different place in heap, all of
them are not in fixed size, so that even when the chunk is freed, it
is difficult for malloc to find the right place for the new chunk. As
the result, there are so many chunks placed in large range of heap, it
is difficult for OS to swap pages out. Then the RSS is always high.

Is that my guessing correct? How can I monitor the memory allocation
of Python? Is there any available tools for this? Or is that modifying
the Python source code and recompiling the only way to monitor
allocation of memory?

Thanks.
Victor Lin.

 

[Dieter Maurer]

Hi,

I encountered an increasing memory usage problem of my twisted server.
I have posted a question on stackoverflow:
http://stackoverflow.com/questions/...f-increasing-memory-usage-of-a-twisted-server

I have read the article "Improving Python's Memory Allocator" (
http://evanjones.ca/memoryallocator/ ) and Python Memory Management
( http://evanjones.ca/python-memory.html ). And I now know little
about how Python manages memory. I am wondering, is that the
increasing memory usage problem of my audio broadcasting caused by the
how python manage memory?

Your careful reading has already told you that Python delegates
memory allocation for larger blocks (>= 256 bytes) to the underlying
C runtime library ("malloc" and friends).

The C runtime library does not use memory compaction, i.e.
it does not relocate used memory blocks in order to free space
in few large chunks. Therefore, it is sensible to memory fragmentation:
the free space gets scattered around in a large number of rather small
blocks.
The fragmentation rate is especially high when the memory request sizes
have a high variance.

....
Is that my guessing correct? How can I monitor the memory allocation
of Python?

Look at "http://guppy-pe.sourceforge.net/"