Controlling memory allocation

[cwoolf]

Let's say I have an OS that can't limit the amount of memory a process
is allowed to consume, but I want to run some Python code in a
simulated low memory environment...

Is there a way to tell Python to limit the size of it's heap? Can you
ask it to use a function other than malloc through the C API? Any
stunts you might recommend to achieve the same results?

Chad

 

[Tommy.Ryding]

Hello.
I have done the thing you are requesting. You can easily create your
own memory pool and have Python to use it.

I added some rows to Pyconfig.h:

#ifdef malloc
#undef malloc
#endif /*malloc*/
#define malloc Python_malloc

Then I have some overhead in my own Python_malloc(size_t nBytes)

 

[cwoolf]

Good to know that works

Did you also do the same for free and realloc to keep up with your
accounting?

The more I thought about it, the more I realized that calls to other
library functions could cause address space expansion if that code
internally called malloc or mmap.

See, I can use the OS to limit stack and data segment growth, but on OS
X/Darwin, it seems malloc uses mmap instead of brk to get more memory,
so setting a ulimit on the data segment doesn't work, because
technically the data segment doesn't grow during a malloc. Setting a
ulimit on the RSS (resident set size) doesn't work either as that limit
seems to only limit the number of pages the process is allowed to touch
(keep resident in memory) and the address range is still allowed to
expand.

Chad

 

[Tommy.Ryding]

Yep!
I redirect Free, Realloc and Malloc. (calloc is never used by Python).

"The more I thoug..."
That could be a possible problem. I will look in to it today and try
and find out if that occurs and in that case how often.


//T