Python embedded in C & memory releasing

[lebo]

Hi

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte

Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?

C code snippet:
#include <Python.h>

int main(int argc, char *argv[])
{
int y = 0;
printf("Started...\n");
for (int count=0;count < 999999999;count++) {
y += 1;
}
printf("Initing Py\n");
Py_Initialize();
PyRun_SimpleString("from time import sleep\n"
"print 'I will sleep for a while'\n"
"sleep(10)\n"
"print 'Exiting Py'\n");
Py_Finalize();

for (int count=0;count < 999999999;count++) {
y += 1;
}
return 0;
}

Thanks, Leonard

 

[Tom Anderson]

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte

Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?

this is how most runtime systems handle memory. they have a certain amount
of space allocated, and if they run out, they ask the OS for more, but
they never give it back. i think this is what most implementations of
malloc/free do; it's certainly what java does.

what you want is some way to tell python to give back some or all of its
spare memory. i'm not aware of such a thing myself.

tom

 

[John J. Lee]

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

[...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.


John

 

[Alex Martelli]

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

[...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.

This is not true any more these days!!! Consider the following program:

#include <stdio.h>
#include <stdlib.h>

int main()
{
char* blobbone;

printf("No blob yet, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

blobbone = malloc(1024*1024);

printf("I have the blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

free(blobbone);

printf("No more blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

printf("Bye bye.\n");
return 0;
}

Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:

[alex@lancelot sae]$ gcc mem.c
[alex@lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


I have the blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


No more blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


Bye bye.
[alex@lancelot sae]$

See? the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again. I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).

And in Python...? *SAME THING*!!! Watch:

print "No blob yet, measure me now, then press enter...",
x = raw_input()
print

blobbone = 1024*256*[None]

print "I have the blob, measure me now, then press enter...",
x = raw_input()
print

del blobbone

print "No more blob, measure me now, then press enter...",
x = raw_input()
print

print "Bye bye."


[alex@lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2504 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py


I have the blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 4892 3536 0.6 python mem.py
[alex@lancelot sae]$ fg
python mem.py


No more blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2508 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py


Bye bye.
[alex@lancelot sae]$

See? Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.


The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back. That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of.


Alex

 

[John J. Lee]

John J. Lee wrote: [...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.

This is not true any more these days!!! Consider the following program: [...]
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out [...]
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out [...]
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out

[...]

Well, you live and learn.


John