Python for embedded systems with memory constraints

[vishnu]

Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/python-list/2001-November/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.

 

[=?iso-8859-1?q?J=FCrgen_Urner?=]

Who else is using python (programmers, scientists, finance)?

Me! Graduated in fine arts. Python is what I do when I am fed up with
all those colors. Much easier to manufacture sense with.

 

[Cameron Laird]

Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/python-list/2001-November/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.

Your report is interesting and important--and surprising! I thought
Python memory allocation is "cleaner" than you seem to be observing.

I hope one of the core Python maintainers can address this. I haven't
worked at this level recently enough to speculate on why it's happen-
ing, nor will I soon be in a position to volunteer to research it on
my own (although I'd eagerly contract to do so on a modestly paid
basis).

Depending on your schedule and technology, there are lots of technical
fixes that might apply:
A. quick-starting Python variations that encourage you
to manage memory on a whole-process level;
B. use of one of the many Python variants (even PyPy?)
that might give you a more favorable memory profile;
C. switch to Lua or Tcl as more easily embeddable
alternative languages;
D. custom memory allocator;
....

 

[vishnu]

Hi,
Thanks Cameron for your suggestions.
In fact I am using custom memory sub-allocator where I preallocate a
pool of memory during initialization of my application and ensure that
Python doesn't make any system mallocs later . With this arrangement,
python seems to run out of preallocated memory (of 10MB) after running
few simple scripts due to huge external fragmentation. My memory
sub-allocator got a good design which uses the best-fit algorithm and
coaelescing the adjacent blocks during each free call.
If anybody out there used their own memory manager and ran Python
without fragmentation , could provide some inputs on this.

Thanks in advance.

 

[MRAB]

Hi,
Thanks Cameron for your suggestions.
In fact I am using custom memory sub-allocator where I preallocate a
pool of memory during initialization of my application and ensure that
Python doesn't make any system mallocs later . With this arrangement,
python seems to run out of preallocated memory (of 10MB) after running
few simple scripts due to huge external fragmentation. My memory
sub-allocator got a good design which uses the best-fit algorithm and
coaelescing the adjacent blocks during each free call.
If anybody out there used their own memory manager and ran Python
without fragmentation , could provide some inputs on this.

From what I remember, the best-fit algorithm isn't a good idea because

unless the free block was exactly the right size you'd tend to get
left with lots of small fragments. (Suppose that the best fit was a
free block only 4 bytes bigger than what you want; what can you do
with a free block of 4 bytes?)

A worst-fit algorithm would leave larger free blocks which are more
useful subsequently, but I think that the recommendation was next-fit
(ie use the first free block that's big enough, starting from where
you found the last one).

 

[Gabriel Genellina]

So now I only see the solution to clear my memory pool and restart Python
without restarting the system (i.e. no power cycle to hardware). I tried
to
do this when my memory pool is 60% used in these steps:
1) Py_Finalize( )
2) Reset my Memory pool (i.e. free list links)
3) Then Restart Python by calling Py_Initialize().

But this resulted in Python crash during Py_Initialize(), where I found
that the static variables within embedded Python source code are still
holding some of the references to my memory pool. So now my question is
how
do I restart Python (i.e. reinitialize Python) without restarting whole
system. Is there a way to reset/re-initilaize those static variables such
that it will be possible to re-Initialize Python.

Ouch... I think this should not happen, but anyway I don't know if this
was a design principle. One should inspect all Python code to locate all
static variable references... Maybe you could instrument your allocator to
see which references are still held?
C extensions may be problematic too - there is no way to "uninitialize"
them, and nothing forbids an extension to hold a reference to any object.