C extension using GSL

[jesse]

I give up. I cannot find my memory leak! I'm hoping that someone out
there has come across something similar. Let me lay out the basic
setup:

I'm performing multiple simulations on a model. Each iteration
involves solving a system of differential equations. For this I use
the GNU Scientific Library (GSL) -- specifically the rk4imp ODE
solver. After the ODE is solved the array is returned to python and is
analyzed. As you may have guessed, my problem is that over the course
of the iterations the memory keeps climbing until python crashes.

Note: *The extension does not keep running.* It returns object (a
list) and is done until the next iteration. AFAIK, any memory
allocated during execution of the extension should be released.
Question: Since the extension was run from within python is memory
allocated within an extension part of python's heap? Would this have
an adverse or unpredictable affect on any memory allocated during the
running of the extension? One hypothesis I have, since most other
possibilities have been eliminated, is that GSL's creation of it's own
data structures (eg., gsl_vector) is messing up Python's control of
the heap. Is this possible? If so, how would one be able to fix this
issue?

It may help some nice folks out there who are good enough to look at
this post if I layout the flow, broken up by where stuff happens
(i.e., in the Python code or C code):

1) Python: Set up the simulation and basic data structures (numpy
arrays with initial conditions, coupling matrices for the ODE's, dicts
with parameters, etc).
2) Python: Pass these to the C extension
3) C: Python objects passed in are converted to C arrays, floats,
etc.
4) C: A PyList object, L, is created (new reference!). This will hold
the solution vector for the ODE
5) C: Initialize GSL ODE solver and stepper functions. Solve the ODE,
at each step use PyList_Append(L, current_state) to append the current
state to L.
6) C: After the ODE solver finishes, free GSL objects, free coupling
matrices, free last state vector.
7) C: Return L to Python with return Py_BuildValue("N", L).
8) Python: Convert returned list L to array A, delete L, work with A.
8.1) Python: Step 8) includes plotting. (I have a small suspicion that
matplotlib holds onto lots of data, but I've used clf() and close(fig)
on all plots, so I think I'm safe here. )
8.2) Python: save analysis results from A, save A. (At this point
there should be no more use of A. In fact, at point 8) in the next
iteration A is replaced by a new array.)
9) Python: Change any parameters or initial conditions and goto 1).

thanks for any help,
-Jesse

 

[sturlamolden]

I give up. I cannot find my memory leak!

That's the penalty for using the Python C API.

http://www.cython.org

 

[jesse]

No, on the normal C heap


If the library passes you data and ownership of a heap block, you'll
need to free it.


Only if GSL is buggy



At every point a memory allocation is made check who owns the memory
and that it is freed through all possible error paths.

Valgrind will help you find the memory leaks.  It works well once
you've jumped the flaming hoops of fire that setting it up is!

Another thing you can try is run your process untill it leaks loads,
then make it dump core.  Examine the core dump with a hex editor and
see what it is full of!  This technique works suprisingly often.

Thanks for the suggestions.
-J

 

[Gabriel Genellina]

I give up. I cannot find my memory leak! I'm hoping that someone out
there has come across something similar. Let me lay out the basic
setup:
[...]
4) C: A PyList object, L, is created (new reference!). This will hold
the solution vector for the ODE
[...]
7) C: Return L to Python with return Py_BuildValue("N", L).

I don't know the "N" format, but if Py_BuildValue returns a new reference,
you're leaking a reference to the L list.
Why don't you return L directly?

You can use sys.getrefcount in Python to see the reference count for an
object:

py> from sys import getrefcount as rc
py> x = object()
py> rc(x)
2 # the name x, and a temporary reference as parameter
py> rc([])
1 # only the temporary reference
py> x = y = []
py> rc(x)
3
py> x = ()
py> rc(x)
954 # the empty tuple is shared