Weird memory consumption problem.

[Bo Peng]

Dear list,

I spent the last 12 hours in catching this bug (?) and what I found out
is very difficult to explain:

Basically, I need to call a user-provided function many times, with a
tuple as parameter.

C/C++ side: (class A, constructed using a python function m_func)

// create in the constructor, used to hold the parameter
m_numArray = PyTuple_New(m_len);
...

// member function, will be called many times
void fun(){
....
// set the tuple
for( j=0; j<m_len; ++j)
PyTuple_SetItem(m_numArray, j, PyInt_FromLong(some_value));
// call function m_func(m_numArray)
PyObject* arglist = Py_BuildValue("(O)", m_numArray );
PyObject* result = PyEval_CallObject(m_func, arglist);
Py_DECREF(arglist);
// check if result == NULL ...
// get value from result, ...
Py_DECREF(result)
return the value of result
}

Python side: pyFun1, pyFun2 will be passed to C/C++ as m_func

def pyFun1(val):
return 0.5

def pyFun2(val):
return 0.5*val[0]

For objects A(pyFun1) and A(pyFun2), member function fun() will be
called many times. The weird thing is that calling A(pyFun1).fun()
endlessly goes well with constant memory consumption; while calling
A(pyFun2).fun() will use more and more memory until core dump. I tried
"valgrind --tool=memcheck --check-leak=yes" but did not find any sign of
memory leak.

The real code is much more complicated and I can not possibly explain
the details here. Can anyone *guess* what might go wrong? My guess is
that access to val[0] increases the ref_cnt of val[0] and make its value
stay in memory, but this is unlikely the case.

Many thanks in advance.
Bo

 

[Steven D'Aprano]

The real code is much more complicated and I can not possibly explain
the details here. Can anyone *guess* what might go wrong?

Sorry, are you saying that the code you posted does NOT
have a memory leak, but you want us to find the memory
leak in your real code sight unseen?

 

[Bo Peng]

Sorry, are you saying that the code you posted does NOT have a memory
leak, but you want us to find the memory leak in your real code sight
unseen?

valgrind does not detect anything so it does not look like memory leak.
I just can not figure out why val[0], readonly access to a tuple
element, can cause this kind of problem.

Anyway, is there a way to list all or newly added python objects? I
guess I can trace the ref_cnt of the tuple element manually.

Bo

 

[Bo Peng]

Problem found. It is hidden in a utility function that converts the
return value to a double. The refcnt of the middle result was not
decreased.

PyObject* res = PyNumber_Float(obj);
val = PyFloat_AsDouble(res);
// missing
Py_DECREF(res);

The problem is not that difficult to find, but it was 2am in the morning
and I was misled by the different behavior of pyFun1 and pyFun2.

Thanks,
Bo

 

[Stephen Kellett]

The problem is not that difficult to find, but it was 2am in the morning and
I was misled by the different behavior of pyFun1 and pyFun2.

Don't know if you were using Windows, but if you were then Python Memory
Validator would have helped you identify that bug. Its in beta at the
moment if you want to try it.

http://www.softwareverify.com

Stephen