Py_DECREF Question:

[Jeremy Moles]

When I add an object created locally to a mapping or sequence (that will
be returned from a function into an running instance of the Python
interpreter), I need to call Py_DECREF on the object, right?
Unfortunately, I really feel like the following code leaks memory...

--- for example ---

PyObject* list = Py_Buildvalue("[]");

while(go) {
PyObject* num = Py_BuildValue("i", 10);

PyList_Append(list, num);

Py_DECREF(num);
}

return list;

/* The responsibility of decrementing the list now lies within the scope
of Python. */

--- end example ---

Secondly, __and most importantly__, does anyone have any tools and/or
recommendations for detecting memory leaks when writing extension
modules in C? I've been using printf() and showing the OBJECT->ob_refcnt
(which, is always 1 when I'm "done" with the pointer?) but this seems
fairly newb.

I'm mostly asking so I can cleanup any craziness in my code before I try
and release pyiw (the Python bindings for Wireless Networking in Linux).
I really want it to be as clean as possible before showing anyone
else.

 

[Scott David Daniels]

When I add an object created locally to a mapping or sequence ...
I need to call Py_DECREF on the object, right? ... for example:
PyObject* list = Py_Buildvalue("[]");
...
PyObject* num = Py_BuildValue("i", 10);
PyList_Append(list, num);
...

... does anyone have any tools and/or recommendations for detecting
memory leaks when writing extension modules in C? ...

Here's how to answer this for yourself:
PyObject* list = Py_Buildvalue("[]");
int precount;
...
PyObject* num = Py_BuildValue("i", 10);
precount = num.ob_refcnt;
PyList_Append(list, num);
<here compare precount and num.ob_refcnt>
...

If you need to do the DECREF, precount should exceed num.ob_refcnt.
If they match, the "PyList_Append" call "stole" your reference.

--Scott David Daniels
(e-mail address removed)

 

[Raymond Hettinger]

[Jeremy Moles]

When I add an object created locally to a mapping or sequence (that will
be returned from a function into an running instance of the Python
interpreter), I need to call Py_DECREF on the object, right?

It depends on the the function or macro, but usually the answer is Yes.

The C API docs always tell you when a function or macro steals a
reference. For instance, the docs for PyList_SET_ITEM specifically
mention reference theft. If the docs don't say anything, then it means
that the call does not affect the ref count of the function parameters.
For instance, the docs for PyList_Append() do not say anything about
ref counts:

http://docs.python.org/api/listObjects.html

This is easy to verify by looking at the source in
Objects/listobject.c:

int n = PyList_GET_SIZE(self);

assert (v != NULL);
if (n == INT_MAX) {
PyErr_SetString(PyExc_OverflowError,
"cannot add more objects to list");
return -1;
}

if (list_resize(self, n+1) == -1)
return -1;

Py_INCREF(v);
PyList_SET_ITEM(self, n, v);
return 0;

The presence of the INCREF confirms that PyList_Append does not steal
your reference.

Secondly, __and most importantly__, does anyone have any tools and/or
recommendations for detecting memory leaks when writing extension
modules in C?

After about two days of coding with the C API, you just get good at
reading the code and being able to track when you've added or consumed
a reference. That desk checking takes care of 95% of the problems.

For the remaining 5%, I run a test suite multiple times in the debug
mode, do a gc.collect(), and print sys.gettotalrefcount() after each
run. For an example, see the last few lines of Lib/test/test_set.py.
If the counts become constant after the first couple of runs, it is a
good bet that your extension doesn't leak. Of course, it helps if your
test suite is thorough and exercises all code paths.

I'm mostly asking so I can cleanup any craziness in my code before I try
and release pyiw (the Python bindings for Wireless Networking in Linux).
I really want it to be as clean as possible before showing anyone
else.

I'm sure your users will appreciate the devotion to quality.



Raymond