Python C Extensions

[aken8062]

Hi,

I have a memory leak problem with my "C" extension module. My C module
returns large dictionaries to python, and the dictionaries never get
deleted, so the memory for my program keeps growing.

I do not know how to delete the dictionary object after it becomes
irrelevant. I do not know if the version of python is relevant, I'm
using the 2.5 !


Here is the "C" code:

PyObject *GetDictionary(PyObject *self, PyObject *args)
{
PyObject *dict = PyDict_New();
PyObject *key;
PyObject *value;

char name[128];

for(int i = 0; i < 60; i++)
{
sprintf(name,"v%d",i);
float number = 1.0 * 0.5*i;

PyDict_SetItem(dict,Py_BuildValue("s",name),Py_BuildValue("f",number));
}
return dict;
}

And here is the Code that I use in a loop, which causes the program
memory to grow:
import libpyTestModule as pyTEST

bankTEST = {}
for j in range(1,100000):
for k in range(1,100000):
bankTEST = pyTEST.GetDictionary()
del bankTEST


Any help will be appreciated.

 

[MRAB]

Hi,

I have a memory leak problem with my "C" extension module. My C module
returns large dictionaries to python, and the dictionaries never get
deleted, so the memory for my program keeps growing.

I do not know how to delete the dictionary object after it becomes
irrelevant. I do not know if the version of python is relevant, I'm
using the 2.5 !


Here is the "C" code:

PyObject *GetDictionary(PyObject *self, PyObject *args)
{
PyObject *dict = PyDict_New();
PyObject *key;
PyObject *value;

char name[128];

for(int i = 0; i< 60; i++)
{
sprintf(name,"v%d",i);
float number = 1.0 * 0.5*i;

PyDict_SetItem(dict,Py_BuildValue("s",name),Py_BuildValue("f",number));
}
return dict;
}

And here is the Code that I use in a loop, which causes the program
memory to grow:
import libpyTestModule as pyTEST

bankTEST = {}
for j in range(1,100000):
for k in range(1,100000):
bankTEST = pyTEST.GetDictionary()
del bankTEST


Any help will be appreciated.

Py_BuildValue(...) returns an object with its refcount set to 1.

PyDict_SetItem(...) increments the refcounts of the key and value
objects when they are added to the dict, so their refcounts will then
be 2.

When the dict is garbage-collected the refcouts of the key and value
objects will be decremented to 1, so they won't be collected, and as
there aren't any other references to them, leading to a memory leak.

You therefore need to decrement the refcounts of the key and value
objects after adding them to the dict:

PyObject *key = Py_BuildValue("s", name);
PyObject *value = Py_BuildValue("f", number);
PyDict_SetItem(dict, key, value);
Py_DECREF(key);
Py_DECREF(value);

 

[aken8062]

Thank you very much, it worked.
I thought the PyDict_SetItem should assume ownership
of the passed object and decrease it's reference count (I do not know
why).

Does this also go for the Lists ? Should anything inserted into list
also
be DECRED-ed ?

Thank you again for reply.

Hi,

I have a memory leak problem with my "C" extension module. My C module
returns large dictionaries to python, and the dictionaries never get
deleted, so the memory for my program keeps growing.

I do not know how to delete the dictionary object after it becomes
irrelevant. I do not know if the version of python is relevant, I'm
using the 2.5 !

Here is the "C" code:

PyObject *GetDictionary(PyObject *self, PyObject *args)
{
   PyObject *dict = PyDict_New();
   PyObject *key;
   PyObject *value;

   char name[128];

   for(int i = 0; i<  60; i++)
     {
       sprintf(name,"v%d",i);
       float number = 1.0 * 0.5*i;

PyDict_SetItem(dict,Py_BuildValue("s",name),Py_BuildValue("f",number));
     }
   return dict;
}

And here is the Code that I use in a loop, which causes the program
memory to grow:
import libpyTestModule as pyTEST

bankTEST = {}
for j in range(1,100000):
     for k in range(1,100000):
         bankTEST = pyTEST.GetDictionary()
         del bankTEST

Any help will be appreciated.

Py_BuildValue(...) returns an object with its refcount set to 1.

PyDict_SetItem(...) increments the refcounts of the key and value
objects when they are added to the dict, so their refcounts will then
be 2.

When the dict is garbage-collected the refcouts of the key and value
objects will be decremented to 1, so they won't be collected, and as
there aren't any other references to them, leading to a memory leak.

You therefore need to decrement the refcounts of the key and value
objects after adding them to the dict:

     PyObject *key = Py_BuildValue("s", name);
     PyObject *value = Py_BuildValue("f", number);
     PyDict_SetItem(dict, key, value);
     Py_DECREF(key);
     Py_DECREF(value);

 

[Carl Banks]

Thank you very much, it worked.
I thought the PyDict_SetItem should assume ownership
of the passed object and decrease it's reference count (I do not know
why).

Does this also go for the Lists ? Should anything inserted into list
also
be DECRED-ed ?

The Python C API documentation has this information--if a function is
documented as borrowing a reference, then it behaves as you were
expecting (it doesn't increase the reference count). If it's
documented as creating a new reference, it does increase the reference
count.

I don't know if there's a simple rule to know of a function borrows or
creates a new reference; I've never noticed one.


Carl Banks

 

[MRAB]

Thank you very much, it worked.
I thought the PyDict_SetItem should assume ownership
of the passed object and decrease it's reference count (I do not know
why).

Does this also go for the Lists ? Should anything inserted into list
also
be DECRED-ed ?

Thank you again for reply.

[snip]
The pattern is that calls which create an object will return that
object with a refcount of 1, and calls which 'store' an object, for
example, PyDict_SetItem(...) and PyList_Append(...) will increment the
refcount of the stored object to ensure that it won't be garbage
collected.

When in doubt, try stepping through the code in a debugger. You'll see
that storing an object will cause its refcount to be incremented.