Auto free Buffer objects in C API?

[Stuart D. Gathman]

I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
It needs to return some large buffers allocated by the library I am
wrapping. The buffers must be freed by the caller when finished. At
present, I do the following to copy, then free the buffer:

PyObject *
toBuffer(char *buf;int len) {
PyObject *sig = PyBuffer_New(len);
if (sig) {
void *data;
int dlen;
if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
memcpy(data,buf,dlen);
else {
Py_DECREF(sig);
sig = 0;
}
}
free(buf);
return sig;
}

Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
way of knowing when to free(buf). Or do I? Am I missing something? Is
there a way for me to know when the Buffer object is being deleted so
that I can free the underlying C data?

 

[David M. Cooke]

I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
It needs to return some large buffers allocated by the library I am
wrapping. The buffers must be freed by the caller when finished. At
present, I do the following to copy, then free the buffer:

PyObject *
toBuffer(char *buf;int len) {
PyObject *sig = PyBuffer_New(len);
if (sig) {
void *data;
int dlen;
if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
memcpy(data,buf,dlen);
else {
Py_DECREF(sig);
sig = 0;
}
}
free(buf);
return sig;
}

Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
way of knowing when to free(buf). Or do I? Am I missing something? Is
there a way for me to know when the Buffer object is being deleted so
that I can free the underlying C data?

Staring at the definition of PyBufferObject in bufferobject.c, I see
it has a member 'b_base' of type PyObject *, which is DECREF'd when
the buffer is destroyed (by the buffer dealloc function). Looks to me
like you could do something using a CObject with a passed destructor:

/**********/
/* WARNING: UNTESTED CODE */

/* the contents of PyBufferObject are supposed to be private, so you
need to include this */
typedef struct {
PyObject_HEAD
PyObject *b_base;
void *b_ptr;
int b_size;
int b_readonly;
long b_hash;
} PyBufferObject;

PyObject *
toBuffer(char *buf, int len) {
PyObject *sig = PyBuffer_FromMemory(buf, len);
if (sig) {
/* pass 'free' as the destructor for the passed memory */
(PyBufferObject *)sig->b_base = PyCObject_FromVoidPtr(buf, free);
}
return sig;
}
/**********/

Now, when the buffer is deallocated, the CObject at b_base will be
DECREF'd, which will free your memory.

This is a bit ugly, as it depends on internal details. Another,
cleaner, method is to define an object which obeys the buffer
protocol, and handles destruction like you want. (This is annoying, as
you'll end up copying all the code in bufferobject.c).

 

[Stuart D. Gathman]

This is a bit ugly, as it depends on internal details. Another, cleaner,
method is to define an object which obeys the buffer protocol, and
handles destruction like you want. (This is annoying, as you'll end up
copying all the code in bufferobject.c).

So annoying, in fact, that I will continue to make extra copies of the
buffer until such time as:

a) Python gets a PyBuffer_FromCObject(PyObject *,int len) method.

b) Profiling reveals the extra copies to be a major bottle neck. The
extra memory allocation is not a problem because the library spends most
of its time moving the data between two buffers while processing it.

BTW, the application is wrapping DSpam for Python. You can download the
dspam module from:

http://bmsi.com/python/milter.html