Auto free Buffer objects in C API?

Discussion in 'Python' started by Stuart D. Gathman, Jul 3, 2003.

  1. I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
    It needs to return some large buffers allocated by the library I am
    wrapping. The buffers must be freed by the caller when finished. At
    present, I do the following to copy, then free the buffer:

    PyObject *
    toBuffer(char *buf;int len) {
    PyObject *sig = PyBuffer_New(len);
    if (sig) {
    void *data;
    int dlen;
    if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
    memcpy(data,buf,dlen);
    else {
    Py_DECREF(sig);
    sig = 0;
    }
    }
    free(buf);
    return sig;
    }

    Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
    way of knowing when to free(buf). Or do I? Am I missing something? Is
    there a way for me to know when the Buffer object is being deleted so
    that I can free the underlying C data?
     
    Stuart D. Gathman, Jul 3, 2003
    #1
    1. Advertising

  2. At some point, "Stuart D. Gathman" <> wrote:

    > I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
    > It needs to return some large buffers allocated by the library I am
    > wrapping. The buffers must be freed by the caller when finished. At
    > present, I do the following to copy, then free the buffer:
    >
    > PyObject *
    > toBuffer(char *buf;int len) {
    > PyObject *sig = PyBuffer_New(len);
    > if (sig) {
    > void *data;
    > int dlen;
    > if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
    > memcpy(data,buf,dlen);
    > else {
    > Py_DECREF(sig);
    > sig = 0;
    > }
    > }
    > free(buf);
    > return sig;
    > }
    >
    > Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
    > way of knowing when to free(buf). Or do I? Am I missing something? Is
    > there a way for me to know when the Buffer object is being deleted so
    > that I can free the underlying C data?


    Staring at the definition of PyBufferObject in bufferobject.c, I see
    it has a member 'b_base' of type PyObject *, which is DECREF'd when
    the buffer is destroyed (by the buffer dealloc function). Looks to me
    like you could do something using a CObject with a passed destructor:

    /**********/
    /* WARNING: UNTESTED CODE */

    /* the contents of PyBufferObject are supposed to be private, so you
    need to include this */
    typedef struct {
    PyObject_HEAD
    PyObject *b_base;
    void *b_ptr;
    int b_size;
    int b_readonly;
    long b_hash;
    } PyBufferObject;

    PyObject *
    toBuffer(char *buf, int len) {
    PyObject *sig = PyBuffer_FromMemory(buf, len);
    if (sig) {
    /* pass 'free' as the destructor for the passed memory */
    (PyBufferObject *)sig->b_base = PyCObject_FromVoidPtr(buf, free);
    }
    return sig;
    }
    /**********/

    Now, when the buffer is deallocated, the CObject at b_base will be
    DECREF'd, which will free your memory.

    This is a bit ugly, as it depends on internal details. Another,
    cleaner, method is to define an object which obeys the buffer
    protocol, and handles destruction like you want. (This is annoying, as
    you'll end up copying all the code in bufferobject.c).

    --
    |>|\/|<
    /--------------------------------------------------------------------------\
    |David M. Cooke
    |cookedm(at)physics(dot)mcmaster(dot)ca
     
    David M. Cooke, Jul 4, 2003
    #2
    1. Advertising

  3. On Thu, 03 Jul 2003 21:44:27 -0400, David M. Cooke wrote:

    > This is a bit ugly, as it depends on internal details. Another, cleaner,
    > method is to define an object which obeys the buffer protocol, and
    > handles destruction like you want. (This is annoying, as you'll end up
    > copying all the code in bufferobject.c).


    So annoying, in fact, that I will continue to make extra copies of the
    buffer until such time as:

    a) Python gets a PyBuffer_FromCObject(PyObject *,int len) method.

    b) Profiling reveals the extra copies to be a major bottle neck. The
    extra memory allocation is not a problem because the library spends most
    of its time moving the data between two buffers while processing it.

    BTW, the application is wrapping DSpam for Python. You can download the
    dspam module from:

    http://bmsi.com/python/milter.html
     
    Stuart D. Gathman, Jul 7, 2003
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Raja
    Replies:
    12
    Views:
    24,441
    John Harrison
    Jun 21, 2004
  2. Replies:
    2
    Views:
    607
    sergejusz
    Mar 26, 2007
  3. linkswanted
    Replies:
    1
    Views:
    938
  4. Neal Becker

    buffer creates only read-only buffer?

    Neal Becker, Jan 8, 2009, in forum: Python
    Replies:
    0
    Views:
    416
    Neal Becker
    Jan 8, 2009
  5. moerchendiser2k3

    buffer objects (C-API)

    moerchendiser2k3, May 13, 2010, in forum: Python
    Replies:
    1
    Views:
    273
    Carl Banks
    May 13, 2010
Loading...

Share This Page