Python C API String Memory Consumption

Discussion in 'Python' started by k3xji, Apr 7, 2009.

  1. k3xji

    k3xji Guest

    When I run the following function, I seem to have a mem leak, a 20 mb
    of memory
    is allocated and is not freed. Here is the code I run:

    >>> import esauth
    >>> for i in range(1000000):


    .... ss = esauth.penc('sumer')
    ....
    >>> for i in range(1000000):


    .... ss = esauth.penc('sumer')
    ....

    And here is the penc() function.

    static PyObject *
    penc(PyObject *self, PyObject *args)
    {
    unsigned char *s= NULL;
    unsigned char *buf = NULL;
    PyObject * result = NULL;
    unsigned int v,len,i = 0;

    if (!PyArg_ParseTuple(args, "s#", &s, &len))
    return NULL;

    buf = strdup(s);
    if (!buf) {
    PyErr_SetString(PyExc_MemoryError,
    "Out of memory: strdup failed");
    return NULL;
    }

    /*string manipulation*/

    result = PyString_FromString(buf);
    free(buf);
    return result;

    }

    Am I doing something wrong?

    Thanks,
     
    k3xji, Apr 7, 2009
    #1
    1. Advertising

  2. k3xji

    Carl Banks Guest

    On Apr 7, 12:01 am, k3xji <> wrote:
    > When I run the following function, I seem to have a mem leak, a 20 mb
    > of memory
    > is allocated and is not freed. Here is the code I run:
    >
    > >>> import esauth
    > >>> for i in range(1000000):

    >
    > ...     ss = esauth.penc('sumer')
    > ...
    >
    > >>> for i in range(1000000):

    >
    > ...     ss = esauth.penc('sumer')
    > ...
    >
    > And here is the penc() function.
    >
    > static PyObject *
    > penc(PyObject *self, PyObject *args)
    > {
    >         unsigned char *s= NULL;
    >         unsigned char *buf = NULL;
    >         PyObject * result = NULL;
    >         unsigned int v,len,i = 0;
    >
    >         if (!PyArg_ParseTuple(args, "s#", &s, &len))
    >         return NULL;
    >
    >         buf = strdup(s);
    >         if (!buf) {
    >                 PyErr_SetString(PyExc_MemoryError,
    >                         "Out of memory: strdup failed");
    >                 return NULL;
    >         }
    >
    >         /*string manipulation*/
    >
    >         result = PyString_FromString(buf);
    >         free(buf);
    >         return result;
    >
    > }
    >
    > Am I doing something wrong?



    It might just be an unfortunate case where malloc keeps allocating
    memory higher and higher on the heap even though it frees all the
    memory. And since it doesn't give it back to the OS, it runs out.

    However, Python apparently does leak a reference if passed a Unicode
    object; PyArg_ParseTuple automatically creates an encoded string but
    never decrefs it. (That might be necessary evil to preserve
    compatibility, though. PyString_AS_STRING does it too.)


    Carl Banks
     
    Carl Banks, Apr 7, 2009
    #2
    1. Advertising

  3. k3xji

    k3xji Guest

    Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
    the problem resolved. However, I really cannot understand why the
    first version does not work. Here is the latest code that has no
    problems at all:

    static PyObject *
    penc(PyObject *self, PyObject *args)
    {
    PyObject * result = NULL;
    unsigned char *s= NULL;
    unsigned char *buf = NULL;
    unsigned int v,len,i = 0;

    if (!PyArg_ParseTuple(args, "s#", &s, &len))
    return NULL;

    buf = (unsigned char *) PyMem_Malloc(len);
    if (buf == NULL) {
    PyErr_NoMemory();
    return NULL;
    }

    /* string manipulation. */

    result = PyString_FromStringAndSize((char *)buf, len);
    PyMem_Free(buf);
    return result;
    }
     
    k3xji, Apr 7, 2009
    #3
  4. k3xji

    MRAB Guest

    k3xji wrote:
    > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
    > the problem resolved. However, I really cannot understand why the
    > first version does not work. Here is the latest code that has no
    > problems at all:
    >
    > static PyObject *
    > penc(PyObject *self, PyObject *args)
    > {
    > PyObject * result = NULL;
    > unsigned char *s= NULL;
    > unsigned char *buf = NULL;
    > unsigned int v,len,i = 0;
    >
    > if (!PyArg_ParseTuple(args, "s#", &s, &len))
    > return NULL;
    >
    > buf = (unsigned char *) PyMem_Malloc(len);
    > if (buf == NULL) {
    > PyErr_NoMemory();
    > return NULL;
    > }
    >
    > /* string manipulation. */
    >
    > result = PyString_FromStringAndSize((char *)buf, len);
    > PyMem_Free(buf);
    > return result;
    > }
    >

    In general I'd say don't mix your memory allocators. I don't know
    whether CPython implements PyMem_Malloc using malloc, but it's better to
    stick with CPython's memory allocators when writing for CPython.
     
    MRAB, Apr 7, 2009
    #4
  5. k3xji

    John Machin Guest

    On Apr 7, 9:19 pm, MRAB <> wrote:
    > k3xji wrote:
    > > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
    > > the problem resolved. However, I really cannot understand why the
    > > first version does not work. Here is the latest code that has no
    > > problems at all:

    >
    > > static PyObject *
    > > penc(PyObject *self, PyObject *args)
    > > {
    > >    PyObject * result = NULL;
    > >    unsigned char *s= NULL;
    > >    unsigned char *buf = NULL;
    > >    unsigned int v,len,i = 0;

    >
    > >    if (!PyArg_ParseTuple(args, "s#", &s, &len))
    > >         return NULL;

    >
    > >    buf = (unsigned char *) PyMem_Malloc(len);
    > >    if (buf == NULL) {
    > >            PyErr_NoMemory();
    > >            return NULL;
    > >    }

    >
    > >         /* string manipulation. */

    >
    > >    result = PyString_FromStringAndSize((char *)buf, len);
    > >    PyMem_Free(buf);
    > >    return result;
    > > }

    >
    > In general I'd say don't mix your memory allocators. I don't know
    > whether CPython implements PyMem_Malloc using malloc,


    The fantastic manual (http://docs.python.org/c-api/
    memory.html#overview) says: """the C allocator and the Python memory
    manager ... implement different algorithms and operate on different
    heaps""".

    > but it's better to
    > stick with CPython's memory allocators when writing for CPython.


    for the reasons given in the last paragraph of the above reference.

    HTH,
    John
     
    John Machin, Apr 7, 2009
    #5
  6. Carl Banks <pavlovevidence <at> gmail.com> writes:
    > However, Python apparently does leak a reference if passed a Unicode
    > object; PyArg_ParseTuple automatically creates an encoded string but
    > never decrefs it. (That might be necessary evil to preserve
    > compatibility, though. PyString_AS_STRING does it too.)


    Unicode objects cache a copy of themselves as default encoded strings. It is
    deallocated when the unicode object its self is.
     
    Benjamin Peterson, Apr 7, 2009
    #6
  7. On Apr 7, 2:10 pm, John Machin <> wrote:
    > On Apr 7, 9:19 pm, MRAB <> wrote:
    >
    >
    >
    > > k3xji wrote:
    > > > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
    > > > the problem resolved. However, I really cannot understand why the
    > > > first version does not work. Here is the latest code that has no
    > > > problems at all:

    >
    > > > static PyObject *
    > > > penc(PyObject *self, PyObject *args)
    > > > {
    > > >    PyObject * result = NULL;
    > > >    unsigned char *s= NULL;
    > > >    unsigned char *buf = NULL;
    > > >    unsigned int v,len,i = 0;

    >
    > > >    if (!PyArg_ParseTuple(args, "s#", &s, &len))
    > > >         return NULL;

    >
    > > >    buf = (unsigned char *) PyMem_Malloc(len);
    > > >    if (buf == NULL) {
    > > >            PyErr_NoMemory();
    > > >            return NULL;
    > > >    }

    >
    > > >         /* string manipulation. */

    >
    > > >    result = PyString_FromStringAndSize((char *)buf, len);
    > > >    PyMem_Free(buf);
    > > >    return result;
    > > > }


    I assume you're doing a memcpy() somewhere in there... This is also
    safer then your first version since the python string can contain an
    embeded \0 and the strdup() of the first version would not copy that.
    But maybe you're sure your input doesn't have NULLs in them so it
    might be fine.

    >
    > > In general I'd say don't mix your memory allocators. I don't know
    > > whether CPython implements PyMem_Malloc using malloc,

    >
    > The fantastic manual (http://docs.python.org/c-api/
    > memory.html#overview) says: """the C allocator and the Python memory
    > manager ... implement different algorithms and operate on different
    > heaps""".
    >
    > > but it's better to
    > > stick with CPython's memory allocators when writing for CPython.

    >
    > for the reasons given in the last paragraph of the above reference.


    That document explictly says you're allowed to use malloc() and free()
    in extensions. There is nothing wrong with allocating things on
    different heaps, I've done and seen it many times and never had
    trouble.

    Why the original problem ocurred I don't understand either tough.

    Regards
    Floris
     
    Floris Bruynooghe, Apr 7, 2009
    #7
  8. k3xji

    Aahz Guest

    In article <>,
    k3xji <> wrote:
    >
    >When I run the following function, I seem to have a mem leak, a 20 mb
    >of memory
    >is allocated and is not freed. Here is the code I run:
    >
    >>>> import esauth
    >>>> for i in range(1000000):

    >
    >... ss = esauth.penc('sumer')
    >...
    >>>> for i in range(1000000):

    >
    >... ss = esauth.penc('sumer')
    >...


    BTW, note that if you're using Python 2.x, range(1000000) will cause a
    "leak" because ints are never freed. Instead, use xrange().
    --
    Aahz () <*> http://www.pythoncraft.com/

    Why is this newsgroup different from all other newsgroups?
     
    Aahz, Apr 10, 2009
    #8
  9. (Aahz) writes:

    > BTW, note that if you're using Python 2.x, range(1000000) will cause
    > a "leak" because ints are never freed. Instead, use xrange().


    Note that using xrange() won't help with that particular problem.
     
    Hrvoje Niksic, Apr 10, 2009
    #9
  10. k3xji

    Carl Banks Guest

    On Apr 9, 11:23 pm, Hrvoje Niksic <> wrote:
    > (Aahz) writes:
    > > BTW, note that if you're using Python 2.x, range(1000000) will cause
    > > a "leak" because ints are never freed.  Instead, use xrange().

    >
    > Note that using xrange() won't help with that particular problem.


    I think it will because with xrange the integers will not all have to
    exist at one time, so Python doesn't have to increase the size of the
    integer pool to a million.


    Carl Banks
     
    Carl Banks, Apr 10, 2009
    #10
  11. Carl Banks <> writes:

    > On Apr 9, 11:23 pm, Hrvoje Niksic <> wrote:
    >> (Aahz) writes:
    >> > BTW, note that if you're using Python 2.x, range(1000000) will cause
    >> > a "leak" because ints are never freed.  Instead, use xrange().

    >>
    >> Note that using xrange() won't help with that particular problem.

    >
    > I think it will because with xrange the integers will not all have
    > to exist at one time, so Python doesn't have to increase the size of
    > the integer pool to a million.


    Good catch! I stand corrected.
     
    Hrvoje Niksic, Apr 10, 2009
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Duccio
    Replies:
    0
    Views:
    537
    Duccio
    Feb 25, 2006
  2. Kiran Kumar

    aspnet_wp.exe memory consumption

    Kiran Kumar, Jul 15, 2003, in forum: ASP .Net
    Replies:
    1
    Views:
    465
    Natty Gur
    Jul 15, 2003
  3. Ervin

    Urgent! GDI+ Memory consumption

    Ervin, Sep 15, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    622
    Ervin
    Sep 15, 2003
  4. Shahriar Shamil Uulu
    Replies:
    5
    Views:
    498
    Olivier Grisel
    Dec 24, 2005
  5. Jonas Maurus
    Replies:
    2
    Views:
    280
    Jeremy Sanders
    Sep 20, 2007
Loading...

Share This Page