Re: API/C memory mananegemnt problem

Discussion in 'Python' started by Tim Peters, Mar 11, 2006.

  1. Tim Peters

    Tim Peters Guest

    []
    > Hi everybody,
    > I have a problem with Python/C API and memory management.
    >
    > I'm using
    > Python 2.3.5 (#1, Jan 4 2006, 16:44:27)
    > [GCC 4.0.2 20050901 (prerelease) (SUSE Linux)] on linux2
    >
    > In my C-module I have a loop like this:
    > ***********************************************
    >
    > int size=10000000;
    >
    > output=(double *) calloc(size, sizeof(double));
    >
    > py_output=PyList_New(0);
    >
    > for(i=0; i<size; i++){
    > tmp=PyFloat_FromDouble(output);
    > PyList_Append(py_output, tmp);
    > }
    >
    > free(outout);
    >
    > return py_output;
    >
    > ***********************************************
    >
    > It returns to python module a (very large) list.
    >
    > Problem: when I delete the list in python module (with python del statement)
    > not all memory is relased.
    >
    > It look like all 10000000 tmp PyFloat allocated in C code
    > remain stored in memory.
    >
    > Somebody can help me?


    As Fredrik noted, your refcounting is off and you need a great deal of
    error-checking code.

    However, none of that will help your underlying problem: CPython
    floats are allocated in a special free list dedicated to floats, and
    that free list is both immortal and unbounded in size. Once Python
    has allocated memory to hold a Python float, that memory is never
    released. BTW, Python has another, distinct immortal and unbounded
    free list dedicated to holding storage for int objects.

    There's nothing you can do about that. It may be possible for you to
    use an array.array to hold "unboxed" floats directly; that depends on
    details of your app.
    Tim Peters, Mar 11, 2006
    #1
    1. Advertising

  2. Tim Peters

    Guest

    I have not looked into C programming and Python so I probably don't
    understand the issues here but the above statement sounds very
    troubling.

    Does this mean that any integer or float created anywhere at anytime in
    a (pure) python program will be stored away for all time (as long as
    the program is running)?

    If the above is true my initial reaction is to recoil in horror...

    Say it ain't so Joe!
    , Mar 11, 2006
    #2
    1. Advertising

  3. Tim Peters

    Guest

    Sorry for responding to my own post.

    I think I understand the original statement now. What you are really
    saying is that there is a pool of Python float objects (which can, at
    different times, wrap different values) which can grow but never
    decrease in size. So the memory held by this pool is dictated by the
    maximum number of floats that have ever been simultaneously active
    (accessible).

    The same goes for integers. All the more reason to avoid range(.) and
    use xrange(.).

    So is this correct?

    Thanks!
    , Mar 11, 2006
    #3
  4. <> wrote:

    > Sorry for responding to my own post.
    >
    > I think I understand the original statement now. What you are really
    > saying is that there is a pool of Python float objects (which can, at
    > different times, wrap different values) which can grow but never
    > decrease in size. So the memory held by this pool is dictated by the
    > maximum number of floats that have ever been simultaneously active
    > (accessible).
    >
    > The same goes for integers. All the more reason to avoid range(.) and
    > use xrange(.).


    Uh? The integers produced as you loop over xrange will be just immortal
    as those that get into the list built by range, so why is int
    immortality any reason to use one over the other?

    If you know you're probably not going to loop over ALL the range or
    xrange, sure, but that's relatively rare -- and, when it does occur,
    xrange is seriously preferable:

    helen:~ alex$ python -mtimeit 'for x in range(100): pass'
    100000 loops, best of 3: 15.9 usec per loop

    helen:~ alex$ python -mtimeit 'for x in xrange(100): pass'
    100000 loops, best of 3: 12.2 usec per loop

    helen:~ alex$ python -mtimeit 'for x in range(100): break'
    100000 loops, best of 3: 7.57 usec per loop

    helen:~ alex$ python -mtimeit 'for x in xrange(100): break'
    1000000 loops, best of 3: 1.5 usec per loop

    The immediate break only halves the time requirements of the range-based
    loop, but it slashes by 8 times those of the xrange-based one -- now
    THAT is big enough to matter, as opposed to the 20% or so difference in
    overhead when you're always looping all the way, which is unlikely to
    make an important difference in overall application speed.


    Alex
    Alex Martelli, Mar 11, 2006
    #4
  5. Tim Peters

    Aahz Guest

    In article <1hc18a8.1u0a6cx14wyhuhN%>,
    Alex Martelli <> wrote:
    ><> wrote:
    >>
    >> I think I understand the original statement now. What you are really
    >> saying is that there is a pool of Python float objects (which can, at
    >> different times, wrap different values) which can grow but never
    >> decrease in size. So the memory held by this pool is dictated by the
    >> maximum number of floats that have ever been simultaneously active
    >> (accessible).
    >>
    >> The same goes for integers. All the more reason to avoid range(.) and
    >> use xrange(.).

    >
    >Uh? The integers produced as you loop over xrange will be just immortal
    >as those that get into the list built by range, so why is int
    >immortality any reason to use one over the other?


    Because unless you save the ints produced by xrange(), you're reusing
    slots in the free list as you go through the loop, whereas range() chews
    up a gob of memory that will never get released even if you never
    otherwise use all the ints it produces.
    --
    Aahz () <*> http://www.pythoncraft.com/

    "19. A language that doesn't affect the way you think about programming,
    is not worth knowing." --Alan Perlis
    Aahz, Mar 11, 2006
    #5
  6. Tim Peters

    Guest

    This exactly what I was thinking.

    Are we wrong Alex?
    , Mar 11, 2006
    #6
  7. <> wrote:

    > This exactly what I was thinking.
    >
    > Are we wrong Alex?


    Nope, you're not -- since ints aren't "immortal" (but rather limited to
    being reused as other ints), then, if at no _other_ place in your
    program do you ever need to be storing N ints at the same time, looping
    on an xrange of N items does reduce maximum memory needed by your
    program compared to looping on an equivalent range.


    Alex
    Alex Martelli, Mar 12, 2006
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shlomo Anglister
    Replies:
    1
    Views:
    405
    Default User
    Aug 2, 2004
  2. Praveen, Tayal (IE10)
    Replies:
    0
    Views:
    372
    Praveen, Tayal (IE10)
    Mar 17, 2005
  3. Replies:
    0
    Views:
    296
  4. Marco Fumana

    API/C memory mananegemnt problem

    Marco Fumana, Mar 10, 2006, in forum: Python
    Replies:
    0
    Views:
    303
    Marco Fumana
    Mar 10, 2006
  5. John123

    Profiling API or Membership API

    John123, Oct 20, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    370
    John123
    Oct 20, 2006
Loading...

Share This Page