Re: Python 2.6 still not giving memory back to the OS...

Discussion in 'Python' started by Mark Dickinson, Aug 15, 2009.

  1. On Aug 15, 12:55 pm, Chris Withers <> wrote:
    > Hi All,
    >
    > I thought this was fixed back in Python 2.5, but I guess not?
    >
    > So, I'm playing in an interactive session:
    >
    >  >>> from xlrd import open_workbook
    >  >>> b = open_workbook('some.xls',pickleable=0,formatting_info=1)
    >
    > At this point, top shows the process usage for python to be about 500Mb.
    > That's okay, I'd expect that, b is big ;-)
    >
    >  >>> del b
    >
    > However, it still does now, maybe the garbage collector needs a kick?
    >
    >  >>> import gc
    >  >>> gc.collect()
    > 702614
    >
    > Nope, still 500Mb. What gives? How can I make Python give the memory its
    > no longer using back to the OS?
    > [...]


    Can you get the same effects without using the xlrd module? I don't
    have xlrd installed on my system (OS X 10.5/Intel), but I just tried
    the following:

    Python 2.6.2 (r262:71600, Jun 17 2009, 09:08:27)
    [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> b = 'x'*(10**9)
    >>> f = open('testfile.txt', 'w')
    >>> f.write(b)
    >>> del b
    >>> f = open('testfile.txt')
    >>> b = f.read()
    >>> del b


    and got the expected memory usage for my Python process, as
    displayed by top: memory usage went up to nearly 1Gb after
    each assignment to b, then dropped down to 19 Mb or so after
    each 'del b'. I get similar results under Python 2.5.

    So maybe there's something in xlrd that's hanging on to all
    that memory?

    Mark
     
    Mark Dickinson, Aug 15, 2009
    #1
    1. Advertising

  2. Mark Dickinson

    John Machin Guest

    On Aug 16, 2:41 am, Mark Dickinson <> wrote:
    > and got the expected memory usage for my Python process, as
    > displayed by top:  memory usage went up to nearly 1Gb after
    > each assignment to b, then dropped down to 19 Mb or so after
    > each 'del b'.  I get similar results under Python 2.5.
    >
    > So maybe there's something in xlrd that's hanging on to all
    > that memory?


    News to me, and news to guppy/heapy v0.1.9 -- unless the new Windows
    support has some deficiencies -- after `del b` heapy can't find any
    trace of the Book object and its contents.

    As far as releasing memory back to the OS is concerned, I have dim
    memories of *x systems where free() would return space to the OS if
    the block was "large" and it was next to the "break" point ... this
    effect could be what you are seeing.
     
    John Machin, Aug 16, 2009
    #2
    1. Advertising

  3. > As far as releasing memory back to the OS is concerned, I have dim
    > memories of *x systems where free() would return space to the OS if
    > the block was "large" and it was next to the "break" point ... this
    > effect could be what you are seeing.


    Today, there are two cases when malloc returns memory on a typical
    Unix system (in particular, in Linux malloc):
    a) if the malloc block block is small (below page size), it is allocated
    from the brk heap, where it can only be returned if the last page of
    that heap is completely free, and
    b) if the block is large (multiple pages), it gets returned to the
    system right away.

    Case b) can only happen if the C malloc uses mmap to allocate large
    blocks. For Python, case b) is realistic, in the sense that most
    allocations go to Python's obmalloc, and those allocate from C malloc
    in chunks of 256kiB (IIRC). So when such an arena is completely free
    (not a single Python object allocated on it anymore), it can get
    returned to the system.

    In some case, Python also allocates smaller blocks from C malloc; in
    this case, a) will trigger. So as long as something at the end of the
    heap stays allocated, C malloc will not return anything from the brk
    heap to the system.

    Regards,
    Martin
     
    Martin v. Löwis, Aug 16, 2009
    #3
  4. Martin v. Löwis wrote:
    > Today, there are two cases when malloc returns memory on a typical
    > Unix system (in particular, in Linux malloc):
    > a) if the malloc block block is small (below page size), it is allocated
    > from the brk heap, where it can only be returned if the last page of
    > that heap is completely free, and
    > b) if the block is large (multiple pages), it gets returned to the
    > system right away.
    >
    > Case b) can only happen if the C malloc uses mmap to allocate large
    > blocks.


    I believe (and John will correct me if I'm wrong) that xlrd uses mmap,
    so why am I not seeing the memory being returned?

    Chris

    --
    Simplistix - Content Management, Batch Processing & Python Consulting
    - http://www.simplistix.co.uk
     
    Chris Withers, Aug 24, 2009
    #4
  5. Mark Dickinson

    John Machin Guest

    On Aug 25, 2:08 am, Chris Withers <> wrote:
    > Martin v. Löwis wrote:
    > > Today, there are two cases when malloc returns memory on a typical
    > > Unix system (in particular, in Linux malloc):
    > > a) if the malloc block block is small (below page size), it is allocated
    > >    from the brk heap, where it can only be returned if the last page of
    > >    that heap is completely free, and
    > > b) if the block is large (multiple pages), it gets returned to the
    > >    system right away.

    >
    > > Case b) can only happen if the C malloc uses mmap to allocate large
    > > blocks.

    >
    > I believe (and John will correct me if I'm wrong) that xlrd uses mmap,
    > so why am I not seeing the memory being returned?


    xlrd uses mmap to access the input file; this seems to have zero
    correlation with "the C malloc uses mmap to allocate large blocks".
     
    John Machin, Aug 25, 2009
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    17
    Views:
    2,545
    Grant Edwards
    Apr 26, 2007
  2. sixteenmillion

    The giving that keeps on giving

    sixteenmillion, Nov 19, 2007, in forum: C Programming
    Replies:
    0
    Views:
    432
    sixteenmillion
    Nov 19, 2007
  3. DR
    Replies:
    0
    Views:
    659
  4. Dave Angel
    Replies:
    0
    Views:
    517
    Dave Angel
    Aug 15, 2009
  5. DR
    Replies:
    0
    Views:
    761
Loading...

Share This Page