How to free memory ( ie garbage collect) at run time with Python 2.5.1(windows)

Discussion in 'Python' started by rfv-370, Aug 27, 2007.

  1. rfv-370

    rfv-370 Guest

    have made the following small test:

    Before starting my test my UsedPhysicalMemory(PF): 555Mb

    >>>tf=range(0,10000000) PF: 710Mb ( so 155Mb for my List)
    >>> tf=[0,1,2,3,4,5] PF: 672Mb (Why? Why the remaining 117Mb is not freed?)
    >>> del tf PF: 672Mb (unused memory not freed)


    So changing the list contents and/or deleting the list changes
    nothing...from a memory point of view.

    This is a problem as I have several applications/threads competing for
    memory share. (ie wxpython app).

    So how can I force Python to clean the memory and free the memory that
    is not used?


    PS: gc.collect() does nothing - To return memory I need to kill my
    process (ie the interpreter).

    Any ideas is welcome.

    Thanx

    Robert
     
    rfv-370, Aug 27, 2007
    #1
    1. Advertising

  2. rfv-370 <> wrote:

    > have made the following small test:
    >
    > Before starting my test my UsedPhysicalMemory(PF): 555Mb
    >
    > >>>tf=range(0,10000000) PF: 710Mb ( so 155Mb for my List)
    > >>>tf=[0,1,2,3,4,5] PF: 672Mb (Why? Why the remaining 117Mb is
    > >>>not freed?) del tf PF: 672Mb (unused memory
    > >>>not freed)


    Integer objects that are once generated are kept around in a "free list"
    against the probability that they might be needed again in the future (a
    few other types of objects similarly keep a per-type free-list, but I
    think int is the only one that keeps an unbounded amount of memory
    there). Like any other kind of "cache", this free-list (in normal
    cases) hoards a bit more memory than needed, but results in better
    runtime performance; anomalous cases like your example can however
    easily "bust" this too-simple heuristic.

    > So how can I force Python to clean the memory and free the memory that
    > is not used?


    On Windows, with Python 2.5, I don't know of a good approach (on Linux
    and other Unix-like systems I've used a strategy based on forking, doing
    the bit that needs a bazillion ints in the child process, ending the
    child process; but that wouldn't work on Win -- no fork).

    I suggest you enter a feature request to let gc grow a way to ask every
    type object to prune its cache, on explicit request from the Python
    program; this will not solve the problem in Python 2.5, but work on 3.0
    is underway and this is just the right time for such requests.


    Alex
     
    Alex Martelli, Aug 27, 2007
    #2
    1. Advertising

  3. rfv-370

    MC Guest

    Hi!

    For windows, I had a soft for this (reduce_memory.exe). Sorry, it's not
    write with Python.

    If you want, I will give a URL for download it.

    --
    @-salutations

    Michel Claveau
     
    MC, Aug 27, 2007
    #3
  4. Re: How to free memory ( ie garbage collect) at run time withPython 2.5.1(windows)

    On Mon, 27 Aug 2007 07:18:26 -0700, rfv-370 wrote:

    > have made the following small test:
    >
    > Before starting my test my UsedPhysicalMemory(PF): 555Mb
    >
    >>>>tf=range(0,10000000) PF: 710Mb ( so 155Mb for my List)
    >>>> tf=[0,1,2,3,4,5] PF: 672Mb (Why? Why the remaining 117Mb is not freed?)
    >>>> del tf PF: 672Mb (unused memory not freed)

    >
    > So changing the list contents and/or deleting the list changes
    > nothing...from a memory point of view.


    From the OS memory point of view to be precise. Although here the integer
    cache applies as Alex pointed out, in the general case memory might be
    kept allocated by Python to be re-used later. The Python allocator is
    better suited for frequent allocation and deallocation of many small
    objects than `malloc()` from the C standard library is. And even a
    `free()` in the C standard library doesn't guarantee that memory is given
    back to the OS and reported as free memory again.

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Aug 27, 2007
    #4
  5. rfv-370

    Peter Otten Guest

    Alex Martelli wrote:

    > Integer objects that are once generated are kept around in a "free list"
    > against the probability that they might be needed again in the future (a


    Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
    [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> x = 1000
    >>> y = 1000
    >>> x is y

    False

    Why don't x and y point to the same object then?

    Peter
     
    Peter Otten, Aug 27, 2007
    #5
  6. rfv-370

    Peter Otten Guest

    Peter Otten wrote:

    > Alex Martelli wrote:
    >
    >> Integer objects that are once generated are kept around in a "free list"
    >> against the probability that they might be needed again in the future (a

    >
    > Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
    > [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    > Type "help", "copyright", "credits" or "license" for more information.
    >>>> x = 1000
    >>>> y = 1000
    >>>> x is y

    > False
    >
    > Why don't x and y point to the same object then?
    >
    > Peter


    Oops, I think I got it now. The actual objects are freed, only the memory is
    kept around for reuse with other ints...

    Peter
     
    Peter Otten, Aug 27, 2007
    #6
  7. Re!

    Sended by direct (private) e-mail

    @+

    Michel Claveau
     
    Méta-MCI \(MVP\), Aug 27, 2007
    #7
  8. Aïe!

    gmail said : "illegal attachment" (because .exe?)
    I will try to send a zipped-file...

    @+

    Michel Claveau
     
    Méta-MCI \(MVP\), Aug 27, 2007
    #8
  9. Re: How to free memory ( ie garbage collect) at run time with Python2.5.1(windows)

    On 8/27/07, Peter Otten <> wrote:
    > Peter Otten wrote:
    >
    > > Alex Martelli wrote:
    > >
    > >> Integer objects that are once generated are kept around in a "free list"
    > >> against the probability that they might be needed again in the future (a

    > >
    > > Python 2.5.1 (r251:54863, May 2 2007, 16:56:35)
    > > [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
    > > Type "help", "copyright", "credits" or "license" for more information.
    > >>>> x = 1000
    > >>>> y = 1000
    > >>>> x is y

    > > False
    > > Why don't x and y point to the same object then?
    > > Peter

    > Oops, I think I got it now. The actual objects are freed, only the memory is
    > kept around for reuse with other ints...


    On my (windows) machine, only integer up to 256 are cached...
    I made two dictionaries with mapping from i to id(i) and then
    compared. They were equal up to 256.
    In short, try your example with 256 and 257 and see what happens :)

    francesco
     
    Francesco Guerrieri, Aug 28, 2007
    #9
  10. Re: How to free memory ( ie garbage collect) at run time with Python2.5.1(windows)

    Alex Martelli a écrit :
    > rfv-370 <> wrote:

    (snip)
    >> So how can I force Python to clean the memory and free the memory that
    >> is not used?

    >
    > On Windows, with Python 2.5, I don't know of a good approach (on Linux
    > and other Unix-like systems I've used a strategy based on forking, doing
    > the bit that needs a bazillion ints in the child process, ending the
    > child process; but that wouldn't work on Win -- no fork).


    IIRC, Windows has it's own way to let you launch other processes, so a
    similar strategy might apply here...
     
    Bruno Desthuilliers, Aug 28, 2007
    #10
  11. Hi!

    I've send the soft.
    But, I have no news ; good news?

    @+

    Michel Claveau
     
    Méta-MCI \(MVP\), Aug 29, 2007
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. John
    Replies:
    1
    Views:
    431
    John C. Bollinger
    Aug 19, 2004
  2. Kevin
    Replies:
    7
    Views:
    560
    Stephen Kellett
    Dec 19, 2005
  3. DR
    Replies:
    0
    Views:
    676
  4. DR
    Replies:
    0
    Views:
    779
  5. DR
    Replies:
    0
    Views:
    772
Loading...

Share This Page