Re: 64 bit memory usage

Discussion in 'Python' started by Dennis Lee Bieber, Dec 9, 2010.

  1. On Wed, 8 Dec 2010 14:44:30 +0000, Rob Randall <>
    declaimed the following in gmane.comp.python.general:

    > I am trying to understand how much memory is available to a 64 bit python
    > process running under Windows XP 64 bit.
    >
    > When I run tests just creating a series of large dictionaries containing
    > string keys and float values I do not seem to be able to grow the process
    > beyond the amount of RAM present.
    >
    > For example, on a box with 2GB RAM and 3 GB pagefile the process stalls at
    > around 2GB.
    >
    > On another machine with 16GB RAM and 24GB pagefile the process stalls at
    > 16GB.
    >


    Probably need to ask M$, but I can understand this behavior as a
    hypothetical...

    The page file can be larger than physical memory because it contains
    memory "images" for multiple processes. However, all those "images" have
    to map into the physically addressable memory -- so a process is likely
    limited to physical memory, but you can have multiple processes adding
    up to physical + pagefile in total.
    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
     
    Dennis Lee Bieber, Dec 9, 2010
    #1
    1. Advertising

  2. Dennis Lee Bieber

    Nobody Guest

    Rob Randall <> wrote:

    > I am trying to understand how much memory is available to a 64 bit python
    > process running under Windows XP 64 bit.
    >
    > When I run tests just creating a series of large dictionaries containing
    > string keys and float values I do not seem to be able to grow the process
    > beyond the amount of RAM present.
    >
    > For example, on a box with 2GB RAM and 3 GB pagefile the process stalls at
    > around 2GB.


    What do you mean by "stalls"? Do you get an exception, or does the program
    just slow to a crawl?
     
    Nobody, Dec 9, 2010
    #2
    1. Advertising

  3. Dennis Lee Bieber

    John Nagle Guest

    On 12/8/2010 10:42 PM, Dennis Lee Bieber wrote:
    > On Wed, 8 Dec 2010 14:44:30 +0000, Rob Randall<>
    > declaimed the following in gmane.comp.python.general:
    >
    >> I am trying to understand how much memory is available to a 64 bit python
    >> process running under Windows XP 64 bit.
    >>
    >> When I run tests just creating a series of large dictionaries containing
    >> string keys and float values I do not seem to be able to grow the process
    >> beyond the amount of RAM present.


    If you get to the point where you need multi-gigabyte Python
    dictionaries, you may be using the wrong tool for the job.
    If it's simply that you need to manage a large amount of data,
    that's what databases are for.

    If this is some super high performance application that needs to
    keep a big database in memory for performance reasons, CPython
    is probably too slow. For that, something like Google's BigTable
    may be more appropriate, and will scale to terabytes if necessary.

    John Nagle
     
    John Nagle, Dec 9, 2010
    #3
  4. Dennis Lee Bieber

    Steve Holden Guest

    On 12/10/2010 2:03 PM, Rob Randall wrote:
    > I manged to get my python app past 3GB on a smaller 64 bit machine.
    > On a test to check memory usage with gc disabled only an extra 6MB was
    > used.
    > The figures were 1693MB to 1687MB.
    >
    > This is great.
    >
    > Thanks again for the help.


    Do remember, though, that with the GC turned off you will "lose" memory
    if you accidentally create cyclic data structures, since they will never
    be reclaimed. It doesn't sound like this is an issue, but I wanted this
    to act as a warning to others who might come across your solution but
    have programmed less carefully.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    PyCon 2011 Atlanta March 9-17 http://us.pycon.org/
    See Python Video! http://python.mirocommunity.org/
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Dec 11, 2010
    #4
  5. Dennis Lee Bieber

    Steve Holden Guest

    On 12/10/2010 2:03 PM, Rob Randall wrote:
    > I manged to get my python app past 3GB on a smaller 64 bit machine.
    > On a test to check memory usage with gc disabled only an extra 6MB was
    > used.
    > The figures were 1693MB to 1687MB.
    >
    > This is great.
    >
    > Thanks again for the help.


    Do remember, though, that with the GC turned off you will "lose" memory
    if you accidentally create cyclic data structures, since they will never
    be reclaimed. It doesn't sound like this is an issue, but I wanted this
    to act as a warning to others who might come across your solution but
    have programmed less carefully.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    PyCon 2011 Atlanta March 9-17 http://us.pycon.org/
    See Python Video! http://python.mirocommunity.org/
    Holden Web LLC http://www.holdenweb.com/
     
    Steve Holden, Dec 11, 2010
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. metfan
    Replies:
    2
    Views:
    4,875
    Robert Olofsson
    Oct 21, 2003
  2. Replies:
    3
    Views:
    1,830
    Timothy Bendfelt
    Jan 19, 2007
  3. hvt
    Replies:
    0
    Views:
    1,231
  4. Replies:
    9
    Views:
    1,033
    Juha Nieminen
    Aug 22, 2007
  5. Jeff.M
    Replies:
    6
    Views:
    199
    Lasse Reichstein Nielsen
    May 4, 2009
Loading...

Share This Page