Re: Python does not take up available physical memory

Discussion in 'Python' started by Emile van Sebille, Oct 20, 2012.

  1. On 10/19/2012 10:08 AM, Pradipto Banerjee wrote:
    > Hi,
    >
    > I am trying to read a file into memory. The size of the file is around 1
    > GB. I have a 3GB memory PC and the Windows Task Manager shows 2.3 GB
    > available physical memory when I was trying to read the file. I tried to
    > read the file as follows:
    >
    >>>> fdata = open(filename, ‘r’).read()

    >
    > I got a “MemoryError”. I was watching the Windows Task Manager while I
    > run the python command, and it appears that python **perhaps** never
    > even attempted to use more memory but gave me this error.
    >
    > Is there any reason why python can’t read a 1GB file in memory even when
    > a 2.3 GB physical memory is available?


    The real issue is likely that there is more than one copy of the file in
    memory somewhere. I had a similar issue years back that I resolved by
    using numeric (now numpy?) as it had a more efficient method of
    importing content from disk.

    Also realize that windows may not allow the full memory to user space.
    I'm not sure what exactly the restrictions are, but a 4Gb windows box
    doesn't always get you 4Gb of memory.

    Emile
     
    Emile van Sebille, Oct 20, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Ian Kelly
    Replies:
    0
    Views:
    165
    Ian Kelly
    Oct 19, 2012
  2. Pradipto Banerjee
    Replies:
    1
    Views:
    146
    Steven D'Aprano
    Oct 19, 2012
  3. Pradipto Banerjee
    Replies:
    6
    Views:
    240
    Pradipto Banerjee
    Oct 21, 2012
  4. MRAB
    Replies:
    0
    Views:
    125
  5. Prasad, Ramit
    Replies:
    0
    Views:
    146
    Prasad, Ramit
    Oct 19, 2012
Loading...

Share This Page