RE: Python does not take up available physical memory

Discussion in 'Python' started by Prasad, Ramit, Oct 19, 2012.

  1. Emile van Sebille wrote:

    > On 10/19/2012 10:08 AM, Pradipto Banerjee wrote:

    > > Hi,
    > >
    > > I am trying to read a file into memory. The size of the file is around 1
    > > GB. I have a 3GB memory PC and the Windows Task Manager shows 2.3 GB
    > > available physical memory when I was trying to read the file. I tried to
    > > read the file as follows:
    > >

    > >>>> fdata = open(filename, 'r').read()

    > >
    > > I got a "MemoryError". I was watching the Windows TaskManager while I
    > > run the python command, and it appears that python **perhaps** never
    > > even attempted to use more memory but gave me this error.
    > >
    > > Is there any reason why python can't read a1GB file in memory even when
    > > a 2.3 GB physical memory is available?

    >
    > The real issue is likely that there is more than one copy of the file in
    > memory somewhere. I had a similar issue years back that I resolved by
    > using numeric (now numpy?) as it had a more efficient method of
    > importing content from disk.
    >
    > Also realize that windows may not allow the full memory to user space.
    > I'm not sure what exactly the restrictions are, but a 4Gb windows box
    > doesn't always get you 4Gb of memory.
    >


    Windows (bydefault) limits user space of a 32 bit machine to 2 GB.
    This is a bit old but I think still applies to pre-Win7.
    (scroll down to "32-bit Client Effective Memory Limits" )
    http://blogs.technet.com/b/markrussinovich/archive/2008/07/21/3092070.aspx

    Offhand, I am not sure how this works in Win7, but for 32-bit clients
    I doubt it has changed much.
    This email is confidential and subject to important disclaimers and
    conditions including on offers for the purchase or sale of
    securities, accuracy and completeness of information, viruses,
    confidentiality, legal privilege, and legal entity disclaimers,
    available at http://www.jpmorgan.com/pages/disclosures/email.
    Prasad, Ramit, Oct 19, 2012
    #1
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Emile van Sebille
    Replies:
    0
    Views:
    219
    Emile van Sebille
    Oct 20, 2012
  2. Ian Kelly
    Replies:
    0
    Views:
    132
    Ian Kelly
    Oct 19, 2012
  3. Pradipto Banerjee
    Replies:
    1
    Views:
    119
    Steven D'Aprano
    Oct 19, 2012
  4. Pradipto Banerjee
    Replies:
    6
    Views:
    209
    Pradipto Banerjee
    Oct 21, 2012
  5. MRAB
    Replies:
    0
    Views:
    104
Loading...

Share This Page