Re: Python garbage collector/memory manager behaving strangely

Discussion in 'Python' started by Robert Miles, Nov 1, 2012.

  1. Robert Miles

    Robert Miles Guest

    On 9/16/2012 9:12 PM, Dave Angel wrote:
    > On 09/16/2012 09:07 PM, Jadhav, Alok wrote:
    >> Hi Everyone,
    >> I have a simple program which reads a large file containing few million
    >> rows, parses each row (`numpy array`) and converts into an array of
    >> doubles (`python array`) and later writes into an `hdf5 file`. I repeat
    >> this loop for multiple days. After reading each file, i delete all the
    >> objects and call garbage collector. When I run the program, First day
    >> is parsed without any error but on the second day i get `MemoryError`. I
    >> monitored the memory usage of my program, during first day of parsing,
    >> memory usage is around **1.5 GB**. When the first day parsing is
    >> finished, memory usage goes down to **50 MB**. Now when 2nd day starts
    >> and i try to read the lines from the file I get `MemoryError`. Following
    >> is the output of the program.

    Is it a 32-bit program? If so, expect the maximum amount of memory it
    can use to hold the program, its current dataspace, and images of all
    the files it has open to be about 3.5 GB, even if it is running on a
    64-bit computer with over 4 GB of memory. It seems that 32-bit
    addresses can only refer to 4 GB of memory, and part of that 4 GB
    must be used for whatever the operating system needs for running
    32-bit programs. With some of the older compilers, only 2 GB can be
    used for the program; the other 2 GB is reserved for the operating system.

    How practical would it be to have that program run twice a day?
    The first time, it should ignore all the data for the second half
    of the day; the second time, it should ignore all the data for the
    first half of the day.
    Robert Miles, Nov 1, 2012
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?iso-8859-1?q?Nils Hedstr=f6m

    Stateserver behaving strangely

    =?iso-8859-1?q?Nils Hedstr=f6m, Feb 11, 2005, in forum: ASP .Net
    =?iso-8859-1?q?Nils Hedstr=f6m
    Feb 11, 2005
  2. Josh Close

    re behaving strangely

    Josh Close, Jun 9, 2005, in forum: Python
    Josh Close
    Jun 9, 2005
  3. Jadhav, Alok
    Thomas Rachel
    Nov 15, 2012
  4. Jadhav, Alok
    Jadhav, Alok
    Sep 17, 2012
  5. Dennis Lee Bieber
    Dennis Lee Bieber
    Sep 17, 2012

Share This Page