Memory Problems in Windows 2003 Server

[amdescombes]

Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes

 

[brad]

Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes

How are you reading the large files? IMO, large files are better read in
chunks:

target_file = open(f, 'rb')
while 1:
data = target_file.read(8192000)
if data:
DO SOMETHING
else:
break

The above reads 8MB at a time until the file has been completely read.
Change the 8MB to whatever you like.

 

[brad]

Hi,

I am using Python 2.5.1
I have an application that reads a file and generates a key in a
dictionary for each line it reads. I have managed to read a 1GB file and
generate more than 8 million keys on an Windows XP machine with only 1GB
of memory and all works as expected. When I use the same program on a
Windows 2003 Server with 2GB of RAM I start getting MemoryError exceptions!
I have tried setting the IMAGE_FILE_LARGE_ADDRESS_AWARE on both
Python.exe and Python25.dll and setting the /3GB flag on the boot.ini
file to no avail. I still get the MemoryError exceptions.

Has anybody encountered this problem before?

Thanks in advance for any ideas/suggestions.

Best Regards,

André M. Descombes

I forgot to mention that the OS itself or other processes may be using a
lot of memory. So, just because you have 2GB, that does not mean you can
access all of that at once. I would guess that 25% of memory is in
constant use by the OS. So, do your IO/reads in smaller chunks similar
to the example I gave earlier.

Brad

 

[AMD]

Hi Brad,

I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.

Andre

 

[Tim Roberts]

I do the reading one line at a time, the problem seems to be with the
dictionary I am creating.

I don't know whether Python dictionaries must live in a contiguous piece of
memory, but if so, that could be the issue. The system DLLs in Server 2003
have been "rebased" in such a way that they chop up the virtual address
space more than XP. Even though there is more virtual memory available, it
is fragmented.

 

[amdescombes]

Yes, I think that might be the issue, perhaps I could implement the
solution using several dictionaries instead of just one.
Are there any classes that implement disk based dictionaries?

Thanks,

Andre

 

[Marc 'BlackJack' Rintsch]

Are there any classes that implement disk based dictionaries?

Take a look at the `shelve` module from the standard library.

Or object databases like ZODB or Durus.

Ciao,
Marc 'BlackJack' Rintsch

 

[AMD]

Thanks Marc,

I just tried shelve but it is very slow
I haven't tried the dbs yet.

Andre

Marc 'BlackJack' Rintsch a écrit :