MemoryError, can I use more?

[Echavarria Gregory, Maria Angelica]

Dear group:

I am developing a program using Python 2.5.4 in windows 32 OS. The amount of data it works with is huge. I have managed to keep memory footprint low, but have found that, independent of the physical RAM of the machine, python always gives the MemoryError message when it has occupied exactly only 2.2 GB. I have tested this in 4 different machines, all with memory of 3 to 4 GB... I'm amazed.

Could any of you please help me to figure out how to change that limit? I typed help(MemoryError) and it is a class itself, but that help told me nothing I can use...

Thanks,
Angelica.

 

[Nobody]

I am developing a program using Python 2.5.4 in windows 32 OS. The amount
of data it works with is huge. I have managed to keep memory footprint
low, but have found that, independent of the physical RAM of the machine,
python always gives the MemoryError message when it has occupied exactly
only 2.2 GB. I have tested this in 4 different machines, all with memory
of 3 to 4 GB... I'm amazed.

Could any of you please help me to figure out how to change that limit? I
typed help(MemoryError) and it is a class itself, but that help told me
nothing I can use...

A single process can't use much more than 2GiB of RAM without a 64-bit CPU
and OS.

If you're stuck with a 32-bit system, you really need to figure out how to
limit the amount of data which is kept in memory at any one time.

 

[Anssi Saari]

A single process can't use much more than 2GiB of RAM without a 64-bit CPU
and OS.

That's not really true. Even Windows XP has the /3GB boot option to
allow 3 GiB per process. On PCs, free operating systems and server
Windows can use PAE to give access to full 4 GB per process.

 

[Diez B. Roggisch]

Am 13.02.10 17:18, schrieb Anssi Saari:

That's not really true. Even Windows XP has the /3GB boot option to
allow 3 GiB per process. On PCs, free operating systems and server
Windows can use PAE to give access to full 4 GB per process.

No, PAE can be used to access much more memory than 4GB - albeit through
paging. AFAIK up to 2^36 Bytes.

Diez

 

[Anssi Saari]

Am 13.02.10 17:18, schrieb Anssi Saari:

No, PAE can be used to access much more memory than 4GB - albeit
through paging. AFAIK up to 2^36 Bytes.

That too. I admit, after checking, that you can't go above 3 GiB per
process even in server Windows. But for Linux there exists (or
existed, since it seems it hasn't been updated since 2004) a kernel
patch which provides a "4GB/4GB" address split. Kernel is in one
segment, userland in another and hence a process can access full 4GB.

 

[Laszlo Nagy]

2010.02.13. 17:40 keltezéssel, Diez B. Roggisch írta:

Am 13.02.10 17:18, schrieb Anssi Saari:

No, PAE can be used to access much more memory than 4GB - albeit
through paging. AFAIK up to 2^36 Bytes.

PAE is for the kernel. Yes, you can use much more memory with 32 bit
kernel + PAE. But the ~3G per 32bit process limit still applies. It is
because the PAE extension allows the kernel to distribute the same
virtual address ranges between different processes, but map them to
different physical memory addresses. However, the process that was
compiled and is running in 32 bit mode, cannot access more than ~3GB
simply because it is not able to address more memory locations with 32
bit addresses. (minus ~1G is because it needs virtual addresses for I/O
devices as well, and those addresses cannot be mapped to physical memory).

So with any Python that is running in 32 bit mode, you cannot use more
than ~3G memory. But you can start many instances of those programs and
use 2G for each process.

L

 

[Ross Ridge]

No, PAE can be used to access much more memory than 4GB - albeit
through paging. AFAIK up to 2^36 Bytes.

That too. I admit, after checking, that you can't go above 3 GiB per
process even in server Windows. But for Linux there exists (or
existed, since it seems it hasn't been updated since 2004) a kernel
patch which provides a "4GB/4GB" address split. Kernel is in one
segment, userland in another and hence a process can access full 4GB.

Windows has a similar feature that allows 32-bit applications running on
64-bit versions of Windows to have a nearly 4Gb virtual address space.
Windows also allows 32-bit applications to use more than 4GB of physical
memory through a paging mechanism called "Address Windowing Extensions".
Also 32-bit applications can effectively use more than 4GB of RAM through
indirect means like multiple processes, the disk cache or video card RAM.

Ross Ridge

 

[Diez B. Roggisch]

Am 14.02.10 12:28, schrieb Laszlo Nagy:

2010.02.13. 17:40 keltezéssel, Diez B. Roggisch írta:
PAE is for the kernel. Yes, you can use much more memory with 32 bit
kernel + PAE. But the ~3G per 32bit process limit still applies. It is
because the PAE extension allows the kernel to distribute the same
virtual address ranges between different processes, but map them to
different physical memory addresses. However, the process that was
compiled and is running in 32 bit mode, cannot access more than ~3GB
simply because it is not able to address more memory locations with 32
bit addresses. (minus ~1G is because it needs virtual addresses for I/O
devices as well, and those addresses cannot be mapped to physical memory).

No. It can access more, through paging, similar like mmap. Specialized
software like databases do that to keep large data in memory. And this
has nothing to do with compilation.

Of course this doesn't help for python, at least not for python-objects,
as these need a flat memory model. But it's not true that only the
kernels benefit from the PAE.

Diez