hard memory limits

[Philippe C. Martin]

Hi,

Why don't you catch the exception and print the trace ?

Regards,

Philippe

It doesn't seems to help. I'm thinking that it might be a SOAPpy
problem. The allocation fails when I grab a list of more than 150k
elements through SOAP but allocating a 1 million element list is fine in
python.

Now I have a performance problem...

Say I have 3 lists (20K elements, 1G elements, and 0 elements), call
them 'a', 'b', and 'c'. I want to filter all that is in 'b' but not in
'a' into 'c'...


a = range(1, 100000, 5)
b = range(0, 1000000)
c = []
for i in b:

... if i not in a: c.append(i)
...

This takes forever to complete. Is there anyway to optimize this?


Checking whether something is in a list may average checking equality
with each element in half the list. Checking for membership in a set
should be much faster for any significant size set/list. I.e., just
changing to

a = set(range(1, 100000, 5))

should help. I assume those aren't examples of your real data ;-)
You must have a lot of memory if you are keeping 1G elements there and
copying a significant portion of them. Do you need to do this
file-to-file, keeping a in memory? Perhaps page-file thrashing is part of
the time problem?

Since when was 1000000 == 1G??

Maurice, is this mucking about with 1M or 1G lists in the same
exercise as the "vm_malloc fails when allocating a 20K-element list"
problem? Again, it might be a good idea if you gave us a little bit
more detail. You haven't even posted the actual *PYTHON* error message
and stack trace that you got from the original problem. In fact,
there's a possible interpretation that the (system?) malloc merely
prints the vm_malloc message and staggers on somehow ...

Regards,
John

This is the exact error message:

*** malloc: vm_allocate(size=9203712) failed (error code=3)
*** malloc[489]: error: Can't allocate region

Nothing else. No stack trace, NOTHING.

maurice

 

[Robert Kern]

Hi,

Why don't you catch the exception and print the trace ?

I don't think a Python exception is ever raised. The error message
quoted below comes from the system, not Python.

Regards,

Philippe
>
Maurice LING wrote:

This is the exact error message:

*** malloc: vm_allocate(size=9203712) failed (error code=3)
*** malloc[489]: error: Can't allocate region

Nothing else. No stack trace, NOTHING.

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[John Roth]

Hi,

I think I've hit a system limit in python when I try to construct a list
of 200,000 elements. My error is

malloc: vm_allocate (size = 2400256) failed......

Just wondering is this specific to my system or what? Will adding more RAM
helps in this case?

Thanks and cheers
Maurice

malloc (which is the memory manager Python uses when it
runs out of its own heap memory) is trying to get another
2.4 megabyte block of memory from the operating system
so it can expand the heap.

The operating system is refusing to fill the request. There are
a lot of reasons why this might happen, ranging from system
limits (too little swap space, too little real memory), to an
inability to find a 2.4 meg block in the user part of the
address space, etc.

I'd check swap space, and then balance redesigning the
application to not try to get large blocks of memory with
spending money on more hardware memory that might not
solve the problem.

John Roth

 

[Bengt Richter]

It doesn't seems to help. I'm thinking that it might be a SOAPpy
problem. The allocation fails when I grab a list of more than 150k
elements through SOAP but allocating a 1 million element list is fine in
python.

Now I have a performance problem...

Say I have 3 lists (20K elements, 1G elements, and 0 elements), call
them 'a', 'b', and 'c'. I want to filter all that is in 'b' but not in
'a' into 'c'...


a = range(1, 100000, 5)
b = range(0, 1000000)
c = []
for i in b:

... if i not in a: c.append(i)
...

This takes forever to complete. Is there anyway to optimize this?


Checking whether something is in a list may average checking equality with
each element in half the list. Checking for membership in a set should
be much faster for any significant size set/list. I.e., just changing to

a = set(range(1, 100000, 5))

should help. I assume those aren't examples of your real data ;-)
You must have a lot of memory if you are keeping 1G elements there and
copying a significant portion of them. Do you need to do this file-to-file,
keeping a in memory? Perhaps page-file thrashing is part of the time problem?

Since when was 1000000 == 1G??

Maurice, is this mucking about with 1M or 1G lists in the same
exercise as the "vm_malloc fails when allocating a 20K-element list"
problem? Again, it might be a good idea if you gave us a little bit
more detail. You haven't even posted the actual *PYTHON* error message
and stack trace that you got from the original problem. In fact,
there's a possible interpretation that the (system?) malloc merely
prints the vm_malloc message and staggers on somehow ...

Regards,
John

This is the exact error message:

*** malloc: vm_allocate(size=9203712) failed (error code=3)
*** malloc[489]: error: Can't allocate region

Nothing else. No stack trace, NOTHING.

1. Can you post minimal exact code that produces the above exact error message?
2. Will you? ;-)

Regards,
Bengt Richter

 

[Maurice LING]

On Sat, 07 May 2005 02:29:48 GMT, (e-mail address removed) (Bengt Richter) wrote:





It doesn't seems to help. I'm thinking that it might be a SOAPpy
problem. The allocation fails when I grab a list of more than 150k
elements through SOAP but allocating a 1 million element list is fine in
python.

Now I have a performance problem...

Say I have 3 lists (20K elements, 1G elements, and 0 elements), call
them 'a', 'b', and 'c'. I want to filter all that is in 'b' but not in
'a' into 'c'...



a = range(1, 100000, 5)
b = range(0, 1000000)
c = []
for i in b:

... if i not in a: c.append(i)
...

This takes forever to complete. Is there anyway to optimize this?


Checking whether something is in a list may average checking equality with
each element in half the list. Checking for membership in a set should
be much faster for any significant size set/list. I.e., just changing to

a = set(range(1, 100000, 5))

should help. I assume those aren't examples of your real data ;-)
You must have a lot of memory if you are keeping 1G elements there and
copying a significant portion of them. Do you need to do this file-to-file,
keeping a in memory? Perhaps page-file thrashing is part of the time problem?


Since when was 1000000 == 1G??

Maurice, is this mucking about with 1M or 1G lists in the same
exercise as the "vm_malloc fails when allocating a 20K-element list"
problem? Again, it might be a good idea if you gave us a little bit
more detail. You haven't even posted the actual *PYTHON* error message
and stack trace that you got from the original problem. In fact,
there's a possible interpretation that the (system?) malloc merely
prints the vm_malloc message and staggers on somehow ...

Regards,
John

This is the exact error message:

*** malloc: vm_allocate(size=9203712) failed (error code=3)
*** malloc[489]: error: Can't allocate region

Nothing else. No stack trace, NOTHING.

1. Can you post minimal exact code that produces the above exact error message?
2. Will you? ;-)

Regards,
Bengt Richter

I've re-tried the minimal code mimicking the error in interactive mode
and got this:
WSDL.Proxy('http://eutils.ncbi.nlm.nih.gov/entrez/eutils/soap/v1.1/eutils.wsdl'
)*** malloc: vm_allocate(size=9121792) failed (error code=3)
*** malloc[901]: error: Can't allocate region
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/sw/lib/python2.3/site-packages/SOAPpy/Client.py", line 453, in
__call__
return self.__r_call(*args, **kw)
File "/sw/lib/python2.3/site-packages/SOAPpy/Client.py", line 475, in
__r_call
self.__hd, self.__ma)
File "/sw/lib/python2.3/site-packages/SOAPpy/Client.py", line 347, in
__call
config = self.config)
File "/sw/lib/python2.3/site-packages/SOAPpy/Client.py", line 212, in
call
data = r.getfile().read(message_len)
File "/sw/lib/python2.3/socket.py", line 301, in read
data = self._sock.recv(recv_size)
MemoryError

When changed retmax to 150000, it works nicely.

 

[Fredrik Lundh]

There is something very non-unixy going on here, though. Why is
vm_malloc exiting with an error message, instead of returning a
failure to the calling application? I've seen other applications
include a FOSS malloc implementation to work around bugs in the
system's malloc. Maybe Python should do that on the Mac?

from what I can tell (by reading the google hits), the malloc implementation
prints a message, but returns NULL as usual (you can find reports of this
message preceeding a MemoryError traceback).

</F>