PC locks up with list operations

[Steven D'Aprano]

Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
script that tries to create a list that is insanely too big.

In the first case, I (stupidly) did something like:

mylist = [0]*12345678901234

After leaving the machine for THREE DAYS (!!!) I eventually was able to get
to a console and kill the Python process. Amazingly, it never raised
MemoryError in that time.

The second time was a little less stupid, but not much:

mylist = []
for x in itertools.combinations_with_replacement(some_big_list, 20):
mylist.append(func(x))

After three hours, the desktop is still locked up. I'm waiting to see what
happens in the morning before rebooting.

Apart from "Then don't do that!", is there anything I can do to prevent this
sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

I am using Linux desktops; both incidents were with Python 2.5. Do newer
versions of Python respond to this sort of situation more gracefully?

 

[Chris Withers]

I am using Linux desktops; both incidents were with Python 2.5. Do newer
versions of Python respond to this sort of situation more gracefully?

Ironically, Windows does better here and dumps you out with a
MemoryError before slowly recovering.

Linux seems to fair badly when programs use more memory than physically
available. Perhaps there's some per-process thing that can be used to
limit things on Linux?

cheers,

Chris

 

[Steven D'Aprano]

Ironically, Windows does better here and dumps you out with a
MemoryError before slowly recovering.

Linux seems to fair badly when programs use more memory than physically
available. Perhaps there's some per-process thing that can be used to
limit things on Linux?

As far as I know, ulimit ("user limit") won't help. It can limit the amount
of RAM available to a process, but that just makes the process start using
virtual memory more quickly. It can also limit the amount of virtual memory
used by the shell, but not of other processes. In other words, Linux will
try really, really, really hard to give you the 84 gigabytes you've asked
for on a 2 GB system, even if it means DOSing your system for a month.

Of course, I would be very happy to learn I'm wrong.

 

[Benjamin Kaplan]

Ironically, Windows does better here and dumps you out with a MemoryErrorbefore slowly recovering.

Linux seems to fair badly when programs use more memory than physically available. Perhaps there's some per-process thing that can be used to limit things on Linux?

cheers,

Chris

--

32-bit or 64-bit Python? A 32-bit program will crash once memory hits
2GB. A 64-bit program will just keep consuming RAM until your computer
starts thrashing. The problem isn't your program using more RAM than
you have, just more RAM than you have free. Last time I faced a
situation like this, I just decided it was better to stick to the
32-bit program and let it crash if it got too big.

 

[Peter Otten]

Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
script that tries to create a list that is insanely too big.

In the first case, I (stupidly) did something like:

mylist = [0]*12345678901234

After leaving the machine for THREE DAYS (!!!) I eventually was able to
get to a console and kill the Python process. Amazingly, it never raised
MemoryError in that time.

The second time was a little less stupid, but not much:

mylist = []
for x in itertools.combinations_with_replacement(some_big_list, 20):
mylist.append(func(x))

After three hours, the desktop is still locked up. I'm waiting to see what
happens in the morning before rebooting.

Apart from "Then don't do that!", is there anything I can do to prevent
this sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

I am using Linux desktops; both incidents were with Python 2.5. Do newer
versions of Python respond to this sort of situation more gracefully?

If you are starting these scripts from the shell, how about ulimit?

$ ulimit -v 40000
$ python -c'print range(10**5)[-1]'
99999
$ python -c'print range(10**6)[-1]'
Traceback (most recent call last):
File "<string>", line 1, in <module>
MemoryError
$

 

[Robert Spanjaard]

Ironically, Windows does better here and dumps you out with a
MemoryError before slowly recovering.

I think it's a little premature to make such a statement based on a single
user experience. I've used Linux for six years now, and it NEVER locked up,
even when a program leaks memory like hell.
I can't duplicate the OP's behaviour because my Python (2.6.5, 64 bit) does
generate an instant MemoryError (which answers the question, ofcourse), but
I've used a VLC version that had a nasty leak. My swap space is on a SSD,
and VLC filled it completely in about 10 seconds. But then, VLC got killed
automatically, and the system recovered gracefully.
On a system with the swapspace on a regular HD, the same will happen, but
it'll take more time. And while VLC is filling the swap space, the system
does not lock up at all. You can still keep working, and (for example) fire
up the process manager and kill VLC yourself.

mylist = [0]*12345678901234

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError

 

[Rodrick Brown]

$ man limits.conf

Sent from my iPhone

 

[Gregory Ewing]

As far as I know, ulimit ("user limit") won't help. It can limit the amount
of RAM available to a process, but that just makes the process start using
virtual memory more quickly.

ulimit -v is supposed to set the maximum amount of virtual
memory the process can use.

It can also limit the amount of virtual memory
used by the shell, but not of other processes.

That doesn't sound right. Not sure about Linux, but the
man page for sh on Darwin says:

Provides control over the resources available to the shell and
to processes started by it, on systems that allow such control.

The Python process should also be able to set its own
limits using resource.setrlimit().

 

[Tim Chase]

The Python process should also be able to set its own
limits using resource.setrlimit().

A new corner of stdlib that I've never poked at. Thanks for the
suggestion. Disappointed though that it doesn't seem to have
docstrings on the functions, so I had to wade back out to the
online docs to probe at it. Granted, after the fact, they were
pretty obvious, but it would be nice if
"help(resource.getrlimit)" gave me a hint as to what that one
expected parameter should have been.

-tim

import resource as r
token = "RLIMIT_"
for item in dir(r):
if item.startswith(token):
print "%s:" % item[len(token):],
print "%s hard/%s soft" % r.getrlimit(getattr(r, item))

 

[Steven D'Aprano]

Twice in a couple of weeks, I have locked up my PC by running a Python 2.5
script that tries to create a list that is insanely too big.

In the first case, I (stupidly) did something like:

mylist = [0]*12345678901234 [...]
Apart from "Then don't do that!", is there anything I can do to prevent
this sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

For anyone who may care, I can report that ulimit under Linux will help with
this situation.

[steve@wow-wow ~]$ ulimit -v 200000
[steve@wow-wow ~]$ python
Python 2.5 (r25:51908, Nov 6 2007, 16:54:01)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError

(Of course, I would have eventually got a MemoryError even without the
ulimit. *Eventually.* After much thrashing and processes crashing and
pain.)


Does anyone else think it would be useful for Python's memory manager to
enforce user-settable limits? Even just a simple thing like "never try to
allocate more than N bytes at once" would probably go a long way to prevent
a certain class of accidental (or deliberate) DOSes.

 

[Roy Smith]

mylist = [0]*12345678901234 [...]
Apart from "Then don't do that!", is there anything I can do to prevent
this sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

For anyone who may care, I can report that ulimit under Linux will help with
this situation.
[...]
Does anyone else think it would be useful for Python's memory manager to
enforce user-settable limits?[/QUOTE]

Not me. You've already discovered that ulimit does exactly what you
want. Why would be gained by having Python duplicate this functionality?

 

[Nobody]

As far as I know, ulimit ("user limit") won't help. It can limit the amount
of RAM available to a process, but that just makes the process start using
virtual memory more quickly. It can also limit the amount of virtual memory
used by the shell, but not of other processes. In other words, Linux will
try really, really, really hard to give you the 84 gigabytes you've asked
for on a 2 GB system, even if it means DOSing your system for a month.

Of course, I would be very happy to learn I'm wrong.

Resource limits set by ulimit are inherited by any processes spawned from
the shell. They also affect the shell itself, but a shell process
shouldn't require a lot of resources. You can use a subshell if you want
to impose limits on a specific process.

For Python, setting the limit on virtual memory (RAM + swap) to no more
than the amount of physical RAM is probably a wise move. Some processes
can use swap effectively, but the typical Python program probably can't.
There are exceptions, e.g. if most of the memory is accounted for by large
NumPy arrays and you're careful about the operations which are performed
upon them. But using large amounts of memory for many small objects is
almost bound to result in swap-thrashing.

One problem with doing this automatically (e.g. in .pythonrc) is the
inheritance issue; any processes spawned from the interpreter will also be
resource limited. Similarly, any binary libraries loaded into the
interpreter will be subject to the process' resource limits. Consequently,
there would be some advantages to the Python interpreter having its own
resource-limiting mechanism.

 

[Terry Reedy]

mylist = [0]*12345678901234 [...]
Apart from "Then don't do that!", is there anything I can do to prevent
this sort of thing in the future? Like instruct Python not to request more
memory than my PC has?

For anyone who may care, I can report that ulimit under Linux will help with
this situation.
[...]
Does anyone else think it would be useful for Python's memory manager to
enforce user-settable limits?

Not me. You've already discovered that ulimit does exactly what you
want. Why would be gained by having Python duplicate this functionality?

Having it on Windows.

 

[Carl Banks]

32-bit or 64-bit Python? A 32-bit program will crash once memory hits
2GB. A 64-bit program will just keep consuming RAM until your computer
starts thrashing. The problem isn't your program using more RAM than
you have, just more RAM than you have free. Last time I faced a
situation like this, I just decided it was better to stick to the
32-bit program and let it crash if it got too big.

On my 64-bit Linux system, I got a memory error in under a second, no thrashing.

I have no swap. It's overrated.


Carl Banks

 

[John Nagle]

Ironically, Windows does better here and dumps you out with a
MemoryError before slowly recovering.

Linux seems to fair badly when programs use more memory than physically
available.

Linux virtual memory has some known design problems. For example,
you can crash many Linux systems with a program which opens large
files and accesses them randomly, combined with another program which
is spawning processes that need a fair amount of memory. The disk
caching system tries to use all unused memory, and when process
startup is consuming pages, it's possible to get into a situation
where a page is needed, the task requesting it can't block, and
a lock preventing freeing a file cache page is set.

Arguably, paging to disk is obsolete. RAM is too cheap and
disk is too slow.

John Nagle