Memory utilization (linux v. openbsd)

[nazgul]

Hi all,

I have an app that runs on multiple boxes. On my slackware box, running
Python 2.5.1, top shows this:

Mem: 1002736k total, 453268k used, 549468k free, 31392k buffers
Swap: 2097136k total, 0k used, 2097136k free, 136876k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2741 dnm 16 0 267m 261m 2676 S 99.3 26.7 14:54.62 python

The same app running on OpenBSD 4.1 (but on Python 2.5P3 from ports)
yields this (actually, I have 2 copies running on a dualcore box):

Memory: Real: 1130M/1989M act/tot Free: 792K Swap: 337M/2048M used/tot

PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU
COMMAND
12380 dnm 64 0 926M 436M run/0 - 22:22 73.58%
python2.5
22956 dnm 64 0 933M 695M onproc/0 - 22:46 68.55%
python2.5

RES was has over 800M earlier in the job.

My problem is that the two process under OpenBSD are going to fail with
a MemoryError becaause the size just keeps getting larger and larger.
ulimit -d is 1G for each process.

Any idea why the memory utilization is so much higher under OpenBSD?

What I also find interesting is that the OpenBSD box seems to grind to a
halt (that's probably due to swapping since there's 500M of swap space
in use). Those processes should both be over 90% since they are CPU
bound, and they are that way at the start of the run. Gradually, the
system % creeps up as does idle, but there's no reason for them to be
idle. It's a CPU bound process. (While I'm typing this top is showing
cpu rats of 63% and 41% for the two jobs).

TIA,
nazgul

 

[Michael Torrie]

My problem is that the two process under OpenBSD are going to fail with
a MemoryError becaause the size just keeps getting larger and larger.
ulimit -d is 1G for each process.

The problem is that you can't get accurate memory use readings from top.

The reality is that your python program is probably using the same
amount of memory on both platforms (in and of itself), but due to the
way the memory management OS system works, the usage is being reported
differently on BSD. Basically you're seeing the memory footprint of the
python executable, it's data (your program) and also all of the shared
libraries that python is linked to. Of course the shared libraries are
shared, so it's not like python's using all this memory just itself.

Any idea why the memory utilization is so much higher under OpenBSD?

What I also find interesting is that the OpenBSD box seems to grind to a
halt (that's probably due to swapping since there's 500M of swap space
in use). Those processes should both be over 90% since they are CPU
bound, and they are that way at the start of the run. Gradually, the
system % creeps up as does idle, but there's no reason for them to be
idle. It's a CPU bound process. (While I'm typing this top is showing
cpu rats of 63% and 41% for the two jobs).

I can't answer your questions or solve your problem. But I just wanted
to point out that the problem is likely somewhere besides python itself.
Maybe there are settings in the OS. Who knows. My guess is the
difference in overall memory footprint is due to differences in the C
library and its dependencies. You probably would want to bring this
issue up on the OpenBSD lists as it's clear that your BSD machine is
having problems generally (trashing to death). This doesn't appear to
be a python problem.