Overhead of individual python apps

Q

Qopit

I'm setting up a system that consists of several small python
applications that all communicate amongst each other on the same pc.

When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.

My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?

One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,

Another possibility I considered is to strip out all but the most
essential imports in each app, but I tested this out and it has
marginal benefits. I demonstrated to myself that a simple one liner
app consisting of 'x = raw_input()' still eats up > 2.7MB .

I also tried -O but it, not surprisingly, did nothing for the
one-liner.

I'm simply running the .py files and I am still on v2.3

All help appreciated!

Thanks,
Russ
 
M

Mike Meyer

Qopit said:
When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.

The first step is to clarify what's being reported. If WTM is
reporting the total memory usage for each process, then it's over
estimating the total usage by a considerable amount. In particular,
all the Python executable code should be shared by all the
processes. Unless you load compiled extensions, anyway - those will
only be shared by the ones that use them. You'll need something that
will give you the stack, heap and code segment sizes separately to
work out what the memory usage really is.
My Question: Is there any way to reduce this per-process overhead? eg:
can you set it somehow so that one python.exe instance handles multiple
processes?

The OS should do that for you by default.
One possibility considered is to run them as threads of a single
process rather than multiple processes, but this has other drawbacks
for my application and I'd rather not,

That shouldn't help memory usage - the data that isn't across
processes would need to be thread-private in any case.

The reason for the uncertainty is that I'm not positive that Windows
behaves sanely in this area. It may be that Windows doesn't have
shared executables. In this case, one solution is to move to a modern
OS - like v6 Unix :).

<mike
 
L

Larry Bates

Several apps using 4Mb each shouldn't be very much
memory (maybe 20Mb at most). You didn't say how
much memory was in your machine, but 256Mb of memory
will cost you no more than $50. Not really worth a
lot of effort.

-Larry
 
P

Paul Rubin

Larry Bates said:
Several apps using 4Mb each shouldn't be very much memory (maybe
20Mb at most). You didn't say how much memory was in your machine,
but 256Mb of memory will cost you no more than $50. Not really
worth a lot of effort.

That is bogus reasoning. I can't add 256mb to the machine I'm typing
on right now for any amount of money, let alone $50. It's an older
laptop, it's maxed out at 512 meg, and I can't add any more to it,
period. An even older laptop that I still use from time to time is
maxed out at 72 meg. And my Sharp Zaurus (Linux handheld) has 32 meg
and is not expandable. If Python claims to be a lightweight language,
it should not present any obstacles to running on machines of that
class. It's already slightly annoying that Python can't run well in
an 8 meg Palm Pilot, since there were many Common Lisp implementations
that ran tolerably in less memory than that.
 
N

ncf

First, please don't get so upset at people's replies (if you weren't
upset, that's how it was coming across, so my apologies). No matter
what newsgroup/forum you go to, there's always someone who's going to
suggest something like that.

Anyways, I'm fairly certain there are some minimalistic Python variants
designed for older computers and such. Maybe something to check into?

HTH,
-Wes
 
D

Dennis Lee Bieber

When running in Windows, launching each application generates a
process, and each of those processes ends up taking up > 4MB of system
memory. This memory usage is as reported by the Windows Task manager
for the python.exe image name.
Ah, but is that physical memory consumed, or virtual memory MAPPED
to the processes.

Most of Python is a DLL; as I recall, most DLLs are shared -- just
the process local data needs to be allocated.
--
 
F

Fredrik Lundh

Paul said:
That is bogus reasoning.

not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.

people are expensive, but hardware is extremely cheap these days.

</F>
 
P

Paul Rubin

Fredrik Lundh said:
not if you're a professional software developer and someone's paying you
to develop an application that is to be run on a platform that they control.

An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.
 
F

Fredrik Lundh

Paul said:
An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.

(and so we go from the OP:s "I'm setting up a system" to the usual c.l.python "but
I can come up with another example" crap...)

But back to the "memory measurements" and your claim that "something is wrong
with Python [if the task manager shows large numbers]"...

Have you looked at a task manager on a lightly loaded W2K or WXP system lately?
On this PC, I currently have a couple of browser windows open, a newsreader, and
a few other programs. The only real app I have that uses less than 4 megabytes
*according to the task manager* is a 2.5 megabyte putty terminal window; most
apps are in the 15-50 megabyte range.

Even a trivial operation, like opening another Firefox tab, can make that process
grow by more than 4 megabytes, and if you browse around a little, you can easily
end up with 30-50 megabytes in "mem usage" -- until you minimize the browser,
and find that Firefox only uses 1.5 megabytes. Maximize it again, and it uses 12
megabytes.

Play the same games with a Python interpreter, and you'll find that the task manager
may report anything from just over 50 kilobytes to just under 5 megabytes.

Here's a random google link that discusses the task manager figures in a little more
detail:

http://www.itwriting.com/dotnetmem.php

"Newcomer to .NET says: I've just been looking at Task Manager. Why
does my simple VB.NET Windows application use 12MB RAM?"

"Don't worry. The app doesn't really use that much RAM."

(for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)

</F>
 
S

Steve Holden

Paul said:
An awful lot of Python targeted users are not in that situation, so if
Python's usability suffers for them when it doesn't have to, then
something is wrong with Python.

If a useful subset of Python can be crammed into a Nokia cell phone then
I really don't think there's much to complain about (except that "it
hasn't been done for *my* machine").

Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!

i-remember-when-we-'ad-ter-code-seven-kilobytes-of-assembly-language-wi'-nowt-bu'-a-teletype-ter-edit-t'-paper-tape-wi'-ly
y'rs - steve
 
P

Paul Boddie

Steve said:
Even embedded systems are much larger now than the minicomputers of
yesteryear. Everything's relative. Just wait three years!

Had you placed such a bet in 2000, you'd have cleaned up at the
"Moore's Law Casino", but there are various factors at work now which
complicate the once-inevitable trends of hardware performance and the
corresponding advice to people wishing to speed up or slim down their
software: the flattening out of the CPU frequency curve and the
tendency of CPU manufacturers to choose multiple core strategies are
two things which prevent applications from speeding up all by
themselves; whilst storage density is still increasing, as far as I
know, I'd imagine other strategies being adopted in system architecture
which could make performance tradeoffs more pronounced in accessing all
that storage.

In other words, expect many more threads about global interpreter locks
in the coming three years than we've seen in the last three or even six
years. ;-)

Paul
 
S

Steve Holden

Paul said:
Had you placed such a bet in 2000, you'd have cleaned up at the
"Moore's Law Casino", but there are various factors at work now which
complicate the once-inevitable trends of hardware performance and the
corresponding advice to people wishing to speed up or slim down their
software: the flattening out of the CPU frequency curve and the
tendency of CPU manufacturers to choose multiple core strategies are
two things which prevent applications from speeding up all by
themselves; whilst storage density is still increasing, as far as I
know, I'd imagine other strategies being adopted in system architecture
which could make performance tradeoffs more pronounced in accessing all
that storage.

In other words, expect many more threads about global interpreter locks
in the coming three years than we've seen in the last three or even six
years. ;-)
I seem to remember blogging about this a while ago, or perhaps just
ranting at c.l.py ... anyway, bottom line is that most people have more
than adequate power for their current needs in a present-day
single-processor computer.

Expect also to see computational methods focus more on memory-intensive
techniques impractical in the past.

regards
Steve
 
Q

Qopit

Thanks - this is all very interesting...
Ah, but is that physical memory consumed, or virtual memory MAPPED
to the processes.
and

for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)

and most useful:

I had no idea that the memory usage reported by Windows Task Manager
(WTM) was so different than what I expected it would be. It doesn't
seem terribly useful to me right now.

After looking into that link (and discovering the perfmon app... can't
believe I never knew about such an amazingly useful tool!) below are
the results of some memory reporting checks that I ran on the simple
one-liner app (x = raw_input()). The "On Launch" is what was reported
immediately after launching the app, and the "Window Min'd" is
following a simple minimization for the window.

Memory report: On Launch Window Min'd
-------------- --------- ------------
WTM Mem Usage: 2,756K 88K
PerfMon Private bytes: 1,540,096 1,540,096
PerfMon Working set: 2,822,144 90,112 **

Basically it looks like the privately allocated memory for a 2.3 app
that cannot be shared by other processes (close to what I want,
neglecting paging possibilities) is the 1.5 MB that Fredrik reported.
Presumably using perfmon as well?

Maybe I'm a stickler, but this still seems pretty high for each and
every application to need to privately hold. Am I still misreading
this somehow? If not, I'd still love for this to be smaller... is
there any way to reduce this further?

Russ

** There was an interesting transient in the Working set during
minimization all the way up to 9 MB... presumably this is because
during minimization the "set of memory pages touched recently" jumps up
because of the transactions needed for the minimization.
 
P

Peter Hansen

Fredrik said:
(for python, the "private" memory use is usually ~1.5 megabytes for a "empty" 2.4
process, and some of that will only occupy space in the paging file... for firefox with
a simple page loaded into a single tab, the private space is ~10 megabytes)

I believe a relatively simple, if crude, way of measuring this sort of
thing (at least for console programs) is to open a handful of command
prompts, check the Physical Memory Available measurement in the Task
Manager, and then load one instance of the program in each console,
checking the drop in physical memory available at each step. After
three or four instances are loaded, the approximate real memory usage
for each should be readily apparent.

Obviously there are a dozen issues that can screw with this, including
having too little memory free at all, and having other programs open
which are very actively allocating or freeing memory.

It does seem to back up the "~1.5MB" number above, and certainly shows
that the total is nowhere near 4MB per Python interpreter.

-Peter
 
D

Dennis Lee Bieber

Maybe I'm a stickler, but this still seems pretty high for each and
every application to need to privately hold. Am I still misreading
this somehow? If not, I'd still love for this to be smaller... is
there any way to reduce this further?
I've not investigated -- but what is the default stack size given to
an application?
--
 
N

Neil Hodgson

Dennis Lee Bieber:
I've not investigated -- but what is the default stack size given to
an application?

Python has a stack reserve of 2000000 bytes but that is an
allocation of virtual space. Pages are realised in that allocation as
required with an initial stack commit of one page, 4096 bytes.

Neil

C:\Python24>dumpbin /headers python.exe
Microsoft (R) COFF/PE Dumper Version 7.10.3077
Copyright (C) Microsoft Corporation. All rights reserved.


Dump of file python.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
14C machine (x86)
4 number of sections
424A51B8 time date stamp Wed Mar 30 17:14:00 2005
0 file pointer to symbol table
0 number of symbols
E0 size of optional header
10F characteristics
Relocations stripped
Executable
Line numbers stripped
Symbols stripped
32 bit word machine

OPTIONAL HEADER VALUES
10B magic # (PE32)
7.10 linker version
400 size of code
A00 size of initialized data
0 size of uninitialized data
1062 entry point (1D001062)
1000 base of code
2000 base of data
1D000000 image base (1D000000 to 1D004FFF)
1000 section alignment
200 file alignment
4.00 operating system version
0.00 image version
4.00 subsystem version
0 Win32 version
5000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
0 DLL characteristics
1E8480 size of stack reserve
1000 size of stack commit
100000 size of heap reserve
1000 size of heap commit
....
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,755
Messages
2,569,536
Members
45,013
Latest member
KatriceSwa

Latest Threads

Top