windows mem leak

B

Bob Smith

Does the Win32 port of Python have a memory leak? I have some code that
runs flawlessly on Linux, but bombs after a few hours on Windows. It's
threaded and uses a lot of memory.

Thanks!
 
P

Peter Hansen

Bob said:
Does the Win32 port of Python have a memory leak? I have some code that
runs flawlessly on Linux, but bombs after a few hours on Windows. It's
threaded and uses a lot of memory.

Let's see what you're missing:

1. platform specifics
2. versions of things involved
3. any sort of detail about the code
4. how you're noticing/measuring the problem
5. what "bombs" means
6. any mention that you've checked Sourceforge to see whether
a similar problem has been reported

There have been memory leaks in various past versions of Python,
and could easily be in the current version, but they're generally
rather specific in terms of the code that can trigger it. Once it
was doing some particular work with a socket, once it was trying to
append (or extend?) to an empty list, and so on.

There are also a few ways you could have written your application
to cause memory to leak as a result of your own code, not as a
result of Python's. And it's even possible that this would happen
only on one platform (though I'm trying hard to think of an example
and can't... maybe it's very unlikely.)

-Peter
 
S

Steve Holden

Bob said:
Does the Win32 port of Python have a memory leak? I have some code that
runs flawlessly on Linux, but bombs after a few hours on Windows. It's
threaded and uses a lot of memory.

Thanks!

Yes, that's a well-known problem. Code that runs with a few errors will
port without any trouble at all to Windows, but once it runs flawlessly
on Linux it starts to leak memory on Windows. The PSU suspects a plot in
Redmond, the basic details of which ar
 
B

Bob Smith

Steve said:
Yes, that's a well-known problem. Code that runs with a few errors will
port without any trouble at all to Windows, but once it runs flawlessly
on Linux it starts to leak memory on Windows. The PSU suspects a plot in
Redmond, the basic details of which ar

Oh, the humor of it all ;)

Attached is the code. Run it yourself and see. You too Peter. Be gentle
with me, this was my first attempt with threads.

import os
import urllib
import socket
import time
import Queue
import threading

##########################################
# Network Section #
##########################################

socket.setdefaulttimeout(30)

networks = []
hosts = []
subnets = []

# Add the network 192.168.0 possibility to networks.
networks.append("192.168.0.")

# Generate and add networks 192.168.1-255 to networks.
n = 0
while n < 255:
n = n + 1
networks.append("192.168.%s." %(n))

# Generate and add hosts 1-254 to hosts.
for network in networks:
h = 1
# Add the n.n.n.0 host possibility to hosts.
hosts.append(network+str(h))
while h < 254:
h = h + 1
hosts.append(network+str(h))

# This should equal 65024 or 256 * 254
# because 256 possibilities are OK in the 3rd octet,
# but only 254 possibilities are OK in the 4th octet...
# we exclude 0 and 255.
print "There are", len(hosts), "total hosts (192.168.256*254) in the hosts list."

a = 0
b = 254

## Add the 192.168.0 net list to the subnets list.
subnets.append(hosts[0:254])
##print subnets
##print len(subnets)

## Add the 192.168.1-254 net lists to the subnets list.
for x in xrange(254):
a = a+254
b = b+254
subnets.append(hosts[a:b])
##print subnets
##print len(subnets)

## Add the 192.168.255 net list to the subnets list.
subnets.append(hosts[64770 :65024])
##print subnets[0]
print "There are", len(subnets), "class C network lists in the subnets list."

##########################################
# Queue Section #
##########################################

# Create a queue of urls to feed the threads
# Make it so that the queue only contains 256 items

nmap_queue = Queue.Queue(256)
for subnet in subnets:
nmap_queue.put(subnet)

###########################################
# Thread Section #
###########################################

class prac(threading.Thread):

def run(self):
net = nmap_queue.get()
for ip in net:
Y = os.popen('nmap -sT -p 80 -P0 -n %s' %ip)
data = Y.read()
Y.close()
if 'open' in data:
O = file('opened.txt', 'a')
print >> O, ip
O.close()
elif 'closed' in data:
C = file('closed.txt', 'a')
print >> C, ip
C.close()
elif 'filtered' in data:
F = file('filtered.txt' , 'a')
print >> F, ip
F.close()
else:
V = file('other.txt', 'a')
print >> V, data, ip
V.close()

threads = []
for i in xrange(256):
go = prac()
threads.append(go)
for thread in threads:
thread.start()
while threading.activeCount() > 1:
print str(threading.activeCount()), "threads running incl. main"
time.sleep(1)
 
B

Bulba!

Yes, that's a well-known problem. Code that runs with a few errors will
port without any trouble at all to Windows, but once it runs flawlessly
on Linux it starts to leak memory on Windows. The PSU suspects a plot in
Redmond, the basic details of which ar

You have my nomination for SOTW*. :)

* skit of the week / short story of the week
 
P

Peter Hansen

Bob said:
Attached is the code. Run it yourself and see. You too Peter. Be gentle
with me, this was my first attempt with threads.

Thanks, Bob, and I will, but not before you answer some of my
questions.

I had good reasons to ask them, one of which is that I don't
feel like wasting my time if, for example, you are using an
older version of Python that *did* have a memory leak.

Python 2.2, for example, (if my memory serves me right) had
a leak in the first release which would have affected pretty
much all software which did network work.

The most important answers you can provide will be versions,
platform (pretty clearly Linux, but please confirm and give
version), and what "bombs" means and how you are measuring
the memory leak.

(I presume you're using a version of nmap that's compiled
for Windows XP then? It's certainly not standard. How have
you proven that it is not *that* program which is at fault?)

-Peter
 
B

Bob Smith

Peter said:
Thanks, Bob, and I will, but not before you answer some of my
questions.

I had good reasons to ask them, one of which is that I don't
feel like wasting my time if, for example, you are using an
older version of Python that *did* have a memory leak.

2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)]
The most important answers you can provide will be versions,
platform (pretty clearly Linux, but please confirm and give
version), and what "bombs" means and how you are measuring
the memory leak.

WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram
Debian Linux Testing (2.4.28 vanilla Kernel) 3GHz P4 proc, 1.5GB Ram
(I presume you're using a version of nmap that's compiled
for Windows XP then?

Yes, I am.
It's certainly not standard.

That's a matter of opinion. Nmap works fine on the WinXP machine.
How have
you proven that it is not *that* program which is at fault?)

I have not. All I know is that on WinXP, the program uses 100% CPU at
times and consumes more Ram than is available (the page file grows to
700 or 800MB). It runs OK for a few hours and then produces a 'not
enough resources' error. And, the machine is generally unuserable. On
Linux, it has no impact whatsoever on resources. Granted, the Linux
machine is much more robust, but one wouldn't expect this great a
difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
if you think that would be a good idea. Perhaps that would at least
remove nmap from the equation.

I can run it if you like and take a screen shot of the error. You'll
have to give me a few hours though ;)
 
P

Peter Hansen

Bob said:
Peter Hansen wrote:
[snip details of Bob's platform]
WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram

That's not really much RAM for a WinXP box. Do you have
lots of physical memory available before running?
Yes, I am.

That's a matter of opinion. Nmap works fine on the WinXP machine.

Perhaps my use of "standard" was obscure, since it's definitely
not a matter of opinion, the way I intended it. What I meant
was "nmap is certainly not included in the standard Win XP
distribution".
I have not. All I know is that on WinXP, the program uses 100% CPU at
times and consumes more Ram than is available (the page file grows to
700 or 800MB). It runs OK for a few hours and then produces a 'not
enough resources' error.

Is it certain that this memory is being consumed by the Python
process? I could imagine, for example, there being dozens of
new processes spawned with os.system. Does the Task Manager
back up the theory that this "python.exe" instance is the one
causing the trouble? Presumably it should be clear even before
the machine grows unusable. Note that you can select which
columns are shown in the "Processes" tab of the Task Manager
window, to get more detail on a given process. (The Performance
Monitor found under Control Panel, Administrative Tools can
be even more useful and precise.)

And, the machine is generally unuserable. On
Linux, it has no impact whatsoever on resources. Granted, the Linux
machine is much more robust, but one wouldn't expect this great a
difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
if you think that would be a good idea. Perhaps that would at least
remove nmap from the equation.

A simpler test might be to change the nmap call to something
guaranteed benign, like a call to "dir", and try with that.
That's assuming you don't actually need the output of nmap
to reproduce the problem, which of course isn't sure. Still,
it would be an easy test, and might show a problem elsewhere.
I can run it if you like and take a screen shot of the error. You'll
have to give me a few hours though ;)

I trust that you are getting the error dialog you say you are. :)
I have a feeling that the message by itself helps little, however,
and that you'll have to try a few different approaches to observe
the problem as it grows, via Task Manager or another tool, rather
than just trying to guess what happened after the machine is
already effective kaput.

-Peter
 
R

Roel Schroeven

It would surprise me: even if it consumes much CPU-time, memory and
other resources, each instances returns all resources when it exits.
I have not. All I know is that on WinXP, the program uses 100% CPU at
times and consumes more Ram than is available (the page file grows to
700 or 800MB). It runs OK for a few hours and then produces a 'not
enough resources' error. And, the machine is generally unuserable. On
Linux, it has no impact whatsoever on resources. Granted, the Linux
machine is much more robust, but one wouldn't expect this great a
difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
if you think that would be a good idea. Perhaps that would at least
remove nmap from the equation.

I wrote a very simple and small fake_nmap that just looks at the
IP-address and prints "open", "closed" or "filtered" to stdout. When I
run your python program (Python 2.4 on Windows XP, like you), the CPU is
utilized 100% (about half of it goes to csrss.exe whatever that may be);
about half of the CPU time is spent in the kernel. The system stays
usable (at least for now, it's been running for about 5 minutes now),
but memory use is increasing, slow but steadily.

The task manager shows, in addition to a number of fake_nmap.exe
processes, a number of cmd.exe processes. I don't understand where these
come from: I know os.system ()uses the shell, but isn't os.popen()
supposed to open the process directly? It seems there are a lot more
instances of cmd.exe than of fake_nmap.exe; no idea what that tells us.

Also, it takes quite some time before "256 threads running incl. main"
is printed the first time, so I think the system needs all that time to
create all the threads. It would be normal for memory use to keep
increasing untill all threads are created, but I'm fairly certain memory
use is still increasing now.
 
P

Peter Hansen

Roel said:
It would surprise me: even if it consumes much CPU-time, memory and
other resources, each instances returns all resources when it exits.

I agree with that statement, but you assume that the program *is*
exiting. And your initial analysis with "fake_nmap" suggests
that, at least to the extent of having leftover cmd.exe's kicking
around, maybe it is not.

-Peter
 
R

Roel Schroeven

Peter said:
I agree with that statement, but you assume that the program *is*
exiting. And your initial analysis with "fake_nmap" suggests
that, at least to the extent of having leftover cmd.exe's kicking
around, maybe it is not.

I see. The number of cmd.exe's running was not *that* big though: about
5-10 I would say. And their PID's kept changing.

I took a look with Process Explorer from sysinternals, which shows the
processes as a tree instead of a simple list. Apparently each fake_nmap
is a child of a cmd.exe, meaning that os.popen indead uses the shell to
run processes. I wouldn't be surprise if cmd.exe would be the culprit here.
 
N

Nick Craig-Wood

Bob Smith said:
Attached is the code. Run it yourself and see.

This seems to run nmap over series of consecutive IP addresses. nmap
can do that all by itself. From its man page::

Nmap also has a more powerful notation which lets you specify an IP
address using lists/ranges for each element. Thus you can scan the
whole class 'B' network 128.210.*.* by specifying '128.210.*.*' or
'128.210.0-255.0-255' or even '128.210.1-50,51-255.1,2,3,4,5-255'. And
of course you can use the mask notation: '128.210.0.0/16'. These are
all equivalent. If you use astericts ('*'), remember that most shells
require you to escape them with back slashes or protect them with
quotes.

This setting might be useful too::

--max_parallelism <number>
Specifies the maximum number of scans Nmap is allowed to perform
in parallel. Setting this to one means Nmap will never try to
scan more than 1 port at a time. It also effects other parallel
scans such as ping sweep, RPC scan, etc.

[sorry not Python related but may solve your problem!]
 
N

Nick Coghlan

Bob said:
Peter said:
Thanks, Bob, and I will, but not before you answer some of my
questions.

I had good reasons to ask them, one of which is that I don't
feel like wasting my time if, for example, you are using an
older version of Python that *did* have a memory leak.


2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)]

Perhaps you could try using the new subprocess module, instead of using os.popen
directly.

A fair amount of work went into making the Windows implementation of that module
as solid as the *nix implementation, whereas there may still be issues with
direct os.popen calls (as Roel's investigation suggests).

Cheers,
Nick.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,769
Messages
2,569,576
Members
45,054
Latest member
LucyCarper

Latest Threads

Top