windows mem leak

Discussion in 'Python' started by Bob Smith, Jan 8, 2005.

  1. Bob Smith

    Bob Smith Guest

    Does the Win32 port of Python have a memory leak? I have some code that
    runs flawlessly on Linux, but bombs after a few hours on Windows. It's
    threaded and uses a lot of memory.

    Thanks!
    Bob Smith, Jan 8, 2005
    #1
    1. Advertising

  2. Bob Smith

    Peter Hansen Guest

    Bob Smith wrote:
    > Does the Win32 port of Python have a memory leak? I have some code that
    > runs flawlessly on Linux, but bombs after a few hours on Windows. It's
    > threaded and uses a lot of memory.


    Let's see what you're missing:

    1. platform specifics
    2. versions of things involved
    3. any sort of detail about the code
    4. how you're noticing/measuring the problem
    5. what "bombs" means
    6. any mention that you've checked Sourceforge to see whether
    a similar problem has been reported

    There have been memory leaks in various past versions of Python,
    and could easily be in the current version, but they're generally
    rather specific in terms of the code that can trigger it. Once it
    was doing some particular work with a socket, once it was trying to
    append (or extend?) to an empty list, and so on.

    There are also a few ways you could have written your application
    to cause memory to leak as a result of your own code, not as a
    result of Python's. And it's even possible that this would happen
    only on one platform (though I'm trying hard to think of an example
    and can't... maybe it's very unlikely.)

    -Peter
    Peter Hansen, Jan 8, 2005
    #2
    1. Advertising

  3. Bob Smith

    Steve Holden Guest

    Bob Smith wrote:

    > Does the Win32 port of Python have a memory leak? I have some code that
    > runs flawlessly on Linux, but bombs after a few hours on Windows. It's
    > threaded and uses a lot of memory.
    >
    > Thanks!


    Yes, that's a well-known problem. Code that runs with a few errors will
    port without any trouble at all to Windows, but once it runs flawlessly
    on Linux it starts to leak memory on Windows. The PSU suspects a plot in
    Redmond, the basic details of which ar
    --
    Steve Holden http://www.holdenweb.com/
    Python Web Programming http://pydish.holdenweb.com/
    Holden Web LLC +1 703 861 4237 +1 800 494 3119
    Steve Holden, Jan 8, 2005
    #3
  4. Bob Smith

    Bob Smith Guest

    Steve Holden wrote:
    > Bob Smith wrote:
    >
    >> Does the Win32 port of Python have a memory leak? I have some code
    >> that runs flawlessly on Linux, but bombs after a few hours on Windows.
    >> It's threaded and uses a lot of memory.
    >>
    >> Thanks!

    >
    >
    > Yes, that's a well-known problem. Code that runs with a few errors will
    > port without any trouble at all to Windows, but once it runs flawlessly
    > on Linux it starts to leak memory on Windows. The PSU suspects a plot in
    > Redmond, the basic details of which ar


    Oh, the humor of it all ;)

    Attached is the code. Run it yourself and see. You too Peter. Be gentle
    with me, this was my first attempt with threads.

    import os
    import urllib
    import socket
    import time
    import Queue
    import threading

    ##########################################
    # Network Section #
    ##########################################

    socket.setdefaulttimeout(30)

    networks = []
    hosts = []
    subnets = []

    # Add the network 192.168.0 possibility to networks.
    networks.append("192.168.0.")

    # Generate and add networks 192.168.1-255 to networks.
    n = 0
    while n < 255:
    n = n + 1
    networks.append("192.168.%s." %(n))

    # Generate and add hosts 1-254 to hosts.
    for network in networks:
    h = 1
    # Add the n.n.n.0 host possibility to hosts.
    hosts.append(network+str(h))
    while h < 254:
    h = h + 1
    hosts.append(network+str(h))

    # This should equal 65024 or 256 * 254
    # because 256 possibilities are OK in the 3rd octet,
    # but only 254 possibilities are OK in the 4th octet...
    # we exclude 0 and 255.
    print "There are", len(hosts), "total hosts (192.168.256*254) in the hosts list."

    a = 0
    b = 254

    ## Add the 192.168.0 net list to the subnets list.
    subnets.append(hosts[0:254])
    ##print subnets
    ##print len(subnets)

    ## Add the 192.168.1-254 net lists to the subnets list.
    for x in xrange(254):
    a = a+254
    b = b+254
    subnets.append(hosts[a:b])
    ##print subnets
    ##print len(subnets)

    ## Add the 192.168.255 net list to the subnets list.
    subnets.append(hosts[64770 :65024])
    ##print subnets[0]
    print "There are", len(subnets), "class C network lists in the subnets list."

    ##########################################
    # Queue Section #
    ##########################################

    # Create a queue of urls to feed the threads
    # Make it so that the queue only contains 256 items

    nmap_queue = Queue.Queue(256)
    for subnet in subnets:
    nmap_queue.put(subnet)

    ###########################################
    # Thread Section #
    ###########################################

    class prac(threading.Thread):

    def run(self):
    net = nmap_queue.get()
    for ip in net:
    Y = os.popen('nmap -sT -p 80 -P0 -n %s' %ip)
    data = Y.read()
    Y.close()
    if 'open' in data:
    O = file('opened.txt', 'a')
    print >> O, ip
    O.close()
    elif 'closed' in data:
    C = file('closed.txt', 'a')
    print >> C, ip
    C.close()
    elif 'filtered' in data:
    F = file('filtered.txt' , 'a')
    print >> F, ip
    F.close()
    else:
    V = file('other.txt', 'a')
    print >> V, data, ip
    V.close()

    threads = []
    for i in xrange(256):
    go = prac()
    threads.append(go)
    for thread in threads:
    thread.start()
    while threading.activeCount() > 1:
    print str(threading.activeCount()), "threads running incl. main"
    time.sleep(1)
    Bob Smith, Jan 8, 2005
    #4
  5. Bob Smith

    Bulba! Guest

    On Sat, 08 Jan 2005 15:00:05 -0500, Steve Holden <>
    wrote:

    >> Does the Win32 port of Python have a memory leak? I have some code that
    >> runs flawlessly on Linux, but bombs after a few hours on Windows. It's
    >> threaded and uses a lot of memory.


    >Yes, that's a well-known problem. Code that runs with a few errors will
    >port without any trouble at all to Windows, but once it runs flawlessly
    >on Linux it starts to leak memory on Windows. The PSU suspects a plot in
    >Redmond, the basic details of which ar


    You have my nomination for SOTW*. :)

    * skit of the week / short story of the week




    --
    It's a man's life in a Python Programming Association.
    Bulba!, Jan 9, 2005
    #5
  6. Bob Smith

    Peter Hansen Guest

    Bob Smith wrote:
    > Attached is the code. Run it yourself and see. You too Peter. Be gentle
    > with me, this was my first attempt with threads.


    Thanks, Bob, and I will, but not before you answer some of my
    questions.

    I had good reasons to ask them, one of which is that I don't
    feel like wasting my time if, for example, you are using an
    older version of Python that *did* have a memory leak.

    Python 2.2, for example, (if my memory serves me right) had
    a leak in the first release which would have affected pretty
    much all software which did network work.

    The most important answers you can provide will be versions,
    platform (pretty clearly Linux, but please confirm and give
    version), and what "bombs" means and how you are measuring
    the memory leak.

    (I presume you're using a version of nmap that's compiled
    for Windows XP then? It's certainly not standard. How have
    you proven that it is not *that* program which is at fault?)

    -Peter
    Peter Hansen, Jan 9, 2005
    #6
  7. Bob Smith

    Bob Smith Guest

    Peter Hansen wrote:
    > Bob Smith wrote:
    >
    >> Attached is the code. Run it yourself and see. You too Peter. Be
    >> gentle with me, this was my first attempt with threads.

    >
    >
    > Thanks, Bob, and I will, but not before you answer some of my
    > questions.
    >
    > I had good reasons to ask them, one of which is that I don't
    > feel like wasting my time if, for example, you are using an
    > older version of Python that *did* have a memory leak.


    2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)]

    >
    > The most important answers you can provide will be versions,
    > platform (pretty clearly Linux, but please confirm and give
    > version), and what "bombs" means and how you are measuring
    > the memory leak.


    WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram
    Debian Linux Testing (2.4.28 vanilla Kernel) 3GHz P4 proc, 1.5GB Ram

    >
    > (I presume you're using a version of nmap that's compiled
    > for Windows XP then?


    Yes, I am.

    > It's certainly not standard.


    That's a matter of opinion. Nmap works fine on the WinXP machine.

    > How have
    > you proven that it is not *that* program which is at fault?)


    I have not. All I know is that on WinXP, the program uses 100% CPU at
    times and consumes more Ram than is available (the page file grows to
    700 or 800MB). It runs OK for a few hours and then produces a 'not
    enough resources' error. And, the machine is generally unuserable. On
    Linux, it has no impact whatsoever on resources. Granted, the Linux
    machine is much more robust, but one wouldn't expect this great a
    difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
    if you think that would be a good idea. Perhaps that would at least
    remove nmap from the equation.

    I can run it if you like and take a screen shot of the error. You'll
    have to give me a few hours though ;)
    Bob Smith, Jan 9, 2005
    #7
  8. Bob Smith

    Peter Hansen Guest

    Bob Smith wrote:
    > Peter Hansen wrote:

    [snip details of Bob's platform]
    > WinXP Home, Service Pack 2, AMD 1400MHz proc, 256MB Ram


    That's not really much RAM for a WinXP box. Do you have
    lots of physical memory available before running?

    >> (I presume you're using a version of nmap that's compiled
    >> for Windows XP then?

    > Yes, I am.
    >
    >> It's certainly not standard.

    > That's a matter of opinion. Nmap works fine on the WinXP machine.


    Perhaps my use of "standard" was obscure, since it's definitely
    not a matter of opinion, the way I intended it. What I meant
    was "nmap is certainly not included in the standard Win XP
    distribution".

    >> How have
    >> you proven that it is not *that* program which is at fault?)

    >
    > I have not. All I know is that on WinXP, the program uses 100% CPU at
    > times and consumes more Ram than is available (the page file grows to
    > 700 or 800MB). It runs OK for a few hours and then produces a 'not
    > enough resources' error.


    Is it certain that this memory is being consumed by the Python
    process? I could imagine, for example, there being dozens of
    new processes spawned with os.system. Does the Task Manager
    back up the theory that this "python.exe" instance is the one
    causing the trouble? Presumably it should be clear even before
    the machine grows unusable. Note that you can select which
    columns are shown in the "Processes" tab of the Task Manager
    window, to get more detail on a given process. (The Performance
    Monitor found under Control Panel, Administrative Tools can
    be even more useful and precise.)


    > And, the machine is generally unuserable. On
    > Linux, it has no impact whatsoever on resources. Granted, the Linux
    > machine is much more robust, but one wouldn't expect this great a
    > difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
    > if you think that would be a good idea. Perhaps that would at least
    > remove nmap from the equation.


    A simpler test might be to change the nmap call to something
    guaranteed benign, like a call to "dir", and try with that.
    That's assuming you don't actually need the output of nmap
    to reproduce the problem, which of course isn't sure. Still,
    it would be an easy test, and might show a problem elsewhere.

    > I can run it if you like and take a screen shot of the error. You'll
    > have to give me a few hours though ;)


    I trust that you are getting the error dialog you say you are. :)
    I have a feeling that the message by itself helps little, however,
    and that you'll have to try a few different approaches to observe
    the problem as it grows, via Task Manager or another tool, rather
    than just trying to guess what happened after the machine is
    already effective kaput.

    -Peter
    Peter Hansen, Jan 9, 2005
    #8
  9. Bob Smith wrote:
    > Peter Hansen wrote:
    >> How have
    >> you proven that it is not *that* program which is at fault?)


    It would surprise me: even if it consumes much CPU-time, memory and
    other resources, each instances returns all resources when it exits.

    > I have not. All I know is that on WinXP, the program uses 100% CPU at
    > times and consumes more Ram than is available (the page file grows to
    > 700 or 800MB). It runs OK for a few hours and then produces a 'not
    > enough resources' error. And, the machine is generally unuserable. On
    > Linux, it has no impact whatsoever on resources. Granted, the Linux
    > machine is much more robust, but one wouldn't expect this great a
    > difference. I can rewrite it so that it's pure Pyhton (no calling nmap)
    > if you think that would be a good idea. Perhaps that would at least
    > remove nmap from the equation.


    I wrote a very simple and small fake_nmap that just looks at the
    IP-address and prints "open", "closed" or "filtered" to stdout. When I
    run your python program (Python 2.4 on Windows XP, like you), the CPU is
    utilized 100% (about half of it goes to csrss.exe whatever that may be);
    about half of the CPU time is spent in the kernel. The system stays
    usable (at least for now, it's been running for about 5 minutes now),
    but memory use is increasing, slow but steadily.

    The task manager shows, in addition to a number of fake_nmap.exe
    processes, a number of cmd.exe processes. I don't understand where these
    come from: I know os.system ()uses the shell, but isn't os.popen()
    supposed to open the process directly? It seems there are a lot more
    instances of cmd.exe than of fake_nmap.exe; no idea what that tells us.

    Also, it takes quite some time before "256 threads running incl. main"
    is printed the first time, so I think the system needs all that time to
    create all the threads. It would be normal for memory use to keep
    increasing untill all threads are created, but I'm fairly certain memory
    use is still increasing now.

    --
    "Codito ergo sum"
    Roel Schroeven
    Roel Schroeven, Jan 9, 2005
    #9
  10. Bob Smith

    Peter Hansen Guest

    Roel Schroeven wrote:
    >> Peter Hansen wrote:
    >>> How have
    >>> you proven that it is not *that* program which is at fault?)

    >
    > It would surprise me: even if it consumes much CPU-time, memory and
    > other resources, each instances returns all resources when it exits.


    I agree with that statement, but you assume that the program *is*
    exiting. And your initial analysis with "fake_nmap" suggests
    that, at least to the extent of having leftover cmd.exe's kicking
    around, maybe it is not.

    -Peter
    Peter Hansen, Jan 9, 2005
    #10
  11. Peter Hansen wrote:
    > Roel Schroeven wrote:
    >
    >>> Peter Hansen wrote:
    >>>
    >>>> How have
    >>>> you proven that it is not *that* program which is at fault?)

    >>
    >>
    >> It would surprise me: even if it consumes much CPU-time, memory and
    >> other resources, each instances returns all resources when it exits.

    >
    >
    > I agree with that statement, but you assume that the program *is*
    > exiting. And your initial analysis with "fake_nmap" suggests
    > that, at least to the extent of having leftover cmd.exe's kicking
    > around, maybe it is not.


    I see. The number of cmd.exe's running was not *that* big though: about
    5-10 I would say. And their PID's kept changing.

    I took a look with Process Explorer from sysinternals, which shows the
    processes as a tree instead of a simple list. Apparently each fake_nmap
    is a child of a cmd.exe, meaning that os.popen indead uses the shell to
    run processes. I wouldn't be surprise if cmd.exe would be the culprit here.

    --
    "Codito ergo sum"
    Roel Schroeven
    Roel Schroeven, Jan 9, 2005
    #11
  12. Bob Smith <> wrote:
    > Attached is the code. Run it yourself and see.


    This seems to run nmap over series of consecutive IP addresses. nmap
    can do that all by itself. From its man page::

    Nmap also has a more powerful notation which lets you specify an IP
    address using lists/ranges for each element. Thus you can scan the
    whole class 'B' network 128.210.*.* by specifying '128.210.*.*' or
    '128.210.0-255.0-255' or even '128.210.1-50,51-255.1,2,3,4,5-255'. And
    of course you can use the mask notation: '128.210.0.0/16'. These are
    all equivalent. If you use astericts ('*'), remember that most shells
    require you to escape them with back slashes or protect them with
    quotes.

    This setting might be useful too::

    --max_parallelism <number>
    Specifies the maximum number of scans Nmap is allowed to perform
    in parallel. Setting this to one means Nmap will never try to
    scan more than 1 port at a time. It also effects other parallel
    scans such as ping sweep, RPC scan, etc.

    [sorry not Python related but may solve your problem!]
    --
    Nick Craig-Wood <> -- http://www.craig-wood.com/nick
    Nick Craig-Wood, Jan 10, 2005
    #12
  13. Bob Smith

    Nick Coghlan Guest

    Bob Smith wrote:
    > Peter Hansen wrote:
    >
    >> Bob Smith wrote:
    >>
    >>> Attached is the code. Run it yourself and see. You too Peter. Be
    >>> gentle with me, this was my first attempt with threads.

    >>
    >>
    >>
    >> Thanks, Bob, and I will, but not before you answer some of my
    >> questions.
    >>
    >> I had good reasons to ask them, one of which is that I don't
    >> feel like wasting my time if, for example, you are using an
    >> older version of Python that *did* have a memory leak.

    >
    >
    > 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)]


    Perhaps you could try using the new subprocess module, instead of using os.popen
    directly.

    A fair amount of work went into making the Windows implementation of that module
    as solid as the *nix implementation, whereas there may still be issues with
    direct os.popen calls (as Roel's investigation suggests).

    Cheers,
    Nick.

    --
    Nick Coghlan | | Brisbane, Australia
    ---------------------------------------------------------------
    http://boredomandlaziness.skystorm.net
    Nick Coghlan, Jan 10, 2005
    #13
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Steffen Krippner
    Replies:
    0
    Views:
    252
    Steffen Krippner
    Nov 14, 2003
  2. Steffen Krippner
    Replies:
    0
    Views:
    349
    Steffen Krippner
    Nov 14, 2003
  3. Ethan

    mem leak in Singleton?

    Ethan, Nov 6, 2004, in forum: C++
    Replies:
    7
    Views:
    358
    Joe Gottman
    Nov 6, 2004
  4. Replies:
    8
    Views:
    302
    jaysome
    Mar 1, 2007
  5. news.aon.at
    Replies:
    11
    Views:
    626
    Ian Collins
    Jan 29, 2011
Loading...

Share This Page