Python 2.4.3 array memory leak

[danmcleran]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

 

[danmcleran]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

I hate to reply to my own thread but I wanted to update it. I tried
the same code using ctypes arrays and see no memory leak:

import commands
import array
import itertools
import sys
import ctypes

from itertools import repeat
from ctypes import *

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = ARRAY(c_uint, int(2E6))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

*** before ***
MemFree: 1416364 kB
*** after ***
MemFree: 1416364 kB

 

[danmcleran]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

I hate to reply to my own thread but I wanted to update it. I tried
the same code using ctypes arrays and see no memory leak:

import commands
import array
import itertools
import sys
import ctypes

from itertools import repeat
from ctypes import *

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = ARRAY(c_uint, int(2E6))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

*** before ***
MemFree: 1416364 kB
*** after ***
MemFree: 1416364 kB

The above code was not correct. I actually do see a memory leak when
using ctypes arrays also. What is going on?


import commands
import array
import itertools
import sys
import ctypes

from itertools import repeat
from ctypes import *

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = ARRAY(c_uint, int(2E6))()
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output
*** before ***
MemFree: 1564096 kB
*** after ***
MemFree: 1556884 kB

 

[danmcleran]

On May 7, 3:31 pm, "(e-mail address removed)" <[email protected]>
wrote:

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:
import commands
import array
import itertools
import sys
from itertools import repeat
print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

I hate to reply to my own thread but I wanted to update it. I tried
the same code using ctypes arrays and see no memory leak:

import commands
import array
import itertools
import sys
import ctypes

from itertools import repeat
from ctypes import *

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = ARRAY(c_uint, int(2E6))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

*** before ***
MemFree: 1416364 kB
*** after ***
MemFree: 1416364 kB

The above code was not correct. I actually do see a memory leak when
using ctypes arrays also. What is going on?

import commands
import array
import itertools
import sys
import ctypes

from itertools import repeat
from ctypes import *

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = ARRAY(c_uint, int(2E6))()
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output
*** before ***
MemFree: 1564096 kB
*** after ***
MemFree: 1556884 kB

Maybe the problem is with the way I was looking at free memory. I
changed my code to look at virtual memory allocated to the process in
question and I do not see any appreciable before/after difference.

import commands
import array
import itertools
import sys
import os

from itertools import repeat

deltas = []
print '*** before ***'
s = commands.getoutput('cat /proc/%d/status' % os.getpid()).split('\n')
[12]
print s
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a

print '*** after ***'
s = commands.getoutput('cat /proc/%d/status' % os.getpid()).split('\n')
[12]
print s

Output:

*** before ***
VmSize: 5952 kB
*** after ***
VmSize: 5956 kB

 

[Terry Reedy]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

What happens if you remove the loop? I would not be surprised if Python
grabs the memory once, reuses it, and does not let go. That is not a
leak. What happens if you put the after inside the loop? Does mem
usage steadily increase, and continue if you increase range to 1000,
10000? That would be a leak.

If there actually is a problem, try a later version of Python.

 

[danmcleran]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
    a = array.array('I', repeat(0, int(2E6)))
    del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree:       1459772 kB
*** after ***
MemFree:       1457688 kB

What happens if you remove the loop?  I would not be surprised if Python
grabs the memory once, reuses it, and does not let go.  That is not a
leak.  What happens if you put the after inside the loop?  Does mem
usage steadily increase, and continue if you increase range to 1000,
10000?  That would be a leak.

If there actually is a problem, try a later version of Python.

I'm not convinced there is a problem anymore. In my latest code, I
record the virtual mem allocated to my specific process and I do not
see it increasing. I would think that if I was leaking memory, I
should see a steady increase in virtual memory consumed by my process,
which I do not.

thanks

 

[Robert Kern]

I am using the array module to instantiate many arrays in my
application. It seems like there is a memory leak in there somewhere.
Can anyone confim this and let me know what, if anything, I can do
about it? I am using Fedora Core 5 Linux:

import commands
import array
import itertools
import sys

from itertools import repeat

print '*** before ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]
for i in range(100):
a = array.array('I', repeat(0, int(2E6)))
del a
print '*** after ***'
print commands.getoutput('cat /proc/meminfo').split('\n')[1]

Output:
*** before ***
MemFree: 1459772 kB
*** after ***
MemFree: 1457688 kB

This is not necessarily indicative of a memory leak, per se. Python sometimes
does not release memory back to the OS. The memory isn't leaked; it will be
reused by Python for later allocations. Now, if you saw Python's memory usage
*grow* with increasing iterations, then that would be evidence of a leak.

Try measuring the "before" memory, then do one iteration, measure the memory
again (let's call it "intermediate"), then do your loop, and then measure the
"after" memory. The "intermediate" measurement should be about your "after"
measurement.

Also, try to measure the process's own memory usage, not the system's free memory.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco