numarray memory leak

[Peter Dobcsanyi]

Calling the following function with a large enough 'n' causes memory leak.

import numarray as N

def loop(n, m=100):
for i in xrange(n):
a = N.zeros((m,m))
N.matrixmultiply(a, a)

If the matrixmultiply line is commented out, there is no leak, the
program has a stable memory size.

I am using numarray 1.1, the result is the same with Python 2.3.4 and
2.4b2.

Peter

 

[Robert Kern]

Calling the following function with a large enough 'n' causes memory leak.

import numarray as N

def loop(n, m=100):
for i in xrange(n):
a = N.zeros((m,m))
N.matrixmultiply(a, a)

If the matrixmultiply line is commented out, there is no leak, the
program has a stable memory size.

I am using numarray 1.1, the result is the same with Python 2.3.4 and
2.4b2.

I don't see such behavior on Mac OSX 10.3, Python 2.3, CVS numarray and
n=10000.

Could you try CVS numarray? What n did you use?

--
Robert Kern
(e-mail address removed)

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

 

[Peter Dobcsanyi]

I don't see such behavior on Mac OSX 10.3, Python 2.3, CVS numarray and
n=10000.

Could you try CVS numarray? What n did you use?

With n a few hundreds (say n>=500), the continuous growth of the
program's memory footprint is quite noticeable. I am working on Linux
(Debian testing), I monitor the memory usage of the program by top.
The allocated memory is never released.

I have noticed this problem of memory leak in connection with numarray a
few months ago in a program of mine. The program processes the incidence
matrices of thousands of combinatorial structures read from a file one
by one. As the number of combinatorial objects went up I started running
out of memory even on a machine with 2 Gbyte memory. It took me a while
to pinpoint that it was caused by matrix multiplication.

I will try the CVS.

Peter

 

[John Hunter]

Peter> With n a few hundreds (say n>=500), the continuous growth
Peter> of the program's memory footprint is quite noticeable. I am
Peter> working on Linux (Debian testing), I monitor the memory
Peter> usage of the program by top. The allocated memory is never
Peter> released.

Peter> I have noticed this problem of memory leak in connection
Peter> with numarray a few months ago in a program of mine. The
Peter> program processes the incidence matrices of thousands of
Peter> combinatorial structures read from a file one by one. As
Peter> the number of combinatorial objects went up I started
Peter> running out of memory even on a machine with 2 Gbyte
Peter> memory. It took me a while to pinpoint that it was caused
Peter> by matrix multiplication.

Peter> I will try the CVS.

Yep, numarray 1.1 has a memory leak in dot, which Matrix multiply
uses. It also has a leak in cumulative boolean functions like
sometrue. Both are fixed in CVS.

JDH

 

[Peter Dobcsanyi]

Peter> I will try the CVS.

Yep, numarray 1.1 has a memory leak in dot, which Matrix multiply
uses. It also has a leak in cumulative boolean functions like
sometrue. Both are fixed in CVS.

Indeed, I tried the CVS version and that fixed the problem; for me, at
least. I am a bit worried about the users of my package depending on
numarray. Most of them would not be able (or willing to) pull down the
CVS tree.

Peter

 

[John Hunter]

Peter> Indeed, I tried the CVS version and that fixed the problem;
Peter> for me, at least. I am a bit worried about the users of my
Peter> package depending on numarray. Most of them would not be
Peter> able (or willing to) pull down the CVS tree.

You might ask on the numpy-discussion list when the release of
numarray 1.2 is expected. My guess is it won't be too long from now.
If it is going to be a while and this is a serious problem for your
package, you have the option of tarring up CVS yourself and/or
building a win32 installer and linking to this on your site for users
concerned about memory leaks.

There is a section of matplotlib code that depends on matrix multiply,
and I have this FAQ for memory leaks

http://matplotlib.sourceforge.net/faq.html#LEAKS

JDH