Memory problem

R

Rolf Wester

Hi,

I have a strange (for me) memory problem. When running a loop in a
Python program memory usage increases from about 4% up to 100%. I do a
gc.collect() every loop cycle but this doesn't help. There are about
67000 objects that are tracked by the garbage collector. This number
does vary a little bit but does not increase on average whereas the
memory usage does. The number of garbage objects (len(gc.garbage()) is
zero. This is a very severe problem for my application, so I would be
very appreciative for any help.

With kind regards

Rolf
 
D

Diez B. Roggisch

Rolf said:
Hi,

I have a strange (for me) memory problem. When running a loop in a
Python program memory usage increases from about 4% up to 100%. I do a
gc.collect() every loop cycle but this doesn't help. There are about
67000 objects that are tracked by the garbage collector. This number
does vary a little bit but does not increase on average whereas the
memory usage does. The number of garbage objects (len(gc.garbage()) is
zero. This is a very severe problem for my application, so I would be
very appreciative for any help.

well, it might be that the relative moon humidity resonates with the
loop-cycle, creating quantum-flux in the main memory banks.

Or not.

Seriously - what help do you expect without showing us _any_ code or at
least getting into details like usage of libs, python-version, possible
C-extensions, platform, ....

Diez
 
R

Rolf Wester

Sorry, of course your are wright. I'm running Python2.5 on Linux, my
program imports numpy, matplotlib, sys and a python module of my own.
This module uses numpy and scipy.weave for imbedded C-code but no
extension modules. I though the code to be to large to show, I hoped you
could give me hint on what I could try.

Thank you in advance

Rolf
 
P

Peter Otten

Sorry, of course your are wright. I'm running Python2.5 on Linux, my
program imports numpy, matplotlib, sys and a python module of my own.
This module uses numpy and scipy.weave for imbedded C-code but no
extension modules. I though the code to be to large to show, I hoped you
could give me hint on what I could try.

I think it is now safe to say that the embedded C code is a more likely
culprit than the relative humidity of the moon...

If you have python prototypes for the C chunks you could temporarily put
them back in. Otherwise it is time to start stripping down the code until
you can isolate the problem -- or at least have something you can post
here.

Peter
 
G

Gabriel Genellina

I think it is now safe to say that the embedded C code is a more likely
culprit than the relative humidity of the moon...

Sure; and among all things that could be wrong, I'd start checking the
reference counts. Get it wrong by -1 and your objects suddenly dissapear;
get it wrong by +1 and your objects will never be destroyed, demanding
more and more memory like in this case.
 
R

Rolf Wester

Hi,

thank you for your comments and your hints (I probably deserve some kind
of subtle irony). I found the problem:

I thought a numpy array A has shape (n,) but actually it had shape
(n,1). In the loop I sampled a value from that array:

v.append(A)

So what happened was that I got a view of A not a single number and
append obviously appended the whole array not just a single element
array. And n is about 64000.

So sorry for bothering you with that stupid fault.

Regards

Rolf
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,481
Members
44,900
Latest member
Nell636132

Latest Threads

Top