Memory leak when creating lots of object

[Godzilla]

Hello,

I have a program that create and pop an object off a queue, but it is
experiencing some memory leakage. I have been unable to detect where
the memory leakage occur. The strange thing is when i replace the
object creation with a plain integer/string, the leak goes away...
Here's the code I used as my test:


import Queue
import threading
import time

q = Queue.Queue(0)

class PDU:
def __init__(self, name=''):
self.name = name

class qTest(threading.Thread):
def __init__(self, name=''):
threading.Thread.__init__(self)
self.name = name
self.threadContinue = True

def run(self):
count = 0
while self.threadContinue:
pdu = q.get(True)
del pdu
count+=1
#print count
if q.qsize() == 0:
print "Total get in bulk", count
count = 0

def stop(self):
self.threadContinue = False
q.put(1)


if __name__ == "__main__":
try:
qt = qTest()
qt.start()
loopCount = 1000
while True:
for i in range(loopCount):
pdu = PDU()
## pdu = 1
q.put(pdu)
time.sleep(5)

except KeyboardInterrupt:
print "Keyboard interrupted. Program exit."
except Exception, why:
print why

if qt.isAlive():
qt.stop()

I can see the memory usage increases slowly in Task Manager under XP,
but do not know why. Anyone help?

Thank you.

 

[Paul Moore]

Hello,

I have a program that create and pop an object off a queue, but it is
experiencing some memory leakage. I have been unable to detect where
the memory leakage occur. The strange thing is when i replace the
object creation with a plain integer/string, the leak goes away...
Here's the code I used as my test: [...]
I can see the memory usage increases slowly in Task Manager under XP,
but do not know why. Anyone help?

I tried your code on my (Windows XP SP2, Python 2.5) system. No memory
leak here - I left it running for over 5 minutes and memory usage was
static at just under 4MB.

Do you see memory growth with precisely this code? Over what period?
How much?

Paul.

 

[Godzilla]

Hello,

I have a program that create and pop an object off a queue, but it is
experiencing some memory leakage. I have been unable to detect where
the memory leakage occur. The strange thing is when i replace the
object creation with a plain integer/string, the leak goes away...
Here's the code I used as my test: [...]
I can see the memory usage increases slowly in Task Manager under XP,
but do not know why. Anyone help?

I tried your code on my (Windows XP SP2, Python 2.5) system. No memory
leak here - I left it running for over 5 minutes and memory usage was
static at just under 4MB.

Do you see memory growth with precisely this code? Over what period?
How much?

Paul.

Hi Paul,

I have it running for more than 1 hour... the main application
software runs for about 50 days non stops and the memory just keep
growing...

I have a pdu.py class which has about 130 methods that's been created
on the fly. If you substitute the above code of PDU() init with the
pdu class from pdu.py, I see a dramatic increase in RAM in python
2.4.4! But not in version 2.5.1. So I upgraded to the latest
version... I still see the main application uses more and more memory
as the time goes by...

What should I do next? Can I force garbage collection manually? If so,
how do I do that?

 

[Terry Reedy]

| What should I do next? Can I force garbage collection manually? If so,
| how do I do that?

Read doc for gc module. I think
import gc
gc.collect()

 

[Paul Moore]

I have it running for more than 1 hour... the main application
software runs for about 50 days non stops and the memory just keep
growing...

I'm sorry. Just to be precise, how long would I need to run the code
you posted to see a memory growth?

What should I do next? Can I force garbage collection manually? If so,
how do I do that?

As Terry suggested, look at the gc module (gc.collect). But on
inspection, I don't see a memory leak that would be cured by forcing a
collection, which is why I'd like to reproduce the problem before
offering suggestions...

Paul.