Program slowing down with greater memory use

D

Dan Stromberg

I have two different python programs that are slowing down quite a bit as
their memory use goes up.

I'm not really sure if this is some sort of CPU cache effect, or if it's
something about python's garbage collector taking more time, or what.

One of the programs is one of those "how fast is data moving through the
pipe" measurement tools, called reblock. I'd initially expected that the
program would run faster with large block sizes, because then you're
wasting less time with in the C library, system call interface and context
switches.

But that turned out to be false. If I use a blocksize of about 2**18, it
runs much faster than if I use a blocksize of 2**22.

Then another program, which is a search engine, will slow way down if I
try to run it against too many search keywords. The program inhales a
bunch of filenames at the beginning, and if I run it against n files, n>m,
and later run it against m files, then the n file run will be quite a bit
slower even after it has handled only m files. The program implements a
memory cache to speed database operations, but even when the size of my
cache is nowhere near the size of the system's physical memory, it still
slows way down in this enigmatic way.

This makes me wonder.

What's the deal here? Is the garbage collector working overtime on these
programs that have a lot of objects? One of the programs above (reblock)
doesn't precisely have a large number of objects - just one huge string.

Thanks!
 
D

Dieter Maurer

Dan Stromberg said:
I have two different python programs that are slowing down quite a bit as
their memory use goes up.

I have seen this too with Zope.

I do not know where it comes from -- maybe from degraded locality.


Dieter
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,743
Messages
2,569,478
Members
44,898
Latest member
BlairH7607

Latest Threads

Top