Memory Leak Problem

[joehust]

I have a crawler program, it fetches html on the internet and a
parser will parse the page.
Because of slow networks, I use many robots to crawl the pages and
one parser.
The program is multi-threaded.
There is a pagestore object used as the bridge between robot and
parser, it's has a List structure,
the parser will remove the item it has parsed.

But as the program runs, its memory keep on inscreasing. I don't know
why. What I see in the taskmgr shows that most memory is in the
virtual memory, and the actual memory occupation is not much.

So what could possibly be the reasons?

Joe

 

[Roedy Green]

But as the program runs, its memory keep on inscreasing. I don't know
why

see http://mindprod.com/jgloss/packratting.html

 

[krumzv]

Hi,

But as the program runs, its memory keep on inscreasing. I don't know
why.

the task manager isn't good to look at the java heap. It gives you
only some info about the memory reserved by the java process. Try
looking at the GC output to see what happends with the java heap. The
easiest option is just
-verbose:gc
add it to your start parameters and look at the console (or wherever
it is redirected).

If you need to analyze a memory leak, make a heap dump and analyze it.
Look at http://www.eclipse.org/mat/ for an good open source tool.

greets,
krum

 

[joehust]

Hi,


the task manager isn't good to look at the java heap. It gives you
only some info about the memory reserved by the java process. Try
looking at the GC output to see what happends with the java heap. The
easiest option is just
-verbose:gc
add it to your start parameters and look at the console (or wherever
it is redirected).

If you need to analyze a memory leak, make a heap dump and analyze it.
Look athttp://www.eclipse.org/mat/for an good open source tool.

greets,
krum

Thanks for your suggestions

 

[joehust]

First, regarding that "its memory keep on increasing", how are you
measuring this?

It is normal for Java programs' memory usage to increase up to a point. Up to
what point does the memory increase? Does the program ever throw an
OutOfMemoryError?

How much memory are you allowing the program to take? It is normal for Java
programs to appear to the operating system to have all their permitted
allocation. It is normal for Java programs' heap to be near the -Xmx value
sometimes.

It is a common and recommended Java idiom to create gobs of very short-lived
objects.

The program does throw an OutOfMemoryError unless I give it a higher
memory. And it will always eat
them up after some time. My program doesn't keep many things in
memory, it seems to me that it will not cost many memories.

Thank you for all you guys' suggestions, I am busy working on some
other problems, and I will try your methods when I get some time.