Memory Leak Problem

Discussion in 'Java' started by joehust@gmail.com, May 13, 2008.

  1. Guest

    I have a crawler program, it fetches html on the internet and a
    parser will parse the page.
    Because of slow networks, I use many robots to crawl the pages and
    one parser.
    The program is multi-threaded.
    There is a pagestore object used as the bridge between robot and
    parser, it's has a List structure,
    the parser will remove the item it has parsed.

    But as the program runs, its memory keep on inscreasing. I don't know
    why. What I see in the taskmgr shows that most memory is in the
    virtual memory, and the actual memory occupation is not much.

    So what could possibly be the reasons?

    Joe
    , May 13, 2008
    #1
    1. Advertising

  2. Roedy Green Guest

    Roedy Green, May 13, 2008
    #2
    1. Advertising

  3. Guest

    Hi,

    > But as the program runs, its memory keep on inscreasing. I don't know
    > why.


    the task manager isn't good to look at the java heap. It gives you
    only some info about the memory reserved by the java process. Try
    looking at the GC output to see what happends with the java heap. The
    easiest option is just
    -verbose:gc
    add it to your start parameters and look at the console (or wherever
    it is redirected).

    If you need to analyze a memory leak, make a heap dump and analyze it.
    Look at http://www.eclipse.org/mat/ for an good open source tool.

    greets,
    krum
    , May 15, 2008
    #3
  4. Guest

    On May 15, 3:13 pm, wrote:
    > Hi,
    >
    > > But as the program runs, its memory keep on inscreasing. I don't know
    > > why.

    >
    > the task manager isn't good to look at the java heap. It gives you
    > only some info about the memory reserved by the java process. Try
    > looking at the GC output to see what happends with the java heap. The
    > easiest option is just
    > -verbose:gc
    > add it to your start parameters and look at the console (or wherever
    > it is redirected).
    >
    > If you need to analyze a memory leak, make a heap dump and analyze it.
    > Look athttp://www.eclipse.org/mat/for an good open source tool.
    >
    > greets,
    > krum


    Thanks for your suggestions
    , May 16, 2008
    #4
  5. Guest

    On May 13, 9:15 pm, Lew <> wrote:
    > wrote:
    > >> I have a crawler program, it fetches html on the internet and a
    > >> parser will parse the page.
    > >> Because of slow networks, I use many robots to crawl the pages and
    > >> one parser.
    > >> The program is multi-threaded.
    > >> There is a pagestore object used as the bridge between robot and
    > >> parser, it's has a List structure,
    > >> the parser will remove the item it has parsed.

    >
    > >> But as the program runs, its memory keep on inscreasing. I don't know
    > >> why. What I see in the taskmgr shows that most memory is in the
    > >> virtual memory, and the actual memory occupation is not much.

    >
    > >> So what could possibly be the reasons?

    >
    > First, regarding that "its memory keep on increasing", how are you
    > measuring this?
    >
    > It is normal for Java programs' memory usage to increase up to a point. Up to
    > what point does the memory increase? Does the program ever throw an
    > OutOfMemoryError?
    >
    > How much memory are you allowing the program to take? It is normal for Java
    > programs to appear to the operating system to have all their permitted
    > allocation. It is normal for Java programs' heap to be near the -Xmx value
    > sometimes.
    >
    > It is a common and recommended Java idiom to create gobs of very short-lived
    > objects.
    >
    > --
    > Lew

    The program does throw an OutOfMemoryError unless I give it a higher
    memory. And it will always eat
    them up after some time. My program doesn't keep many things in
    memory, it seems to me that it will not cost many memories.

    Thank you for all you guys' suggestions, I am busy working on some
    other problems, and I will try your methods when I get some time.
    , May 16, 2008
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Spur
    Replies:
    1
    Views:
    388
    Howard Hinnant
    May 9, 2004
  2. s.subbarayan

    Dynamic memory allocation and memory leak...

    s.subbarayan, Mar 18, 2005, in forum: C Programming
    Replies:
    10
    Views:
    682
    Eric Sosman
    Mar 22, 2005
  3. Richard Heathfield

    Leak or no leak ??

    Richard Heathfield, Jul 10, 2006, in forum: C Programming
    Replies:
    4
    Views:
    340
    Richard Heathfield
    Jul 10, 2006
  4. cham
    Replies:
    5
    Views:
    758
  5. Mark Probert
    Replies:
    4
    Views:
    318
    Mark Probert
    Feb 9, 2005
Loading...

Share This Page