Handling OutOfMemory Error

Discussion in 'Java' started by ruds, Sep 16, 2008.

  1. ruds

    ruds Guest

    Hi,
    I'm operating on a 600MB file to extract some data from it.
    I'm using ArrayList and HashTable for storing the values extracted
    from the file and manipulating the data obtained.
    When I try executing the program I'm getting the following error:

    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown
    Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuffer.append(Unknown Source)
    at
    Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
    at
    Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)

    How should I increase the JVM's memory to handle such large amount of
    data?
    Is there any other way around it?
     
    ruds, Sep 16, 2008
    #1
    1. Advertising

  2. On Sep 16, 8:49 am, ruds <> wrote:
    > Hi,
    > I'm operating on a 600MB file to extract some data from it.
    > I'm using ArrayList and HashTable for storing the values extracted
    > from the file and manipulating the data obtained.
    > When I try executing the program I'm getting the following error:
    >
    > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    >         at java.util.Arrays.copyOf(Unknown Source)
    >         at java.lang.AbstractStringBuilder.expandCapacity(Unknown
    > Source)
    >         at java.lang.AbstractStringBuilder.append(Unknown Source)
    >         at java.lang.StringBuffer.append(Unknown Source)
    >         at
    > Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
    >         at
    > Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)
    >
    > How should I increase the JVM's memory to handle such large amount of
    > data?
    > Is there any other way around it?


    The -Xmx command line flag lets you set the heap size. See java.exe
    docs dor details..
     
    softwarepearls_com, Sep 16, 2008
    #2
    1. Advertising

  3. ruds

    EJP Guest

    ruds wrote:
    > I'm using ArrayList and HashTable for storing the values extracted
    > from the file and manipulating the data obtained.


    Don't.

    Use a database.
     
    EJP, Sep 16, 2008
    #3
  4. ruds

    Tom Anderson Guest

    On Mon, 15 Sep 2008, ruds wrote:

    > I'm operating on a 600MB file to extract some data from it. I'm using
    > ArrayList and HashTable for storing the values extracted from the file
    > and manipulating the data obtained. When I try executing the program I'm
    > getting the following error:
    >
    > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    > at java.util.Arrays.copyOf(Unknown Source)
    > at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    > at java.lang.AbstractStringBuilder.append(Unknown Source)
    > at java.lang.StringBuffer.append(Unknown Source)
    > at Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
    > at Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)
    >
    > How should I increase the JVM's memory to handle such large amount of
    > data?


    -Xmx, as others have mentioned.

    > Is there any other way around it?


    Firstly, the stacktrace there is from a StringBuffer which is trying to
    expand itself. If you can work out ahead of time how big that StringBuffer
    eventually needs to be, or even put a useful upper bound on it, then you
    can create the StringBuffer with that much capacity in the first place,
    which will avoid the need to expand it, and might avoid that failure. It
    should certainly improve performance.

    I'd also look at whether you need to use that buffer at all - you're not
    doing something like reading the entire file line by line and putting it
    in the buffer, are you? If you are, find an alternative! In general, a
    great way to reduce memory use is to find ways of doing things
    incrementally, so you don't need to have all your data in memory at once.
    For instance, if you were adding up all the numbers in a file, you might
    do:

    make an empty list of numbers
    for line in file:
    parse the line to a number
    put the number in the list
    set the total to 0
    for number in list:
    add the number to the total
    report the total

    But you'd use far less memory like this:

    set the total to 0
    for line in file:
    parse the line to a number
    add the number to the total
    report the total

    That's an obvious and trivial and example, and i imagine your example does
    not admit such easy improvements. However, with the application of a
    sufficient amount of cleverness, some degree of incrementalisation is
    often possible.

    Otherwise, whilst ArrayList is fine, the HashMap would worry me slightly
    in terms of space. How are you using it? Is the set of keys the same or
    similar between records? Could you use an object instead of a hashmap?

    If you tell us more about your program, we can give you more specific
    help.

    tom

    --
    Any problem in computer science can be solved with another layer of
    indirection. -- David Wheeler
     
    Tom Anderson, Sep 16, 2008
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. mani

    OutOfMemory error

    mani, Sep 12, 2003, in forum: Java
    Replies:
    2
    Views:
    428
    Michael Borgwardt
    Sep 16, 2003
  2. Replies:
    25
    Views:
    9,519
  3. Lenny Wintfeld
    Replies:
    0
    Views:
    742
    Lenny Wintfeld
    May 17, 2006
  4. Replies:
    1
    Views:
    725
    Juha Laiho
    Jul 3, 2007
  5. Replies:
    0
    Views:
    501
Loading...

Share This Page