java.lang.OutOfMemory Error

Discussion in 'Java' started by puneet.bansal@wipro.com, Jul 11, 2005.

  1. Guest

    Hi,

    In a servlet, I am reading from the database and writing to PrintWriter
    stream. The database returns around 50,000 rows. I create a
    StringBuffer object just once outside the ResultSet loop and then
    append all my database data to it and write it to PrintWriter at the
    end of each iteration. In the beginning of each iteration I set the
    length of the StringBuffer to zero. I think this is an optimized code
    as there are no object creations inside the loop and I am re-using the
    same object for each iteration. But still when that loop is executed
    the memory consumption goes up by 50MB. I am using StringBuffers
    everywhere in the loop. Any idea what could be wrong. I am running my
    code on Websphere 5.

    Thanks.
    Puneet
     
    , Jul 11, 2005
    #1
    1. Advertising

  2. Ben_ Guest

    You can activate verbose GC logs (checkbox in the Administrative Console in
    the Process Definition).

    You'll then get some details about how memory increases (continuous slow
    increase or peaks).

    What OS are you running ?
     
    Ben_, Jul 11, 2005
    #2
    1. Advertising

  3. shakah Guest

    wrote:
    > Hi,
    >
    > In a servlet, I am reading from the database and writing to PrintWriter
    > stream. The database returns around 50,000 rows. I create a
    > StringBuffer object just once outside the ResultSet loop and then
    > append all my database data to it and write it to PrintWriter at the
    > end of each iteration. In the beginning of each iteration I set the
    > length of the StringBuffer to zero. I think this is an optimized code
    > as there are no object creations inside the loop and I am re-using the
    > same object for each iteration. But still when that loop is executed
    > the memory consumption goes up by 50MB. I am using StringBuffers
    > everywhere in the loop. Any idea what could be wrong. I am running my
    > code on Websphere 5.
    >
    > Thanks.
    > Puneet


    Is your servlet engine set to buffer the output? If so, try overriding
    that -- in the JSP world it would be with a page directive like the
    following, not sure what it would be for a servlet:
    <%@ page buffer="none" %>
     
    shakah, Jul 11, 2005
    #3
  4. Guest

    I am running IBM AIX 4.3. I can't do much on the server as I don't have
    control over it. I can read Websphere logs though. They just say
    java.lang.OutOfMemory error. I read a lot on the net, but couldn't find
    any concrete conclusion as to what might cause the error. Some people
    are debating that if a loop has

    String a = "b" + "c"
    and this loop is run over a million iterations then it can consume
    significant amount of memory. And that such things should be made
    static and outside the loop. Is this true, as I do have a couple of
    such expressions. Rest I am using StringBuffer to write the output and
    also estimating and ensuring the capacity.

    Thanks.
    Puneet
     
    , Jul 11, 2005
    #4
  5. On 11 Jul 2005 10:30:39 -0700, wrote:

    > Some people
    > are debating


    Debating?!?

    >..that if a loop has
    >
    > String a = "b" + "c"
    > and this loop is run over a million iterations then it can consume
    > significant amount of memory. And that such things should be made
    > static and outside the loop. Is this true,


    Sure is!

    >..as I do have a couple of such expressions.


    That will almost certainly be contributing to the problem.

    Try this way instead..

    public String makeTheString(String[] bigResult) {
    // better way - uses a StringBuffer
    StringBuffer sb = new StringBuffer();
    for (int ii=0; ii<bigResult.length; ii++) {
    sb.append();
    }
    return sb.toString();
    }

    HTH

    --
    Andrew Thompson
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.PhySci.org/ Open-source software suite
    http://www.1point1C.org/ Science & Technology
    http://www.LensEscapes.com/ Images that escape the mundane
     
    Andrew Thompson, Jul 11, 2005
    #5
  6. Pete Barrett Guest

    On 11 Jul 2005 08:14:33 -0700, wrote:

    >Hi,
    >
    >In a servlet, I am reading from the database and writing to PrintWriter
    >stream. The database returns around 50,000 rows. I create a
    >StringBuffer object just once outside the ResultSet loop and then
    >append all my database data to it and write it to PrintWriter at the
    >end of each iteration. In the beginning of each iteration I set the
    >length of the StringBuffer to zero. I think this is an optimized code
    >as there are no object creations inside the loop and I am re-using the
    >same object for each iteration. But still when that loop is executed
    >the memory consumption goes up by 50MB. I am using StringBuffers
    >everywhere in the loop. Any idea what could be wrong. I am running my
    >code on Websphere 5.
    >

    The PrintWriter constructors have versions which take an extra boolean
    value called autoFlush. If you set this to true, writeln() will flush
    the stream, but if not, not. If you're not setting the PrintWriter to
    autoflush already, either do that or flush it occasionally, and see if
    that helps.

    Pete Barrett
     
    Pete Barrett, Jul 11, 2005
    #6
  7. On 11 Jul 2005 08:14:33 -0700, wrote:

    > I create a
    > StringBuffer object just once outside the ResultSet loop and then
    > append all my database data to it and write it to PrintWriter at the
    > end of each iteration.


    Why not writing to the PrintWriter directly?


    Regards, Lothar
    --
    Lothar Kimmeringer E-Mail:
    PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

    Always remember: The answer is forty-two, there can only be wrong
    questions!
     
    Lothar Kimmeringer, Jul 11, 2005
    #7
  8. Andrew Thompson wrote:
    > On 11 Jul 2005 10:30:39 -0700, wrote:
    >
    >
    >>Some people
    >>are debating

    >
    >
    > Debating?!?
    >
    >
    >>..that if a loop has
    >>
    >>String a = "b" + "c"
    >>and this loop is run over a million iterations then it can consume
    >>significant amount of memory. And that such things should be made
    >>static and outside the loop. Is this true,

    >
    >
    > Sure is!
    >


    Surely you do not mean this. I assume you read the code as:

    String a = b + c;

    Since

    String a = "b" + "c";

    is transformed into at compile time and equivalent to

    String a = "bc";

    which will simply re-use the same interned string representation of "bc"
    over and over again.

    >
    >>..as I do have a couple of such expressions.

    >
    >
    > That will almost certainly be contributing to the problem.
    >
    > Try this way instead..
    >
    > public String makeTheString(String[] bigResult) {
    > // better way - uses a StringBuffer
    > StringBuffer sb = new StringBuffer();
    > for (int ii=0; ii<bigResult.length; ii++) {
    > sb.append();
    > }
    > return sb.toString();
    > }
    >
    > HTH
    >


    Ray

    --
    XML is the programmer's duct tape.
     
    Raymond DeCampo, Jul 11, 2005
    #8
  9. Guest

    Thanks for all your responses. I have removed all string concatenation
    from my loop now. I am using only sb.append(x) to do any concatenation.
    I am manually flushing the PrintWriter at the end of each iteration. So
    that should take care of buffering if the servlet engine is trying to
    buffer. And am re-using the same StringBuffer in each iteration by
    setting its length to zero in the beginning of loop.
    But still the problem persists. I just generated an output with 22000
    rows and 40 columns and the server memory consumption went up from 90MB
    to 308 MB. What is more surprising is, that after this method has
    returned, then at least the server should release the memory. It does
    not. I can't figure out where is the memory being used. Please advise.
    I have tried all the above posted suggestions.

    Puneet
     
    , Jul 11, 2005
    #9
  10. Guest

    Also, will directly writing to PrintWriter object help?
     
    , Jul 11, 2005
    #10
  11. On Mon, 11 Jul 2005 18:10:01 GMT, Raymond DeCampo wrote:

    > Andrew Thompson wrote:
    >> On 11 Jul 2005 10:30:39 -0700, wrote:


    >> Sure is!

    ...
    > Surely you do not mean this.


    Err..

    >..I assume you read the code as:
    >
    > String a = b + c;


    (chuckles) Actually, I read it wrong, but a third way!

    Please disregard my message - it was based on a completely
    wrong understanding of what the OP meant. My bad.

    --
    Andrew Thompson
    http://www.PhySci.org/codes/ Web & IT Help
    http://www.PhySci.org/ Open-source software suite
    http://www.1point1C.org/ Science & Technology
    http://www.LensEscapes.com/ Images that escape the mundane
     
    Andrew Thompson, Jul 11, 2005
    #11
  12. Ben_ Guest

    (1)

    The IBM JDK can produce full Heap Dumps.

    You can then review them with the Heap Analyzer
    (http://www.alphaworks.ibm.com/tech/heapanalyzer).

    With this, you could see the largest objects in the heap at that moment. I'd
    suggest you review the IBM JVM Diagnostic Guide
    (http://www.ibm.com/developerworks/java/jdk/diagnosis).

    I'm not familiar with the AIX platform, but according to the guide (Ch. 14),
    they are produced automatically in the working directory in files like
    heapdumpPID.TIME.txt.

    (2)

    You've been focusing on the String usage in your loop. But what about the
    ResultSet and the data retrieved from the DB ? Are you sure it's not that
    piece that cause the application to exhaust its heap ?
     
    Ben_, Jul 11, 2005
    #12
  13. Guest

    I am posting the code that I am running (just in case anybody wants to
    evaluate it). It's fairly simple. The arrays used in the code are
    static arrays that contain list of columns to be read from database.


    while (rs.next())
    {
    sb.setLength(0);
    sb.ensureCapacity(550);
    sb.append("<tr>");
    sb.append("<td>").append(rs.getString("CB_NM")).append("</td>");
    sb.append("<td>").append(rs.getString("BP_ENT_ID")).
    append("/").append(rs.getString("BP_ROLE_ID")).append("</td>");
    sb.append("<td>").append(rs.getString("CNGLMRTE")).append("</td>");
    sb.append("<td>").append(rs.getString("BP_NM")).append("</td>");
    sb.append("<td>").append(rs.getString("BP_MKT_CTY_ST_ABRV")).
    append("</td>");

    sb.append("<td>").append(rs.getString("BP_FAX_NBR")).append("</td>");
    sb.append("<td>").append(rs.getString("CNTCT_NM")).append("</td>");
    sb.append("<td>").append((rs.getString("CNTCT_FAX_NBR")==null?"":
    rs.getString("CNTCT_FAX_NBR"))).append("</td>");
    sb.append("<td>").append((rs.getString("CNTCT_TEL_NBR")==null?"":
    rs.getString("CNTCT_TEL_NBR"))).append("</td>");
    sb.append("<td>").append((rs.getString("CNTCT_EMAIL_ADDR")==null?"":
    rs.getString("CNTCT_EMAIL_ADDR"))).append("</td>");
    sb.append("<td>").append(rs.getString("STR_NBR")).append("</td>");

    sb.append("<td>").append(rs.getString("STR_ADDR_LINE_1")).append("</td>");
    sb.append("<td>").append(rs.getString("STR_ADDR_CTY")).
    append("</td>");
    sb.append("<td>").append(rs.getString("STR_STATE_ABRV")).
    append("</td>");
    sb.append("<td>").append(rs.getString("STR_ADDR_ZIP")).
    append("</td>");
    sb.append("<td>").append(rs.getString("STR_CNTY_NM")).
    append("</td>");

    for (j=0; j < brandCount; j++)
    {
    //Suppress zero if store count is zero
    sb.append("<td>").append((rs.getString(brands[j]).equals("0"))?"":
    rs.getString(brands[j])).append("</td>");
    }
    sb.append("</tr>");

    // Print the business partner total if business partner has changed
    String currBPEntId = rs.getString("BP_ENT_ID");
    String currBPRoleId = rs.getString("BP_ROLE_ID");
    String nextBPEntId = null;
    String nextBPRoleId = null;
    String nextRegion = null;
    String nextDivision = null;
    String nextZone = null;
    String nextSystem = null;
    String nextCC = null;

    //Peep into the next row and get the hierarchy for next row
    if (rs.next())
    {
    nextBPEntId = rs.getString("BP_ENT_ID");
    nextBPRoleId = rs.getString("BP_ROLE_ID");
    nextRegion = rs.getString("REGION_NM");
    nextDivision = rs.getString("DIVISION_NM");
    nextZone = rs.getString("ZONE_NM");
    nextSystem = rs.getString("SYSTEM_NM");
    nextCC = rs.getString("CC_NM");
    }
    rs.previous();
    //If BP has changed print BP Total
    if (!currBPEntId.equals(nextBPEntId)
    &&(!currBPRoleId.equals(nextBPRoleId)))
    {
    sb.append("<tr >");
    sb.append(colouredTD).append(
    "<b>Business Partner Total </b></td>");
    sb.append(oneBlankCell).append(fourteenBlankCells);

    for (j=0; j<brandCount; j++)
    {
    sb.append(colouredTD).
    append("<B>").append(rs.getString(bpTotalStrArr[j])).
    append("</b></td>");
    }
    }
    //End of business partner total printing

    } //end of while loop
     
    , Jul 11, 2005
    #13
  14. Guest

    You could be right Ben. My focus has totally been on String usage. But
    what's the alternative of doing rs.getString(). I have to do it for
    every value I retrieve. Isn't it?
     
    , Jul 11, 2005
    #14
  15. Ben_ Guest

    My point is that loading 50K rows will take memory anyhow. And maybe it's
    that part that is resource consuming.

    Can you make the test of browse the 50K rows without building the HTML table
    at all (just count them and print the total, for example) ? This would give
    an indication as to who of the Strings or the ResultSet is eating the
    memory.
     
    Ben_, Jul 11, 2005
    #15
  16. wrote:
    > I am posting the code that I am running (just in case anybody wants to
    > evaluate it). It's fairly simple. The arrays used in the code are
    > static arrays that contain list of columns to be read from database.
    >
    >
    > while (rs.next())
    > {
    >

    [snip]
    > //Peep into the next row and get the hierarchy for next row
    > if (rs.next())
    > {

    [snip]
    > }
    > rs.previous();
    >

    [snip]
    > } //end of while loop
    >


    If your rs.next() fails, you will still do rs.previous(). You should
    skip the call to previous also if the next fails, so put it inside that
    'if' statement.
    If you get an OutOfMemory error, the problem is an infinite loop +
    dynamic allocation ('append' in this case) with a 99% chance IME.
    --Paul
     
    Paul Bilnoski, Jul 11, 2005
    #16
  17. Guest

    You are right Ben. I commented all the code inside the loop and had
    just one int variable incrementing to count the rows and the memory
    consumption still went up from 90MB to 272MB. With all the code it goes
    from 90MB to 308 MB. So the problem is not the code, it's the amount of
    data. For this test I am fetching 22000 rows with 40 columns each. That
    makes it 0.88 million data items and all of it gets loaded into the
    ResultSet object. It is bound to consume memory. It should have struck
    us before.
    But then what is the solution for this? This seems to be a dead end. If
    I have to show this much data, then what do I do?

    And Paul, the call to previous() is out of the if condition because
    /*This call to previous needs to be out of if
    condition because
    * on the last row, call to next() will still move the
    pointer to
    * the end of the resultset but will return false. The
    pointer
    * still has to be moved back to point to the last row
    * /
    (I had taken off this comment to make the code compact for posting)
     
    , Jul 11, 2005
    #17
  18. Ben_ Guest

    Why this:
    sb.append("<tr>");
    sb.append("<td>").append(rs.getString("CB_NM")).append("</td>");
    instead of this:
    out.print("<tr><td>");
    out.print(rs.getString("CB_NM"));
    out.print("</td>");

    With your code, you're building an ever-increasing StringBuffer (and thus
    array of char) before printing all the stuff at once somewhere later on.
    Arrays have to fit in a contiguous (non fragmented) area of the heap. The
    larger the array, the smaller the chance to find a large enough heap block.
    And if no heap is found, then OutOfMemory is thrown.
     
    Ben_, Jul 11, 2005
    #18
  19. wrote:
    > You are right Ben. I commented all the code inside the loop and had
    > just one int variable incrementing to count the rows and the memory
    > consumption still went up from 90MB to 272MB. With all the code it goes
    > from 90MB to 308 MB. So the problem is not the code, it's the amount of
    > data. For this test I am fetching 22000 rows with 40 columns each. That
    > makes it 0.88 million data items and all of it gets loaded into the
    > ResultSet object. It is bound to consume memory. It should have struck
    > us before.
    > But then what is the solution for this? This seems to be a dead end. If
    > I have to show this much data, then what do I do?
    >
    > And Paul, the call to previous() is out of the if condition because
    > /*This call to previous needs to be out of if
    > condition because
    > * on the last row, call to next() will still move the
    > pointer to
    > * the end of the resultset but will return false. The
    > pointer
    > * still has to be moved back to point to the last row
    > * /
    > (I had taken off this comment to make the code compact for posting)
    >


    Okay, thanks for clearing that up.
    --Paul
     
    Paul Bilnoski, Jul 11, 2005
    #19
  20. Guest

    Ben,

    I have another report to output where I am not using the StringBuffer
    technique but am directly writing to PrintWriter as you hav suggested.
    It still consumes 10MB memory to show 1200 rows with 30 columns each.
    Basically, it's the ResultSet object that takes all the memory.

    Puneet
     
    , Jul 11, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Michael Borgwardt

    java.lang.OutOfMemory

    Michael Borgwardt, May 7, 2004, in forum: Java
    Replies:
    8
    Views:
    3,530
    Boudewijn Dijkstra
    May 15, 2004
  2. Rune
    Replies:
    2
    Views:
    557
    Kevin McMurtrie
    Feb 19, 2005
  3. Rajesh.Rapaka

    Busting "java.lang.outofmemory"

    Rajesh.Rapaka, Jul 13, 2005, in forum: Java
    Replies:
    17
    Views:
    945
    Martin Jost
    Jul 27, 2005
  4. Replies:
    0
    Views:
    4,588
  5. Ananth
    Replies:
    8
    Views:
    4,213
    Jason Cavett
    Mar 5, 2008
Loading...

Share This Page