java.lang.OutOfMemory Error

[puneet.bansal]

Hi,

In a servlet, I am reading from the database and writing to PrintWriter
stream. The database returns around 50,000 rows. I create a
StringBuffer object just once outside the ResultSet loop and then
append all my database data to it and write it to PrintWriter at the
end of each iteration. In the beginning of each iteration I set the
length of the StringBuffer to zero. I think this is an optimized code
as there are no object creations inside the loop and I am re-using the
same object for each iteration. But still when that loop is executed
the memory consumption goes up by 50MB. I am using StringBuffers
everywhere in the loop. Any idea what could be wrong. I am running my
code on Websphere 5.

Thanks.
Puneet

 

[Ben_]

You can activate verbose GC logs (checkbox in the Administrative Console in
the Process Definition).

You'll then get some details about how memory increases (continuous slow
increase or peaks).

What OS are you running ?

 

[shakah]

Hi,

In a servlet, I am reading from the database and writing to PrintWriter
stream. The database returns around 50,000 rows. I create a
StringBuffer object just once outside the ResultSet loop and then
append all my database data to it and write it to PrintWriter at the
end of each iteration. In the beginning of each iteration I set the
length of the StringBuffer to zero. I think this is an optimized code
as there are no object creations inside the loop and I am re-using the
same object for each iteration. But still when that loop is executed
the memory consumption goes up by 50MB. I am using StringBuffers
everywhere in the loop. Any idea what could be wrong. I am running my
code on Websphere 5.

Thanks.
Puneet

Is your servlet engine set to buffer the output? If so, try overriding
that -- in the JSP world it would be with a page directive like the
following, not sure what it would be for a servlet:
<%@ page buffer="none" %>

 

[puneet.bansal]

I am running IBM AIX 4.3. I can't do much on the server as I don't have
control over it. I can read Websphere logs though. They just say
java.lang.OutOfMemory error. I read a lot on the net, but couldn't find
any concrete conclusion as to what might cause the error. Some people
are debating that if a loop has

String a = "b" + "c"
and this loop is run over a million iterations then it can consume
significant amount of memory. And that such things should be made
static and outside the loop. Is this true, as I do have a couple of
such expressions. Rest I am using StringBuffer to write the output and
also estimating and ensuring the capacity.

Thanks.
Puneet

 

[Andrew Thompson]

Some people
are debating
Debating?!?

..that if a loop has

String a = "b" + "c"
and this loop is run over a million iterations then it can consume
significant amount of memory. And that such things should be made
static and outside the loop. Is this true,

Sure is!

..as I do have a couple of such expressions.

That will almost certainly be contributing to the problem.

Try this way instead..

public String makeTheString(String[] bigResult) {
// better way - uses a StringBuffer
StringBuffer sb = new StringBuffer();
for (int ii=0; ii<bigResult.length; ii++) {
sb.append();
}
return sb.toString();
}

HTH

 

[Pete Barrett]

Hi,

In a servlet, I am reading from the database and writing to PrintWriter
stream. The database returns around 50,000 rows. I create a
StringBuffer object just once outside the ResultSet loop and then
append all my database data to it and write it to PrintWriter at the
end of each iteration. In the beginning of each iteration I set the
length of the StringBuffer to zero. I think this is an optimized code
as there are no object creations inside the loop and I am re-using the
same object for each iteration. But still when that loop is executed
the memory consumption goes up by 50MB. I am using StringBuffers
everywhere in the loop. Any idea what could be wrong. I am running my
code on Websphere 5.

The PrintWriter constructors have versions which take an extra boolean
value called autoFlush. If you set this to true, writeln() will flush
the stream, but if not, not. If you're not setting the PrintWriter to
autoflush already, either do that or flush it occasionally, and see if
that helps.

Pete Barrett

 

[Lothar Kimmeringer]

I create a
StringBuffer object just once outside the ResultSet loop and then
append all my database data to it and write it to PrintWriter at the
end of each iteration.

Why not writing to the PrintWriter directly?


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Raymond DeCampo]

Sure is!

Surely you do not mean this. I assume you read the code as:

String a = b + c;

Since

String a = "b" + "c";

is transformed into at compile time and equivalent to

String a = "bc";

which will simply re-use the same interned string representation of "bc"
over and over again.

..as I do have a couple of such expressions.

That will almost certainly be contributing to the problem.

Try this way instead..

public String makeTheString(String[] bigResult) {
// better way - uses a StringBuffer
StringBuffer sb = new StringBuffer();
for (int ii=0; ii<bigResult.length; ii++) {
sb.append();
}
return sb.toString();
}

HTH

Ray

 

[puneet.bansal]

Thanks for all your responses. I have removed all string concatenation
from my loop now. I am using only sb.append(x) to do any concatenation.
I am manually flushing the PrintWriter at the end of each iteration. So
that should take care of buffering if the servlet engine is trying to
buffer. And am re-using the same StringBuffer in each iteration by
setting its length to zero in the beginning of loop.
But still the problem persists. I just generated an output with 22000
rows and 40 columns and the server memory consumption went up from 90MB
to 308 MB. What is more surprising is, that after this method has
returned, then at least the server should release the memory. It does
not. I can't figure out where is the memory being used. Please advise.
I have tried all the above posted suggestions.

Puneet

 

[puneet.bansal]

Also, will directly writing to PrintWriter object help?

 

[Andrew Thompson]

...
Surely you do not mean this.
Err..

..I assume you read the code as:

String a = b + c;

(chuckles) Actually, I read it wrong, but a third way!

Please disregard my message - it was based on a completely
wrong understanding of what the OP meant. My bad.

 

[Ben_]

(1)

The IBM JDK can produce full Heap Dumps.

You can then review them with the Heap Analyzer
(http://www.alphaworks.ibm.com/tech/heapanalyzer).

With this, you could see the largest objects in the heap at that moment. I'd
suggest you review the IBM JVM Diagnostic Guide
(http://www.ibm.com/developerworks/java/jdk/diagnosis).

I'm not familiar with the AIX platform, but according to the guide (Ch. 14),
they are produced automatically in the working directory in files like
heapdumpPID.TIME.txt.

(2)

You've been focusing on the String usage in your loop. But what about the
ResultSet and the data retrieved from the DB ? Are you sure it's not that
piece that cause the application to exhaust its heap ?

 

[puneet.bansal]

I am posting the code that I am running (just in case anybody wants to
evaluate it). It's fairly simple. The arrays used in the code are
static arrays that contain list of columns to be read from database.


while (rs.next())
{
sb.setLength(0);
sb.ensureCapacity(550);
sb.append("<tr>");
sb.append("<td>").append(rs.getString("CB_NM")).append("</td>");
sb.append("<td>").append(rs.getString("BP_ENT_ID")).
append("/").append(rs.getString("BP_ROLE_ID")).append("</td>");
sb.append("<td>").append(rs.getString("CNGLMRTE")).append("</td>");
sb.append("<td>").append(rs.getString("BP_NM")).append("</td>");
sb.append("<td>").append(rs.getString("BP_MKT_CTY_ST_ABRV")).
append("</td>");

sb.append("<td>").append(rs.getString("BP_FAX_NBR")).append("</td>");
sb.append("<td>").append(rs.getString("CNTCT_NM")).append("</td>");
sb.append("<td>").append((rs.getString("CNTCT_FAX_NBR")==null?"":
rs.getString("CNTCT_FAX_NBR"))).append("</td>");
sb.append("<td>").append((rs.getString("CNTCT_TEL_NBR")==null?"":
rs.getString("CNTCT_TEL_NBR"))).append("</td>");
sb.append("<td>").append((rs.getString("CNTCT_EMAIL_ADDR")==null?"":
rs.getString("CNTCT_EMAIL_ADDR"))).append("</td>");
sb.append("<td>").append(rs.getString("STR_NBR")).append("</td>");

sb.append("<td>").append(rs.getString("STR_ADDR_LINE_1")).append("</td>");
sb.append("<td>").append(rs.getString("STR_ADDR_CTY")).
append("</td>");
sb.append("<td>").append(rs.getString("STR_STATE_ABRV")).
append("</td>");
sb.append("<td>").append(rs.getString("STR_ADDR_ZIP")).
append("</td>");
sb.append("<td>").append(rs.getString("STR_CNTY_NM")).
append("</td>");

for (j=0; j < brandCount; j++)
{
//Suppress zero if store count is zero
sb.append("<td>").append((rs.getString(brands[j]).equals("0"))?"":
rs.getString(brands[j])).append("</td>");
}
sb.append("</tr>");

// Print the business partner total if business partner has changed
String currBPEntId = rs.getString("BP_ENT_ID");
String currBPRoleId = rs.getString("BP_ROLE_ID");
String nextBPEntId = null;
String nextBPRoleId = null;
String nextRegion = null;
String nextDivision = null;
String nextZone = null;
String nextSystem = null;
String nextCC = null;

//Peep into the next row and get the hierarchy for next row
if (rs.next())
{
nextBPEntId = rs.getString("BP_ENT_ID");
nextBPRoleId = rs.getString("BP_ROLE_ID");
nextRegion = rs.getString("REGION_NM");
nextDivision = rs.getString("DIVISION_NM");
nextZone = rs.getString("ZONE_NM");
nextSystem = rs.getString("SYSTEM_NM");
nextCC = rs.getString("CC_NM");
}
rs.previous();
//If BP has changed print BP Total
if (!currBPEntId.equals(nextBPEntId)
&&(!currBPRoleId.equals(nextBPRoleId)))
{
sb.append("<tr >");
sb.append(colouredTD).append(
"<b>Business Partner Total </b></td>");
sb.append(oneBlankCell).append(fourteenBlankCells);

for (j=0; j<brandCount; j++)
{
sb.append(colouredTD).
append("<B>").append(rs.getString(bpTotalStrArr[j])).
append("</b></td>");
}
}
//End of business partner total printing

} //end of while loop

 

[puneet.bansal]

You could be right Ben. My focus has totally been on String usage. But
what's the alternative of doing rs.getString(). I have to do it for
every value I retrieve. Isn't it?

 

[Ben_]

My point is that loading 50K rows will take memory anyhow. And maybe it's
that part that is resource consuming.

Can you make the test of browse the 50K rows without building the HTML table
at all (just count them and print the total, for example) ? This would give
an indication as to who of the Strings or the ResultSet is eating the
memory.

 

[Paul Bilnoski]

I am posting the code that I am running (just in case anybody wants to
evaluate it). It's fairly simple. The arrays used in the code are
static arrays that contain list of columns to be read from database.


while (rs.next())
{
[snip]
//Peep into the next row and get the hierarchy for next row
if (rs.next())
{ [snip]

}
rs.previous();

[snip]
} //end of while loop

If your rs.next() fails, you will still do rs.previous(). You should
skip the call to previous also if the next fails, so put it inside that
'if' statement.
If you get an OutOfMemory error, the problem is an infinite loop +
dynamic allocation ('append' in this case) with a 99% chance IME.
--Paul

 

[puneet.bansal]

You are right Ben. I commented all the code inside the loop and had
just one int variable incrementing to count the rows and the memory
consumption still went up from 90MB to 272MB. With all the code it goes
from 90MB to 308 MB. So the problem is not the code, it's the amount of
data. For this test I am fetching 22000 rows with 40 columns each. That
makes it 0.88 million data items and all of it gets loaded into the
ResultSet object. It is bound to consume memory. It should have struck
us before.
But then what is the solution for this? This seems to be a dead end. If
I have to show this much data, then what do I do?

And Paul, the call to previous() is out of the if condition because
/*This call to previous needs to be out of if
condition because
* on the last row, call to next() will still move the
pointer to
* the end of the resultset but will return false. The
pointer
* still has to be moved back to point to the last row
* /
(I had taken off this comment to make the code compact for posting)

 

[Ben_]

Why this:
sb.append("<tr>");
sb.append("<td>").append(rs.getString("CB_NM")).append("</td>");
instead of this:
out.print("<tr><td>");
out.print(rs.getString("CB_NM"));
out.print("</td>");

With your code, you're building an ever-increasing StringBuffer (and thus
array of char) before printing all the stuff at once somewhere later on.
Arrays have to fit in a contiguous (non fragmented) area of the heap. The
larger the array, the smaller the chance to find a large enough heap block.
And if no heap is found, then OutOfMemory is thrown.

 

[Paul Bilnoski]

You are right Ben. I commented all the code inside the loop and had
just one int variable incrementing to count the rows and the memory
consumption still went up from 90MB to 272MB. With all the code it goes
from 90MB to 308 MB. So the problem is not the code, it's the amount of
data. For this test I am fetching 22000 rows with 40 columns each. That
makes it 0.88 million data items and all of it gets loaded into the
ResultSet object. It is bound to consume memory. It should have struck
us before.
But then what is the solution for this? This seems to be a dead end. If
I have to show this much data, then what do I do?

And Paul, the call to previous() is out of the if condition because
/*This call to previous needs to be out of if
condition because
* on the last row, call to next() will still move the
pointer to
* the end of the resultset but will return false. The
pointer
* still has to be moved back to point to the last row
* /
(I had taken off this comment to make the code compact for posting)

Okay, thanks for clearing that up.
--Paul

 

[puneet.bansal]

Ben,

I have another report to output where I am not using the StringBuffer
technique but am directly writing to PrintWriter as you hav suggested.
It still consumes 10MB memory to show 1200 rows with 30 columns each.
Basically, it's the ResultSet object that takes all the memory.

Puneet