Servlet OutOfMemory on images. Please help me!

[Animanera]

Hi to all.

I developed a small servlet that give me the possibility to insert news
with photo. Each inserted images is resized to two resolutions (640x480
and 100x133) and saved to disk, discarding the original image.

The problem is that, after a certain number of upload of images files of
just 350kb, the Tomcat give me error of OutOfMemory and the upload of
files of such dimension come impossible (but it allows resizing of small
images). Restarting the server solve the problem, but only for few
uploads since after inserting again a certain number of file of the
aforesaid dimensions the problem come back.

You should know that:
- my ISP doesn't give to me the possibility to increase the maximum RAM
dedicated to the servlet container;
- the problem also occurs with sequential upload (therefore NOT
concurrent upload) alternated each other for a certain time (from 1 to 5
minutes);
- after some tests results that the problem appears after around 7
insertions of an image of 329kb;

Is there a way I can force the cleaning of images in memory immediately
(since I think that doing that resolves the probelm)?

The incriminated code is (as you will see I've tried to assign null to
image vars and call System.gc()...but without solving the problem):


public static void resizeImage(File imageFile, int width, int height,
float quality, int hints) throws IOException {
// Get input image

Image inputImage =
Toolkit.getDefaultToolkit().getImage(imageFile.getPath());



waitForImage(inputImage);

int inputWidth = inputImage.getWidth(null);
int inputHeight = inputImage.getHeight(null);

if ((width<=0) && (height<=0)) {
width=inputWidth;
height=inputHeight;
} else {
if (width==0)
width=inputWidth;
else if (width<0)
width=inputWidth*height/inputHeight;

if (height==0)
height=inputWidth;
else if (height<0)
height=inputHeight*width/inputWidth;

}

// Create output image.
Image outputImage = inputImage.getScaledInstance(width, height, hints);
waitForImage(outputImage);

// Encode JPEG file.
writeJPEG(outputImage, imageFile, quality);

inputImage = null;
outputImage = null;

System.gc();
}

public static int[] imageSize(File imageFile) throws IOException {
Image inputImage =
Toolkit.getDefaultToolkit().getImage(imageFile.getPath());
waitForImage(inputImage);

int[] result = new int[2];
result[0] = inputImage.getWidth(null);
result[1] = inputImage.getHeight(null);
inputImage = null;

System.gc();
return result;
}




public static void waitForImage( Image image ) {
try {
tracker.addImage(image, 0);

tracker.waitForID(0);

tracker.removeImage(image, 0);
} catch(InterruptedException exc) {
exc.printStackTrace();
} catch(NullPointerException exc) {
// Sometime appens. Why?
}
}


public static void writeJPEG(Image outputImage, File imageFile,
float outputQuality) throws java.io.IOException {
FileOutputStream os = new FileOutputStream(imageFile);
int outputWidth = outputImage.getWidth(null);
int outputHeight = outputImage.getHeight(null);


BufferedImage bi = new BufferedImage(outputWidth, outputHeight,
PJABufferedImage.TYPE_INT_RGB);
Graphics2D biContext = bi.createGraphics();
biContext.drawImage(outputImage, 0, 0, null);



JPEGImageEncoder encoder = JPEGCodec.createJPEGEncoder(os);

JPEGEncodeParam jep = JPEGCodec.getDefaultJPEGEncodeParam(bi);
jep.setQuality(outputQuality, true);
encoder.encode(bi, jep);

os.flush();
os.close();
}

Thank you very much in advance.
Bye, Animanera.

 

[Michael Borgwardt]

Is there a way I can force the cleaning of images in memory immediately
(since I think that doing that resolves the probelm)?

Most likely it would only delay the problem, since you'd still
have a memory leak, albeit with smaller objects.

The incriminated code is

No, it's not. You only posted some methods, not the entire class
definition. There is, for example a (presumably instance) variable
"tracker" that looks like a top contender for hanging on to image
references. Especially since you throw away Exceptions in your
waitForImage() method and possibly don't remove images from the
tracker in that case.

(as you will see I've tried to assign null to
image vars and call System.gc()...but without solving the problem):

Neither will achieve anything.

Memory leaks are fixed using a tool called a "heap profiler".

 

[Animanera]

No, it's not. You only posted some methods, not the entire class
definition. There is, for example a (presumably instance) variable
"tracker" that looks like a top contender for hanging on to image
references. Especially since you throw away Exceptions in your
waitForImage() method and possibly don't remove images from the
tracker in that case.

Oh yes, excuse me, I didn't post the entire class since it is a toolkit
with various functions.

However, reset all what I say: I'm starting from basics. Why this code
prints "Size is -1 -1"?!?

try {
File f = new File("c:\\temp\\file.jpg");
System.out.println(f.exists()); //Prints true!
MediaTracker t = new MediaTracker(new Frame());
Image im = Toolkit.getDefaultToolkit().getImage (f.getPath());

t.addImage(im, 0);
t.waitForID(0);
System.out.println("Size is" + im.getWidth(null) + " " +
im.getHeight(null));

} catch (Exception exc) {
exc.printStackTrace();
}

Thanks, bye, Animanera.

 

[Animanera]

However, reset all what I say: I'm starting from basics. Why this code
prints "Size is -1 -1"?!?

OOooops! Just seen that the image file used for test was
corrupted....sorry

Continuing to try to resolve the 1st problem...

Thanks, Bye, Animanera.

 

[Michael Borgwardt]

Continuing to try to resolve the 1st problem...

As I wrote: get a profiler. It might a bit difficult to set up, but
once it works you'll see immediately what's using up your memory
and where it's referenced. From there, it's usually pretty easy to
find out why those references are kept.

 

[Andrew Thompson]

Oh yes, excuse me, I didn't post the entire class since it is a toolkit
with various functions.

Your current code supports *exactly* what Michael was saying. [1]
But the code has a wider problem. [2]

However, reset all what I say: I'm starting from basics.

Even though you've changed the code, both of these are
problems with the current code.

try {
File f = new File("c:\\temp\\file.jpg");
System.out.println(f.exists()); //Prints true!
MediaTracker t = new MediaTracker(new Frame());
Image im = Toolkit.getDefaultToolkit().getImage (f.getPath());

[2] Your images are cached.

t.addImage(im, 0);
t.waitForID(0);
System.out.println("Size is" + im.getWidth(null) + " " +
im.getHeight(null));

} catch (Exception exc) {
exc.printStackTrace();

[1] You should free the image from the tracker in the catch.

}

HTH

 

[Animanera]

As I wrote: get a profiler. It might a bit difficult to set up, but
once it works you'll see immediately what's using up your memory
and where it's referenced. From there, it's usually pretty easy to
find out why those references are kept.

Just tried JProbe (freeware edition): It's true, the memory usage
increases as I upload images. As I seen, if I call the Garbage
collection immediatley after the upload (via my code or via Jprobe), or
after 1-2 seconds the memory doesn't go down. But if I call it after
5-10 seconds, the memory goes down. How I can interpret this fact?

I'm not a great knower of GC, but why this doesn't work:

<code>
resizeImage(filename, ..., ...);
System.gc();
</code>

However this version of Jprobe (as I see) doesn't give me the
possibility to view the memory amount used from each Object, but only
the total amount.

Can you suggest me a more detailed (but simple to install) heap profiler?

Many thanks, bye, Animanera.

 

[Animanera]

Your current code supports *exactly* what Michael was saying. [1]
[1] You should free the image from the tracker in the catch.

Oh yes, I'll do it. But I've to specify that there wasn't exceptions in
my tests.

But the code has a wider problem. [2]
[2] Your images are cached.

Why you say that image caching (in my case) is a problem?

Thanks, Bye, Animanera

 

[Andrew Thompson]

[2] Your images are cached.

Why you say that image caching (in my case) is a problem?

I base that upon a word in your subject 'OutOfMemory'.

 

[Animanera]

[2] Your images are cached.

Why you say that image caching (in my case) is a problem?

I base that upon a word in your subject 'OutOfMemory'.

Great! In a fast reading, I read it as "you are loading a chached image"
and not as "you are storing your images in a cache".

Using createImage instead of getImage resolved my problem! Thank you
very much!

But...I think that the problem will come back if users do a concurrent
access to the function. Therefore, what you can suggest to me to resolve
this possible problem? I think that I can't catch OutOfMemory (and
insert a sleep) since it is thrown at a thread level and not at a method
level, right?

Or, what do you think if I declare the static resizeImage method as
synchronized?

Many thanks Andrew and Michael
Bye, Animanera.

 

[Thomas Schodt]

if I call the Garbage
collection immediately after the upload (via my code or via Jprobe), or
after 1-2 seconds the memory doesn't go down. But if I call it after
5-10 seconds, the memory goes down. How I can interpret this fact?

Calling the System.gc() method only *suggests* that the JVM expend
effort toward recycling unused objects in order to make the memory they
currently occupy available for quick reuse.

When control returns from the System.gc() call, the JVM has made a best
effort to reclaim space from all discarded objects.

 

[Andrew Thompson]

Using createImage instead of getImage resolved my problem! Thank you
very much!

You're welcome.

But...I think that the problem will come back if users do a concurrent
access to the function. Therefore, what you can suggest to me to resolve
this possible problem?

I do not quite understand what you mean. I suggest you check
if there are problems before you worry about it. If problems
become apparent, perhaps start a new thread (referencing this one).

..I think that I can't catch OutOfMemory (and
insert a sleep) since it is thrown at a thread level and not at a method
level, right?

AFAIU, that is not the case. In fact I have a vague recollection* that
I managed to 'catch' OOM errors before, there are two points to note

1) java.lang.OutOfMemoryError sub-classes Error, not Exception, so

try {
// ...
} catch(Exception e) {
// this will not catch an OOMError!
} catch(OutOfMemoryError oome) {
// this will catch an OutOfMemoryError
// YOU SHOULD NOT EVEN TRY TO CATCH OTHER ERRORS
// BUT THE ONE YOU ARE DEALING WITH. EVEN THAT
// IS MORE 'WISHFUL THINKING' THAN 'USEFUL' see 2)
}

2) The application's out of memory, so what are you going to do now?

The only strategy I found that was even half workable was
to prepare a 'failure - ran out of memory' Dialog and
*have it ready* to setVisible(true) if you hit an OOME.

As far as I recall, even *that* was not entirely reliable.

It is because my attempts with 2) were not reliable that I
abandoned even attempting to inform the user and simply
dumped a trace to stderr and called System.exit(-1)*.

[ * Rough translation. "I threw up my hands in despair." ]

 

[Michael Borgwardt]

There is no "thread level" or "method level". All errors and exceptions
are thread-specific.

AFAIU, that is not the case. In fact I have a vague recollection* that
I managed to 'catch' OOM errors before, there are two points to note

1) java.lang.OutOfMemoryError sub-classes Error, not Exception, so

2) The application's out of memory, so what are you going to do now?

Well, theoretically if you catch an OOME several methods back into
the stack, then a lot of objects created by the method calls inbetween
should be egligible for garbage collection, so there should be some
memory available. Of course you don't know how much...

 

[Animanera]

Well, theoretically if you catch an OOME several methods back into
the stack, then a lot of objects created by the method calls inbetween
should be egligible for garbage collection, so there should be some
memory available. Of course you don't know how much...

Sorry, probabily I do not explain my self very well (Please excuse me
for my english) :-\

My environment is a SERVLET, therefore I've to deal with concurrent user
access. The 1st code didn't work since it loaded (we say) 7 image in
memory (cache) also if the operations were SEQUENTIAL. Do not charging
the cache resolved the problem.

BUT if 7 users CONCURRENTLY access to the servlet and execute the same
code, the 7 images are *contemporarily* in memory, saturating it.

My (ugly) idea on catching OOME was "if a user try to execute the
resizing and this operation throw an OOME, it means that too user are
executing this code, therefore for THIS user I sleep for a while loop
since the operation doesn't throw an OOME".

Simpler (and, maybe, better) solution should be declaring the resize
method as SYNCHRONIZED therefore only ONE user at a time can execute the
method (is it right, also if the method is static?). = no possibilities
to saturate the memory;

What do you think about it?

Thank you, Bye, Animanera.

 

[Michael Borgwardt]

BUT if 7 users CONCURRENTLY access to the servlet and execute the same
code, the 7 images are *contemporarily* in memory, saturating it.

My (ugly) idea on catching OOME was "if a user try to execute the
resizing and this operation throw an OOME, it means that too user are
executing this code, therefore for THIS user I sleep for a while loop
since the operation doesn't throw an OOME".

Very ugly.

Simpler (and, maybe, better) solution should be declaring the resize
method as SYNCHRONIZED therefore only ONE user at a time can execute the
method (is it right, also if the method is static?). = no possibilities
to saturate the memory;

Should work, but limits your throughput more than necessary.

A clean alternative would be to keep count on how many calls are inside
the method at any one time and permit only a limited number (e.g. 5).

java.util.concurrent.Semaphore does exactly that, but is available only
in Java 1.5, or you can use the package it was based on:
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html

 

[Animanera]

Should work, but limits your throughput more than necessary.

A clean alternative would be to keep count on how many calls are inside
the method at any one time and permit only a limited number (e.g. 5).

java.util.concurrent.Semaphore does exactly that, but is available only
in Java 1.5, or you can use the package it was based on:
http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html

Yes I think that I'll use this package (there is Java 1.4 on server of
my ISP).

Thank you very much.
Bye, Animanera