Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PN

[dy/dx]

Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-any-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
SoftReference b = new SoftReference(ImageIO.read(f));
SoftReference c = new SoftReference(ImageIO.read(f));
SoftReference d = new SoftReference(ImageIO.read(f));
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

It should be easy for any of you with a digital camera to adapt this --
just change the filename string to point to a 24-megapixel image you have
laying around. Failing that, there's one linked at the bottom left of
http://aerialphotographysandiego.com/samples-aerial-photography-san-diego.html

The above will work fine with jpegs and noninterlaced pngs, reporting
falsefalsefalsefalsefalse if you have more than a few hundred megs of mem
and the -server VM. Convert the image to an interlaced png and point the
above at the png, though, and it seems to behave as if System.exit was
called, at least on my system, which is clearly incorrect behavior. (I
tested it with the file from that link, converted to interlaced png with
Photoshop CS2, in case that somehow makes a difference -- with a decoder
bug, who knows? With the png created as described, it crashes with five
copies loaded, but not with four.)

Curiously, this change seems to prevent it:

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-a-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference b = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference c = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference d = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

That's clearly buggy, because System.gc() added or removed is not supposed
to alter program semantics, only maybe performance; PLUS if it was running
out of memory some SoftReferences should have been cleared to make more
room without anything else in the way of consequences; PLUS if it somehow
ran out of memory anyway it should have thrown an OOME rather than
pretended the code called System.exit.

As near as I can tell from this, the ImageIO png decoder in Java 1.6.0_13
contains a crash-inducing bug that requires the png it's decoding to be
interlaced *and* requires heap space to be running low to trigger it.

I'm curious to know what other Java versions reproduce this buggy behavior.

If it's present in 1.6.0_13 but absent in a later version, then obviously
I'd especially like to know that.

But I don't feel like going to a huge effort downloading a hundred megs of
later-Java-version, installing it, rebooting, fixing everything I'd need to
fix to make stuff use the later version, fixing broken links because the
binary pathname changed, and so forth, only to find out that the bug's
still there in the current version. So I'd like confirmation that it's
gone in some later version before I spend an hour or two of my life on such
a task.

 

[dy/dx]

Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-any-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
SoftReference b = new SoftReference(ImageIO.read(f));
SoftReference c = new SoftReference(ImageIO.read(f));
SoftReference d = new SoftReference(ImageIO.read(f));
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

It should be easy for any of you with a digital camera to adapt this --
just change the filename string to point to a 24-megapixel image you have
laying around. Failing that, there's one linked at the bottom left of
http://aerialphotographysandiego.com/samples-aerial-photography-san-diego.html

The above will work fine with jpegs and noninterlaced pngs, reporting
falsefalsefalsefalsefalse if you have more than a few hundred megs of mem
and the -server VM. Convert the image to an interlaced png and point the
above at the png, though, and it seems to behave as if System.exit was
called, at least on my system, which is clearly incorrect behavior. (I
tested it with the file from that link, converted to interlaced png with
Photoshop CS2, in case that somehow makes a difference -- with a decoder
bug, who knows? With the png created as described, it crashes with five
copies loaded, but not with four.)

Curiously, this change seems to prevent it:

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-a-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference b = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference c = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference d = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

That's clearly buggy, because System.gc() added or removed is not supposed
to alter program semantics, only maybe performance; PLUS if it was running
out of memory some SoftReferences should have been cleared to make more
room without anything else in the way of consequences; PLUS if it somehow
ran out of memory anyway it should have thrown an OOME rather than
pretended the code called System.exit.

As near as I can tell from this, the ImageIO png decoder in Java 1.6.0_13
contains a crash-inducing bug that requires the png it's decoding to be
interlaced *and* requires heap space to be running low to trigger it.

Addendum: if the png is *either* interlaced *or* 32bpp (alpha channel) that
seems to suffice. Encoding a problem png in Photoshop as a 24bpp
non-interlaced png seems to make it "clean", i.e. non-bug-triggering for
Java use. In Photoshop CS2 that involves "flatten image" and then saving to
another directory and choosing a "none" radio button on a save options
popup. YMMV with other Photoshop versions -- you're probably all using CS4
or later.

Similarly, taking a non-troublesome png (or non-png) and reencoding it as a
png that's interlaced or 32bpp seems to make it crash ImageIO's decoder
*if* the heap space is low enough at the time of decoding. In particular it
makes the above code exhibit the crash. The size of the png matters, at
least insofar as how quickly the above code gets the heap space low enough
to enable the bug to strike. I pngcrushed a problem png and the number of
loads I could have without a crash went up from 3 to 5; pngcrush reported a
27% reduction in size. 5*0.73 = 3.65 so the bug enabling threshold was
somewhere between 3*original size and 3.65*original size with that png.
Moreover this was the *same image*; the BufferedImage object would have
been about 72 megs and identical down to the last byte for both cases.

So it's not the BufferedImage alone, it's also whatever temporary objects
the decoder makes that affect the bug on subsequent decodes, through their
lingering memory use as uncollected-as-yet garbage or some other mechanism,
and this effect is proportional to the problem png's file size, not its
uncompressed size, pointing to data structures created early in the
decoding -- likely, the byte arrays holding successive chunks of the file
itself.

Changing the decoder to recycle one array instead of constantly making and
discarding them might "fix" the bug, then, though it would really only be
working around it. I'd have to guess that ImageIO's png decoder contains
native code, and that native code does something to allocate memory on the
Java heap for something, likely the output's WritableRaster, in a way that
bypasses some safeguards. In particular, perhaps it doesn't check for heap
exhaustion, run a stop-the-world collection, try again, and then throw OOME
on failure like a normal allocation in non-native code, and some idiot put
if (buff == NULL) { /* Can't happen */ exit(0); } or something similar. In
any event, the bug should be found and fixed, if it hasn't been already,
and not simply papered over by finding a way to avoid as easily triggering
it. It would just end up happening with even
larger-but-should-still-fit-in-the-heap-space pngs, or even with smaller
pngs with big enough other data structures lying about.

 

[dy/dx]

I pngcrushed a problem png and the number of loads I could have without a
crash went up from 3 to 5; pngcrush reported a 27% reduction in size.
5*0.73 = 3.65 so the bug enabling threshold was somewhere between 3*original
size and 3.65*original size with that png.

Bah. Late night. 4*0.73 = 2.92 (no crash) while it crashed at 3, so the
threshold was between 2.92*original size and 3*original size -- a pretty
narrow range. That file was about 10MB on disk before crushing and about
7.3MB afterward, so between 29.2 and 30 megs of interlaced-or-32-bpp png
lies the triggering threshold, at least in my system's case. Again, that
might vary even on systems that have the bug: try replacing the individual
SoftReference variable initializers and println with something like List a
= new ArrayList(); for (int i = 0; i < 50; i++) { System.out.println("" +
i); a.add(ImageIO.read(f)); } and run it. If you get OOME, the bug isn't
happening for you; if Java just exits, it is.

 

[markspace]

Bah. Late night. 4*0.73 = 2.92 (no crash) while it crashed at 3, so the
threshold was between 2.92*original size and 3*original size -- a pretty
narrow range. That file was about 10MB on disk before crushing and about
7.3MB afterward, so between 29.2 and 30 megs of interlaced-or-32-bpp png
lies the triggering threshold, at least in my system's case. Again, that
might vary even on systems that have the bug: try replacing the individual
SoftReference variable initializers and println with something like List a
= new ArrayList(); for (int i = 0; i < 50; i++) { System.out.println("" +
i); a.add(ImageIO.read(f)); } and run it. If you get OOME, the bug isn't
happening for you; if Java just exits, it is.

Could you load the offending files on a photo sharing service? I'd like
to check them out.

 

[Fredrik Jonson]

Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?

[...] I don't feel like going to a huge effort downloading a hundred megs of
later-Java-version, installing it, rebooting, fixing everything I'd need to
fix to make stuff use the later version, fixing broken links because the
binary pathname changed, and so forth, only to find out that the bug's
still there in the current version.

Please confirm that your internet connection is a 110 baud telex line on
Antarctica, and that you need to copy every byte by hand from the telprinter
paper to your terminal prompt to write them to disk. I'd be happy to assist if
I knew downloading 70 - not a couple of hundred - megabytes was a substantial
chore for you.

Also, tell me you're not running a 6u13 based service that exposed to the
internet? There are remotely triggerable DOS issues that has been resolved
since u13. The latest patch release is update 37, that's a whopping 24
security and stability improving patch releases ahead of your environment.

Besides, if your source code, build system, and service configuration is so
fragile it requires several hours of work just to upgrade the JDK, I recommend
that you take some time to fix that. Changing your $PATH and $JAVA_HOME
variables shouldn't be that much work. And while you're at it, consider
upgrading to JDK 7, as JDK 6 (non-for-pay) has a scheduled EOL in November 2012.

http://www.oracle.com/technetwork/java/eol-135779.html
https://blogs.oracle.com/henrik/entry/updated_java_6_eol_date

 

[Fredrik Jonson]

And while you're at it, consider upgrading to JDK 7, as JDK 6
(non-for-pay) has a scheduled EOL in November 2012.

http://www.oracle.com/technetwork/java/eol-135779.html
https://blogs.oracle.com/henrik/entry/updated_java_6_eol_date

I apologise. You've got another three month of time to upgrade to
JDK 7. Public releases of JDK 6 wont go away until February 2013.

 

[dy/dx]

Could you load the offending files on a photo sharing service? I'd like
to check them out.

I already provided an exact recipe for creating a problem png: download the
24-megapixel image linked from

http://aerialphotographysandiego.com/samples-aerial-photography-san-diego.html

and use Photoshop to create a copy that is an interlaced png. (As long as
you don't /distribute/ the copy, it shouldn't be copyright infringement, as
private format-shifting of copyrighted content has been found legal. But
I'm not about to risk getting sued by uploading the results to imageshack
or wherever, and the other problem pngs I have are part of some
confidential work, so...)

 

[dy/dx]

Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?

[...] I don't feel like going to a huge effort downloading a hundred megs of
later-Java-version, installing it, rebooting, fixing everything I'd need to
fix to make stuff use the later version, fixing broken links because the
binary pathname changed, and so forth, only to find out that the bug's
still there in the current version.

Please confirm that your internet connection is a 110 baud telex line on
Antarctica, and that you need to copy every byte by hand from the telprinter
paper to your terminal prompt to write them to disk. I'd be happy to assist if
I knew downloading 70 - not a couple of hundred - megabytes was a substantial
chore for you.

Also, tell me you're not running a 6u13 based service that exposed to the
internet?

Nope. Private development machine. And we are making desktop apps --
shocker, I know.

Besides, if your source code, build system, and service configuration is so
fragile it requires several hours of work just to upgrade the JDK, I recommend
that you take some time to fix that. Changing your $PATH and $JAVA_HOME
variables shouldn't be that much work. And while you're at it, consider
upgrading to JDK 7, as JDK 6 (non-for-pay) has a scheduled EOL in November 2012.

http://www.oracle.com/technetwork/java/eol-135779.html
https://blogs.oracle.com/henrik/entry/updated_java_6_eol_date

Nothing is ever just straightforward plug-and-play, whatever is advertised.
Simply downloading and running an installer for JDK 7 will not be
sufficient. Either stuff will just chug along merrily using 1.6.0_13 or
stuff will break. It happened before when our shop finally updated to Java
6 from Java 1.3, a few years ago. Without a compelling reason it just
doesn't seem worth the hassle.

 

[Joerg Meier]

I already provided an exact recipe for creating a problem png: download the
24-megapixel image linked from

http://aerialphotographysandiego.com/samples-aerial-photography-san-diego.html

and use Photoshop to create a copy that is an interlaced png.

You think people will buy (or pirate) a $700 product because you're too
lazy to find an example image for the problem you want people to spend
their time on for you ?

Good luck with that ;-)

Liebe Gruesse,
Joerg

 

[dy/dx]

You think people will buy (or pirate) a $700 product because you're too
lazy to find an example image for the problem you want people to spend
their time on for you ?

Good luck with that ;-)

Liebe Gruesse,
Joerg

Who said anything about buying or pirating anything? I gave a recipe I knew
was guaranteed to make a problem png. I doubt very much it's the only one.
Surely you have access to image conversion tools that can make an
interlaced png from a jpg.

 

[Daniel Pitts]

Who said anything about buying or pirating anything? I gave a recipe I knew
was guaranteed to make a problem png. I doubt very much it's the only one.
Surely you have access to image conversion tools that can make an
interlaced png from a jpg.

Surely you want to help us help you as much as possible. Provide for us
a problem image.

 

[dy/dx]

Surely you want to help us help you as much as possible. Provide for us
a problem image.

How? Even supposing I was willing to either expose confidential data or
infringe San Diego Photos and Prints's copyright, where am I supposed to
host a 50MB png file on short notice? You might not have noticed, but
Imageshack and similar such sites emphatically do NOT support hosting
images that big. I have not been able to find a problem png on the web. And
is "download this 13MB jpg and then convert it to interlaced png" really
much more onerous than just "download this 50MB png"?

 

[Knute Johnson]

How? Even supposing I was willing to either expose confidential data or
infringe San Diego Photos and Prints's copyright, where am I supposed to
host a 50MB png file on short notice? You might not have noticed, but
Imageshack and similar such sites emphatically do NOT support hosting
images that big. I have not been able to find a problem png on the web. And
is "download this 13MB jpg and then convert it to interlaced png" really
much more onerous than just "download this 50MB png"?

Clearly nobody is going to go to the effort to attempt to diagnose your
problem without a known bad image file. Surely you can obfuscate one of
your faulty images and make it available somewhere? The problem is
intriguing, that's why I haven't Kd this thread. But it is not
intriguing enough to get me to buy Adobe or to attempt to create a
defective file without some actual knowledge that it is in fact
defective. Your choice.

 

[dy/dx]

Clearly nobody is going to go to the effort to attempt to diagnose your
problem without a known bad image file. Surely you can obfuscate one of
your faulty images and make it available somewhere?

Once more, with feeling:

Make it available /where/? Again, the consumer image hosting sites don't
take images that big. They'll resize it to something much smaller if
they'll even let me upload it to begin with.

 

[Knute Johnson]

Once more, with feeling:

Make it available /where/? Again, the consumer image hosting sites don't
take images that big. They'll resize it to something much smaller if
they'll even let me upload it to begin with.

Send me an email and I'll give you a place to send it. I can make it
available to anyone else that wants to work on it.

 

[Andreas Leitgeb]

How? Even supposing I was willing to either expose confidential data or
infringe San Diego Photos and Prints's copyright, where am I supposed to
host a 50MB png file on short notice?

Fwiw., have you looked at dropbox.com ?

Here's an example of how download of a dropbox'ed image works for
others: https://www.dropbox.com/s/ig5yt2v1thsgncd/IMG_0241.JPG

According to the "help center":
< Files uploaded through the website [...] have a 300 MB cap.

that should be enough for the case at hand.

Now, that the webspace-issue should be solved, you can tile
up any d-cam photo (if you like, also the one I linked as example)
to the necessary size, so no need to infringe anyone's copyrights.

PS: Can't promise any help on the actual bug, though.

 

[Robert Klemme]

Make it available /where/? Again, the consumer image hosting sites don't
take images that big. They'll resize it to something much smaller if

If you have a GMail account you should also have Google Drive.

Cheers

robert