Is Java 1.6 number crunching slower than 1.5?

[Arne Vajhøj]

No.

But then the results from SUN Java is not very relevant for Apple Java.

Arne

 

[Lew]

But then the results from SUN Java is not very relevant for Apple Java.

I never said they were.

 

[Lew]

But then the results from SUN Java is not very relevant for Apple Java.

OK, let me try it this way. I provided the results of a Sun JVM because
people in this thread were discussing submitting a bug report to Sun, which is
as you yourself said even less relevant to Apple Java, and to provide a point
of comparison. It is relevant to see if the situation is common to more than
one implementation of Java 6, or if it is limited to the Apple version. As in
any other troubleshooting, you try to eliminate variables that do not affect
the outcome. Evidence that other implementations don't suffer the performance
regression helps to eliminate variables.

 

[Arne Vajhøj]

OK, let me try it this way. I provided the results of a Sun JVM because
people in this thread were discussing submitting a bug report to Sun,
which is as you yourself said even less relevant to Apple Java, and to
provide a point of comparison. It is relevant to see if the situation
is common to more than one implementation of Java 6, or if it is limited
to the Apple version. As in any other troubleshooting, you try to
eliminate variables that do not affect the outcome. Evidence that other
implementations don't suffer the performance regression helps to
eliminate variables.

OK. It is starting to sink in now.



Arne

 

[Kevin McMurtrie]

Lew wrote: [numbers cut]

As you can see, not nearly the extent of difference - the timing ranges
overlap.

You are assuming that the Apple is reusing the SUN JIT compiler
unchanged ?

(MacOS X Java is from Apple, this Linux Java is from SUN)

Arne

Sun numbers are good to see, as that's what the final environment is.
I've heard that 1.6 is supposed to be much faster than 1.5 but it's
looking like each class of server would need to be tested individually
because they use different features of Java. That, and it's always a
challenge to find a release where there's not one server class hitting a
CMS GC bug.

(It's quite possible that Apple is using Sun's x86 compiler.)

 

[Tom Anderson]

There are two good reasons for why that will not accomplish anything:
* the Java on MacOS X is Apple's responsibility not SUN's (the fact
that Apple is buying Java technology from SUN as the basis for
their Java does not mean that SUN has a responsibility for
Apple's end users)

True - it would be better to report this to Apple.

I would be quite surprised if the compiler in Apple's JVM was different to
Sun's; i don't see why Apple would invest engineering effort in improving
it, given that it took a huge effort to make it as good as it is, and so
would take a lot of expertise and work to make it better, and Apple has no
pressing business need for a java which is faster than Sun's. But still,
this is something that should be reported to Apple if it can't be
reproduced on non-Apple hardware.

* neither SUN nor Apple has claimed that there will not exist code
where the newer version perform worse than the old version (I don't
think any compiler vendor has done that - it happens frequently for
C compilers)

True, but entirely unimportant. Performance *is* a goal for Sun, and so
they are likely to find performance problems interesting. It might be that
the OP's code is so unusual that it isn't worth them bothering with, but
it might very well not be. Filing a bug report about it is easy to do, so
why not?

tom

 

[Lew]

True, but entirely unimportant. Performance *is* a goal for Sun, and so
they are likely to find performance problems interesting. It might be
that the OP's code is so unusual that it isn't worth them bothering
with, but it might very well not be. Filing a bug report about it is
easy to do, so why not?

But is there an actual performance problem? If there's no bug, there is a
very good reason not to file a bug report. You may find it easy to file one,
but then someone has to analyze it and confirm it and determine if there is a
fix if a bug is confirmed. Why put someone else through all that effort just
because one is too lazy to check for themselves if there really is a problem?

My results with the OP's benchmark as published upthread:

Java 5
mean 11722.1096
stdev 472.9497

Java 6 w/ Java 5 classes
mean 11986.0732
stdev 487.6812

Java 6 w/ Java 6 classes
mean 12278.0133
stdev 1144.2802

 

[Tom Anderson]

But is there an actual performance problem? If there's no bug, there is a
very good reason not to file a bug report. You may find it easy to file one,
but then someone has to analyze it and confirm it and determine if there is a
fix if a bug is confirmed. Why put someone else through all that effort just
because one is too lazy to check for themselves if there really is a problem?

My results with the OP's benchmark as published upthread:

Java 5
mean 11722.1096
stdev 472.9497

Java 6 w/ Java 5 classes
mean 11986.0732
stdev 487.6812

Java 6 w/ Java 6 classes
mean 12278.0133
stdev 1144.2802

So do you have a Mac now?

tom

 

[Lew]

So do you have a Mac now?

Huh?

 

[Tom Anderson]

Huh?

The problem was reported on a Mac. From your data, it looks like it's not
a problem on whatever non-lickable platform you use, but that doesn't mean
it's not a problem on a Mac.

As we discussed (ie as i was told!), that means it should be reported to
Apple rather than Sun, of course.

I don't immediately see why it would be a problem on a Mac but not an
architecturally very similar PC (which i'm making a wild assumption is
what you're using), but numbers don't lie (except for the 14% that are
made up on the spot). Some kind of OS effect, i would guess.

tom

 

[John B. Matthews]

For reference, I tried the benchmark on Ubuntu 9.0.4 running in a 32-bit
VirtualBox <http://www.virtualbox.org/>. I used the latest repository
versions; IIUC, more recent version require manual installation:

$ make clean run
rm -f *.class
javac Benchmark.java
java -version
java version "1.5.0_18"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_18-b02)
Java HotSpot(TM) Client VM (build 1.5.0_18-b02, mixed mode, sharing)
java Benchmark
Millis: 14581.717
Millis: 12933.799
Millis: 11833.187
Millis: 12715.087
Millis: 11672.402

$ make clean run
rm -f *.class
javac Benchmark.java
java -version
java version "1.6.0_14"
Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) Client VM (build 14.0-b16, mixed mode, sharing)
java Benchmark
Millis: 8084.638996
Millis: 8357.479729
Millis: 8014.00301
Millis: 9221.138071
Millis: 9712.828442

The problem was reported on a Mac. From your data, it looks like it's not
a problem on whatever non-lickable platform you use, but that doesn't mean
it's not a problem on a Mac.

As we discussed (ie as i was told!), that means it should be reported to
Apple rather than Sun, of course.

I don't immediately see why it would be a problem on a Mac but not an
architecturally very similar PC (which i'm making a wild assumption is
what you're using), but numbers don't lie (except for the 14% that are
made up on the spot). Some kind of OS effect, i would guess.

As the versions cited by the OP are not generally available on Mac OS, I
suspect that this may have been pre-release software.

 

[Lew]

The problem was reported on a Mac. From your data, it looks like it's
not a problem on whatever non-lickable platform you use, but that
doesn't mean it's not a problem on a Mac.

The OP's post was about a Mac. Your post was about Sun, and it was that to
which I responded most directly. I also explained upthread how comparing to
other platforms is useful to isolate whether the problem is Mac-specific.

Regardless, my post of statistical summaries was directly about your
suggestion that we post a bug report to Sun because it wasn't difficult to do
so.

Performance *is* a goal for Sun,
and so they are likely to find performance problems interesting.
...
Filing a bug report about it is easy to do, so why not?

My post was also intended as a model for how to investigate prior to reporting
a bug to Apple, and a contravention of the extension of the "easy so why not?"
philosophy to organizations other than Sun.

Sun, Apple or whomever, it doesn't make sense to report a bug unless you have
some pretty good evidence that there is one to report. Running a
micro-benchmark through five loops on one machine with an unknown load, even
if it is a might Mac, might not really be enough evidence to report a bug no
matter how easy it might be for the *reporter* of the bug.

Can you see how my example might generalize, or are we going to quibble more
about its relevance?

 

[Tom Anderson]

The OP's post was about a Mac. Your post was about Sun, and it was that to
which I responded most directly. I also explained upthread how comparing to
other platforms is useful to isolate whether the problem is Mac-specific.

Regardless, my post of statistical summaries was directly about your
suggestion that we post a bug report to Sun because it wasn't difficult to do
so.

My suggestion of reporting a bug to Sun was mistaken - i was thoroughly
corrected on that point. I certainly agree with you about not reporting
this to Sun.

My post was also intended as a model for how to investigate prior to
reporting a bug to Apple, and a contravention of the extension of the "easy
so why not?" philosophy to organizations other than Sun.

Sun, Apple or whomever, it doesn't make sense to report a bug unless you have
some pretty good evidence that there is one to report. Running a
micro-benchmark through five loops on one machine with an unknown load, even
if it is a might Mac, might not really be enough evidence to report a bug no
matter how easy it might be for the *reporter* of the bug.

Can you see how my example might generalize, or are we going to quibble more
about its relevance?

I certainly endorse making more thorough measurements before reporting -
you make a good point there.

tom

 

[Kevin McMurtrie]

It was originally a question about whether it was a "Java" issue or a
"Mac" issue. It makes a difference if I develop on Mac but deploy on
various Solaris servers. Tests run on non-Mac systems have been
helpful. (Thanks!)

Yes, micro-benchmarks are terrible. I do set up sessions to throw real
customer traffic at servers but it's an ordeal and it could be a bad
customer experience for a few people when things go wrong. Java 1.6 is
starting to look attractive because CMS GC in Java 1.5 has never been
completely robust. It becomes erratic under fluctuating loads and
sometimes it throws out a live object or two.

 

[Arne Vajhøj]

True - it would be better to report this to Apple.

I would be quite surprised if the compiler in Apple's JVM was different
to Sun's; i don't see why Apple would invest engineering effort in
improving it, given that it took a huge effort to make it as good as it
is, and so would take a lot of expertise and work to make it better, and
Apple has no pressing business need for a java which is faster than
Sun's. But still, this is something that should be reported to Apple if
it can't be reproduced on non-Apple hardware.


True, but entirely unimportant. Performance *is* a goal for Sun, and so
they are likely to find performance problems interesting. It might be
that the OP's code is so unusual that it isn't worth them bothering
with, but it might very well not be. Filing a bug report about it is
easy to do, so why not?

Because it is not a bug !?!?

I consider it rather important for filing a bug whether it is
a bug or not a bug.

Arne

 

[Arne Vajhøj]

Lew wrote: [numbers cut]

As you can see, not nearly the extent of difference - the timing ranges
overlap.

You are assuming that the Apple is reusing the SUN JIT compiler
unchanged ?

(MacOS X Java is from Apple, this Linux Java is from SUN)

Sun numbers are good to see, as that's what the final environment is.
I've heard that 1.6 is supposed to be much faster than 1.5 but it's
looking like each class of server would need to be tested individually
because they use different features of Java. That, and it's always a
challenge to find a release where there's not one server class hitting a
CMS GC bug.

(It's quite possible that Apple is using Sun's x86 compiler.)

If the final destination is Linux, then you should test
performance on Linux.

1.6 is supposed to a bit faster than 1.5. If you created
1000 benchmarks with different types of code I would
expect 1.6 to be maybe 10% faster than 1.5 at average, but I would
expect the individual tests to be anywhere from let us
say -25% to +50%.

Arne