performance question

  • Thread starter Olivier Scalbert
  • Start date
L

Lew

Jon said:
Yeah, like the rubbish Steve Wampler just posted. There are lots of numbers
out there but nothing objective that I can reproduce.

One minute of search produced
<http://www.idiom.com/~zilla/Computer/javaCbenchmark.html>

These are results from ca. 2001-2004, and Java was already reasonably close on
a number of the benchmarks, 9x slower on some, faster on others.

A few more minutes turned up
<http://www.ddj.com/cpp/184401976>
from 2005, which includes source code and operating conditions for the
benchmarks. Pretty freaking reproducible, eh?

There's more.
 
S

Steve Wampler

Jon said:
There is absolutely no evidence whatsoever that Hotspot has improved
anything in these results. You first run warms the cache (just as it does
in C) and you second run in Java jumped virtually to the fastest of all of
your results (that was even identical to your last run).

Good guess, but wrong. Here's a test with Hotspot turned off (I went back
to 1,000,000 output lines because I'm impatient):

--------------------------------------------------------------
->java -Xint -server T1 1000000 5 | cat >/dev/null
Bench 0
Took 2.497 seconds
Bench 1
Took 2.453 seconds
Bench 2
Took 2.432 seconds
Bench 3
Took 2.432 seconds
Bench 4
Took 2.427 seconds
->
--------------------------------------------------------------

Notice that:
(a) the times are a *lot* slower, and
(b) there is no significant performance change.

The HotSpot optimizer comes into play quite quickly, which is
why even the *first* benchmark run is faster with it turned on -
it has *already* sped things up (the first run is slower because
the first few 'un-optimized' loop iterations are dragging its average
down.

Somehow, though, I suspect I'm wasting my time... Maybe people
other than Jon will find this illuminating.
 
S

Steve Wampler

Jon said:
The claim can equally well be taken as "Java is prohibitively difficult to
optimize". Real programs often won't afford 10x code bloat to get decent
performance.

In both the C and Java examples, the performance bottleneck is in the
code that does the output. Printf is a library function
that does all sorts of fascinating things to get good performance.
If you moved that code into the main routine you'd see a remarkable
amount of code bloat. In the Java case, you could just as well take
the code bloat you complain about and put into a separate package,
just as printf is found in a library. Apparently there hasn't been
much demand that this be done. I suspect one could get it so the
Java main could make a single method call on this package, just
as the printf is a single function call - should one find it
worthwhile to do so.
 
J

Jon Harrop

Lew said:
One minute of search produced
<http://www.idiom.com/~zilla/Computer/javaCbenchmark.html>

These are results from ca. 2001-2004, and Java was already reasonably
close on a number of the benchmarks, 9x slower on some, faster on others.

I would love to try to verify their results but the test codes are not
available AFAICT.
A few more minutes turned up
<http://www.ddj.com/cpp/184401976>
from 2005, which includes source code and operating conditions for the
benchmarks. Pretty freaking reproducible, eh?

C++ code runs twice as fast as the Java on my machine. However, the
experiment has design flaws which make the results useless.

The essence of a good benchmark is that the task it performs must not be
trivially reducible. This is exactly why my ray tracer is a much better
benchmark than most of the tasks on the shootout. In contrast, this
benchmark simply prints a few constants that could be hardcoded.

Here is an interested quote from one of their citations:

"I found some tests that claim Java is faster than C++... found the test
& results to be pretty bogus."
http://www.w3sys.com/pages.meta/benchmarks.html

He concludes that Java was much slower but the benchmark was rigged to show
Java to be fast by deliberately measuring the wrong thing.

Every time I find a benchmark that shows Java favorably, further
investigation shows it to be a fraud.
 
J

Jon Harrop

Lew said:
It was in response to a benchmark that purported to show a much larger
difference also writing to the screen. Maybe it was wrong, but it was
necessary to keep the conversation on an even playing field.

Piped, as you suggest, Java was about 10x slower than C on that benchmark.

Yes. With enough effort you should be able to get Java within 2x the
performance of C on that benchmark. The problem is that you have to put in
the effort yourself rather than simply letting the compiler get on with it.
 
R

Roedy Green

System.out.println("abcdefghijk " + i);

the problem is the console. For some reason Sun's Java console native
method is unusually slow. Try that benchmark with Jet which has a
better console interface.

You can see what I mean by changing the benchmark to write to a
buffered file.

Sun seemed to presume the console would never have more than a few
pages of output.

Consider how that console works -- most of it has nothing to do with
Java. It simulates an REGEN buffer on the old 80286. Periodically
the OS scans the character buffer looking for changes and paints the
equivalent dots on the screen buffer.
 
S

Steve Wampler

Jon said:
I was stating a fact.


There was so significant performance change beyond cache warming last time
either, which was my point.

So, you're claiming that if someone instrumented the C version to match the
Java one (i.e. allow multiple runs of the 'benchmark' from within the same
process, with times for each internal run) you'd see the same performance profile
for the 1000000xN line case as with the Java?

Out of curiosity, what is your explanation for the fact that the performance is
almost an order of magnitude slower when HotSpot is turned off?
 
L

Lew

Jon said:
C++ code runs twice as fast as the Java on my machine. However, the
experiment has design flaws which make the results useless.

The essence of a good benchmark is that the task it performs must not be
trivially reducible. This is exactly why my ray tracer is a much better
benchmark than most of the tasks on the shootout. In contrast, this
benchmark simply prints a few constants that could be hardcoded.

Here is an interested quote from one of their citations:

"I found some tests that claim Java is faster than C++... found the test
& results to be pretty bogus."
http://www.w3sys.com/pages.meta/benchmarks.html

He concludes that Java was much slower but the benchmark was rigged to show
Java to be fast by deliberately measuring the wrong thing.

Every time I find a benchmark that shows Java favorably, further
investigation shows it to be a fraud.


Blah, blah, blah. A benchmark shows that Java is twice as slow as C++, when
before you said it was a hundred times as slow, and all of a sudden you deny
the validity of the benchmark. Then you bring up one which you know will beat
Java, and yet ignore all the use cases that favor Java.

I am not saying "Java is faster than C++" as a blanket statement. I'm saying
Java is a lot closer to C++ in performance than naysayers such as yourself
would have us believe. I say there's a lot of evidence out there, you say it
isn't reproducible. I find reproducible results that show Java is comparable
to C++, you say, "Oh, but those benchmarks aren't valid." Well, the fact that
they are reproducible means that people can decide for themselves.

The fact remains that Java is not the slowpoke you claim. The fact is that it
is, for many tasks Java is comparable, and for most, in the same ballpark as
C++. The fact is that you have not countered with any reasonable evidence for
*your* claims, while the evidence just mounts and mounts and mounts for the
"Java is near enough to C++" side. I run the benchmarks, too, you know. I
read the articles with the same eye to reproducibility and comparability you
claim to use. I'm convinced by the hard evidence, enough to discount your
flamebait rants.

At this point I am aware that Jon is only going to continue his polemical
approach not grounded in tests and facts, for what agenda I can only
speculate. My intent is to counteract the misrepresentations and provide
evidence for the objective-minded to decide for themselves.

FWIW, my conclusion is that Java is generally somewhat slower than C++ for the
types of scenarios micro-benchmarks validate. I have yet to find out if it is
for the claimed "Java wins" scenarios: long-running server processes.
Nevertheless, in every test I've read or performed, Java is only somewhat
slower, not dramatically, and on several benchmarks * it actually outperforms
C++ * even in this micro-benchmark scenario.

Those are facts. Now let Jon rant.
 
S

Steve Wampler

Gordon said:
Warning: cat used with no arguments. Costs a process, gains you
nothing.

Do this instead:

java -Xint -server T1 1000000 5 >/dev/null

I understand. However, I was trying to mimic what might be
one use of the java app - as the front end to a pipe. So
I used the pipe to avoid people complaining that sending
the output directly to /dev/null was different than
sending it to a file [which is true, but not relevant].
 
L

Lew

Jon said:
Yes. With enough effort you should be able to get Java within 2x the
performance of C on that benchmark. The problem is that you have to put in
the effort yourself rather than simply letting the compiler get on with it.

The problem isn't with Java, but with creating a level playing field, which in
other posts you seem to favor, but I guess not when there's a risk it might
show Java in a favorable light.
 
S

Steve Wampler

Lew said:
I am not saying "Java is faster than C++" as a blanket statement. I'm
saying Java is a lot closer to C++ in performance than naysayers such as
yourself would have us believe. I say there's a lot of evidence out
there, you say it isn't reproducible. I find reproducible results that
show Java is comparable to C++, you say, "Oh, but those benchmarks
aren't valid." Well, the fact that they are reproducible means that
people can decide for themselves.

This friend speaks my mind.
 
G

George Neuner

This hardly refutes Steve Wampler's point, indeed, it validates it. He was
pointing out that one can easily construct a benchmark that exploits the
difference(s) between languages to prove whatever point, either side, one
wishes to make. A good knowledge of the different tradeoffs between languages
lets one construct a benchmark that exploits a strength in one language (Java
in his case) that is a weakness in another (C), with respect to what the
benchmark measures.

Absolutely. I understood that Steve's program was intentionally
constructed to favor Java over C - I was redundantly pointing out that
it was mirroring the mistake made by the OP.

I like Java not for its speed, which is plenty fast enough in the real world,
but for its features as a language and the richness of the API, and not
incidentally, for the abundance of job opportunities.

Java is OK. I have a number of issues with it including syntax,
organization, and the fact that it religiously enforces OO which IMO
is not an optimal programming model for many problems. While I am
perfectly comfortable working in OO (pure or not), I prefer languages
that let me mix different programming styles more freely.

network awareness (anyone remember "The network is the computer"?)

Actually I do. AFAIC Java is only mediocre in this regard. Its
marshalling support is good, but support for easy computation
migration is lacking - replicating an entire application is not always
desirable and dynamically loading a shitload of classes to relocate a
computation is often impractical.


George
 
A

Andreas Leitgeb

That may be true, but keep in mind also that you can run g++ with
maximum optimization today, and it will get the benefit of todays
optimizer. ...

I'm not sure if I got you right. Your argument seems to be:
If I compile the C-program with g++ and full-optimization today,
then in a few years it will appear like an unoptimized compiled
binary compiled at that time, thus an unoptimized compile now
is like a binary that was compiled few years ago with optimization.

If you tried compiling the C-prog with, say, version 2.9.1 (now current
is 4.x) and -O6, and comparing that binary with both the unoptimized
contemporate g++ compiled one and with java, then there would be some
substance to that comparision, especially, if the gcc-2.9.1 -O6 compiled
binary were indeed slower, but without this extra contestant the
comparision is rather worthless, IMHO.

Now, I also don't have gcc-2.9.1 available anymore, but I did the
experiment, anyway, copied the source-samples from your page, and
compiled them and ran them:
Funnily, for this particular code sample
gcc-3.3: without -O: 210ms
with any -O: 275ms
gcc-3,4: without -O: 210ms
with any -O: 270ms
gcc-4.1: w/o 270ms
with: 280ms
java: 275ms
(timings are averages over these 10 values each run.)

So, it seems as if for your example, gcc's performance
is dropping over time, and it's optimization contra-productive.
So, preserving the binary with old gcc-3.3 appears like a good
choice. :)

This of course depends on the particular benchmark,
because I've run C-programs where the difference
between no opt, and -O2 / -O6 was impressively
in favour of optimization. (that was back when
gcc-3.3 was brand new)
The real point of the post is that the micro-benchmark can prove Java is
fast enough.
The point surely stands nowadays (It didn't a few java-versions earlier),
but it is in no way proven nor shown by your comparision.

PS: gcc and g++ were actually equivalent, but for gcc I had to
declare the loop-variables at block start, since C didn't
yet allow "for(long i=..."-syntax.
 
S

Steve Wampler

I think he pointing out that the HotSpot approach means that you always
get the 'latest' optimization out of newer JVMs without having to
retranslate the Java into bytecodes again. With compile-time optimizers,
the optimization you get is the one you're stuck with until you recompile.

It's an interesting point - with closed source 3rd party code, for example,
each advance in HotSpot buys you better performance, but advances in
your C/C++ compiler have no effect unless the developer does a recompile.
With open-source code, presumably you could 'keep up' yourself, and it's
of course possible a closed-source developer has access to a better
compiler than you do [*maybe!*].
 
J

Jon Harrop

Lew said:
The problem isn't with Java, but with creating a level playing field,
which in other posts you seem to favor, but I guess not when there's a
risk it might show Java in a favorable light.

I'm more that happy to show Java in a good light. Compared to OCaml, for
example, Java has more users, more libraries and more commercial backing.
However, one thing Java does not have is good performance.
 
J

Jon Harrop

Steve said:
It's an interesting point - with closed source 3rd party code, for
example, each advance in HotSpot buys you better performance, but advances
in your C/C++ compiler have no effect unless the developer does a
recompile. With open-source code, presumably you could 'keep up' yourself,
and it's of course possible a closed-source developer has access to a
better compiler than you do [*maybe!*].

Again, this is pure speculation.
 
J

Jon Harrop

Steve said:
So, you're claiming that if someone instrumented the C version to match
the Java one (i.e. allow multiple runs of the 'benchmark' from within the
same process, with times for each internal run) you'd see the same
performance profile for the 1000000xN line case as with the Java?
Exactly.

Out of curiosity, what is your explanation for the fact that the
performance is almost an order of magnitude slower when HotSpot is turned
off?

Missing optimizations.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,483
Members
44,902
Latest member
Elena68X5

Latest Threads

Top