performance question

[Olivier Scalbert]

Hello,

I would like to show to somebody how different languages look like by
doing a very simple case.
I was very surprised by the poor performance of the java version !

Here are the programs in different languages:

in C:

#include <stdio.h>

int main()
{
int i;

for(i = 0; i < 1000000; i++)
{
printf("abcdefghijk %d\n", i);
}

return 0;
}

time ./test1 > out.txt

real 0m0.710s
user 0m0.576s
sys 0m0.124s

in java:
public class Test
{
public static void main(String[] args)
{
for (int i = 0; i < 1000000; i++)
{
System.out.println("abcdefghijk " + i);
}
}
}

time java Test > out.txt

real 0m12.364s
user 0m4.180s
sys 0m7.676s

time java -server Test > out.txt

real 0m10.537s
user 0m3.120s
sys 0m6.544s


That is not good at all !
ols@tatooine:~/projects/ruby$ java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Client VM (build 1.6.0_02-b05, mixed mode, sharing)

ols@tatooine:~/projects/ruby$ uname -a
Linux tatooine 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007
i686 GNU/Linux

In python:
i=0
while i < 1000000:
print "abcdefghijk", i
i=i+1

time python test.py > out.txt

real 0m2.292s
user 0m2.064s
sys 0m0.112s

In perl:
for ($count = 0; $count < 1000000; $count++)
{
print "abcdefghijk $count\n";
}
time perl test.pl > out.txt

real 0m1.243s
user 0m1.060s
sys 0m0.160s

In ruby:
counter = 0
while counter < 1000000
puts("abcdefghijk #{counter}")
counter+=1
end

time ruby test.rb > out.txt

real 0m4.731s
user 0m4.452s
sys 0m0.100s

As you can see the java program performance is far from the other one. I
was very surprised! Before the tests, I was sure that the winner will be
C followed by java, but it is not the case ...

Of course, I can improve the performance of the java version, by using a
StringBuffer, and print this buffer when it is bigger than a given size,
but it is not fair !

If you have any ideas, there are welcomed !

Thanks,

Olivier

 

[Real Gagnon]

time java -server Test > out.txt

real 0m10.537s
user 0m3.120s
sys 0m6.544s

I bet that the JVM loading and startup time is very important with your
test.

So it's not this particular code which is slow, the timing technique is not
adequate.

Bye.

 

[Eric Sosman]

Hello,

I would like to show to somebody how different languages look like by
doing a very simple case.
I was very surprised by the poor performance of the java version !
[...]

Congratulations! You have discovered that Java is a
poor choice of language if you plan to do something silly,
pointless, and stupid. Please report your findings to Sun,
and file a Request For Enhancement to optimize Java for
better performance on silly, pointless, and stupid programs.
(Don't be too surprised, though, if the Javafolk attach a
relatively low priority to the RFE; they're more interested
in getting Java to work well for useful programs. That's
a clear case of bias against the average programmer, don't
you think?)

"And the wheel," said the Captain, "what about this
wheel thingy? It sounds a terribly interesting project."

"Ah," said the marketing girl, "well, we're having a
little difficulty there."

"Difficulty?" exclaimed Ford. "Difficulty? What do you
mean, difficulty? It's the single simplest machine in
the entire Universe!"

The marketing girl soured him with a look. "All right
Mr. Wiseguy," she said, "you're so clever, you tell us
what color it should be."

 

[Olivier Scalbert]

I bet that the JVM loading and startup time is very important with your
test.

Not sure !

With:
public class Test
{
public static void main(String[] args)
{
for (int i = 0; i < 1; i++)
{
System.out.println("abcdefghijk " + i);
}
}
}

time java Test > out.txt

real 0m0.125s
user 0m0.040s
sys 0m0.032s

So it's not this particular code which is slow, the timing technique is not
adequate.

Do you have another technique ?

Bye.

See you

 

[Stefan Ram]

So it's not this particular code which is slow, the timing
technique is not adequate.

May be it is. For example, when one wants to write an
echo command for the command line, to be used like:

echo "Hello world!"

one does often not use Java for a reason.

On the other hand, people start to consider to write a
JVM in Java, because C is too slow:

http://www.adam-bien.com/roller/abien/entry/when_c_becomes_too_slow

And then:

»if you run across benchmarks of any kind that show JRuby
running slower than Ruby 1.8.x, we'd appreciate you filing
them as bugs«

http://groups.google.com/group/jrub...g/9fd1ab9664a1a36b?dmode=source&output=gplain

 

[Olivier Scalbert]

Hi,

Thanks for your congratulations !

But do you have a real explanation ?

And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

For me, it is only a small program, nothing else.

Hello,

I would like to show to somebody how different languages look like by
doing a very simple case.
I was very surprised by the poor performance of the java version !
[...]

Congratulations! You have discovered that Java is a
poor choice of language if you plan to do something silly,
pointless, and stupid. Please report your findings to Sun,
and file a Request For Enhancement to optimize Java for
better performance on silly, pointless, and stupid programs.
(Don't be too surprised, though, if the Javafolk attach a
relatively low priority to the RFE; they're more interested
in getting Java to work well for useful programs. That's
a clear case of bias against the average programmer, don't
you think?)

"And the wheel," said the Captain, "what about this
wheel thingy? It sounds a terribly interesting project."

"Ah," said the marketing girl, "well, we're having a
little difficulty there."

"Difficulty?" exclaimed Ford. "Difficulty? What do you
mean, difficulty? It's the single simplest machine in
the entire Universe!"

The marketing girl soured him with a look. "All right
Mr. Wiseguy," she said, "you're so clever, you tell us
what color it should be."

 

[Lew]

I bet that the JVM loading and startup time is very important with your
test.

Not sure !

With:
public class Test
{
public static void main(String[] args)
{
for (int i = 0; i < 1; i++)
{
System.out.println("abcdefghijk " + i);

Note: This has the overhead of character encoding. Your C example does not.
So you are comparing very different activities.

}
}
}

time java Test > out.txt

real 0m0.125s
user 0m0.040s
sys 0m0.032s


Do you have another technique ?

Yes - time the loop, not the load time. With Java, you should run the loop
for a while, then run it again with timing instrumentation. This lets the JVM
optimize the loop.

Your benchmark as presented is pretty useless. Apples and oranges.

 

[Olivier Scalbert]

Lew wrote:
;

Note: This has the overhead of character encoding. Your C example does
not. So you are comparing very different activities.

Good point, but I find that the char encoding overhead is quite expensive !
Thanks.

 

[Eric Sosman]

[...]
And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

Let's apply this test: How valuable is the output of
this program? How many backup copies did you make? How
many other programs used this output as their input?

If you made no backups and fed the output to no other
programs, as I suspect, then by your own actions you have
declared the value of the program's output to be zero. A
program that produces nothing of value meets my definition
of silly, pointless, and stupid.

 

[Mark Thornton]

Hi,

Thanks for your congratulations !

But do you have a real explanation ?

And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

For me, it is only a small program, nothing else.

1. Modern Java systems monitor the execution of the program for a little
while before compiling. This allows a better choice of optimisation
strategy, but doesn't produce the fastest results on applications that
take much less than a second to run anyway (but if a program takes less
than a second why do you care how much less).

2. Unicode vs ASCII. Java always works with Unicode and converts the
characters to whatever encoding is in use on the machine. Your C
equivalent assumes ASCII. Try including a £ character in your test; Java
will get this right so long as the character encoding is set correctly
(this can be done at execution time).

3. Output buffering. By default Java output to System.out is line
buffered. That is the output buffer is flushed at the end of every line.
The C 'equivalent' usually uses a much larger buffer by default (512
bytes or more). You can change Java to use a larger buffer and not flush
at EOL or you could change C to use line buffered output. Unless you
change one or the other your comparison is flawed.

4. You could look back a week or two and find that someone else proposed
a similar test, with similar flaws. After seeing similar mistakes on an
almost weekly basis for at least 10 years, it is not surprising that
people get a bit tired of providing detailed explanations of what is wrong.

Mark Thornton

 

[Jon Harrop]

Lew wrote:
;

Good point, but I find that the char encoding overhead is quite expensive
! Thanks.

No, char encoding is not an adequate explanation of a 15x slowdown. This is
Java being very poor at buffering.

 

[Jon Harrop]

I bet that the JVM loading and startup time is very important with your
test.

Not sure !

With:
public class Test
{
public static void main(String[] args)
{
for (int i = 0; i < 1; i++)
{
System.out.println("abcdefghijk " + i);
}
}
}

time java Test > out.txt

real 0m0.125s
user 0m0.040s
sys 0m0.032s

Indeed, startup time is the number one mis-attribution posted by Java
advocates as a way around explaining very poor performance. As you can see,
it was quite wrong to blame startup time on this benchmark.

 

[Jon Harrop]

Let's apply this test: How valuable is the output of
this program?

How common is IO?

 

[Mark Thornton]

Hi,

Thanks for your congratulations !

But do you have a real explanation ?

And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

For me, it is only a small program, nothing else.

Search for the thread "Why is Java so slow???" starting on 19 November,
posted by "java.performance.expert" (who was clearly nothing of the sort).

Mark Thornton

 

[Olivier Scalbert]

[...]
And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

Let's apply this test: How valuable is the output of
this program? How many backup copies did you make? How
many other programs used this output as their input?

If you made no backups and fed the output to no other
programs, as I suspect, then by your own actions you have
declared the value of the program's output to be zero. A
program that produces nothing of value meets my definition
of silly, pointless, and stupid.

Ok, still no technical info, just blabla on what is silly, pointless.
I do not appreciate people, that are "donneurs de leçons" (moralist
people that give lesson).

Using java daily since many many years from embedded platform to
application server, I have never noticed that java can detect silly
programs from very intelligent one !

For your info, by adding few lines of code, this program will generate
huge csv files (several billions of lines). Theses files will be used to
test the import functionality of different databases.

 

[Jon Harrop]

Aside from the emotionally-loaded pejoratives and all the /ad hominem/
flaming, the fact remains that your benchmark did not compare the same
behaviors and is not indicative of the relative speeds of Java vs. the
other platforms.

On the contrary, Olivier's test is simple and flawless. Can you make the
Java nearly as fast as the C? Feel free to not use unicode, or does Java
have poor support for everything else?

 

[Lew]

No, char encoding is not an adequate explanation of a 15x slowdown. This is
Java being very poor at buffering.

A point also made by Mark Thornton, and more evidence that my point is true:
that the benchmark is not comparing the same algorithms and is therefore not
valid.

 

[Jon Harrop]

Disclaimer: I am not a fan of Java.

As you can see the java program performance is far from the other one. I
was very surprised! Before the tests, I was sure that the winner will be
C followed by java, but it is not the case ...

Here's what I learned from my ray tracer language comparison:

http://www.ffconsultancy.com/languages/ray_tracer/

For a compiled language, Java is slow. However, this effect is compounded in
two important ways:

1. Getting accurate information about how to optimize Java code is almost
impossible.

2. Actually implementing the optimizations is typically unnecessarily slow
and tedious in Java.

In this case, you have already seen the two standard responses to a valid
question from the Java community:

1. This is "only" a microbenchmark.

2. You're measuring startup time.

In reality, IO is a very important aspect of many programs (including my ray
tracer benchmark, which was hopelessly IO bound _only_ in Java until some
kind soul taught me how to work around these deficiencies in Java) and you
are a long way from measuring the startup time, which will be a fraction of
a second in this case.

Of course, I can improve the performance of the java version, by using a
StringBuffer, and print this buffer when it is bigger than a given size,
but it is not fair !

Actually the best solution is to use buffered output rather than a string
buffer to accumulate an intermediate result. Don't ask me why the Java
implementors aren't competent enough to buffer automatically but they
aren't.

So, you start by importing a namespace for doing non-sucky IO:

import java.io.*;

Then you create an unnecessarily long-named BufferedOutputStream:

BufferedOutputStream out = new BufferedOutputStream(System.out);

Finally, you write to this instead of System.out:

out.write(("abcdefghijk " + i + "\n").getBytes());

Holy mackerel, your Java code now takes only 1.275s instead of the original
3.135! That's a lot better than it was before but it is still over and
order of magnitude (!) slower than any decent language, e.g. C at 0.196s.

You see, Java is what the functional programming community refer to as
a "low-level language" because it simply fails to convey high-level
information to the compiler for optimization. In this case, the Java
compiler is probably being amazingly retarded in that it actually performs
two string concatenations in unicode for absolutely no reason whatsoever.

So let's try manually unrolling the code:

for (int i = 0; i < 1000000; i++) {
out.write("abcdefghijk ".getBytes());
out.write(s1);
out.write(Integer.toString(i).getBytes());
out.write("\n".getBytes());
out.write(s2);
}

Wow, this is actually slower than it was before at 1.492s. But this is going
in the right direction. As it happens, Java is so crap that it can't even
hoist loop invariants as well as the C compilers from the 70s.

Hoisting manually, we finally arrive at:

import java.io.*;

public class Test
{
public static void main(String[] args) throws java.io.IOException
{
BufferedOutputStream out = new BufferedOutputStream(System.out);

byte[] s1 = "abcdefghijk ".getBytes();
byte[] s2 = "\n".getBytes();

for (int i = 0; i < 1000000; i++) {
out.write(s1);
out.write(Integer.toString(i).getBytes());
out.write(s2);
}
}
}

Now this only takes 0.947s! Incredible, several times longer now and still
several times slower than any decent language!

In summary, what we have learned is that for the love of God (TM) don't use
Java if performance is at all important for your work. This doesn't just
apply to IO either: Java is very slow at a wide variety of vitally-
important tasks, e.g. allocation.

 

[Lew]

[...]
And by the way, can you explain me, why it was a silly, pointless and
stupid program ?

Let's apply this test: How valuable is the output of
this program? How many backup copies did you make? How
many other programs used this output as their input?

If you made no backups and fed the output to no other
programs, as I suspect, then by your own actions you have
declared the value of the program's output to be zero. A
program that produces nothing of value meets my definition
of silly, pointless, and stupid.

Ok, still no technical info, just blabla on what is silly, pointless.
I do not appreciate people, that are "donneurs de leçons" (moralist
people that give lesson).

Using java daily since many many years from embedded platform to
application server, I have never noticed that java can detect silly
programs from very intelligent one !

For your info, by adding few lines of code, this program will generate
huge csv files (several billions of lines). Theses files will be used to
test the import functionality of different databases.

Aside from the emotionally-loaded pejoratives and all the /ad hominem/
flaming, the fact remains that your benchmark did not compare the same
behaviors and is not indicative of the relative speeds of Java vs. the other
platforms.

 

[Olivier Scalbert]

A point also made by Mark Thornton, and more evidence that my point is
true: that the benchmark is not comparing the same algorithms and is
therefore not valid.

Ok, but how can I do the same thing in Java ?
By the way, I do not want to attack java (that I use daily). I just want
to understand ...