performance question

Discussion in 'Java' started by Olivier Scalbert, Dec 8, 2007.

  1. Hello,

    I would like to show to somebody how different languages look like by
    doing a very simple case.
    I was very surprised by the poor performance of the java version !

    Here are the programs in different languages:

    in C:

    #include <stdio.h>

    int main()
    {
    int i;

    for(i = 0; i < 1000000; i++)
    {
    printf("abcdefghijk %d\n", i);
    }

    return 0;
    }

    time ./test1 > out.txt

    real 0m0.710s
    user 0m0.576s
    sys 0m0.124s

    in java:
    public class Test
    {
    public static void main(String[] args)
    {
    for (int i = 0; i < 1000000; i++)
    {
    System.out.println("abcdefghijk " + i);
    }
    }
    }

    time java Test > out.txt

    real 0m12.364s
    user 0m4.180s
    sys 0m7.676s

    time java -server Test > out.txt

    real 0m10.537s
    user 0m3.120s
    sys 0m6.544s


    That is not good at all !
    ols@tatooine:~/projects/ruby$ java -version
    java version "1.6.0_02"
    Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
    Java HotSpot(TM) Client VM (build 1.6.0_02-b05, mixed mode, sharing)

    ols@tatooine:~/projects/ruby$ uname -a
    Linux tatooine 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007
    i686 GNU/Linux

    In python:
    i=0
    while i < 1000000:
    print "abcdefghijk", i
    i=i+1

    time python test.py > out.txt

    real 0m2.292s
    user 0m2.064s
    sys 0m0.112s

    In perl:
    for ($count = 0; $count < 1000000; $count++)
    {
    print "abcdefghijk $count\n";
    }
    time perl test.pl > out.txt

    real 0m1.243s
    user 0m1.060s
    sys 0m0.160s

    In ruby:
    counter = 0
    while counter < 1000000
    puts("abcdefghijk #{counter}")
    counter+=1
    end

    time ruby test.rb > out.txt

    real 0m4.731s
    user 0m4.452s
    sys 0m0.100s

    As you can see the java program performance is far from the other one. I
    was very surprised! Before the tests, I was sure that the winner will be
    C followed by java, but it is not the case ...

    Of course, I can improve the performance of the java version, by using a
    StringBuffer, and print this buffer when it is bigger than a given size,
    but it is not fair !

    If you have any ideas, there are welcomed !

    Thanks,

    Olivier
     
    Olivier Scalbert, Dec 8, 2007
    #1
    1. Advertising

  2. Olivier Scalbert

    Real Gagnon Guest

    > time java -server Test > out.txt
    >
    > real 0m10.537s
    > user 0m3.120s
    > sys 0m6.544s
    >


    I bet that the JVM loading and startup time is very important with your
    test.

    So it's not this particular code which is slow, the timing technique is not
    adequate.

    Bye.
    --
    Real Gagnon from Quebec, Canada
    * Java, Javascript, VBScript and PowerBuilder code snippets
    * http://www.rgagnon.com/howto.html
    * http://www.rgagnon.com/bigindex.html
     
    Real Gagnon, Dec 8, 2007
    #2
    1. Advertising

  3. Olivier Scalbert

    Eric Sosman Guest

    Olivier Scalbert wrote:
    > Hello,
    >
    > I would like to show to somebody how different languages look like by
    > doing a very simple case.
    > I was very surprised by the poor performance of the java version !
    > [...]


    Congratulations! You have discovered that Java is a
    poor choice of language if you plan to do something silly,
    pointless, and stupid. Please report your findings to Sun,
    and file a Request For Enhancement to optimize Java for
    better performance on silly, pointless, and stupid programs.
    (Don't be too surprised, though, if the Javafolk attach a
    relatively low priority to the RFE; they're more interested
    in getting Java to work well for useful programs. That's
    a clear case of bias against the average programmer, don't
    you think?)

    "And the wheel," said the Captain, "what about this
    wheel thingy? It sounds a terribly interesting project."

    "Ah," said the marketing girl, "well, we're having a
    little difficulty there."

    "Difficulty?" exclaimed Ford. "Difficulty? What do you
    mean, difficulty? It's the single simplest machine in
    the entire Universe!"

    The marketing girl soured him with a look. "All right
    Mr. Wiseguy," she said, "you're so clever, you tell us
    what color it should be."

    --
    Eric Sosman
    lid
     
    Eric Sosman, Dec 8, 2007
    #3
  4. Real Gagnon wrote:
    >> time java -server Test > out.txt
    >>
    >> real 0m10.537s
    >> user 0m3.120s
    >> sys 0m6.544s
    >>

    >
    > I bet that the JVM loading and startup time is very important with your
    > test.

    Not sure !

    With:
    public class Test
    {
    public static void main(String[] args)
    {
    for (int i = 0; i < 1; i++)
    {
    System.out.println("abcdefghijk " + i);
    }
    }
    }

    time java Test > out.txt

    real 0m0.125s
    user 0m0.040s
    sys 0m0.032s


    >
    > So it's not this particular code which is slow, the timing technique is not
    > adequate.

    Do you have another technique ?

    >
    > Bye.

    See you
     
    Olivier Scalbert, Dec 8, 2007
    #4
  5. Olivier Scalbert

    Stefan Ram Guest

    Real Gagnon <> writes:
    >So it's not this particular code which is slow, the timing
    >technique is not adequate.


    May be it is. For example, when one wants to write an
    echo command for the command line, to be used like:

    echo "Hello world!"

    one does often not use Java for a reason.

    On the other hand, people start to consider to write a
    JVM in Java, because C is too slow:

    http://www.adam-bien.com/roller/abien/entry/when_c_becomes_too_slow

    And then:

    »if you run across benchmarks of any kind that show JRuby
    running slower than Ruby 1.8.x, we'd appreciate you filing
    them as bugs«

    http://groups.google.com/group/jrub...g/9fd1ab9664a1a36b?dmode=source&output=gplain
     
    Stefan Ram, Dec 8, 2007
    #5
  6. Hi,

    Thanks for your congratulations !

    But do you have a real explanation ?

    And by the way, can you explain me, why it was a silly, pointless and
    stupid program ?

    For me, it is only a small program, nothing else.


    Eric Sosman wrote:
    > Olivier Scalbert wrote:
    >> Hello,
    >>
    >> I would like to show to somebody how different languages look like by
    >> doing a very simple case.
    >> I was very surprised by the poor performance of the java version !
    >> [...]

    >
    > Congratulations! You have discovered that Java is a
    > poor choice of language if you plan to do something silly,
    > pointless, and stupid. Please report your findings to Sun,
    > and file a Request For Enhancement to optimize Java for
    > better performance on silly, pointless, and stupid programs.
    > (Don't be too surprised, though, if the Javafolk attach a
    > relatively low priority to the RFE; they're more interested
    > in getting Java to work well for useful programs. That's
    > a clear case of bias against the average programmer, don't
    > you think?)
    >
    > "And the wheel," said the Captain, "what about this
    > wheel thingy? It sounds a terribly interesting project."
    >
    > "Ah," said the marketing girl, "well, we're having a
    > little difficulty there."
    >
    > "Difficulty?" exclaimed Ford. "Difficulty? What do you
    > mean, difficulty? It's the single simplest machine in
    > the entire Universe!"
    >
    > The marketing girl soured him with a look. "All right
    > Mr. Wiseguy," she said, "you're so clever, you tell us
    > what color it should be."
    >
     
    Olivier Scalbert, Dec 8, 2007
    #6
  7. Olivier Scalbert

    Lew Guest

    Olivier Scalbert wrote:
    > Real Gagnon wrote:
    >>> time java -server Test > out.txt
    >>>
    >>> real 0m10.537s
    >>> user 0m3.120s
    >>> sys 0m6.544s
    >>>

    >> I bet that the JVM loading and startup time is very important with your
    >> test.

    > Not sure !
    >
    > With:
    > public class Test
    > {
    > public static void main(String[] args)
    > {
    > for (int i = 0; i < 1; i++)
    > {
    > System.out.println("abcdefghijk " + i);


    Note: This has the overhead of character encoding. Your C example does not.
    So you are comparing very different activities.

    > }
    > }
    > }
    >
    > time java Test > out.txt
    >
    > real 0m0.125s
    > user 0m0.040s
    > sys 0m0.032s
    >
    >
    >> So it's not this particular code which is slow, the timing technique is not
    >> adequate.

    > Do you have another technique ?


    Yes - time the loop, not the load time. With Java, you should run the loop
    for a while, then run it again with timing instrumentation. This lets the JVM
    optimize the loop.

    Your benchmark as presented is pretty useless. Apples and oranges.

    --
    Lew
     
    Lew, Dec 8, 2007
    #7
  8. Lew wrote:
    ;
    >
    > Note: This has the overhead of character encoding. Your C example does
    > not. So you are comparing very different activities.
    >


    Good point, but I find that the char encoding overhead is quite expensive !
    Thanks.
     
    Olivier Scalbert, Dec 8, 2007
    #8
  9. Olivier Scalbert

    Eric Sosman Guest

    Olivier Scalbert wrote:
    > [...]
    > And by the way, can you explain me, why it was a silly, pointless and
    > stupid program ?


    Let's apply this test: How valuable is the output of
    this program? How many backup copies did you make? How
    many other programs used this output as their input?

    If you made no backups and fed the output to no other
    programs, as I suspect, then by your own actions you have
    declared the value of the program's output to be zero. A
    program that produces nothing of value meets my definition
    of silly, pointless, and stupid.

    --
    Eric Sosman
    lid
     
    Eric Sosman, Dec 8, 2007
    #9
  10. Olivier Scalbert wrote:
    > Hi,
    >
    > Thanks for your congratulations !
    >
    > But do you have a real explanation ?
    >
    > And by the way, can you explain me, why it was a silly, pointless and
    > stupid program ?
    >
    > For me, it is only a small program, nothing else.
    >
    >


    1. Modern Java systems monitor the execution of the program for a little
    while before compiling. This allows a better choice of optimisation
    strategy, but doesn't produce the fastest results on applications that
    take much less than a second to run anyway (but if a program takes less
    than a second why do you care how much less).

    2. Unicode vs ASCII. Java always works with Unicode and converts the
    characters to whatever encoding is in use on the machine. Your C
    equivalent assumes ASCII. Try including a £ character in your test; Java
    will get this right so long as the character encoding is set correctly
    (this can be done at execution time).

    3. Output buffering. By default Java output to System.out is line
    buffered. That is the output buffer is flushed at the end of every line.
    The C 'equivalent' usually uses a much larger buffer by default (512
    bytes or more). You can change Java to use a larger buffer and not flush
    at EOL or you could change C to use line buffered output. Unless you
    change one or the other your comparison is flawed.

    4. You could look back a week or two and find that someone else proposed
    a similar test, with similar flaws. After seeing similar mistakes on an
    almost weekly basis for at least 10 years, it is not surprising that
    people get a bit tired of providing detailed explanations of what is wrong.

    Mark Thornton
     
    Mark Thornton, Dec 8, 2007
    #10
  11. Olivier Scalbert

    Jon Harrop Guest

    Olivier Scalbert wrote:
    > Lew wrote:
    > ;
    >>
    >> Note: This has the overhead of character encoding. Your C example does
    >> not. So you are comparing very different activities.
    >>

    >
    > Good point, but I find that the char encoding overhead is quite expensive
    > ! Thanks.


    No, char encoding is not an adequate explanation of a 15x slowdown. This is
    Java being very poor at buffering.

    --
    Dr Jon D Harrop, Flying Frog Consultancy Ltd.
    http://www.ffconsultancy.com/products/?u
     
    Jon Harrop, Dec 8, 2007
    #11
  12. Olivier Scalbert

    Jon Harrop Guest

    Olivier Scalbert wrote:
    > Real Gagnon wrote:
    >> I bet that the JVM loading and startup time is very important with your
    >> test.

    > Not sure !
    >
    > With:
    > public class Test
    > {
    > public static void main(String[] args)
    > {
    > for (int i = 0; i < 1; i++)
    > {
    > System.out.println("abcdefghijk " + i);
    > }
    > }
    > }
    >
    > time java Test > out.txt
    >
    > real 0m0.125s
    > user 0m0.040s
    > sys 0m0.032s


    Indeed, startup time is the number one mis-attribution posted by Java
    advocates as a way around explaining very poor performance. As you can see,
    it was quite wrong to blame startup time on this benchmark.

    --
    Dr Jon D Harrop, Flying Frog Consultancy Ltd.
    http://www.ffconsultancy.com/products/?u
     
    Jon Harrop, Dec 8, 2007
    #12
  13. Olivier Scalbert

    Jon Harrop Guest

    Eric Sosman wrote:
    > Let's apply this test: How valuable is the output of
    > this program?


    How common is IO?

    --
    Dr Jon D Harrop, Flying Frog Consultancy Ltd.
    http://www.ffconsultancy.com/products/?u
     
    Jon Harrop, Dec 8, 2007
    #13
  14. Olivier Scalbert wrote:
    > Hi,
    >
    > Thanks for your congratulations !
    >
    > But do you have a real explanation ?
    >
    > And by the way, can you explain me, why it was a silly, pointless and
    > stupid program ?
    >
    > For me, it is only a small program, nothing else.
    >


    Search for the thread "Why is Java so slow???" starting on 19 November,
    posted by "java.performance.expert" (who was clearly nothing of the sort).

    Mark Thornton
     
    Mark Thornton, Dec 8, 2007
    #14
  15. Eric Sosman wrote:
    > Olivier Scalbert wrote:
    >> [...]
    >> And by the way, can you explain me, why it was a silly, pointless and
    >> stupid program ?

    >
    > Let's apply this test: How valuable is the output of
    > this program? How many backup copies did you make? How
    > many other programs used this output as their input?
    >
    > If you made no backups and fed the output to no other
    > programs, as I suspect, then by your own actions you have
    > declared the value of the program's output to be zero. A
    > program that produces nothing of value meets my definition
    > of silly, pointless, and stupid.
    >


    Ok, still no technical info, just blabla on what is silly, pointless.
    I do not appreciate people, that are "donneurs de leçons" (moralist
    people that give lesson).

    Using java daily since many many years from embedded platform to
    application server, I have never noticed that java can detect silly
    programs from very intelligent one !

    For your info, by adding few lines of code, this program will generate
    huge csv files (several billions of lines). Theses files will be used to
    test the import functionality of different databases.
     
    Olivier Scalbert, Dec 8, 2007
    #15
  16. Olivier Scalbert

    Jon Harrop Guest

    Lew wrote:
    > Aside from the emotionally-loaded pejoratives and all the /ad hominem/
    > flaming, the fact remains that your benchmark did not compare the same
    > behaviors and is not indicative of the relative speeds of Java vs. the
    > other platforms.


    On the contrary, Olivier's test is simple and flawless. Can you make the
    Java nearly as fast as the C? Feel free to not use unicode, or does Java
    have poor support for everything else?

    --
    Dr Jon D Harrop, Flying Frog Consultancy Ltd.
    http://www.ffconsultancy.com/products/?u
     
    Jon Harrop, Dec 8, 2007
    #16
  17. Olivier Scalbert

    Lew Guest

    Jon Harrop wrote:
    > Olivier Scalbert wrote:
    >> Lew wrote:
    >> ;
    >>> Note: This has the overhead of character encoding. Your C example does
    >>> not. So you are comparing very different activities.
    >>>

    >> Good point, but I find that the char encoding overhead is quite expensive
    >> ! Thanks.

    >
    > No, char encoding is not an adequate explanation of a 15x slowdown. This is
    > Java being very poor at buffering.


    A point also made by Mark Thornton, and more evidence that my point is true:
    that the benchmark is not comparing the same algorithms and is therefore not
    valid.

    --
    Lew
     
    Lew, Dec 8, 2007
    #17
  18. Olivier Scalbert

    Jon Harrop Guest

    Disclaimer: I am not a fan of Java.

    Olivier Scalbert wrote:
    > As you can see the java program performance is far from the other one. I
    > was very surprised! Before the tests, I was sure that the winner will be
    > C followed by java, but it is not the case ...


    Here's what I learned from my ray tracer language comparison:

    http://www.ffconsultancy.com/languages/ray_tracer/

    For a compiled language, Java is slow. However, this effect is compounded in
    two important ways:

    1. Getting accurate information about how to optimize Java code is almost
    impossible.

    2. Actually implementing the optimizations is typically unnecessarily slow
    and tedious in Java.

    In this case, you have already seen the two standard responses to a valid
    question from the Java community:

    1. This is "only" a microbenchmark.

    2. You're measuring startup time.

    In reality, IO is a very important aspect of many programs (including my ray
    tracer benchmark, which was hopelessly IO bound _only_ in Java until some
    kind soul taught me how to work around these deficiencies in Java) and you
    are a long way from measuring the startup time, which will be a fraction of
    a second in this case.

    > Of course, I can improve the performance of the java version, by using a
    > StringBuffer, and print this buffer when it is bigger than a given size,
    > but it is not fair !


    Actually the best solution is to use buffered output rather than a string
    buffer to accumulate an intermediate result. Don't ask me why the Java
    implementors aren't competent enough to buffer automatically but they
    aren't.

    So, you start by importing a namespace for doing non-sucky IO:

    import java.io.*;

    Then you create an unnecessarily long-named BufferedOutputStream:

    BufferedOutputStream out = new BufferedOutputStream(System.out);

    Finally, you write to this instead of System.out:

    out.write(("abcdefghijk " + i + "\n").getBytes());

    Holy mackerel, your Java code now takes only 1.275s instead of the original
    3.135! That's a lot better than it was before but it is still over and
    order of magnitude (!) slower than any decent language, e.g. C at 0.196s.

    You see, Java is what the functional programming community refer to as
    a "low-level language" because it simply fails to convey high-level
    information to the compiler for optimization. In this case, the Java
    compiler is probably being amazingly retarded in that it actually performs
    two string concatenations in unicode for absolutely no reason whatsoever.

    So let's try manually unrolling the code:

    for (int i = 0; i < 1000000; i++) {
    out.write("abcdefghijk ".getBytes());
    out.write(s1);
    out.write(Integer.toString(i).getBytes());
    out.write("\n".getBytes());
    out.write(s2);
    }

    Wow, this is actually slower than it was before at 1.492s. But this is going
    in the right direction. As it happens, Java is so crap that it can't even
    hoist loop invariants as well as the C compilers from the 70s.

    Hoisting manually, we finally arrive at:

    import java.io.*;

    public class Test
    {
    public static void main(String[] args) throws java.io.IOException
    {
    BufferedOutputStream out = new BufferedOutputStream(System.out);

    byte[] s1 = "abcdefghijk ".getBytes();
    byte[] s2 = "\n".getBytes();

    for (int i = 0; i < 1000000; i++) {
    out.write(s1);
    out.write(Integer.toString(i).getBytes());
    out.write(s2);
    }
    }
    }

    Now this only takes 0.947s! Incredible, several times longer now and still
    several times slower than any decent language!

    In summary, what we have learned is that for the love of God (TM) don't use
    Java if performance is at all important for your work. This doesn't just
    apply to IO either: Java is very slow at a wide variety of vitally-
    important tasks, e.g. allocation.

    --
    Dr Jon D Harrop, Flying Frog Consultancy Ltd.
    http://www.ffconsultancy.com/products/?u
     
    Jon Harrop, Dec 8, 2007
    #18
  19. Olivier Scalbert

    Lew Guest

    Olivier Scalbert wrote:
    > Eric Sosman wrote:
    >> Olivier Scalbert wrote:
    >>> [...]
    >>> And by the way, can you explain me, why it was a silly, pointless and
    >>> stupid program ?

    >> Let's apply this test: How valuable is the output of
    >> this program? How many backup copies did you make? How
    >> many other programs used this output as their input?
    >>
    >> If you made no backups and fed the output to no other
    >> programs, as I suspect, then by your own actions you have
    >> declared the value of the program's output to be zero. A
    >> program that produces nothing of value meets my definition
    >> of silly, pointless, and stupid.
    >>

    >
    > Ok, still no technical info, just blabla on what is silly, pointless.
    > I do not appreciate people, that are "donneurs de leçons" (moralist
    > people that give lesson).
    >
    > Using java daily since many many years from embedded platform to
    > application server, I have never noticed that java can detect silly
    > programs from very intelligent one !
    >
    > For your info, by adding few lines of code, this program will generate
    > huge csv files (several billions of lines). Theses files will be used to
    > test the import functionality of different databases.


    Aside from the emotionally-loaded pejoratives and all the /ad hominem/
    flaming, the fact remains that your benchmark did not compare the same
    behaviors and is not indicative of the relative speeds of Java vs. the other
    platforms.

    --
    Lew
     
    Lew, Dec 8, 2007
    #19
  20. Lew wrote:
    > Jon Harrop wrote:
    >> Olivier Scalbert wrote:
    >>> Lew wrote:
    >>> ;
    >>>> Note: This has the overhead of character encoding. Your C example does
    >>>> not. So you are comparing very different activities.
    >>>>
    >>> Good point, but I find that the char encoding overhead is quite
    >>> expensive
    >>> ! Thanks.

    >>
    >> No, char encoding is not an adequate explanation of a 15x slowdown.
    >> This is
    >> Java being very poor at buffering.

    >
    > A point also made by Mark Thornton, and more evidence that my point is
    > true: that the benchmark is not comparing the same algorithms and is
    > therefore not valid.
    >


    Ok, but how can I do the same thing in Java ?
    By the way, I do not want to attack java (that I use daily). I just want
    to understand ...
     
    Olivier Scalbert, Dec 8, 2007
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Don Beal
    Replies:
    13
    Views:
    843
    Richard Grimes [MVP]
    Sep 29, 2003
  2. jm
    Replies:
    1
    Views:
    512
    alien2_51
    Dec 12, 2003
  3. Cris Rock

    Performance related Question.....

    Cris Rock, Feb 12, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    313
    Stefano Mostarda
    Feb 12, 2004
  4. cjl
    Replies:
    3
    Views:
    991
    John Nagle
    May 21, 2007
  5. Software Engineer
    Replies:
    0
    Views:
    335
    Software Engineer
    Jun 10, 2011
Loading...

Share This Page