Re: benchmarks? java vs .net

Discussion in 'Java' started by Jon Skeet [C# MVP], Jun 3, 2008.

  1. Razii <> wrote:
    > $ time partialsums 2500000 (.NET)
    > 3.000000000 (2/3)^k
    > 3160.817621887 k^-0.5
    > 0.999999600 1/k(k+1)
    > 30.314541510 Flint Hills
    > 42.995233998 Cookson Hills
    > 15.309017155 Harmonic
    > 1.644933667 Riemann Zeta
    > 0.693146981 Alternating Harmonic
    > 0.785398063 Gregory
    >
    > real 0m1.530s
    > user 0m0.000s
    > sys 0m0.031s


    That looks very much like Unix output rather than Windows. Are you
    running under cygwin or something similar?

    It would really help if you gave a full explanation of how *you* (as
    opposed to the shoot-out) site are running your tests.

    (I'd also argue that benchmarks taking only a second and a half to
    complete aren't likely to have a good startup vs running program
    balance.)

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 3, 2008
    #1
    1. Advertising

  2. Razii <> wrote:
    > >That looks very much like Unix output rather than Windows. Are you
    > >running under cygwin or something similar?

    >
    > Yes, I am running cygwin.


    Has it occurred to you that that may have an effect on the results?

    The benchmarks would be *much* better in my view if each benchmark
    started up a single process and ran with it for a significant amount of
    time. After all, that's far more realistic in terms of how almost all
    software is actually run in the real world.

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 3, 2008
    #2
    1. Advertising

  3. On Jun 3, 11:15 pm, Razii <> wrote:
    > <> wrote:
    > >Has it occurred to you that that may have an effect on the results?

    >
    > No, it won't have a major effect.


    And you know this because...?

    Just claiming that it won't have an effect isn't exactly compelling
    evidence.

    Jon
    Jon Skeet [C# MVP], Jun 4, 2008
    #3
  4. On Jun 4, 5:43 pm, Razii <> wrote:
    > <> wrote:
    > >And you know this because...?

    >
    > The one who asserts must prove.


    I absolutely agree. Here's what I wrote:

    <quote>Has it occurred to you that that may have an effect on the
    results?</quote>

    I don't see an assertion there - merely speculation that it *may* have
    an effect.

    Now here's what you wrote:

    <quote>No, it won't have a major effect.</quote>

    *That's* an assertion. Therefore by your own instructions, it's up to
    you to prove your assertion.

    If you're running a program in a non-standard way (and running .NET
    under Cygwin certainly isn't the usual environment) then it's up to
    you to investigate whether or not that has any effect.

    Jon
    Jon Skeet [C# MVP], Jun 4, 2008
    #4
  5. Razii <> wrote:
    > ><quote>No, it won't have a major effect.</quote>
    > >
    > >*That's* an assertion. Therefore by your own instructions, it's up to
    > >you to prove your assertion.

    >
    > There is no reason to believe that it "may have" any effect.


    There absolutely is. What will be feeding the programs their standard
    input? Cygwin. What will be launching the processes? Cygwin.

    Will it definitely have a difference? Who knows. I'm not claiming it
    definitely will - just that until you've proved that it doesn't, your
    tests are invalid when considering how Java and .NET are *normally* run
    on Windows.

    > Cygwin offers easy way to time with time command. If you can time all
    > the becnhmarks as easily on Windows and get different results, let us
    > know.


    What I *will* do is amend the sumcol test to be sensible in the first
    place - to run *once* with a suitably large input. That way you can
    easily measure the time taken within the app, discounting the
    (massively repeated) startup costs.

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 4, 2008
    #5
  6. Jon Skeet [C# MVP] <> wrote:
    > > Cygwin offers easy way to time with time command. If you can time all
    > > the becnhmarks as easily on Windows and get different results, let us
    > > know.

    >
    > What I *will* do is amend the sumcol test to be sensible in the first
    > place - to run *once* with a suitably large input. That way you can
    > easily measure the time taken within the app, discounting the
    > (massively repeated) startup costs.


    Okay, done - and it's interesting.

    First here's a program to duplicate the original input file:
    using System.IO;

    class DuplicateSumCol
    {
    static void Main(string[] args)
    {
    string text = File.ReadAllText("sumcol-input.txt");
    int iterations = int.Parse(args[0]);
    File.Delete("sumcol-large.txt");
    for (int i=0; i < iterations; i++)
    {
    File.AppendAllText("sumcol-large.txt", text);
    }
    }
    }

    I ran that with an input parameter of 10000 to get a 43MB file, which
    appears to be large enough to give interesting results.

    Then I refactored both the Java and C# versions to allow input either
    from a file or from standard input. The bulk of the time is spent in a
    single method, so it shouldn't make much difference - and if *either*
    language is significantly hampered by this sort of refactoring, then
    frankly it deserves all it gets. Here's the C# version:

    using System;
    using System.Diagnostics;
    using System.IO;

    class SumCol
    {
    static void Main(string[] args)
    {
    int sum;
    Stopwatch sw = Stopwatch.StartNew();
    if (args.Length == 0)
    {
    sum = Parse(Console.In);
    }
    else
    {
    using (TextReader reader = File.OpenText(args[0]))
    {
    sum = Parse(reader);
    }
    }
    long elapsed = sw.ElapsedMilliseconds;
    Console.WriteLine("Result: {0}", sum);
    Console.WriteLine("Elapsed millis: {0}", elapsed);
    }

    static int Parse (TextReader reader)
    {
    int sum = 0;
    string line;
    while ( (line=reader.ReadLine()) != null)
    {
    sum += int.Parse(line);
    }
    return sum;
    }
    }

    And here's the Java version:

    import java.io.*;

    public class SumCol
    {
    public static void main(String[] args) throws Exception
    {
    int sum;
    long start = System.currentTimeMillis();

    if (args.length==0)
    {
    sum = parse(System.in);
    }
    else
    {
    // For the sake of brevity, don't bother closing...
    sum = parse(new BufferedInputStream
    (new FileInputStream(args[0])));

    }
    long end = System.currentTimeMillis();

    System.out.println("Result: " + sum);
    System.out.println("Elapsed millis: " + (end-start));
    }

    static int parse(InputStream stream) throws IOException
    {
    int sum = 0;
    StreamTokenizer lineTokenizer = new StreamTokenizer(stream);
    while (lineTokenizer.nextToken() != StreamTokenizer.TT_EOF)
    {
    sum += (int) lineTokenizer.nval;
    }
    return sum;
    }
    }

    I'm running .NET 3.5, and here's all the Java information:
    java version "1.6.0_05"
    Java(TM) SE Runtime Environment (build 1.6.0_05-b13)
    Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode, sharing)

    The C# version was compiled with:
    csc -o+ -debug- SumCol.cs

    The Java version was compiled with:
    javac SumCol.java

    I'm running on an Intel Core 2 Duo with 32-bit Vista.

    Results (in ms):

    SumCol.exe < sumcol-large.txt: 4833
    SumCol.exe sumcol-large.txt: 3327
    java SumCol < sumcol-large.txt: 3321
    java SumCol sumcol-large.txt: 3315


    Feel free to suggest changes to how the Java version is run, of course
    - and please test on your own machine.

    My conclusion: Java and .NET are effectively neck and neck here in
    terms of actual *processing*, but .NET's console handling slows it down
    significantly. No great loss IMO - I very rarely pipe large input from
    the console.

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 4, 2008
    #6
  7. Razii <> wrote:
    > On Wed, 4 Jun 2008 19:05:19 +0100, Jon Skeet [C# MVP]
    > <> wrote:
    >
    > >Will it definitely have a difference? Who knows.

    >
    > Dowsnload Cygwin
    >
    > http://www.cygwin.com/
    >
    > Time it and let us know if there is a difference.


    I'd rather time a more *useful* benchmark (one that doesn't involve
    starting thousands of processes) - so I've done exactly that.

    My previous experiences with Cygwin have been far from great - I'd
    rather not install it at all. As I say, if you want to justify running
    ..NET in a somewhat alien environment (you must admit that it's not
    exactly the norm), you should check that it doesn't affect results.

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 4, 2008
    #7
  8. Jon Skeet [C# MVP]

    Barry Kelly Guest

    Jon Skeet wrote:

    > My previous experiences with Cygwin have been far from great - I'd
    > rather not install it at all.


    I'm not addressing Razzi, as I don't see his posts anymore, but I do
    want to pipe up about Cygwin.

    Cygwin works very well as long as you treat it almost like a terminal to
    another operating system. Using the command-line utilities in a console
    window running cmd.exe is usually a waste of time. Running bash in rxvt,
    with an appropriately-configured ~/.bash{rc,_profile} etc. and
    ~/.Xdefaults (for rxvt settings), and it's a pleasure - so long as you
    enjoy the Unix experience and want to replicate it on Windows.

    With an appropriately constructed user/profile directory, you can get
    almost identical shell experiences across most OSes.

    -- Barry

    --
    http://barrkel.blogspot.com/
    Barry Kelly, Jun 4, 2008
    #8
  9. Razii <> wrote:
    > On Wed, 04 Jun 2008 22:13:49 +0100, Barry Kelly
    > <> wrote:
    >
    > >I'm not addressing Razzi, as I don't see his posts anymore, but I do
    > >want to pipe up about Cygwin.

    >
    > You definitely see my posts because my email changes with every post.


    And it's completely inconceivable to you that a newsreader could filter
    just on name, and not on email address?

    Forte Agent, the newsreader Barry uses (by his headers, at least) has
    precisely this capability. I've just tried it, and verified that it
    filters out all of your posts.

    Now, of course you *could* change your name as well as your email
    address for every post - but there's really no legitimate reason for
    doing that, and no point in irritating people just for the sake of it.
    Heck, why do you even change email address each time?

    --
    Jon Skeet - <>
    Web site: http://www.pobox.com/~skeet
    Blog: http://www.msmvps.com/jon.skeet
    C# in Depth: http://csharpindepth.com
    Jon Skeet [C# MVP], Jun 4, 2008
    #9
  10. Jon Skeet [C# MVP]

    Arne Vajhøj Guest

    Jon Skeet [C# MVP] wrote:
    > On Jun 3, 11:15 pm, Razii <> wrote:
    >> <> wrote:
    >>> Has it occurred to you that that may have an effect on the results?

    >> No, it won't have a major effect.

    >
    > And you know this because...?
    >
    > Just claiming that it won't have an effect isn't exactly compelling
    > evidence.


    It probably is not.

    time just note counters, execute the command and note counters
    again. I don't know which API's it use to execute the command,
    but it should not effect the programs speed whether is 3 Unix
    emulation calls and ShellExecute or something else.

    Arne
    Arne Vajhøj, Jun 8, 2008
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mark Thornton

    Re: benchmarks? java vs .net

    Mark Thornton, Jun 3, 2008, in forum: Java
    Replies:
    35
    Views:
    880
    Arne Vajhøj
    Jun 9, 2008
  2. Mark Thornton

    Re: benchmarks? java vs .net

    Mark Thornton, Jun 3, 2008, in forum: C++
    Replies:
    35
    Views:
    913
    Arne Vajhøj
    Jun 9, 2008
  3. Jon Skeet [C# MVP]

    Re: benchmarks? java vs .net

    Jon Skeet [C# MVP], Jun 3, 2008, in forum: C++
    Replies:
    10
    Views:
    515
    Arne Vajhøj
    Jun 8, 2008
  4. Mark Thornton
    Replies:
    3
    Views:
    350
    Arne Vajhøj
    Jun 9, 2008
  5. Mark Thornton
    Replies:
    3
    Views:
    467
    Arne Vajhøj
    Jun 9, 2008
Loading...

Share This Page