New browsers : SunSpider JavaScript Benchmark Results.

Discussion in 'Javascript' started by Jorge, Jun 18, 2008.

  1. Jorge

    Jorge Guest

    Webkit r34469 vs. Opera 9.50 :

    3.00x as fast 6339.6ms(Opera) 2109.8ms (Webkit)

    -----

    FF3.0 (final) vs. Opera 9.50 :

    1.94x as fast 6339.6ms (Opera) 3269.6ms (FF3)

    -----

    Webkit r34469 vs. FF3.0 (final) :

    1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)

    -----

    http://webkit.org/perf/sunspider-0.9/sunspider.html

    --Jorge.
    Jorge, Jun 18, 2008
    #1
    1. Advertising

  2. Jorge meinte:

    > Webkit r34469 vs. FF3.0 (final) :
    >
    > 1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)


    Safari 3.1.1 vs. FF3 (WinXP): 4909ms vs. 5602ms

    Anyway, how useful is a JS benchmark nowadays, which deliberately leaves
    out DOM perfomance? Since the benchmark is by Apple, I don't expect
    anything but "lightning fast" performance by their in-house browser.

    Gregor




    --
    http://photo.gregorkofler.at ::: Landschafts- und Reisefotografie
    http://web.gregorkofler.com ::: meine JS-Spielwiese
    http://www.image2d.com ::: Bildagentur für den alpinen Raum
    Gregor Kofler, Jun 18, 2008
    #2
    1. Advertising

  3. Jorge <> writes:

    > 3.00x as fast 6339.6ms(Opera) 2109.8ms (Webkit)

    ....
    > 1.94x as fast 6339.6ms (Opera) 3269.6ms (FF3)

    ....
    > 1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)

    ....
    > http://webkit.org/perf/sunspider-0.9/sunspider.html


    Numbers on their own are meaningless. What were the configurations
    used to perform the test? OS, CPU, etc? Any other pages open in the
    browser? Was Opera's mail client enabled? Was WebKit used in
    a browser (e.g. Safari) or run as a GUI-less batch job?

    And sorry, but I don't trust a benchmark hosted on the site of the one
    performing best at that benchmark. For all we know, Webkit could have
    been microoptimized for exactly the tested tasks, and suck at
    everything else. Probably not, but it wouldn't be the first time
    something like that happened.
    Is the sunspider test suite available for download and perusal, or
    should we just read the page source?

    /L
    --
    Lasse Reichstein Nielsen -
    DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html>
    'Faith without judgement merely degrades the spirit divine.'
    Lasse Reichstein Nielsen, Jun 18, 2008
    #3
  4. Jorge

    Jorge Guest

    On Jun 18, 10:13 am, Lasse Reichstein Nielsen <> wrote:

    > Numbers on their own are meaningless.


    Why ? The results show that it runs x times faster/slower on my
    computer.
    All the browsers were tested on the same computer.

    > What were the configurations
    > used to perform the test? OS, CPU, etc? Any other pages open in the
    > browser?  Was Opera's mail client enabled? Was WebKit used in
    > a browser (e.g. Safari) or run as a GUI-less batch job?


    Browser : a browser is a browser. Webkit is a browser as well.
    Open the browser, run the test. That's it. Nothing special. Try it
    yourself.

    >
    > And sorry, but I don't trust a benchmark hosted on the site of the one
    > performing best at that benchmark. For all we know, Webkit could have
    > been microoptimized for exactly the tested tasks, and suck at
    > everything else. Probably not, but it wouldn't be the first time
    > something like that happened.
    > Is the sunspider test suite available for download and perusal, or
    > should we just read the page source?
    >


    Hey, what's up ?
    Take it easy...

    Thanks,
    --Jorge.
    Jorge, Jun 18, 2008
    #4
  5. Jorge

    Jorge Guest

    On Jun 18, 10:12 am, Gregor Kofler <> wrote:

    > Anyway, how useful is a JS benchmark nowadays, which deliberately leaves
    > out DOM perfomance?


    Is JS core performance a don't care, then ?

    > Since the benchmark is by Apple, I don't expect
    > anything but "lightning fast" performance by their in-house browser.


    Not so.
    They must have been working hard lately,
    these tests show no lightning-fast-performance here : look :

    http://mozillalinks.org/wp/2008/02/firefox-3-ultimate-feature-performance/

    That was in February. 15 days later :

    http://mozillalinks.org/wp/2008/03/updated-web-browsers-javascript-benchmarks/

    And today we are where we are :

    Webkit r34469 vs. FF3.0 (final) :
    1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)

    --Jorge.
    Jorge, Jun 18, 2008
    #5
  6. Jorge meinte:
    > On Jun 18, 10:12 am, Gregor Kofler <> wrote:
    >
    >> Anyway, how useful is a JS benchmark nowadays, which deliberately leaves
    >> out DOM perfomance?

    >
    > Is JS core performance a don't care, then ?


    No. But only a fraction of the speed of a JS application running in a
    browser window will be contributed by the core performance. Typical "Web
    2.0" applications rely heavily on DOM performance.

    >> Since the benchmark is by Apple, I don't expect
    >> anything but "lightning fast" performance by their in-house browser.

    >
    > Not so.
    > They must have been working hard lately,
    > these tests show no lightning-fast-performance here : look :
    >
    > http://mozillalinks.org/wp/2008/02/firefox-3-ultimate-feature-performance/
    >
    > That was in February. 15 days later :
    >
    > http://mozillalinks.org/wp/2008/03/updated-web-browsers-javascript-benchmarks/
    >
    > And today we are where we are :
    >
    > Webkit r34469 vs. FF3.0 (final) :
    > 1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)


    One could argue, that they've tweaked their browser to outperform
    competitors with their home-made benchmark...
    Anyway, on WinXP the current Safari version is 10 to 15% faster than FF3
    running a benchmark that measures core JS performance. IMO nothing to
    write home about.

    Gregor


    --
    http://photo.gregorkofler.at ::: Landschafts- und Reisefotografie
    http://web.gregorkofler.com ::: meine JS-Spielwiese
    http://www.image2d.com ::: Bildagentur für den alpinen Raum
    Gregor Kofler, Jun 18, 2008
    #6
  7. Jorge

    Jorge Guest

    On Jun 18, 11:33 am, Gregor Kofler <> wrote:

    > Anyway, on WinXP the current Safari version is 10 to 15% faster than FF3
    > running a benchmark that measures core JS performance. IMO nothing to
    > write home about.
    >


    But you might be glad to learn that

    -FF is almost as fast as the fastest.

    -all of them are performing much better than before. (all but IE).

    --Jorge.
    Jorge, Jun 18, 2008
    #7
  8. Jorge <> writes:

    > On Jun 18, 10:13 am, Lasse Reichstein Nielsen <> wrote:
    >
    >> Numbers on their own are meaningless.

    >
    > Why ? The results show that it runs x times faster/slower on my
    > computer.


    That *what* runs faster? To understand the numbers, we need to know
    how they were produced. E.g., how many other applications were running
    at the time, how many other pages were open in the same browser, etc.
    In other words, we need to be able to reproduce the setting and test
    the numbers.
    Sorry, I'm a trained scientist, pedantry is an occupational hazerd.

    > All the browsers were tested on the same computer.


    Good. That's the least to require.

    >> What were the configurations
    >> used to perform the test? OS, CPU, etc? Any other pages open in the
    >> browser?  Was Opera's mail client enabled? Was WebKit used in
    >> a browser (e.g. Safari) or run as a GUI-less batch job?

    >
    > Browser : a browser is a browser.


    Meh. A browser component running in a thin wrapper doesn't have the
    memory footprint or other concurrent threads to handle that a full
    fledged, highly skinnable browser+mail agent+news reader+irc
    client+download manager does.

    > Webkit is a browser as well.


    Webkit, if I read correctly, is a browser component. I.e., it only
    hadles the display of the page. Safari is a browser that is build
    on Webkit, just as Internet Explorer is a browser build on the
    Microsoft Browser Component, and Firefox is build on the

    > Open the browser, run the test. That's it. Nothing special. Try it
    > yourself.


    If I did that on my browser, I would open a browser with 30+ tabs,
    a mail client checking mail and an IRC client connecting to a channel.
    It would be likely to give worse results than if I opened a clean profile
    with everything fancy disabled and no pages open.

    I don't know that what you did, which is why I say that the number is
    meaningless.

    You could easily have done everything perfectly, eliminating as many
    spurious influences as at all possible, but then again, you might
    not. It would be equally wrong of me to assume either, which is why I
    merely say that the numbers, by themselves, are meaningless.

    >> And sorry, but I don't trust a benchmark hosted on the site of the one
    >> performing best at that benchmark.

    .... [and more of the same] ...

    > Hey, what's up ?


    Distrust of benchmarks, mainly :)
    I'm remembering things like
    http://www.geek.com/ati-driver-caught-optimizing-for-3dmark03-as-well/
    and have a natural distrust of any benchmark that comes from the same
    source as the products that scores highest on the benchmark.
    It just smells like selective sampling. In science, procedures to avoid
    selective sampling are introduced not only to prevent cheating, but
    also, just as importantly, to remove subcontious bias.
    In that case, the producers should make a serious effort to convince
    people that is a balanced and generic benchmark. That hasn't happened
    here (to my knowledge).

    > Take it easy...


    Oh, I do :)

    /L
    --
    Lasse Reichstein Nielsen -
    DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html>
    'Faith without judgement merely degrades the spirit divine.'
    Lasse Reichstein Nielsen, Jun 18, 2008
    #8
  9. Lasse Reichstein Nielsen <> writes:

    ....
    > on Webkit, just as Internet Explorer is a browser build on the
    > Microsoft Browser Component, and Firefox is build on the
    >


    Gecko browser component.

    (Note to self: remember to finish sentences before moving on)
    /L
    --
    Lasse Reichstein Nielsen -
    DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html>
    'Faith without judgement merely degrades the spirit divine.'
    Lasse Reichstein Nielsen, Jun 18, 2008
    #9
  10. Jorge

    Jorge Guest

    On Jun 18, 8:26 pm, Lasse Reichstein Nielsen <> wrote:

    > That *what* runs faster? To understand the numbers, we need to know
    > how they were produced. E.g., how many other applications were running
    > at the time, how many other pages were open in the same browser, etc.
    > In other words, we need to be able to reproduce the setting and test
    > the numbers.


    On the same computer, in the same circumstances, open a browser, run
    the test, close it. Open another browser, run the test, close it.
    Repeat for the third browser. Paste in the results and note the
    difference.

    The numbers obtained serve very well to *compare* one browser against
    the other. I posted : browser a ran x times faster than browser b,
    etc.
    That's not meaningless, I think.
    It does not matter whether the actual number was 5000mS, what matters
    is that 2xmS is twice as much as xmS... no ?
    And (x times faster) applies most likely to most other computers
    running the same OS, as well.
    The numbers themselves are meaningless if you intend to compare them
    against results obtained in another computer/OS/whatever, yes.

    I'm glad to see that JS performance for the last generation of
    browsers is several times better. Don't you ?

    Thanks,
    --Jorge.
    Jorge, Jun 18, 2008
    #10
  11. Jorge

    Jorge Guest

    On Jun 18, 8:26 pm, Lasse Reichstein Nielsen <> wrote:

    > Webkit, if I read correctly, is a browser component. I.e., it only
    > hadles the display of the page. Safari is a browser that is build
    > on Webkit, just as Internet Explorer is a browser build on the
    > Microsoft Browser Component, and Firefox is build on the Gecko browser.


    The non-release versions Safari are usually called "Webkit nightly
    builds". If you go to nightly.webkit.org what you download is there an
    app, that is a browser, that looks and behaves like Safari, but it's
    called WebKit not Safari (it's name is WebKit). I used the name in
    that sense.

    Thanks,
    --Jorge.
    Jorge, Jun 18, 2008
    #11
  12. On Jun 18, 12:51 pm, Jorge <> wrote:
    > On Jun 18, 8:26 pm, Lasse Reichstein Nielsen <> wrote:
    >
    > > Webkit, if I read correctly, is a browser component. I.e., it only
    > > hadles the display of the page. Safari is a browser that is build
    > > on Webkit, just as Internet Explorer is a browser build on the
    > > Microsoft Browser Component, and Firefox is build on the Gecko browser.

    >
    > The non-release versions Safari are usually called "Webkit nightly
    > builds". If you go to nightly.webkit.org what you download is there an
    > app, that is a browser, that looks and behaves like Safari, but it's
    > called WebKit not Safari (it's name is WebKit). I used the name in
    > that sense.
    >
    > Thanks,
    > --Jorge.


    I recently found this article: http://www.sitepen.com/blog/2008/05/09/string-performance-an-analysis/

    I think it highlights the concerns people have expressed. Something
    innocuous in one browser is a horrible kludge in another. JS tests
    can be very sensitive, too. Simple things like adding a setTimeout()
    call between test runs can make a difference.

    That isn't to say the tests are completely useless, but saying safari
    is 2x as opera, and 50% faster than firefox becomes questionable.
    The best you can really do is say Safari >= FF > Opera, but without
    any quantitative measurements.

    Richard
    Richard Levasseur, Jun 19, 2008
    #12
  13. Jorge

    Jorge Guest

    On Jun 19, 10:22 am, Richard Levasseur <> wrote:
    >
    > I recently found this article:http://www.sitepen.com/blog/2008/05/09/string-performance-an-analysis/
    >
    > I think it highlights the concerns people have expressed.  Something
    > innocuous in one browser is a horrible kludge in another.  JS tests
    > can be very sensitive, too.  Simple things like adding a setTimeout()
    > call between test runs can make a difference.
    >


    And the getTime() accuracy as well. There's a browser out there that
    increments it in 16mS steps... I heard.

    Do you say "take it with a grain of salt" ?

    Yes, the results shown are a mean value, and all the consequences of
    being an average apply.
    But if you want you can dig deeper and see "what really happenned" and
    have a look at the invidual tests results as well :


    ============================================
    RESULTS (means and 95% confidence intervals)
    --------------------------------------------
    Total: 2247.6ms +/- 7.6%
    --------------------------------------------

    3d: 321.6ms +/- 23.2%
    cube: 106.4ms +/- 23.3%
    morph: 109.0ms +/- 36.4%
    raytrace: 106.2ms +/- 10.4%

    access: 325.6ms +/- 9.0%
    binary-trees: 39.6ms +/- 10.8%
    fannkuch: 105.2ms +/- 4.0%
    nbody: 143.2ms +/- 19.3%
    nsieve: 37.6ms +/- 21.6%

    bitops: 223.2ms +/- 4.3%
    3bit-bits-in-byte: 33.0ms +/- 6.0%
    bits-in-byte: 43.8ms +/- 22.7%
    bitwise-and: 63.0ms +/- 3.9%
    nsieve-bits: 83.4ms +/- 12.1%

    controlflow: 24.4ms +/- 12.8%
    recursive: 24.4ms +/- 12.8%

    crypto: 148.8ms +/- 12.7%
    aes: 52.0ms +/- 5.6%
    md5: 48.6ms +/- 23.6%
    sha1: 48.2ms +/- 10.5%

    date: 206.8ms +/- 8.4%
    format-tofte: 120.8ms +/- 6.0%
    format-xparb: 86.0ms +/- 12.2%

    math: 240.0ms +/- 13.7%
    cordic: 82.4ms +/- 11.3%
    partial-sums: 109.8ms +/- 15.4%
    spectral-norm: 47.8ms +/- 15.9%

    regexp: 214.8ms +/- 0.8%
    dna: 214.8ms +/- 0.8%

    string: 542.4ms +/- 1.9%
    base64: 84.0ms +/- 1.5%
    fasta: 95.4ms +/- 1.5%
    tagcloud: 130.2ms +/- 2.8%
    unpack-code: 135.8ms +/- 3.9%
    validate-input: 97.0ms +/- 3.4%

    There there's and empty field to paste in other browser's results so
    that you can compare them side-by-side, line-by-line, test-by-test.

    I doubt that the tests themselves are architected to benefit WebKit's
    results.
    As JS is necessarily open-source, how would you hide such a thing ?

    Thanks,
    --Jorge.
    Jorge, Jun 19, 2008
    #13
  14. In comp.lang.javascript message <870207f7-b830-409e-bc94-cd694ceea498@m3
    6g2000hse.googlegroups.com>, Thu, 19 Jun 2008 02:27:01, Jorge
    <> posted:
    >
    >And the getTime() accuracy as well. There's a browser out there that
    >increments it in 16mS steps... I heard.


    Standard is that 16mS means 16 milliSiemens, which is a conductance
    corresponding to about 63 Ohms. Doubtless you mean what should be
    written as 16 ms - the approximate duration of one cycle of the mains in
    hasty countries.

    In Win98, I got intervals of 54.9 ms, rounded to 50 or 60. The
    underlying interval in WinXP is generally 15.625 ms, rounded to 1 ms
    unless something changes it. But in Firefox 3.0 I seem to see 1 ms
    intervals.

    --
    (c) John Stockton, near London. *@merlyn.demon.co.uk/?.?
    Web <URL:http://www.merlyn.demon.co.uk/> - FAQish topics, acronyms, & links.
    Correct <= 4-line sig. separator as above, a line precisely "-- " (SoRFC1036)
    Do not Mail News to me. Before a reply, quote with ">" or "> " (SoRFC1036)
    Dr J R Stockton, Jun 19, 2008
    #14
  15. Jorge

    Jorge Guest

    On Jun 19, 9:25 pm, Dr J R Stockton <> wrote:
    >
    > Standard is that 16mS means 16 milliSiemens, which is a conductance
    > corresponding to about 63 Ohms.  Doubtless you mean what should be
    > written as 16 ms - the approximate duration of one cycle of the mains in
    > hasty countries.
    >


    Sure, yep.
    "mS" is what I get when my brain executes
    "milisegundos".abbreviate().toCamelCase(); :)

    > The
    > underlying interval in WinXP is generally 15.625 ms, rounded to 1 ms
    > unless something changes it.


    That's ~ a tick, in MacOS's tongue : TickCount().

    --Jorge.
    Jorge, Jun 20, 2008
    #15
  16. Jorge wrote:
    > On Jun 18, 11:33 am, Gregor Kofler <> wrote:
    >> Anyway, on WinXP the current Safari version is 10 to 15% faster than FF3
    >> running a benchmark that measures core JS performance. IMO nothing to
    >> write home about.

    >
    > But you might be glad to learn that
    >
    > -FF is almost as fast as the fastest.
    >
    > -all of them are performing much better than before. (all but IE).


    ISTM you have yet to learn that comparisons, especially but not excluded to
    be those made in numbers, need a reliable reference base for them becoming
    possible to be taken under serious consideration. For example, "almost as
    fast as the fastest" (what?) alone is an utterly useless and pointless
    statement about everything. There are really much better ways to waste
    bandwidth than posting it here.


    PointedEars
    --
    realism: HTML 4.01 Strict
    evangelism: XHTML 1.0 Strict
    madness: XHTML 1.1 as application/xhtml+xml
    -- Bjoern Hoehrmann
    Thomas 'PointedEars' Lahn, Jun 22, 2008
    #16
  17. Jorge

    Jorge Guest

    On Jun 22, 7:21 pm, Thomas 'PointedEars' Lahn <>
    wrote:

    > ISTM you have yet to learn that comparisons, especially but not excluded to
    > be those made in numbers, need a reliable reference base for them becoming
    > possible to be taken under serious consideration.  For example, "almostas
    > fast as the fastest" (what?) alone is an utterly useless and pointless
    > statement about everything.


    Might be.
    But you would be asking "What the hell ?" if FF3 ran 2x slower than
    FF2.

    >  There are really much better ways to waste
    > bandwidth than posting it here.


    You're welcome too.

    --Jorge.
    Jorge, Jun 22, 2008
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Dang Griffith

    Re: any benchmark results for python?

    Dang Griffith, Jun 25, 2003, in forum: Python
    Replies:
    0
    Views:
    1,278
    Dang Griffith
    Jun 25, 2003
  2. Hans Mull

    Benchmark results unrealistic?

    Hans Mull, Feb 11, 2008, in forum: C++
    Replies:
    3
    Views:
    420
    James Kanze
    Feb 12, 2008
  3. Daniel Berger

    StringIO affecting Benchmark results

    Daniel Berger, Aug 25, 2004, in forum: Ruby
    Replies:
    0
    Views:
    152
    Daniel Berger
    Aug 25, 2004
  4. Juan Alvarez

    Help interpreting benchmark results

    Juan Alvarez, Feb 24, 2009, in forum: Ruby
    Replies:
    3
    Views:
    157
    Sandor Szücs
    Feb 25, 2009
  5. Replies:
    3
    Views:
    153
Loading...

Share This Page