J
Jorge
Webkit r34469 vs. Opera 9.50 :
3.00x as fast 6339.6ms(Opera) 2109.8ms (Webkit)
3.00x as fast 6339.6ms(Opera) 2109.8ms (Webkit)
G
Gregor Kofler
Jorge meinte:
Safari 3.1.1 vs. FF3 (WinXP): 4909ms vs. 5602ms
Anyway, how useful is a JS benchmark nowadays, which deliberately leaves
out DOM perfomance? Since the benchmark is by Apple, I don't expect
anything but "lightning fast" performance by their in-house browser.
Gregor
Safari 3.1.1 vs. FF3 (WinXP): 4909ms vs. 5602ms
Anyway, how useful is a JS benchmark nowadays, which deliberately leaves
out DOM perfomance? Since the benchmark is by Apple, I don't expect
anything but "lightning fast" performance by their in-house browser.
Gregor
L
Lasse Reichstein Nielsen
Jorge said:
Numbers on their own are meaningless. What were the configurations
used to perform the test? OS, CPU, etc? Any other pages open in the
browser? Was Opera's mail client enabled? Was WebKit used in
a browser (e.g. Safari) or run as a GUI-less batch job?
And sorry, but I don't trust a benchmark hosted on the site of the one
performing best at that benchmark. For all we know, Webkit could have
been microoptimized for exactly the tested tasks, and suck at
everything else. Probably not, but it wouldn't be the first time
something like that happened.
Is the sunspider test suite available for download and perusal, or
should we just read the page source?
/L
J
Jorge
Why ? The results show that it runs x times faster/slower on my
computer.
All the browsers were tested on the same computer.
Browser : a browser is a browser. Webkit is a browser as well.
Open the browser, run the test. That's it. Nothing special. Try it
yourself.
Hey, what's up ?
Take it easy...
Thanks,
--Jorge.
J
Jorge
Is JS core performance a don't care, then ?
Not so.
They must have been working hard lately,
these tests show no lightning-fast-performance here : look :
http://mozillalinks.org/wp/2008/02/firefox-3-ultimate-feature-performance/
That was in February. 15 days later :
http://mozillalinks.org/wp/2008/03/updated-web-browsers-javascript-benchmarks/
And today we are where we are :
Webkit r34469 vs. FF3.0 (final) :
1.55x as fast 3269.6ms(FF3) 2109.8ms(Webkit)
--Jorge.
G
Gregor Kofler
Jorge meinte:
No. But only a fraction of the speed of a JS application running in a
browser window will be contributed by the core performance. Typical "Web
2.0" applications rely heavily on DOM performance.
One could argue, that they've tweaked their browser to outperform
competitors with their home-made benchmark...
Anyway, on WinXP the current Safari version is 10 to 15% faster than FF3
running a benchmark that measures core JS performance. IMO nothing to
write home about.
Gregor
No. But only a fraction of the speed of a JS application running in a
browser window will be contributed by the core performance. Typical "Web
2.0" applications rely heavily on DOM performance.
One could argue, that they've tweaked their browser to outperform
competitors with their home-made benchmark...
Anyway, on WinXP the current Safari version is 10 to 15% faster than FF3
running a benchmark that measures core JS performance. IMO nothing to
write home about.
Gregor
J
Jorge
But you might be glad to learn that
-FF is almost as fast as the fastest.
-all of them are performing much better than before. (all but IE).
--Jorge.
L
Lasse Reichstein Nielsen
Jorge said:
That *what* runs faster? To understand the numbers, we need to know
how they were produced. E.g., how many other applications were running
at the time, how many other pages were open in the same browser, etc.
In other words, we need to be able to reproduce the setting and test
the numbers.
Sorry, I'm a trained scientist, pedantry is an occupational hazerd.
Good. That's the least to require.
Meh. A browser component running in a thin wrapper doesn't have the
memory footprint or other concurrent threads to handle that a full
fledged, highly skinnable browser+mail agent+news reader+irc
client+download manager does.
Webkit, if I read correctly, is a browser component. I.e., it only
hadles the display of the page. Safari is a browser that is build
on Webkit, just as Internet Explorer is a browser build on the
Microsoft Browser Component, and Firefox is build on the
If I did that on my browser, I would open a browser with 30+ tabs,
a mail client checking mail and an IRC client connecting to a channel.
It would be likely to give worse results than if I opened a clean profile
with everything fancy disabled and no pages open.
I don't know that what you did, which is why I say that the number is
meaningless.
You could easily have done everything perfectly, eliminating as many
spurious influences as at all possible, but then again, you might
not. It would be equally wrong of me to assume either, which is why I
merely say that the numbers, by themselves, are meaningless.
.... [and more of the same] ...
Distrust of benchmarks, mainly
I'm remembering things like
http://www.geek.com/ati-driver-caught-optimizing-for-3dmark03-as-well/
and have a natural distrust of any benchmark that comes from the same
source as the products that scores highest on the benchmark.
It just smells like selective sampling. In science, procedures to avoid
selective sampling are introduced not only to prevent cheating, but
also, just as importantly, to remove subcontious bias.
In that case, the producers should make a serious effort to convince
people that is a balanced and generic benchmark. That hasn't happened
here (to my knowledge).
Oh, I do
/L
L
Lasse Reichstein Nielsen
....
Gecko browser component.
(Note to self: remember to finish sentences before moving on)
/L
Gecko browser component.
(Note to self: remember to finish sentences before moving on)
/L
J
Jorge
On the same computer, in the same circumstances, open a browser, run
the test, close it. Open another browser, run the test, close it.
Repeat for the third browser. Paste in the results and note the
difference.
The numbers obtained serve very well to *compare* one browser against
the other. I posted : browser a ran x times faster than browser b,
etc.
That's not meaningless, I think.
It does not matter whether the actual number was 5000mS, what matters
is that 2xmS is twice as much as xmS... no ?
And (x times faster) applies most likely to most other computers
running the same OS, as well.
The numbers themselves are meaningless if you intend to compare them
against results obtained in another computer/OS/whatever, yes.
I'm glad to see that JS performance for the last generation of
browsers is several times better. Don't you ?
Thanks,
--Jorge.
J
Jorge
The non-release versions Safari are usually called "Webkit nightly
builds". If you go to nightly.webkit.org what you download is there an
app, that is a browser, that looks and behaves like Safari, but it's
called WebKit not Safari (it's name is WebKit). I used the name in
that sense.
Thanks,
--Jorge.
R
Richard Levasseur
I recently found this article: http://www.sitepen.com/blog/2008/05/09/string-performance-an-analysis/
I think it highlights the concerns people have expressed. Something
innocuous in one browser is a horrible kludge in another. JS tests
can be very sensitive, too. Simple things like adding a setTimeout()
call between test runs can make a difference.
That isn't to say the tests are completely useless, but saying safari
is 2x as opera, and 50% faster than firefox becomes questionable.
The best you can really do is say Safari >= FF > Opera, but without
any quantitative measurements.
Richard
J
Jorge
And the getTime() accuracy as well. There's a browser out there that
increments it in 16mS steps... I heard.
Do you say "take it with a grain of salt" ?
Yes, the results shown are a mean value, and all the consequences of
being an average apply.
But if you want you can dig deeper and see "what really happenned" and
have a look at the invidual tests results as well :
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 2247.6ms +/- 7.6%
--------------------------------------------
3d: 321.6ms +/- 23.2%
cube: 106.4ms +/- 23.3%
morph: 109.0ms +/- 36.4%
raytrace: 106.2ms +/- 10.4%
access: 325.6ms +/- 9.0%
binary-trees: 39.6ms +/- 10.8%
fannkuch: 105.2ms +/- 4.0%
nbody: 143.2ms +/- 19.3%
nsieve: 37.6ms +/- 21.6%
bitops: 223.2ms +/- 4.3%
3bit-bits-in-byte: 33.0ms +/- 6.0%
bits-in-byte: 43.8ms +/- 22.7%
bitwise-and: 63.0ms +/- 3.9%
nsieve-bits: 83.4ms +/- 12.1%
controlflow: 24.4ms +/- 12.8%
recursive: 24.4ms +/- 12.8%
crypto: 148.8ms +/- 12.7%
aes: 52.0ms +/- 5.6%
md5: 48.6ms +/- 23.6%
sha1: 48.2ms +/- 10.5%
date: 206.8ms +/- 8.4%
format-tofte: 120.8ms +/- 6.0%
format-xparb: 86.0ms +/- 12.2%
math: 240.0ms +/- 13.7%
cordic: 82.4ms +/- 11.3%
partial-sums: 109.8ms +/- 15.4%
spectral-norm: 47.8ms +/- 15.9%
regexp: 214.8ms +/- 0.8%
dna: 214.8ms +/- 0.8%
string: 542.4ms +/- 1.9%
base64: 84.0ms +/- 1.5%
fasta: 95.4ms +/- 1.5%
tagcloud: 130.2ms +/- 2.8%
unpack-code: 135.8ms +/- 3.9%
validate-input: 97.0ms +/- 3.4%
There there's and empty field to paste in other browser's results so
that you can compare them side-by-side, line-by-line, test-by-test.
I doubt that the tests themselves are architected to benefit WebKit's
results.
As JS is necessarily open-source, how would you hide such a thing ?
Thanks,
--Jorge.
D
Dr J R Stockton
In comp.lang.javascript message <870207f7-b830-409e-bc94-cd694ceea498@m3
6g2000hse.googlegroups.com>, Thu, 19 Jun 2008 02:27:01, Jorge
Standard is that 16mS means 16 milliSiemens, which is a conductance
corresponding to about 63 Ohms. Doubtless you mean what should be
written as 16 ms - the approximate duration of one cycle of the mains in
hasty countries.
In Win98, I got intervals of 54.9 ms, rounded to 50 or 60. The
underlying interval in WinXP is generally 15.625 ms, rounded to 1 ms
unless something changes it. But in Firefox 3.0 I seem to see 1 ms
intervals.
6g2000hse.googlegroups.com>, Thu, 19 Jun 2008 02:27:01, Jorge
Standard is that 16mS means 16 milliSiemens, which is a conductance
corresponding to about 63 Ohms. Doubtless you mean what should be
written as 16 ms - the approximate duration of one cycle of the mains in
hasty countries.
In Win98, I got intervals of 54.9 ms, rounded to 50 or 60. The
underlying interval in WinXP is generally 15.625 ms, rounded to 1 ms
unless something changes it. But in Firefox 3.0 I seem to see 1 ms
intervals.
J
Jorge
Sure, yep.
"mS" is what I get when my brain executes
"milisegundos".abbreviate().toCamelCase();
That's ~ a tick, in MacOS's tongue : TickCount().
--Jorge.
T
Thomas 'PointedEars' Lahn
Jorge said:
ISTM you have yet to learn that comparisons, especially but not excluded to
be those made in numbers, need a reliable reference base for them becoming
possible to be taken under serious consideration. For example, "almost as
fast as the fastest" (what?) alone is an utterly useless and pointless
statement about everything. There are really much better ways to waste
bandwidth than posting it here.
PointedEars
J
Jorge
Might be.
But you would be asking "What the hell ?" if FF3 ran 2x slower than
FF2.
You're welcome too.
--Jorge.