Ruby Performance

Daniel Amelang · Aug 16, 2005

Hi Isaac,

Austin 'lead' quite an involved and lengthy discussion about
benchmarks (including the Alioth and Ackermann) just two months ago.
Before you leap into the fray, take a look:

http://www.ruby-talk.org/cgi-bin/scat.rb/ruby/ruby-talk/145390

If you do a google search of ruby-talk.org for 'austin alioth', or
just 'austin benchmarks' you can get even more on his opinion. I
highly recommend reading through the arguments and counter arguments
before we do a complete reproduction here.

But, hey, if you've got something to say beyond all that has already
been said, don't let me stop you

I'm just trying to save us (and
you) from more of the same-old-same-old discussions.

Dan

Isaac Gouy · Aug 16, 2005

Daniel Amelang wrote:
-snip-

I highly recommend reading through the arguments and counter arguments
before we do a complete reproduction here.

Thank you Daniel, you might notice my name in those discussions.

As you've mentioned, other folk on ruby-talk presented
counter-arguments to Austin's claims - afaict those counter-arguments
were never rebutted.

-snip-

I'm just trying to save us (and
you) from more of the same-old-same-old discussions.

If there's any substance behind the name-calling we should be able to
point-out specific errors on the shootout.

http://shootout.alioth.debian.org/f...y-0&p2=php-0&p3=psyco-0&p4=lua-0&sort=fullcpu

Isaac Gouy · Aug 16, 2005

Wilson said:
I hate to bring up the actual thread topic.. but isn't the original
poster basically just benchmarking startup time, something that is
currently one of Ruby's weak points?

Do you think startup time is large compared to a runtime of nearly a
minute?

Austin Ziegler · Aug 16, 2005

Daniel Amelang wrote:
-snip-
Thank you Daniel, you might notice my name in those discussions.

Only as an uncritical booster who doesn't actually listen. Just like now, n=
o?

As you've mentioned, other folk on ruby-talk presented
counter-arguments to Austin's claims - afaict those counter-arguments
were never rebutted.

Then you simply can't tell. They were. Basically, the shootout you run
is -- always has been -- and always will be -- crap. You can take
steps to make it honest crap, instead of what it is, though.

-snip-
If there's any substance behind the name-calling we should be able to
point-out specific errors on the shootout.

When the problem with the shootout is the methodology and the
presentation -- without getting into more substantial errors
(statistical problems) -- and the general unwillingness of the people
who run it to do anything to improve the methodology and presentation,
then there's not a specific error. The whole thing is wrong. Until you
understand that there's problems with your whole damned methodology
and even worse problems with your presentation, you're never going to
get it.

Even if Ruby were to become the best performing language tomorrow, I
wouldn't recommend that anyone pay attention to the pile of garbage
known as the "Great Computer Language Shootout." The only thing
"great" about it is the amount of sheer ignorance that it and its
maintainers willingly perpetuate.

--=20
Austin Ziegler * (e-mail address removed)
* Alternate: (e-mail address removed)

Matthew Desmarais · Aug 16, 2005

Hi Isaac,

I've paid attention to some of the discussions on the shootout here,
but I'm not sure that I've ever had any of my basic questions
answered. Maybe you can answer them.

The site lists as its goal, "to learn about programming languages,
compare their performance in various (possibly meaningless) ways and,
most importantly, have some fun!"

What is it that we are learning about programming languages?

What do we gain from comparing performance in "various (possibly
meaningless) ways?

I don't think that I've learned much from the shootout, but it may
well be that I'm missing something. I do that a lot.

William James · Aug 16, 2005

Austin said:
Then you simply can't tell. They were. Basically, the shootout you run
is -- always has been -- and always will be -- crap. You can take
steps to make it honest crap, instead of what it is, though.

Doug Bagley, the creator of the original "Great Computer Language
Shootout", called his system for ranking the languages "CRAPS".

It allowed him easily to tinker with the weight assigned to each test
and hence to make his favorite language come out on top.

Isaac Gouy · Aug 16, 2005

Austin said:
Isaac,

I'm done responding to you. None of my statements about the
fundamental problems of the shootout and your (apparent)
administration (or boosterism, at a minimum) are false or baseless.

Just a few days ago, you made claims which are baseless.

http://groups.google.com/group/comp.lang.ruby/msg/9716aeda66f54a17?hl=en&

They are well documented over the last several months -- pretty much
every time someone brings up the question of the Alioth shootout.

You're repeated claims are well documented - that doesn't make them
true.

Brock Weaver · Aug 16, 2005

------=_Part_4465_649850.1124218999858
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

I feel a lot of love on this thread.

Should I invoke Godwin's Law and put this argument to rest?

http://en.wikipedia.org/wiki/Godwin's_law

Okay.

All benchmarks are the brainchild of Hitler and his nazis.

=20
=20
Austin Ziegler wrote:
=20
=20
Doug Bagley, the creator of the original "Great Computer Language
Shootout", called his system for ranking the languages "CRAPS".
=20
It allowed him easily to tinker with the weight assigned to each test
and hence to make his favorite language come out on top.
=20
=20
=20

--=20
Brock Weaver
http://www.circaware.com

------=_Part_4465_649850.1124218999858--

Joe Van Dyk · Aug 16, 2005

=20
Only as an uncritical booster who doesn't actually listen. Just like now,= no?
=20
Then you simply can't tell. They were. Basically, the shootout you run
is -- always has been -- and always will be -- crap. You can take
steps to make it honest crap, instead of what it is, though.
=20
=20
When the problem with the shootout is the methodology and the
presentation -- without getting into more substantial errors
(statistical problems) -- and the general unwillingness of the people
who run it to do anything to improve the methodology and presentation,
then there's not a specific error. The whole thing is wrong. Until you
understand that there's problems with your whole damned methodology
and even worse problems with your presentation, you're never going to
get it.
=20
Even if Ruby were to become the best performing language tomorrow, I
wouldn't recommend that anyone pay attention to the pile of garbage
known as the "Great Computer Language Shootout." The only thing
"great" about it is the amount of sheer ignorance that it and its
maintainers willingly perpetuate.

Please, tell us how you *really* feel. I feel like there's some
bottled up emotions in there somewhere that you aren't telling us.

Isaac Gouy · Aug 16, 2005

Matthew said:
Hi Isaac,

I've paid attention to some of the discussions on the shootout here,
but I'm not sure that I've ever had any of my basic questions
answered. Maybe you can answer them.

The site lists as its goal, "to learn about programming languages,
compare their performance in various (possibly meaningless) ways and,
most importantly, have some fun!"

What is it that we are learning about programming languages?

What do we gain from comparing performance in "various (possibly
meaningless) ways?

I don't think that I've learned much from the shootout, but it may
well be that I'm missing something. I do that a lot.

Interesting questions Matthew.

imo What we learn depends on how actively we participate, and what we
already know or assume we know, and many other factors.

So I should ask
- did you try to make any of the programs faster or more elegant?
- did you try to write programs in a new language?
- did you know there was a language Clean?
- did you understand that relative performance would vary so much from
tcp-request-reply to k-nucleotide to n-body and mandelbrot?
- did you notice any patterns: slow for io, fast for number crunching?
- did you wonder why the Lua program used 100K for nsieve?
etc

Isaac Gouy · Aug 16, 2005

Matthew said:
Hi Isaac,

I've paid attention to some of the discussions on the shootout here,
but I'm not sure that I've ever had any of my basic questions
answered. Maybe you can answer them.

The site lists as its goal, "to learn about programming languages,
compare their performance in various (possibly meaningless) ways and,
most importantly, have some fun!"

What is it that we are learning about programming languages?

What do we gain from comparing performance in "various (possibly
meaningless) ways?

I don't think that I've learned much from the shootout, but it may
well be that I'm missing something. I do that a lot.

Interesting questions Matthew.

imo What we learn depends on how actively we participate, and what we
already know or assume we know, and many other factors.

So I should ask
- did you try to make any of the programs faster or more elegant?
- did you try to write programs in a new language?
- did you know there was a language Clean?
- did you understand that relative performance would vary so much from
tcp-request-reply to k-nucleotide to n-body and mandelbrot?
- did you notice any patterns: slow for io, fast for number crunching?
- did you wonder why the Lua program used ~100MB for nsieve?
etc

ogilthorpe · Aug 16, 2005

Interesting questions Matthew.

imo What we learn depends on how actively we participate, and what we
already know or assume we know, and many other factors.

So I should ask
- did you try to make any of the programs faster or more elegant?
- did you try to write programs in a new language?
- did you know there was a language Clean?
- did you understand that relative performance would vary so much from
tcp-request-reply to k-nucleotide to n-body and mandelbrot?
- did you notice any patterns: slow for io, fast for number crunching?
- did you wonder why the Lua program used ~100MB for nsieve?
etc

I don't think that you answered either of my questions. If you have and
someone can tell me what the answers were, I'd appreciate it.

I guess I'm not sure why the site is called "The Computer Language
Shootout". To me, a shootout feels like a thing that pits two entities
against one another in a competition. The result of a shootout is
definitely a winner and a loser, depending on the definition of victory.

When you discuss the site, you talk about it as an educational tool. I'm
all for that kind of thing, and your response to me is reasonable and if =
I
were looking for help it would be helpful.

The problem is the use of the word benchmark. By claiming to offer
"benchmarks", the site is purporting to offer measured and standard
methods of comparisons between programming languages. The many
disclaimers show that these numbers should be taken with a grain of salt,
but the site's keywords, "benchmarking fast programming language benchmar=
k
performance benchmarks shootout program" show otherwise.

And so I think that the site, or more specifically its design, is very
disingenuous. The site offers itself as a home for objective comparison
of the performance of programming languages. There is no such thing, and
so the site should not claim that there is.

Kirk Haines · Aug 16, 2005

Interesting questions Matthew.

imo What we learn depends on how actively we participate, and what we
already know or assume we know, and many other factors.

So I should ask

[List of questions that don't answer the prior questions deleted.]

I've been watching this debate with some interest. I think Austin does some
harm to his position with the anger with which he seems to present it, but on
the flipside, you seem to be taking a lot of long, lazy walks around some
clear questions without actually getting to an answer. This last reply that
you sent to Matthew's questions is a perfect example. Answering a question
with a question isn't an answer. It's a dance, a circumlocution, an evasion.
Answer the man's question, one at a time. Then pose questions of your own.
That's how a debate is supposed to take place; answer a question with a
question is just weak.

Kirk Haines

Austin Ziegler · Aug 16, 2005

Please, tell us how you *really* feel. I feel like there's some
bottled up emotions in there somewhere that you aren't telling us.

Darn. I was aiming for the "perfectly clear that I have no patience
for pseudo-intellectuals" effect. I'll try harder next time to be
clearer. It doesn't help that said pseudo-intellectual is impervious
to reality.

-austin
--=20
Austin Ziegler * (e-mail address removed)
* Alternate: (e-mail address removed)

Sy · Aug 16, 2005

Isaac Gouy · Aug 16, 2005

I don't think that you answered either of my questions. If you have and
someone can tell me what the answers were, I'd appreciate it.

I was trying to answer "I don't think that I've learned much from the
shootout".

Let's try again - "What is it that we are learning about programming
languages?"

I don't know what you learn about programming languages from shootout,
the website states "Our goals are to learn about programming
languages..." - the goals of the folk administering the shootout are to
learn about programming languages...

Let's try again - "What do we gain from comparing performance in
"various (possibly meaningless) ways?"

Some perspective on how performance varies between programming language
implementations and tasks.

I guess I'm not sure why the site is called "The Computer Language
Shootout". To me, a shootout feels like a thing that pits two entities
against one another in a competition. The result of a shootout is
definitely a winner and a loser, depending on the definition of victory.

History - Doug Bagley's Shootout begat Aldo Calpini's Win32 Shootout
begat Brent Fulgham's Shootout.

When you discuss the site, you talk about it as an educational tool. I'm
all for that kind of thing, and your response to me is reasonable and if I
were looking for help it would be helpful.

The problem is the use of the word benchmark. By claiming to offer
"benchmarks", the site is purporting to offer measured and standard
methods of comparisons between programming languages. The many
disclaimers show that these numbers should be taken with a grain of salt,
but the site's keywords, "benchmarking fast programming language benchmark
performance benchmarks shootout program" show otherwise.

And so I think that the site, or more specifically its design, is very
disingenuous. The site offers itself as a home for objective comparison
of the performance of programming languages. There is no such thing, and
so the site should not claim that there is.

The site provides multiple comparison programs, which show various
different language implementations "winning".

The site shows different ways to "win" - by CPU time, by memory use, by
LOCs.

The site provides a synthetic overall score and invites you to
"manipulate the multipliers and weights to make your favourite language
the fastest programming language in the Shootout" for "a solution that
is simple, neat, and wrong".

Does the site proclaim A is faster than B, or subvert that simplistic
notion?

Isaac Gouy · Aug 17, 2005

Kirk said:
Interesting questions Matthew.

imo What we learn depends on how actively we participate, and what we
already know or assume we know, and many other factors.

So I should ask

Click to expand...

[List of questions that don't answer the prior questions deleted.]

I've been watching this debate with some interest. I think Austin does some
harm to his position with the anger with which he seems to present it, but on
the flipside, you seem to be taking a lot of long, lazy walks around some
clear questions without actually getting to an answer. This last reply that
you sent to Matthew's questions is a perfect example. Answering a question
with a question isn't an answer. It's a dance, a circumlocution, an evasion.

Sometimes it's none of those things, sometimes it takes a couple of
conversational turns before we are all clear what the real questions
are.

http://groups.google.com/group/comp.lang.ruby/msg/613e239d096f7138?hl=en&

Answer the man's question, one at a time. Then pose questions of your own.
That's how a debate is supposed to take place; answer a question with a
question is just weak.

Is debate the only form of conversation that's acceptable here, or can
there be discussion without winners and losers?

Kirk Haines · Aug 17, 2005

Sometimes it's none of those things, sometimes it takes a couple of
conversational turns before we are all clear what the real questions
are.

http://groups.google.com/group/comp.lang.ruby/msg/613e239d096f7138?hl=en&

"Real questions"? A real question is a question that someone asks. It's
generally safe to assume that the question someone asks is the one that they
want to have answered. You may think you know better than they do what
question they truly want answered, but that's a different issue.

Is debate the only form of conversation that's acceptable here, or can
there be discussion without winners and losers?

I said nothing of winners or losers. And I stand by my statement that
answering a question with a question, as in your prior post is a
circumlocution. It's useful if you don't have a good answer, especially if
you want to imply that you have a superior understanding without actually
saying anything useful, but it's a lousy way to have a conversation.

And that's about as far off topic as I personally want to carry this. It's
time to make dinner for the kids and then go write more Ruby code (had to get
Ruby into the post somewhere).

Kirk Haines

Bradley Kite · Aug 17, 2005

[in reply to no single post in particular]

Well I really had no idea of how much emotional ettatchment people had
to their languages of choice.

When I posted my original question, I had no idea that the later
conversation would
contain such passion, humor, sarcasim and personal insult.

Any way, my orginal post stemmed from my curiosity with regards
to what Ruby was trying to achieve:
s).

The highly non-real-world and simplistic benchmarks were not meant to
provoke the zeolots, but rather well displayed my lack of understanding
for what Ruby was trying to achieve. Of which I still dont quite understand=

Robert Klemme · Aug 17, 2005

Bradley said:
[in reply to no single post in particular]

Well I really had no idea of how much emotional ettatchment people had
to their languages of choice.

When I posted my original question, I had no idea that the later
conversation would
contain such passion, humor, sarcasim and personal insult.

There has always been heated debate about which programming language was
best (or better than another). "better" is not easily defined (and it
also changes with context) so there's plenty room for personal taste and
preference. These in turn increase the likelyhood of heated debate...

Any way, my orginal post stemmed from my curiosity with regards
to what Ruby was trying to achieve:

The highly non-real-world and simplistic benchmarks were not meant to
provoke the zeolots, but rather well displayed my lack of
understanding
for what Ruby was trying to achieve. Of which I still dont quite
understand

Hm... Maybe you should just use it for a while and see for yourself?

IMHO Ruby tries to be a pure OO dynamic language with clean syntax that
doesn't make simple tasks more complicated than necessary. And it does
this just great. My 0.02 EUR.

Kind regards

robert

Ruby performance	9	Sep 28, 2007
Ruby vs Perl performance	62	Feb 7, 2009
performance question	129	Dec 8, 2007
Is ruby's regex slower?	32	Jan 4, 2010
Concurrent code performance	10	Oct 9, 2011
rake aborted! Validation failed:	4	Oct 12, 2010
For performance, write it in C - Part 2, comparing C, Ruby and Java	48	Jul 28, 2006
Wow: jruby vs. MRE performance...	3	Apr 24, 2009

Ruby Performance

Daniel Amelang

Isaac Gouy

Isaac Gouy

Austin Ziegler

Matthew Desmarais

William James

Isaac Gouy

Brock Weaver

Joe Van Dyk

Isaac Gouy

Isaac Gouy

ogilthorpe

Kirk Haines

Austin Ziegler

Sy

Isaac Gouy

Isaac Gouy

Kirk Haines

Bradley Kite

Robert Klemme

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads