Benchmarking, part 2

J

Jerry Coffin

As promised, more benchmarking results of comparing C++ to Java. This
time around, our first target will be the strcat program. This is one
that the C++ version rather bothered me -- I'm reasonably convinced
that anybody who could have written it would have immediately realized
that there was a drastically better way to do it. Here's some code:

#include <iostream>
#include <string>
#include <cstdlib>

int main(int argc, char *argv[])
{
int i, n = ((argc == 2) ? std::atoi(argv[1]) : 3000000);
std::string str;
str.reserve(6*n);
for (i=0; i<n; i++)
str += "hello\n";
std::cout << str.length() << std::endl;
return 0;
}

Despite being noticeably shorter and simpler than the original code,
this is still pretty fast.

Unfortunately, Java seems unwilling (or perhaps unable) to take
advantage of all the available memory, causing a bit of a timing
problem -- I normally try to get the fastest program to take at least
one second, but in this case Java ran out of memory before even the
slowest version too a second. Fortunately, the difference in speed is
large enough that the winner is pretty clear, though we only have one
significant digit in the times:

Java client: .9 seconds
Java Server: .7 seconds
C++: .1 seconds

A quick check shows that even when n was increased to 4000000, the
time for C++ stayed at .1 seconds. That indicates that the margin of
victory for C++ is probably closer to 10:1 than the 7:1 shown above. I
may start adding a better timing framework into the code to reduce
problems like this.

I was going to give the results for the sieve of Eratosthenes next,
but I've decided to delay that -- if I was trying to be partial, I'd
just publish some code with results like:

Java Client: 5.6 seconds
Java Server: 5.5 seconds
C++: 1.1 seconds

and leave it at that. OTOH, this test gives a chance to show quite a
bit more, especially about how the benchmarker can affect the results
while seeming to remain impartial.
 
T

Thomas Matthews

Why do we need to know if Java, Fortran, Basic, Ada, Modula 2,
Snobol, Cobol, Lisp, Pascal, etc. are better or faster than
C++?

The problem with all of these benchmarks is that they are
platform and issue specific. Nobody is going to write
Sieve of Erasthnes {sp!} into their application / product.
If one language is better at sorting than another, nobody
is going to write only the sorting portion in that language
and the rest in another. Also, much of the performance
is based on the translation tool used. Unfortunately,
there is no single translation tool that is optimized
to compiler every language.

Many translation tools that can translate different
languages are not designed to bring about the most
efficient translation for a single language. They are
not designed for that purpose.

I have a project. If the management allows, I choose a
language for that project otherwise I'm stuck with the
house language. These benchmark tests will not help
me if I have no choice for my project.

I have a project. I am allowed to choose the language.
Most of my developers know language X. Some know
language Y. However, language Z has great benchmarks
proving it is the fastest language. Nobody knows
language Z. If I use language Z, I take a huge schedule
hit because all of my developers must be taught the
language and become accustomed to it. Also, they will
not code as efficiently as the benchmark was because
they are new to the language. However, if I choose
language X, I can get the project out the door quickly
because all my developers know it. It will be moderate
to very efficient because they are familiar with the
language. In this case, the benchmarks don't help.
The language comparisons don't help because I can't
afford the long schedule required to train the developers.

I have a project. I am allowed to choose the language.
My developers are familiar and efficient with the fastest
language. However, the customers don't notice the fantastic
speed rates due to slow hardware or the program sits and
waits for user input. Have I gained anything by using this
fast language? Not really.

I have a project. I am allowed to choose the language.
I have requirements that the language must address. One
example might be number crunching. Another might be the
ability to easily access the hardware. I may have all
the computing power I need, but I need a language that
requires less time of my developers. Perhaps I need a
language that can support custom built libraries. Or
I need a language that can run on many different platforms
with the least amount of changes to the software. Do
I care about the language being the faster than another
on sorting? Nope, not on this project. I need a language
that can handle most of my requirements, the more the
better.

I do not care about how fast a language sorts, Sam-I-am.
I do not care about how fast a language calculates PI.
If I am allowed to choose a language, it must satisfy
most of my project's requirements for a language. Most
likely, the customer will not care about the language
I use. If the performance is too slow, I will optimize
the program, not choose another language. If the
project is still too slow, I will change the platform.
Perhaps, if all else fails and I have no other recourse,
I will convert pieces over to a "faster" language, most
likely assembly.

Can people stop posting these language comparisons?
Please.


--
Thomas Matthews

C++ newsgroup welcome message:
http://www.slack.net/~shiva/welcome.txt
C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.learn.c-c++ faq:
http://www.raos.demon.uk/acllc-c++/faq.html
Other sites:
http://www.josuttis.com -- C++ STL Library book
 
G

Gianni Mariani

Thomas said:
Why do we need to know if Java, Fortran, Basic, Ada, Modula 2,
Snobol, Cobol, Lisp, Pascal, etc. are better or faster than
C++?

The concensus (or lack of) is that with C++ you are able to create
faster code because you have more control of exactly how the compiler
generates code. The assertion for years has been that Java VM's will be
able to generate the same or better performance as C++.

The classic pissing contest. Sit back and enjoy !
 
J

Jerry Coffin

Thomas Matthews said:
Why do we need to know if Java, Fortran, Basic, Ada, Modula 2,
Snobol, Cobol, Lisp, Pascal, etc. are better or faster than
C++?

We don't need to -- but from the size of threads devoted to such
things, it's obvious that quite a few people CARE, even if they don't
need to know. When you get down to it, people obviously got along
without computers for a long time; nothing related to computer
programming can possibly be classified as a need.
The problem with all of these benchmarks is that they are
platform and issue specific. Nobody is going to write
Sieve of Erasthnes {sp!} into their application / product.

I've used a number of programs that included a sieve of Eratosthenes.
You may not have, and you may not realize when, where or how it would
be useful, but your ignorance implies nothing about the algorithm
itself.

[ LONG list of examples of why one person doesn't care elided ... ]

From the sound of things, 1) your environment is highly constrained,
and 2) you never write anything where speed really matters.

That's fine -- but the same is not true of everybody else in the
world.

Fact: people DO mix languages to get the best capabilties of each.
People do write code where performance matters -- and when it does, it
usually matters a LOT.
Can people stop posting these language comparisons?

Look in the mirror and you'll see who was posting off-topic.

OOTC: while it's true that use of the Sieve of Eratosthenes in a
program is _fairly_ unusual, it's also true that the Sieve uses
operations that are used heavily in MANY other areas. I know of no
current C++ compiler that has special code to recognize the Sieve and
produce special code just for it. A compiler that produces good code
for the sieve can generally be depended upon to produce at least
pretty decent code for quite a few other algorithms as well.
 
K

Karl Heinz Buchegger

Jerry said:
[ LONG list of examples of why one person doesn't care elided ... ]

From the sound of things, 1) your environment is highly constrained,
and 2) you never write anything where speed really matters.

That's fine -- but the same is not true of everybody else in the
world.

Fact: people DO mix languages to get the best capabilties of each.
People do write code where performance matters -- and when it does, it
usually matters a LOT.

But do you agree:
Those for whom speed really matters are not those posting such
questions in newgroups. Those are experienced enough to know
what to do.
 
J

Jerry Coffin

[ ... ]
But do you agree:
Those for whom speed really matters are not those posting such
questions in newgroups. Those are experienced enough to know
what to do.

No, not necessarily. I could wish there was a strong correlation, but
experience indicates otherwise.

I guess I could see objections if I was posting code that included
every dirty trick possible to eke out a few cycles here and there, but
exactly the opposite is the case -- the code I've posted has been
simple and straightforward.

Looking at the other big thread on the subject, it struck me that
there was a nearly lot of arguing, but little or nothing in the way of
facts to support any of it. I thought some verifiable, factual data
might be of interest, but apparently I was wrong. Nobody's shown any
positive interest in this thread, so I guess I'll drop it for lack of
interest.
 
J

John Harrison

Looking at the other big thread on the subject, it struck me that
there was a nearly lot of arguing, but little or nothing in the way of
facts to support any of it. I thought some verifiable, factual data
might be of interest, but apparently I was wrong. Nobody's shown any
positive interest in this thread, so I guess I'll drop it for lack of
interest.

Facts get in the way of a good argument.

john
 
D

David Harmon

On 22 Jun 2004 08:00:47 -0700 in comp.lang.c++, (e-mail address removed) (Jerry
Coffin) wrote,
might be of interest, but apparently I was wrong. Nobody's shown any
positive interest in this thread, so I guess I'll drop it for lack of
interest.

I have been reading with interest.

Let's see, if the Java guys do not count the time that it takes the JVM
to load, but only from the time when main() begins, then it is only fair
if the C++ program does all the work in the constructor of a static
object and doesn't count that, right?

See, you may be better off without my contribution.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,764
Messages
2,569,567
Members
45,042
Latest member
icassiem

Latest Threads

Top