Benchmarking C compilers for embedded systems

  • Thread starter Philipp Klaus Krause
  • Start date
P

Philipp Klaus Krause

I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp
 
A

Abhishek

I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions athttps://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp

Hi Philipp,

Have you tried Coremark from EEMBC (www.coremark.org)?
It is available for free.
The other benchmarking suites from EEMBC require you to pay small
membership fee i guess.

Regards,
Abhishek
 
P

Philipp Klaus Krause

Am 20.09.2011 09:50, schrieb Abhishek:
Have you tried Coremark from EEMBC (www.coremark.org)?
It is available for free.
The other benchmarking suites from EEMBC require you to pay small
membership fee i guess.

I now did, however the license is a problem:
1) The license. I can understand when benchmarking organizations don't
distribute older versions of their benchmarks and no longer accept
reults generated using an old version, like SPEC does. However the
Coremark license explicitly forbids you from just running or even
keeping a copy of older versions: You loose all rights to them each time
a new version is released.

I'll probably use it anyway and just hope they don't release a new
version anytime soon. Still, I'm looking for further benchmarks. The
Contiki OS seems perfect, but I'd like to find more of that kind.

Philipp
 
I

Ian Collins

Am 20.09.2011 09:50, schrieb Abhishek:


I now did, however the license is a problem:
1) The license. I can understand when benchmarking organizations don't
distribute older versions of their benchmarks and no longer accept
reults generated using an old version, like SPEC does. However the
Coremark license explicitly forbids you from just running or even
keeping a copy of older versions: You loose all rights to them each time
a new version is released.

I'll probably use it anyway and just hope they don't release a new
version anytime soon. Still, I'm looking for further benchmarks. The
Contiki OS seems perfect, but I'd like to find more of that kind.

What about your own code?
 
P

Philipp Klaus Krause

Am 21.09.2011 11:06, schrieb Ian Collins:
What about your own code?

I'd like to get results that are useful to other people as well. I
intend to use some of my code as one of the benchmarks benchmark, but in
general I'd prefer to use more generally accepted stuff.

Philipp
 
P

Philipp Klaus Krause

Am 19.09.2011 22:10, schrieb David Brown:
There are no good general benchmarks for small embedded systems - partly
because it would be a great deal of work (standard C is not standard for
many compilers - you need to use compiler-specific features to get the
best out of them), and partly because it would be of very little use.

What you want to know is which tools are the best for /your/ code, not
some general code written for completely different uses. Most tool
vendors provide free evaluation software - test them out on your own code.

I don't just want to know which tool is best for my code. I want to
know, which tool is best for typical code.
I am one of the developers of the free sdcc compiler and currently
working on register and stack allcoation. I see that what I implemented
results in better code when compiling my own programs, Contiki,
dhrystone, Coremark. But I'd like to know if there is any code for which
the new approach performs worse than the on epreviously used in sdcc.
And I'd like to know it know, instead of having to wait for bug reports
about code size regressions coming in after the next release.

Philipp
 
I

Ian Collins

Am 21.09.2011 11:06, schrieb Ian Collins:

I'd like to get results that are useful to other people as well. I
intend to use some of my code as one of the benchmarks benchmark, but in
general I'd prefer to use more generally accepted stuff.

Every target and code base combination will behave differently, even on
different members of the same processor family. This makes general
benchmarks useless for embedded systems.
 
A

Arlet Ottens

I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Elm's FAT filesystem implementation:

http://elm-chan.org/fsw/ff/00index_e.html
 
W

Walter Banks

Philipp said:
I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp,

There are tons of embedded systems benchmarks on the internet. In our
compiler work we have found the most informative benchmarks are application
type benchmarks close to the actual intended application.

The application fragment type benchmarks (fft, filters, math packs) are
useful generally is seeing how focused a compiler is in optimizing a
specific application area.

Generic benchmarks would likely need to also have some definition of the
intended application area.

There isn't much meaning in comparing the performance of processors used in
mice and processors used in home appliance controllers. Both are non hosted
embedded controllers with a few K of code but very different requirements.
They may even be members of the same family.

Walter..
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top