Benchmarking C compilers for embedded systems

Discussion in 'C Programming' started by Philipp Klaus Krause, Sep 19, 2011.

  1. I'd like to compare C compilers targetting embedded systems - the kind
    that only have a few KB of RAM, etc.

    However there seems to be a lack of software I could use. Waht I'm
    looking for should be free, written (mostly) in standard C, be not
    totally different from typical embedded applications.

    So far I see:
    - dhrystone
    - The Contiki OS
    - the files I've been using to track sdcc over revisions at
    https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list

    Are there any other suitable benchmarks I could look at? They don't have
    to be designed as benchmarks, other software, like the Contiki OS
    mentioned above is OK, too. THey don't even have to "run", so libraries
    would be OK, too.

    Philipp
     
    Philipp Klaus Krause, Sep 19, 2011
    #1
    1. Advertising

  2. Philipp Klaus Krause

    Abhishek Guest

    On Sep 19, 10:43 pm, Philipp Klaus Krause <> wrote:
    > I'd like to compare C compilers targetting embedded systems - the kind
    > that only have a few KB of RAM, etc.
    >
    > However there seems to be a lack of software I could use. Waht I'm
    > looking for should be free, written (mostly) in standard C, be not
    > totally different from typical embedded applications.
    >
    > So far I see:
    > - dhrystone
    > - The Contiki OS
    > - the files I've been using to track sdcc over revisions athttps://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
    >
    > Are there any other suitable benchmarks I could look at? They don't have
    > to be designed as benchmarks, other software, like the Contiki OS
    > mentioned above is OK, too. THey don't even have to "run", so libraries
    > would be OK, too.
    >
    > Philipp


    Hi Philipp,

    Have you tried Coremark from EEMBC (www.coremark.org)?
    It is available for free.
    The other benchmarking suites from EEMBC require you to pay small
    membership fee i guess.

    Regards,
    Abhishek
     
    Abhishek, Sep 20, 2011
    #2
    1. Advertising

  3. Am 20.09.2011 09:50, schrieb Abhishek:

    >
    > Have you tried Coremark from EEMBC (www.coremark.org)?
    > It is available for free.
    > The other benchmarking suites from EEMBC require you to pay small
    > membership fee i guess.


    I now did, however the license is a problem:
    1) The license. I can understand when benchmarking organizations don't
    distribute older versions of their benchmarks and no longer accept
    reults generated using an old version, like SPEC does. However the
    Coremark license explicitly forbids you from just running or even
    keeping a copy of older versions: You loose all rights to them each time
    a new version is released.

    I'll probably use it anyway and just hope they don't release a new
    version anytime soon. Still, I'm looking for further benchmarks. The
    Contiki OS seems perfect, but I'd like to find more of that kind.

    Philipp
     
    Philipp Klaus Krause, Sep 21, 2011
    #3
  4. Philipp Klaus Krause

    Ian Collins Guest

    On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
    > Am 20.09.2011 09:50, schrieb Abhishek:
    >
    >>
    >> Have you tried Coremark from EEMBC (www.coremark.org)?
    >> It is available for free.
    >> The other benchmarking suites from EEMBC require you to pay small
    >> membership fee i guess.

    >
    > I now did, however the license is a problem:
    > 1) The license. I can understand when benchmarking organizations don't
    > distribute older versions of their benchmarks and no longer accept
    > reults generated using an old version, like SPEC does. However the
    > Coremark license explicitly forbids you from just running or even
    > keeping a copy of older versions: You loose all rights to them each time
    > a new version is released.
    >
    > I'll probably use it anyway and just hope they don't release a new
    > version anytime soon. Still, I'm looking for further benchmarks. The
    > Contiki OS seems perfect, but I'd like to find more of that kind.


    What about your own code?

    --
    Ian Collins
     
    Ian Collins, Sep 21, 2011
    #4
  5. Am 21.09.2011 11:06, schrieb Ian Collins:
    > On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
    >> Am 20.09.2011 09:50, schrieb Abhishek:
    >>
    >>>
    >>> Have you tried Coremark from EEMBC (www.coremark.org)?
    >>> It is available for free.
    >>> The other benchmarking suites from EEMBC require you to pay small
    >>> membership fee i guess.

    >>
    >> I now did, however the license is a problem:
    >> 1) The license. I can understand when benchmarking organizations don't
    >> distribute older versions of their benchmarks and no longer accept
    >> reults generated using an old version, like SPEC does. However the
    >> Coremark license explicitly forbids you from just running or even
    >> keeping a copy of older versions: You loose all rights to them each time
    >> a new version is released.
    >>
    >> I'll probably use it anyway and just hope they don't release a new
    >> version anytime soon. Still, I'm looking for further benchmarks. The
    >> Contiki OS seems perfect, but I'd like to find more of that kind.

    >
    > What about your own code?
    >


    I'd like to get results that are useful to other people as well. I
    intend to use some of my code as one of the benchmarks benchmark, but in
    general I'd prefer to use more generally accepted stuff.

    Philipp
     
    Philipp Klaus Krause, Sep 22, 2011
    #5
  6. Am 19.09.2011 22:10, schrieb David Brown:
    > On 19/09/11 19:43, Philipp Klaus Krause wrote:
    >> I'd like to compare C compilers targetting embedded systems - the kind
    >> that only have a few KB of RAM, etc.
    >>
    >> However there seems to be a lack of software I could use. Waht I'm
    >> looking for should be free, written (mostly) in standard C, be not
    >> totally different from typical embedded applications.
    >>
    >> So far I see:
    >> - dhrystone
    >> - The Contiki OS
    >> - the files I've been using to track sdcc over revisions at
    >> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list
    >>
    >> Are there any other suitable benchmarks I could look at? They don't have
    >> to be designed as benchmarks, other software, like the Contiki OS
    >> mentioned above is OK, too. THey don't even have to "run", so libraries
    >> would be OK, too.
    >>
    >> Philipp

    >
    > There are no good general benchmarks for small embedded systems - partly
    > because it would be a great deal of work (standard C is not standard for
    > many compilers - you need to use compiler-specific features to get the
    > best out of them), and partly because it would be of very little use.
    >
    > What you want to know is which tools are the best for /your/ code, not
    > some general code written for completely different uses. Most tool
    > vendors provide free evaluation software - test them out on your own code.
    >


    I don't just want to know which tool is best for my code. I want to
    know, which tool is best for typical code.
    I am one of the developers of the free sdcc compiler and currently
    working on register and stack allcoation. I see that what I implemented
    results in better code when compiling my own programs, Contiki,
    dhrystone, Coremark. But I'd like to know if there is any code for which
    the new approach performs worse than the on epreviously used in sdcc.
    And I'd like to know it know, instead of having to wait for bug reports
    about code size regressions coming in after the next release.

    Philipp
     
    Philipp Klaus Krause, Sep 22, 2011
    #6
  7. Philipp Klaus Krause

    Ian Collins Guest

    On 09/22/11 07:12 PM, Philipp Klaus Krause wrote:
    > Am 21.09.2011 11:06, schrieb Ian Collins:
    >> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
    >>> Am 20.09.2011 09:50, schrieb Abhishek:
    >>>
    >>>>
    >>>> Have you tried Coremark from EEMBC (www.coremark.org)?
    >>>> It is available for free.
    >>>> The other benchmarking suites from EEMBC require you to pay small
    >>>> membership fee i guess.
    >>>
    >>> I now did, however the license is a problem:
    >>> 1) The license. I can understand when benchmarking organizations don't
    >>> distribute older versions of their benchmarks and no longer accept
    >>> reults generated using an old version, like SPEC does. However the
    >>> Coremark license explicitly forbids you from just running or even
    >>> keeping a copy of older versions: You loose all rights to them each time
    >>> a new version is released.
    >>>
    >>> I'll probably use it anyway and just hope they don't release a new
    >>> version anytime soon. Still, I'm looking for further benchmarks. The
    >>> Contiki OS seems perfect, but I'd like to find more of that kind.

    >>
    >> What about your own code?
    >>

    >
    > I'd like to get results that are useful to other people as well. I
    > intend to use some of my code as one of the benchmarks benchmark, but in
    > general I'd prefer to use more generally accepted stuff.


    Every target and code base combination will behave differently, even on
    different members of the same processor family. This makes general
    benchmarks useless for embedded systems.

    --
    Ian Collins
     
    Ian Collins, Sep 22, 2011
    #7
  8. Philipp Klaus Krause

    Arlet Ottens Guest

    On 09/19/2011 07:43 PM, Philipp Klaus Krause wrote:
    > I'd like to compare C compilers targetting embedded systems - the kind
    > that only have a few KB of RAM, etc.
    >
    > However there seems to be a lack of software I could use. Waht I'm
    > looking for should be free, written (mostly) in standard C, be not
    > totally different from typical embedded applications.
    >
    > So far I see:
    > - dhrystone
    > - The Contiki OS
    > - the files I've been using to track sdcc over revisions at
    > https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list
    >
    > Are there any other suitable benchmarks I could look at? They don't have
    > to be designed as benchmarks, other software, like the Contiki OS
    > mentioned above is OK, too. THey don't even have to "run", so libraries
    > would be OK, too.


    Elm's FAT filesystem implementation:

    http://elm-chan.org/fsw/ff/00index_e.html
     
    Arlet Ottens, Sep 22, 2011
    #8
  9. Philipp Klaus Krause

    Walter Banks Guest

    Philipp Klaus Krause wrote:

    > I'd like to compare C compilers targetting embedded systems - the kind
    > that only have a few KB of RAM, etc.
    >
    > However there seems to be a lack of software I could use. Waht I'm
    > looking for should be free, written (mostly) in standard C, be not
    > totally different from typical embedded applications.
    >
    > So far I see:
    > - dhrystone
    > - The Contiki OS
    > - the files I've been using to track sdcc over revisions at
    > https://sourceforge.net/apps/trac/sdcc/wiki/Philipp's TODO list
    >
    > Are there any other suitable benchmarks I could look at? They don't have
    > to be designed as benchmarks, other software, like the Contiki OS
    > mentioned above is OK, too. THey don't even have to "run", so libraries
    > would be OK, too.
    >


    Philipp,

    There are tons of embedded systems benchmarks on the internet. In our
    compiler work we have found the most informative benchmarks are application
    type benchmarks close to the actual intended application.

    The application fragment type benchmarks (fft, filters, math packs) are
    useful generally is seeing how focused a compiler is in optimizing a
    specific application area.

    Generic benchmarks would likely need to also have some definition of the
    intended application area.

    There isn't much meaning in comparing the performance of processors used in
    mice and processors used in home appliance controllers. Both are non hosted
    embedded controllers with a few K of code but very different requirements.
    They may even be members of the same family.

    Walter..
     
    Walter Banks, Sep 30, 2011
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Mike
    Replies:
    2
    Views:
    636
  2. Martin Schoeberl

    Benchmarking embedded Java

    Martin Schoeberl, Sep 24, 2004, in forum: Java
    Replies:
    1
    Views:
    487
    Martin Schoeberl
    Sep 25, 2004
  3. Replies:
    3
    Views:
    459
    Jack Klein
    May 12, 2005
  4. geletine

    commercial c compilers vs free c compilers

    geletine, Jul 2, 2006, in forum: C Programming
    Replies:
    33
    Views:
    1,321
  5. Tomás Ó hÉilidhe

    Should embedded systems compilers be trusted

    Tomás Ó hÉilidhe, Apr 20, 2008, in forum: C Programming
    Replies:
    16
    Views:
    457
    Ian Collins
    Apr 23, 2008
Loading...

Share This Page