Testing a compiler for standards compliance

Discussion in 'C++' started by Matt Gregory, Jun 13, 2014.

  1. Matt Gregory

    Matt Gregory Guest

    Is there some sort of test suite that checks how standards-compliant a particular C++ compiler is?
     
    Matt Gregory, Jun 13, 2014
    #1
    1. Advertisements

  2. On 6/13/2014 3:46 PM, Matt Gregory wrote:
    > Is there some sort of test suite that checks how standards-compliant
    > a particular C++ compiler is?


    I think this question comes up now and again, but I don't remember it
    answered positively, ever.

    There is a reason for absence of such a test suite, and it's pretty
    simple, really. There is nobody (besides compiler vendors themselves)
    interested enough in investing what would turn out a serious amount of
    work into creating such a suite. So, independently it is not likely to
    exist, and those suites that compiler creators have for their internal
    consumption, aren't published because they are proprietary information,
    not to mention that they are likely specific to the testing tools in use
    (which are probably proprietary as well).

    V
    --
    I do not respond to top-posted replies, please don't ask
     
    Victor Bazarov, Jun 13, 2014
    #2
    1. Advertisements

  3. On 6/16/2014 10:43 AM, Scott Lurndal wrote:
    > Victor Bazarov <> writes:
    >> On 6/13/2014 3:46 PM, Matt Gregory wrote:
    >>> Is there some sort of test suite that checks how standards-compliant
    >>> a particular C++ compiler is?

    >>
    >> I think this question comes up now and again, but I don't remember it
    >> answered positively, ever.
    >>
    >> There is a reason for absence of such a test suite, and it's pretty
    >> simple, really. There is nobody (besides compiler vendors themselves)
    >> interested enough in investing what would turn out a serious amount of
    >> work into creating such a suite. So, independently it is not likely to
    >> exist, and those suites that compiler creators have for their internal
    >> consumption, aren't published because they are proprietary information,
    >> not to mention that they are likely specific to the testing tools in use
    >> (which are probably proprietary as well).
    >>

    >
    > FWIW, GCC's test suite is available, but specific to GCC.


    So, if one wanted to create something independent from the vendor, one
    might use GCC's test suite as the starting point.

    V
    --
    I do not respond to top-posted replies, please don't ask
     
    Victor Bazarov, Jun 16, 2014
    #3
  4. Matt Gregory

    David Brown Guest

    On 16/06/14 17:43, Victor Bazarov wrote:
    > On 6/16/2014 10:43 AM, Scott Lurndal wrote:
    >> Victor Bazarov <> writes:
    >>> On 6/13/2014 3:46 PM, Matt Gregory wrote:
    >>>> Is there some sort of test suite that checks how standards-compliant
    >>>> a particular C++ compiler is?
    >>>
    >>> I think this question comes up now and again, but I don't remember it
    >>> answered positively, ever.
    >>>
    >>> There is a reason for absence of such a test suite, and it's pretty
    >>> simple, really. There is nobody (besides compiler vendors themselves)
    >>> interested enough in investing what would turn out a serious amount of
    >>> work into creating such a suite. So, independently it is not likely to
    >>> exist, and those suites that compiler creators have for their internal
    >>> consumption, aren't published because they are proprietary information,
    >>> not to mention that they are likely specific to the testing tools in use
    >>> (which are probably proprietary as well).
    >>>

    >>
    >> FWIW, GCC's test suite is available, but specific to GCC.

    >
    > So, if one wanted to create something independent from the vendor, one
    > might use GCC's test suite as the starting point.
    >


    gcc's test suite is rather specific to gcc, but it might be possible to
    adapt it. Certainly gdb and a few other gnu projects use the same basic
    test system. llvm may have a similar test suite, but I know very little
    about that.

    However, gcc's test suite is not for standards compliance as such - it
    is a feature and regression test suite. It will indirectly test
    standards compliance to some extent (in the sense that compiling legal
    code is a feature to test), but it is certainly not a full compliance
    test suite. In particular, if the gcc folks have misunderstood the
    standards somewhere then they will use the same interpretation in the
    compiler and the test suite. For serious compliance testing, you need
    an independent test suite.

    There are a few big commercial compiler compliance test suites around.
    "Plum Hall" is one that I have heard of (though I have never used these
    myself). These sorts of suites are expensive, running them is hard and
    time-consuming, they are often a bit behind the latest language
    standards, and they impose restrictions on the toolset (such as
    disabling compiler extensions). But they are used by big commercial
    toolchain vendors such as Green Hills and IAR, and in industries like
    automotive and aeronautic software development they are used as legal
    props (to show "due diligence" regarding tool choice). They don't prove
    the compiler is bug-free, of course - merely that it correctly
    interprets the test source code. Compiler vendors will give you copies
    of their Plum Hall compliance certificates if you need them - for a fee,
    of course, along with an NDA covering any non-compliance issues found.
    (CodeSourcery will provide such reports for gcc.)
     
    David Brown, Jun 17, 2014
    #4
    1. Advertisements

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.
Similar Threads
  1. JavaEnquirer

    Eclipse compiler compliance

    JavaEnquirer, May 24, 2005, in forum: Java
    Replies:
    2
    Views:
    7,291
    Dale King
    May 31, 2005
  2. Michael Andersson

    Compiler standard compliance comparison

    Michael Andersson, Jul 31, 2003, in forum: C++
    Replies:
    3
    Views:
    553
    Greg P.
    Jul 31, 2003
  3. christopher diggins
    Replies:
    9
    Views:
    609
    Claudio Puviani
    Apr 16, 2004
  4. Matthew Del Buono
    Replies:
    7
    Views:
    1,735
    Matthew Del Buono
    May 14, 2004
  5. ozbear

    Standards compliance

    ozbear, Nov 26, 2003, in forum: C Programming
    Replies:
    5
    Views:
    555
    Dan Pop
    Nov 27, 2003
  6. Steve
    Replies:
    2
    Views:
    450
  7. Mike Owen

    Web Standards Compliance - Autopostback / Javascript off

    Mike Owen, May 2, 2006, in forum: ASP .Net Web Controls
    Replies:
    5
    Views:
    489
    Steven Cheng[MSFT]
    May 8, 2006
  8. John Doe

    Standards compliance and show-hide div

    John Doe, Jan 27, 2005, in forum: Javascript
    Replies:
    2
    Views:
    277
    John Doe
    Jan 27, 2005
Loading...