Testing a compiler for standards compliance

Discussion in 'C++' started by Matt Gregory, Jun 13, 2014.

  1. Matt Gregory

    Matt Gregory Guest

    Is there some sort of test suite that checks how standards-compliant a particular C++ compiler is?
    Matt Gregory, Jun 13, 2014
    1. Advertisements

  2. I think this question comes up now and again, but I don't remember it
    answered positively, ever.

    There is a reason for absence of such a test suite, and it's pretty
    simple, really. There is nobody (besides compiler vendors themselves)
    interested enough in investing what would turn out a serious amount of
    work into creating such a suite. So, independently it is not likely to
    exist, and those suites that compiler creators have for their internal
    consumption, aren't published because they are proprietary information,
    not to mention that they are likely specific to the testing tools in use
    (which are probably proprietary as well).

    Victor Bazarov, Jun 13, 2014
    1. Advertisements

  3. So, if one wanted to create something independent from the vendor, one
    might use GCC's test suite as the starting point.

    Victor Bazarov, Jun 16, 2014
  4. Matt Gregory

    David Brown Guest

    gcc's test suite is rather specific to gcc, but it might be possible to
    adapt it. Certainly gdb and a few other gnu projects use the same basic
    test system. llvm may have a similar test suite, but I know very little
    about that.

    However, gcc's test suite is not for standards compliance as such - it
    is a feature and regression test suite. It will indirectly test
    standards compliance to some extent (in the sense that compiling legal
    code is a feature to test), but it is certainly not a full compliance
    test suite. In particular, if the gcc folks have misunderstood the
    standards somewhere then they will use the same interpretation in the
    compiler and the test suite. For serious compliance testing, you need
    an independent test suite.

    There are a few big commercial compiler compliance test suites around.
    "Plum Hall" is one that I have heard of (though I have never used these
    myself). These sorts of suites are expensive, running them is hard and
    time-consuming, they are often a bit behind the latest language
    standards, and they impose restrictions on the toolset (such as
    disabling compiler extensions). But they are used by big commercial
    toolchain vendors such as Green Hills and IAR, and in industries like
    automotive and aeronautic software development they are used as legal
    props (to show "due diligence" regarding tool choice). They don't prove
    the compiler is bug-free, of course - merely that it correctly
    interprets the test source code. Compiler vendors will give you copies
    of their Plum Hall compliance certificates if you need them - for a fee,
    of course, along with an NDA covering any non-compliance issues found.
    (CodeSourcery will provide such reports for gcc.)
    David Brown, Jun 17, 2014
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.