Testing a compiler for standards compliance

M

Matt Gregory

Is there some sort of test suite that checks how standards-compliant a particular C++ compiler is?
 
V

Victor Bazarov

Is there some sort of test suite that checks how standards-compliant
a particular C++ compiler is?

I think this question comes up now and again, but I don't remember it
answered positively, ever.

There is a reason for absence of such a test suite, and it's pretty
simple, really. There is nobody (besides compiler vendors themselves)
interested enough in investing what would turn out a serious amount of
work into creating such a suite. So, independently it is not likely to
exist, and those suites that compiler creators have for their internal
consumption, aren't published because they are proprietary information,
not to mention that they are likely specific to the testing tools in use
(which are probably proprietary as well).

V
 
V

Victor Bazarov

FWIW, GCC's test suite is available, but specific to GCC.

So, if one wanted to create something independent from the vendor, one
might use GCC's test suite as the starting point.

V
 
D

David Brown

So, if one wanted to create something independent from the vendor, one
might use GCC's test suite as the starting point.

gcc's test suite is rather specific to gcc, but it might be possible to
adapt it. Certainly gdb and a few other gnu projects use the same basic
test system. llvm may have a similar test suite, but I know very little
about that.

However, gcc's test suite is not for standards compliance as such - it
is a feature and regression test suite. It will indirectly test
standards compliance to some extent (in the sense that compiling legal
code is a feature to test), but it is certainly not a full compliance
test suite. In particular, if the gcc folks have misunderstood the
standards somewhere then they will use the same interpretation in the
compiler and the test suite. For serious compliance testing, you need
an independent test suite.

There are a few big commercial compiler compliance test suites around.
"Plum Hall" is one that I have heard of (though I have never used these
myself). These sorts of suites are expensive, running them is hard and
time-consuming, they are often a bit behind the latest language
standards, and they impose restrictions on the toolset (such as
disabling compiler extensions). But they are used by big commercial
toolchain vendors such as Green Hills and IAR, and in industries like
automotive and aeronautic software development they are used as legal
props (to show "due diligence" regarding tool choice). They don't prove
the compiler is bug-free, of course - merely that it correctly
interprets the test source code. Compiler vendors will give you copies
of their Plum Hall compliance certificates if you need them - for a fee,
of course, along with an NDA covering any non-compliance issues found.
(CodeSourcery will provide such reports for gcc.)
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top