So, if one wanted to create something independent from the vendor, one
might use GCC's test suite as the starting point.
gcc's test suite is rather specific to gcc, but it might be possible to
adapt it. Certainly gdb and a few other gnu projects use the same basic
test system. llvm may have a similar test suite, but I know very little
However, gcc's test suite is not for standards compliance as such - it
is a feature and regression test suite. It will indirectly test
standards compliance to some extent (in the sense that compiling legal
code is a feature to test), but it is certainly not a full compliance
test suite. In particular, if the gcc folks have misunderstood the
standards somewhere then they will use the same interpretation in the
compiler and the test suite. For serious compliance testing, you need
an independent test suite.
There are a few big commercial compiler compliance test suites around.
"Plum Hall" is one that I have heard of (though I have never used these
myself). These sorts of suites are expensive, running them is hard and
time-consuming, they are often a bit behind the latest language
standards, and they impose restrictions on the toolset (such as
disabling compiler extensions). But they are used by big commercial
toolchain vendors such as Green Hills and IAR, and in industries like
automotive and aeronautic software development they are used as legal
props (to show "due diligence" regarding tool choice). They don't prove
the compiler is bug-free, of course - merely that it correctly
interprets the test source code. Compiler vendors will give you copies
of their Plum Hall compliance certificates if you need them - for a fee,
of course, along with an NDA covering any non-compliance issues found.
(CodeSourcery will provide such reports for gcc.)