Is there any new C++ compiler standard conformance test result?

Discussion in 'C++' started by Tao Wang, Nov 6, 2005.

  1. Tao Wang

    Tao Wang Guest

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Hi,

    I saw cuj's conformance roundup, but the result is quite old. I think
    many people like me want to know newer c++ standard conformance test
    result. Especially, I want to know the compiler conformance of vc7.1,
    vc8.0, gcc 4.x and intel c++, and also each implementation of STL. The
    world is moving on, right? :)

    Thanks.

    Dancefire
    ???
    CCNA
    http://www.dancefires.com/
    http://blog.csdn.net/dancefire/
    MSN:
    I am interested in Operating System, Embedded System and Network Security.
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.1 (GNU/Linux)
    Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

    iD8DBQFDbpaERS5AkKgtcCcRAhLaAKCqdH2hhqpNNeGG4y8it6b4qYdbqQCggSAH
    ofGLUJUDdVQsPvsPK5ncb4k=
    =WQkH
    -----END PGP SIGNATURE-----
     
    Tao Wang, Nov 6, 2005
    #1
    1. Advertising

  2. Tao Wang

    Greg Guest

    Tao Wang wrote:
    > -----BEGIN PGP SIGNED MESSAGE-----
    > Hash: SHA1
    >
    > Hi,
    >
    > I saw cuj's conformance roundup, but the result is quite old. I think
    > many people like me want to know newer c++ standard conformance test
    > result. Especially, I want to know the compiler conformance of vc7.1,
    > vc8.0, gcc 4.x and intel c++, and also each implementation of STL. The
    > world is moving on, right? :)


    Well, it's always possible that cuj will publish another survey. In the
    meantime, a fairly good barometer of compiler compliance would probably
    be the Boost Release summary: (http://tinyurl.com/7aq2t - be patient,
    page takes awhile to load), The information though has to be mined from
    these build reports. Figuring out the limitations of particular
    compiler requires sifting through the failures and error messages to
    understand the problem.

    Alternately, much can be learned from most compilers' documentation -
    although again it is often necessary to consult extensive release notes
    for the latest information. Compiler vendors for competitive reasons
    may not always wish to highlight areas that put them at a competitive
    disadvantage vis-a-vis other compilers. I'm not suggesting that there
    is anything untoward in doing so, and indeed compiler makers tend to be
    remarkably forthcoming about the features and limitations of their
    products, at least in my limited experience.

    And lastly I would note that the best conformance suites are likely to
    be commercial. When I worked for a C embedded systems compiler company,
    each release of the compiler was tested against the Plum Hall
    validation suite - a series of tortuous tests designed to thoroughly
    vet the compiler's ANSI C's conformance. Needless to say, the results
    of a run were used to fix bugs in the compiler before its release; so
    the company had every reason to keep these results private.
    Furthermore, Plum Hall put a lot of work and a lot of expertise into
    writing their test suites and, given the small size of the market for C
    and C++ test suites in general, they are probably too expensive for
    anyone other than a compiler maker to license.

    Greg
     
    Greg, Nov 7, 2005
    #2
    1. Advertising

  3. Tao Wang

    Greg Comeau Guest

    In article <8Ewbf.10445$>,
    Tao Wang <> wrote:
    >I saw cuj's conformance roundup, but the result is quite old. I think
    >many people like me want to know newer c++ standard conformance test
    >result. Especially, I want to know the compiler conformance of vc7.1,
    >vc8.0, gcc 4.x and intel c++, and also each implementation of STL. The
    >world is moving on, right? :)


    I have prodded a number of publications to do said, but
    as with any ongoing project, that can take considerable work,
    and so it never materializes. Even when smaller scopes,
    and different themes to consider, are mentioned.
    --
    Greg Comeau / Celebrating 20 years of Comeauity!
    Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
    World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
    Comeau C/C++ with Dinkumware's Libraries... Have you tried it?
     
    Greg Comeau, Nov 7, 2005
    #3
  4. Tao Wang

    Greg Comeau Guest

    In article <>,
    Greg <> wrote:
    >a fairly good barometer of compiler compliance would probably
    >be the Boost Release summary....


    The Boost folks themselves disagree with such a statement,
    so we should too.
    --
    Greg Comeau / Celebrating 20 years of Comeauity!
    Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
    World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
    Comeau C/C++ with Dinkumware's Libraries... Have you tried it?
     
    Greg Comeau, Nov 7, 2005
    #4
  5. Tao Wang

    Greg Guest

    Greg Comeau wrote:
    > In article <>,
    > Greg <> wrote:
    > >a fairly good barometer of compiler compliance would probably
    > >be the Boost Release summary....

    >
    > The Boost folks themselves disagree with such a statement,
    > so we should too.


    Granted, the purpose of the Release pages is not to test compiler
    compliance, nor do the Boost libraries provide any type of systematic
    test coverage. Furthermore the compiler is not the only tool used in a
    build, so a failure may be a problem with the linker, for example. As I
    tried to communicate in my original post, the information here is quite
    raw. To find data that would help answer the poster's question would
    require carefully sifting through all of these results and would
    necessarily entail discarding a lot of irrelevant information.

    Nevertheless, it would be difficult to assemble another set of C++
    source files on one's own that would make use of so many advanced
    language features as the do the Boost libraries. So it must be possible
    to learn something about a compiler by having it build this set of
    libraries. How useful that information may be is another matter.
    Certainly, it's unlikely that one's own code would look much like a
    Boost library, so it is valid to argue that any information gleaned
    from the Boost build results is of academic interest. In strictly
    practical terms, I doubt it makes much difference to the average C++
    programmer whether:


    boost::wave::cpplexer::lex_input_interface<boost::wave::cpplexer::lex_token<T2>>
    *boost::wave::cpplexer::new_lexer_gen<T1, T2>::new_lexer(
    const T1 &,
    const T1 &,
    const T2 &,
    boost::wave::language_support)
    [with T1=char *,
    T2=boost::wave::util::file_position<boost::wave::util::
    flex_string<char, std::char_traits<char>,
    std::allocator<char>,
    boost::wave::util::CowString<boost::wave::util::
    AllocatorStringStorage<char,std::allocator<char>>, char
    *>>>

    is multiply-defined or not.

    But with the maturity of today's C++ compilers, the issue of language
    compliance really has been pushed to the more esoteric language
    constructs. The days when compiling "hello, world" in order to
    reasonably assess a C++ compiler have passed. Granted there are easier
    ways to answer the question than sifting through reams of build logs,
    but if those methods are not readily available, then answering the
    question can still be done I believe, but it would require a lot of
    work.

    Greg
     
    Greg, Nov 7, 2005
    #5
  6. Tao Wang

    Greg Comeau Guest

    In article <>,
    Greg <> wrote:
    >Greg Comeau wrote:
    >> In article <>,
    >> Greg <> wrote:
    >> >a fairly good barometer of compiler compliance would probably
    >> >be the Boost Release summary....

    >>
    >> The Boost folks themselves disagree with such a statement,
    >> so we should too.

    >
    >Granted, the purpose of the Release pages is not to test compiler
    >compliance, nor do the Boost libraries provide any type of systematic
    >test coverage. Furthermore the compiler is not the only tool used in a
    >build, so a failure may be a problem with the linker, for example.


    Yes, that can be so. But...

    >As I
    >tried to communicate in my original post, the information here is quite
    >raw. To find data that would help answer the poster's question would
    >require carefully sifting through all of these results and would
    >necessarily entail discarding a lot of irrelevant information.


    .... you did communicate that originally, and it's a leap that
    does not exist.

    >Nevertheless, it would be difficult to assemble another set of C++
    >source files on one's own that would make use of so many advanced
    >language features as the do the Boost libraries.


    Probably, but that's a very different issue.

    >So it must be possible
    >to learn something about a compiler by having it build this set of
    >libraries.


    Sure, but again, a different issue.

    >How useful that information may be is another matter.


    Exactly.

    >The days when compiling "hello, world" in order to
    >reasonably assess a C++ compiler have passed.


    Even though that drags in "tons" of stuff,
    that could never reasonable access a C++ compiler.
    What it did was give a quick and gentle indication
    if there was even a fighting chance in that direction.

    >Granted there are easier
    >ways to answer the question than sifting through reams of build logs,
    >but if those methods are not readily available, then answering the
    >question can still be done I believe, but it would require a lot of
    >work.


    The Boost folks themselves disagree with such a statement,
    so we should too.
    --
    Greg Comeau / Celebrating 20 years of Comeauity!
    Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
    World Class Compilers: Breathtaking C++, Amazing C99, Fabulous C90.
    Comeau C/C++ with Dinkumware's Libraries... Have you tried it?
     
    Greg Comeau, Nov 7, 2005
    #6
  7. Tao Wang

    Tao Wang Guest

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Greg Comeau wrote:
    > In article <>,
    > Greg <> wrote:
    >
    >>Greg Comeau wrote:
    >>
    >>>In article <>,
    >>>Greg <> wrote:
    >>>
    >>>>a fairly good barometer of compiler compliance would probably
    >>>>be the Boost Release summary....
    >>>
    >>>The Boost folks themselves disagree with such a statement,
    >>>so we should too.

    >>
    >>Granted, the purpose of the Release pages is not to test compiler
    >>compliance, nor do the Boost libraries provide any type of systematic
    >>test coverage. Furthermore the compiler is not the only tool used in a
    >>build, so a failure may be a problem with the linker, for example.

    >
    >
    > Yes, that can be so. But...
    >
    >
    >>As I
    >>tried to communicate in my original post, the information here is quite
    >>raw. To find data that would help answer the poster's question would
    >>require carefully sifting through all of these results and would
    >>necessarily entail discarding a lot of irrelevant information.

    >
    >
    > ... you did communicate that originally, and it's a leap that
    > does not exist.
    >
    >
    >>Nevertheless, it would be difficult to assemble another set of C++
    >>source files on one's own that would make use of so many advanced
    >>language features as the do the Boost libraries.

    >
    >
    > Probably, but that's a very different issue.
    >
    >
    >>So it must be possible
    >>to learn something about a compiler by having it build this set of
    >>libraries.

    >
    >
    > Sure, but again, a different issue.
    >
    >
    >>How useful that information may be is another matter.

    >
    >
    > Exactly.
    >
    >
    >>The days when compiling "hello, world" in order to
    >>reasonably assess a C++ compiler have passed.

    >
    >
    > Even though that drags in "tons" of stuff,
    > that could never reasonable access a C++ compiler.
    > What it did was give a quick and gentle indication
    > if there was even a fighting chance in that direction.
    >
    >
    >>Granted there are easier
    >>ways to answer the question than sifting through reams of build logs,
    >>but if those methods are not readily available, then answering the
    >>question can still be done I believe, but it would require a lot of
    >>work.

    >
    >
    > The Boost folks themselves disagree with such a statement,
    > so we should too.


    Thanks Greg and Greg Comeau for answers.

    Boost compiler release page is good for overview of the compiler but not
    good enough I think.

    In cuj's conformance roundup, Herb Sutter used Dinkumware, Perennial and
    Plum Hall's test case tools for help to get the result. I think they are
    more objective to the truth. I prefer the style of Perennial and Plum
    Hall, they try to make a whole list of C++ feature according to the C++
    Standard Document.

    If there is a test case set provided by C++ Standard committee, it
    should make developer much easizer to have more clear concept of current
    using compiler ability. What it can do, what is cannot do.

    But today, it's not open, maybe only linguist know the real different
    between each compiler or library. For user or normal developer ,it's not
    clear enough. Only way to know the limits of compiler is I do something
    which I really sure it should correct, and it doesn't work. or it worked
    on some other compiler, but this time it doesn't work. It will comfuse
    some beginners to learn the modern features of C++.

    If we have a open c++ standard conformance test tools, it will much
    clear to know the current using compiler, and will help us choosing
    between the compiler.

    Thanks.

    Dancefire
    - --
    CCNA
    http://www.dancefires.com/
    http://blog.csdn.net/dancefire/
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.1 (GNU/Linux)
    Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

    iD8DBQFDb0+eRS5AkKgtcCcRAn5YAKCw2t8+CfKngfEa522wxWlqnSZ/xACfYSoV
    D5n+Ky9uAMKUoVqS7GPLTRk=
    =dV4s
    -----END PGP SIGNATURE-----
     
    Tao Wang, Nov 7, 2005
    #7
  8. Tao Wang

    P.J. Plauger Guest

    "Tao Wang" <> wrote in message
    news:JcIbf.10984$...

    > Thanks Greg and Greg Comeau for answers.
    >
    > Boost compiler release page is good for overview of the compiler but not
    > good enough I think.
    >
    > In cuj's conformance roundup, Herb Sutter used Dinkumware, Perennial and
    > Plum Hall's test case tools for help to get the result. I think they are
    > more objective to the truth. I prefer the style of Perennial and Plum
    > Hall, they try to make a whole list of C++ feature according to the C++
    > Standard Document.


    Agreed. It's difficult to get any published results of Plum Hall and
    Perennial tests, but you can see the results of our library tests at
    our web site. They're nearly three years old now, and we're about to
    update them, but the results have changed remarkably little. STLport
    and Rogue Wave have been stagnant for years, while libstdc++ is
    improving slowly. Only the Dinkumware C99 and C++ libraries score
    anywhere near 100 per cent on conformance. (We fail none of our own
    tests, of course, and just a handful of Perennial C++ tests -- because
    of compiler bugs.) Since the other C++ libraries depends on existing
    C libraries, they fare particularly poorly on tests for the C
    enhancements required by the C++ Standard.

    As for the C++ language, it should be well known by now that gcc and
    VC++ V7.1/V8.0 score high on conformance, except that neither implements
    separate compilation of templates. Only the Edison Design Group front
    end does, and it gets essentially perfect scores on the validation
    suites. You'll find the EDG front end in quite a few commercial compilers
    out there.

    > If there is a test case set provided by C++ Standard committee, it
    > should make developer much easizer to have more clear concept of current
    > using compiler ability. What it can do, what is cannot do.


    There are a gazillion reasons why the C and C++ committees will never
    issue validation suites. The sheer amount of work involved is daunting,
    and the politics is nearly insurmountable.

    > But today, it's not open, maybe only linguist know the real different
    > between each compiler or library. For user or normal developer ,it's not
    > clear enough. Only way to know the limits of compiler is I do something
    > which I really sure it should correct, and it doesn't work. or it worked
    > on some other compiler, but this time it doesn't work. It will comfuse
    > some beginners to learn the modern features of C++.


    That is indeed one of the drawbacks of a complex language.

    > If we have a open c++ standard conformance test tools, it will much
    > clear to know the current using compiler, and will help us choosing
    > between the compiler.


    There *are* test suites available for things like gcc, libstdc++,
    and STLport. Unfortunately, they're more in the nature of regression
    tests for their respective products. Try them on a different
    compiler or library and you'll get all sorts of false failures
    because of dialect issues. They're also a bit thin and spotty, IMO.

    P.J. Plauger
    Dinkumware, Ltd.
    http://www.dinkumware.com
     
    P.J. Plauger, Nov 7, 2005
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. T.J.

    Triple-A conformance?

    T.J., Dec 14, 2003, in forum: HTML
    Replies:
    20
    Views:
    788
    brucie
    Dec 16, 2003
  2. EN
    Replies:
    45
    Views:
    1,148
    Mark Parnell
    May 5, 2006
  3. Amarendra GODBOLE
    Replies:
    25
    Views:
    692
    Dan Pop
    Jan 21, 2004
  4. Merciadri Luca
    Replies:
    5
    Views:
    1,092
    Merciadri Luca
    Oct 14, 2010
  5. Michael Tan
    Replies:
    32
    Views:
    1,006
    Ara.T.Howard
    Jul 21, 2005
Loading...

Share This Page