Evaluating static analysis and Dynamic analysis tools for C/C++

Discussion in 'C Programming' started by ssubbarayan, Oct 29, 2009.

  1. ssubbarayan

    ssubbarayan Guest

    Dear all,
    Whats the general norm people use to evaluate static code analysis
    and dynamic code analysis tools in your experience.I am confused on
    the best tool to choose from,given that so many tools are available
    over the net.I believe
    following are the criteria to test the usefullness of these tools:
    1)User friendliness
    2)Ability to detect bugs
    3)Ability to enforce coding guidelines
    4)Ability to generate userfriendly reports
    5)Speed of detection
    6)Should not be providing lot of false defects.
    7)Easily customisable

    Are there any test suites available to evaluate these analysis tools?
    How do people who use them evaluate?Are there any test programs
    available for eg a sample C code which can be run with analysis tool
    to see how it reports?

    Please let me know your thoughts.Advance thanks for all your inputs.
    Are there any bench marks available for these analysis tools?

    Looking farward for your inputs and advanced thanks,
    Regards,
    s.subbarayan
     
    ssubbarayan, Oct 29, 2009
    #1
    1. Advertising

  2. ssubbarayan

    Guest

    On Oct 29, 8:07 am, ssubbarayan <> wrote:
    > Dear all,
    > Whats the general norm people use to evaluate static code  analysis
    > and dynamic code analysis tools in your experience.I am confused on
    > the best tool to choose from,given that so many tools are available
    > over the net.I believe
    > following are the criteria to test the usefullness of these tools:
    > 1)User friendliness
    > 2)Ability to detect bugs
    > 3)Ability to enforce coding guidelines
    > 4)Ability to generate userfriendly reports
    > 5)Speed of detection
    > 6)Should not be providing lot of false defects.
    > 7)Easily customisable
    >
    > Are there any test suites available to evaluate these analysis tools?
    > How do people who use them evaluate?Are there any test programs
    > available for eg a sample C code which can be run with analysis tool
    > to see how it reports?
    >
    > Please let me know your thoughts.Advance thanks for all your inputs.
    > Are there any bench marks available for these analysis tools?
    >
    > Looking farward for your inputs and advanced thanks,
    > Regards,
    > s.subbarayan


    In regards of static analysis tools:

    NIST SAMATE project has some test suites for C, C++, Java...
    You can reach it here:
    http://samate.nist.gov/SRD

    These are syntactic test cases, so they do not represent properly the
    result of a tool on your code base. It just gives you an idea of the
    weaknesses coverage of the tool (tools should also provide a list of
    weaknesses they support, you can make sense of it with the CWE -
    http://cwe.mitre.org)

    As criteria to select a tool, I think it depends on lot on how you
    plan to use to tool.
    For example, if only few people (software security folks) use the
    tool, then usability shouldn't be such a big deal; it is if many
    developers will use the tool.

    Otherwise, I would recommend few things:
    - proper detection with few false-positive rate on selected test cases
    - take some of your code (restrict the scope of the scan), and compare
    tool results and look for false-negative/false-positive on your
    code... (tools are sensitive to the code constructs/API used in the
    code)
    - customization (especially if you see an important FP/FN rate) might
    be considered as important too; I suppose it depends on how you want
    to use the tool...

    Romain
     
    , Oct 29, 2009
    #2
    1. Advertising

  3. Op Thu, 29 Oct 2009 13:07:55 +0100 schreef ssubbarayan <>:
    > Dear all,
    > Whats the general norm people use to evaluate static code analysis
    > and dynamic code analysis tools in your experience.I am confused on
    > the best tool to choose from,given that so many tools are available
    > over the net.I believe
    > following are the criteria to test the usefullness of these tools:
    > 1)User friendliness
    > 2)Ability to detect bugs
    > 3)Ability to enforce coding guidelines
    > 4)Ability to generate userfriendly reports
    > 5)Speed of detection
    > 6)Should not be providing lot of false defects.
    > 7)Easily customisable
    >
    > Are there any test suites available to evaluate these analysis tools?
    > How do people who use them evaluate?Are there any test programs
    > available for eg a sample C code which can be run with analysis tool
    > to see how it reports?


    Most companies who evaluate such a tool, use their own code base or a
    representative, well-known portion of it.

    > Please let me know your thoughts.Advance thanks for all your inputs.
    > Are there any bench marks available for these analysis tools?


    1)You cannot benchmark user friendliness.
    2)There is no list of a respectable portion of all known bugs.
    3)Coding guidelines have test suites, but sometimes interpretation differs.
    4)You cannot benchmark user friendliness.
    5)Speed of detection depends on code structure and style and is nearly
    irrelevant for night-time testing.
    6)There is no list of a respectable portion of all known false positives
    of all coding guidelines.
    7)You cannot benchmark user experience.


    --
    Gemaakt met Opera's revolutionaire e-mailprogramma:
    http://www.opera.com/mail/
    (remove the obvious prefix to reply by mail)
     
    Boudewijn Dijkstra, Oct 29, 2009
    #3
  4. ssubbarayan wrote:

    > Whats the general norm people use to evaluate static code analysis
    > and dynamic code analysis tools in your experience.


    There isn't one. People ask different things of such tools, and thus
    value their properties differently. One person's strictly necessary
    feature is another's gratuitous gimmick. One person's show-stopping
    limitation is the next one's barely noticeable glitch.

    > 1)User friendliness


    As the saying goes: <insert program name here> is extremely
    user-friendly. It's just a little picky who it makes friends with.

    > 2)Ability to detect bugs


    Depends on what kinds of bugs there are for the detecting.

    > 3)Ability to enforce coding guidelines


    Depends on the coding guideline in question.

    > 4)Ability to generate userfriendly reports


    There's no such thing as a user-friendly report.

    > 5)Speed of detection


    Depends on usage pattern. Unless the processing time runs considerably
    north of one whole day, usage patterns will adapt to the tool, once its
    usage has been prescribed.

    > 6)Should not be providing lot of false defects.


    See answer to 2).

    > 7)Easily customisable


    Depending on what you're trying to do, there's a point to be made that
    the tools should not _need_ any "customization" ... it should find any
    an all problems it can, period.

    > Are there any test suites available to evaluate these analysis tools?


    I rather much doubt it.

    > How do people who use them evaluate?


    Initially by throwing at it a significant amount of the worst, the best,
    and the most typical code they can find at it. The real evaluation only
    comes from actual long-time usage. The proof of the pudding, as they
    say, is in the eating.
     
    Hans-Bernhard Bröker, Oct 29, 2009
    #4
  5. ssubbarayan

    ssubbarayan Guest

    On Oct 30, 4:11 am, Hans-Bernhard Bröker <>
    wrote:
    > ssubbarayan wrote:
    > > Whats the general norm people use to evaluate static code  analysis
    > > and dynamic code analysis tools in your experience.

    >
    > There isn't one.  People ask different things of such tools, and thus
    > value their properties differently.  One person's strictly necessary
    > feature is another's gratuitous gimmick.  One person's show-stopping
    > limitation is the next one's barely noticeable glitch.
    >
    > > 1)User friendliness

    >
    > As the saying goes: <insert program name here> is extremely
    > user-friendly.  It's just a little picky who it makes friends with.
    >
    > > 2)Ability to detect bugs

    >
    > Depends on what kinds of bugs there are for the detecting.
    >
    > > 3)Ability to enforce coding guidelines

    >
    > Depends on the coding guideline in question.
    >
    > > 4)Ability to generate userfriendly reports

    >
    > There's no such thing as a user-friendly report.
    >
    > > 5)Speed of detection

    >
    > Depends on usage pattern.  Unless the processing time runs considerably
    > north of one whole day, usage patterns will adapt to the tool, once its
    > usage has been prescribed.
    >
    > > 6)Should not be providing lot of false defects.

    >
    > See answer to 2).
    >
    > > 7)Easily customisable

    >
    > Depending on what you're trying to do, there's a point to be made that
    > the tools should not _need_ any "customization" ... it should find any
    > an all problems it can, period.
    >
    > > Are there any test suites available to evaluate these analysis tools?

    >
    > I rather much doubt it.
    >
    > > How do people who use them evaluate?

    >
    > Initially by throwing at it a significant amount of the worst, the best,
    > and the most typical code they can find at it.  The real evaluation only
    > comes from actual long-time usage.  The proof of the pudding, as they
    > say, is in the eating.


    Hi,
    I love your quote "As the saying goes: <insert program name here> is
    extremely
    user-friendly. It's just a little picky who it makes friends with. "
    Thanks for making me smile.

    I realise like what others say,It depends on the targeted audience and
    no single tool can satisfy all.

    Regards,
    s.subbarayan
     
    ssubbarayan, Oct 30, 2009
    #5
  6. ssubbarayan

    Dave Hansen Guest

    On Oct 29, 6:07 am, ssubbarayan <> wrote:
    > Dear all,
    > Whats the general norm people use to evaluate static code  analysis
    > and dynamic code analysis tools in your experience.I am confused on
    > the best tool to choose from,given that so many tools are available
    > over the net.I believe
    > following are the criteria to test the usefullness of these tools:


    First question: do you lint your code?

    Some of the new static analysis tools are wondrous things, that detect
    subtle bugs in convoluted code. But they tend to be expensive,
    require extensive setup and tuning, and can take hours to run.

    Lint, on the other hand, is a wondrous thing, detects subtle bugs in
    convoluted, code, is cheap, easy to use, and faster than your
    compiler. <insert standard praise of Gimpel's PC-lint here>.

    The newer tools can do more, certainly, but if you can't be bothered
    to lint your code, you're not going to be willing to put in the effort
    required by the new tools to get the best results. No tool can give
    good results if you don't use it.

    On the other hand, if you're already an enthusiastic user of lint, and
    you are looking for a tool that digs deeper, carry on with your
    search.

    Lint early. Lint often. Lint is your friend.

    Regards,

    -=Dave
     
    Dave Hansen, Nov 3, 2009
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Derek
    Replies:
    2
    Views:
    437
    Ron Natalie
    Mar 4, 2005
  2. Don

    Static Code Analysis Tools

    Don, Jul 25, 2003, in forum: C Programming
    Replies:
    1
    Views:
    399
    Eric Sosman
    Jul 25, 2003
  3. Richard
    Replies:
    10
    Views:
    704
    Francesco
    Sep 6, 2009
  4. Brian Jiang
    Replies:
    11
    Views:
    2,200
    Jorgen Grahn
    Dec 30, 2010
  5. Scott Meyers
    Replies:
    13
    Views:
    1,102
    Balog Pal
    Jun 29, 2011
Loading...

Share This Page