unit testing guidelines

Discussion in 'Java' started by Jacob, Mar 17, 2006.

  1. Jacob

    Jacob Guest

    Jacob, Mar 17, 2006
    #1
    1. Advertising

  2. -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
    NotDashEscaped: You need GnuPG to verify this message

    Jacob uitte de volgende tekst op 03/18/2006 12:03 AM:
    > I have compiled a set og unit testing
    > recommendations based on my own experience
    > on the concept.
    >
    > Feedback and suggestions for improvements
    > are appreciated:
    >
    > http://geosoft.no/development/unittesting.html


    Nice work.

    I don't totally agree with point 16: a throws statement means an
    exception *might* be thrown, and the circumstances under which this can
    happen should be documented. It is seldom that an exception must be thrown.

    You might want to give some explanation about what you assertX methods do.

    H.
    --
    Hendrik Maryns

    ==================
    www.lieverleven.be
    http://aouw.org
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.2 (GNU/Linux)

    iD4DBQFEHEUxe+7xMGD3itQRAsFwAKCBXX77fVjeMGJz3+AU0Bxe/pYnoQCY5z2F
    Ti5C4PCNnHJBSHbz3tp2jQ==
    =VsD2
    -----END PGP SIGNATURE-----
    Hendrik Maryns, Mar 18, 2006
    #2
    1. Advertising

  3. Jacob

    Daniel T. Guest

    In article <dvhgcf$opd$03$-online.com>,
    Hendrik Maryns <> wrote:

    > -----BEGIN PGP SIGNED MESSAGE-----
    > Hash: SHA1
    > NotDashEscaped: You need GnuPG to verify this message
    >
    > Jacob uitte de volgende tekst op 03/18/2006 12:03 AM:
    > > I have compiled a set og unit testing
    > > recommendations based on my own experience
    > > on the concept.
    > >
    > > Feedback and suggestions for improvements
    > > are appreciated:
    > >
    > > http://geosoft.no/development/unittesting.html

    >
    > Nice work.
    >
    > I don't totally agree with point 16: a throws statement means an
    > exception *might* be thrown, and the circumstances under which this can
    > happen should be documented. It is seldom that an exception must be thrown.


    I agree. Only test what you actually want the client code to rely on.
    Now if you want the client code to rely on the method throwing an
    exception...


    --
    Magic depends on tradition and belief. It does not welcome observation,
    nor does it profit by experiment. On the other hand, science is based
    on experience; it is open to correction by observation and experiment.
    Daniel T., Mar 18, 2006
    #3
  4. Jacob

    Ian Collins Guest

    Jacob wrote:
    > I have compiled a set og unit testing
    > recommendations based on my own experience
    > on the concept.
    >
    > Feedback and suggestions for improvements
    > are appreciated:
    >
    > http://geosoft.no/development/unittesting.html
    >

    I'd add point 0 - write the tests first.

    8 - names should be more expressive, rather than testSaveAs(), how about
    a series of tests, testSaveAsCreatesANewFile(),
    testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad name
    attempt to test too much ad don't express their intent.

    Point 0 covers point 11.

    13 - take care with random numbers, they can lead to failures that are
    hard to reproduce. I'd use a pseudo-random sequence that is repeatable
    with a given seed.

    0 and 8 covers 14.

    0 covers 17.

    0 covers 20.

    --
    Ian Collins.
    Ian Collins, Mar 18, 2006
    #4
  5. Jacob

    Timo Stamm Guest

    Jacob schrieb:
    > I have compiled a set og unit testing
    > recommendations based on my own experience
    > on the concept.
    >
    > Feedback and suggestions for improvements
    > are appreciated:



    | 7. Keep tests close to the class being tested
    |
    | If the class to test is Foo the test class should be called FooTest
    | and kept in the same package (directory) as Foo. The build environment
    | must be configured so that the test classes doesn't make its way into
    | production code.

    It is necessary to have test classes in the same package as the tested
    class in order to test package private methods.

    But you don't have to put the classes in the same directory. Most IDEs
    support several source folders. You can setup two source folders. For
    example: "src" for your application source, "test" for your test source.
    If you use the same package structure in the test source folder, you can
    test package private methods and it is very easy to deploy only
    application code.


    Timo
    Timo Stamm, Mar 18, 2006
    #5
  6. Jacob

    Ian Collins Guest

    Timo Stamm wrote:
    > Jacob schrieb:
    >
    >> I have compiled a set og unit testing
    >> recommendations based on my own experience
    >> on the concept.
    >>
    >> Feedback and suggestions for improvements
    >> are appreciated:

    >
    >
    >
    > | 7. Keep tests close to the class being tested
    > |
    > | If the class to test is Foo the test class should be called FooTest
    > | and kept in the same package (directory) as Foo. The build environment
    > | must be configured so that the test classes doesn't make its way into
    > | production code.
    >
    > It is necessary to have test classes in the same package as the tested
    > class in order to test package private methods.
    >

    Another view that tests that require access to private methods are a
    design smell. Often these can be refactored into objects that can be
    tested in isolation.

    In C++, it's very tempting to make the test class a friend of the class
    under test. I've found that I end up with a better design by resisting
    this temptation.

    --
    Ian Collins.
    Ian Collins, Mar 18, 2006
    #6
  7. -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
    NotDashEscaped: You need GnuPG to verify this message

    Ian Collins uitte de volgende tekst op 03/19/2006 12:13 AM:
    > Timo Stamm wrote:
    >> Jacob schrieb:
    >>
    >>> I have compiled a set og unit testing
    >>> recommendations based on my own experience
    >>> on the concept.
    >>>
    >>> Feedback and suggestions for improvements
    >>> are appreciated:

    >>
    >>
    >>
    >> | 7. Keep tests close to the class being tested
    >> |
    >> | If the class to test is Foo the test class should be called FooTest
    >> | and kept in the same package (directory) as Foo. The build environment
    >> | must be configured so that the test classes doesn't make its way into
    >> | production code.
    >>
    >> It is necessary to have test classes in the same package as the tested
    >> class in order to test package private methods.
    >>

    > Another view that tests that require access to private methods are a
    > design smell. Often these can be refactored into objects that can be
    > tested in isolation.


    I was about to answer the same: shouldn't problems in package private
    methods spill through to public methods? Then why test them separately?
    Find an error in a public method and retrace it with you favorite
    debugger to the package private method, I'd say (without much
    experience, so correct me if I'm wrong).

    H.

    --
    Hendrik Maryns

    ==================
    www.lieverleven.be
    http://aouw.org
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.2 (GNU/Linux)

    iD8DBQFEHJyze+7xMGD3itQRAuV5AJ9RMb89yiGxxknFqT7XCaTjHnJJ9QCfXrZ3
    7iuZP99HYR0uR9FuffXLeLk=
    =KKUv
    -----END PGP SIGNATURE-----
    Hendrik Maryns, Mar 18, 2006
    #7
  8. In article <dvi687$bgi$03$-online.com>,
    Hendrik Maryns <> wrote:
    >
    >I was about to answer the same: shouldn't problems in package private
    >methods spill through to public methods? Then why test them separately?


    It makes it more time-consuming to find out where the error is.

    > Find an error in a public method and retrace it with you favorite
    >debugger to the package private method, I'd say (without much
    >experience, so correct me if I'm wrong).


    I prefer my unit tests to have obvious failure modes so that I can
    basically tell from which test failed, exactly where in my source the
    bug is. This means I don't have to muck around with a debugger, I can
    just fix it and get on with things.

    For this to be the case, however, the methods that I test need to be
    reasonably small and not do a whole lot. These are my private helper
    methods that I invoke from my more involved algorithm methods. Many
    are one or two liners and they generally don't make sense to have
    publicly accessible since they're really just internal building blocks
    for constructing other more interesting methods.

    Cheers
    Bent D
    --
    Bent Dalager - - http://www.pvv.org/~bcd
    powered by emacs
    Bent C Dalager, Mar 19, 2006
    #8
  9. Jacob

    Timo Stamm Guest

    Ian Collins schrieb:
    > Timo Stamm wrote:
    >> It is necessary to have test classes in the same package as the tested
    >> class in order to test package private methods.
    >>

    > Another view that tests that require access to private methods are a
    > design smell. Often these can be refactored into objects that can be
    > tested in isolation.


    Not "private", but "package private".

    Package private classes are only visible within the same package (same
    directory). They are useful in large APIs where you have a lot of
    functionality, but only want to expose a small interface.


    Timo
    Timo Stamm, Mar 19, 2006
    #9
  10. Jacob

    Timo Stamm Guest

    Timo Stamm schrieb:
    > | 7. Keep tests close to the class being tested
    > |
    > | If the class to test is Foo the test class should be called FooTest
    > | and kept in the same package (directory) as Foo. The build environment
    > | must be configured so that the test classes doesn't make its way into
    > | production code.
    >
    > It is necessary to have test classes in the same package as the tested
    > class in order to test package private methods.
    >
    > But you don't have to put the classes in the same directory. Most IDEs
    > support several source folders. You can setup two source folders. For
    > example: "src" for your application source, "test" for your test source.
    > If you use the same package structure in the test source folder, you can
    > test package private methods and it is very easy to deploy only
    > application code.



    Oops, I didn't realize that the guidelines aren't java-specific and that
    this thread is on c.l.java.p as well as c.l.c++.

    My objection is specific to java. I doubt that the same applies to c++.
    Timo Stamm, Mar 19, 2006
    #10
  11. Jacob

    Ian Collins Guest

    Timo Stamm wrote:
    > Ian Collins schrieb:
    >
    >> Timo Stamm wrote:
    >>
    >>> It is necessary to have test classes in the same package as the
    >>> tested class in order to test package private methods.
    >>>

    >> Another view that tests that require access to private methods are a
    >> design smell. Often these can be refactored into objects that can be
    >> tested in isolation.

    >
    >
    > Not "private", but "package private".
    >
    > Package private classes are only visible within the same package (same
    > directory). They are useful in large APIs where you have a lot of
    > functionality, but only want to expose a small interface.
    >

    I see, a concept not shared with C++.

    --
    Ian Collins.
    Ian Collins, Mar 19, 2006
    #11
  12. Jacob

    Ian Collins Guest

    Hendrik Maryns wrote:
    >>>
    >>>| 7. Keep tests close to the class being tested
    >>>|
    >>>| If the class to test is Foo the test class should be called FooTest
    >>>| and kept in the same package (directory) as Foo. The build environment
    >>>| must be configured so that the test classes doesn't make its way into
    >>>| production code.
    >>>
    >>>It is necessary to have test classes in the same package as the tested
    >>>class in order to test package private methods.
    >>>

    >>
    >>Another view that tests that require access to private methods are a
    >>design smell. Often these can be refactored into objects that can be
    >>tested in isolation.

    >
    >
    > I was about to answer the same: shouldn't problems in package private
    > methods spill through to public methods? Then why test them separately?
    > Find an error in a public method and retrace it with you favorite
    > debugger to the package private method, I'd say (without much
    > experience, so correct me if I'm wrong).
    >

    As Brent said, you are testing too much with your tests. A golden rule
    is not to rely on indirect tests.

    I've recently come to the conclusion (while working with PHP which
    doesn't have a handy debugger) that resorting to the debugger is a
    strong indicator that your tests aren't fine grained enough. Try working
    without one for a while and see your tests improve!

    --
    Ian Collins.
    Ian Collins, Mar 19, 2006
    #12
  13. Jacob

    Adam Maass Guest

    "Jacob" <> wrote:
    >I have compiled a set og unit testing
    > recommendations based on my own experience
    > on the concept.
    >
    > Feedback and suggestions for improvements
    > are appreciated:
    >
    > http://geosoft.no/development/unittesting.html
    >
    > Thanks.


    I strongly object to number 13. Unit-tests, especially in an automated
    framework, should be repeatable. (When a test fails, you need to know on
    what inputs it failed. Once you fix the failure, you should hard-code the
    inputs it failed on so that subsequent changes do not cause a regression of
    the error.)

    I don't necessarily object to looping over large numbers of inputs and
    testing each one for expected outputs. But a unit test should contain no
    randomness at all. (Or at least should have a way of specifying the seed for
    the randomness generator(s).)


    -- Adam Maass
    Adam Maass, Mar 20, 2006
    #13
  14. Jacob

    Jacob Guest

    Hendrik Maryns wrote:

    > I don't totally agree with point 16: a throws statement means an
    > exception *might* be thrown, and the circumstances under which this can
    > happen should be documented. It is seldom that an exception must be thrown.


    I assume the conditions for when an exception is thrown
    is deterministic and well documented (though it to a large
    extent depend on *documentation* rather than language syntax
    which is a problem as documentation is inherently inaccurate).

    The simple example is the java List.get(int index) method that
    is documented to throw an exception if index < 0. This is the
    contract, and this is one of the things I want to test in a
    unit test.

    Recommendation 16 just indicate how this is done in practice.
    Jacob, Mar 20, 2006
    #14
  15. Jacob

    Jacob Guest

    Ian Collins wrote:

    > I'd add point 0 - write the tests first.


    Personaly find the XP approach to unit testing a bit too restrictive
    and therefore left the issue intentionally open. I really like
    more feedback on it though, as though I have practiced unit testing
    for years, I never adopted this practice myself.

    > 8 - names should be more expressive, rather than testSaveAs(), how about
    > a series of tests, testSaveAsCreatesANewFile(),
    > testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad name
    > attempt to test too much ad don't express their intent.


    Agree. I think this is basically what's in #8 without being to
    verbose.

    > Point 0 covers point 11.


    I am not sure it does, and I wanted to define the two concepts
    "execution coverage" and "test coverage" anyway. There is a blurred
    distinction between the two in the literature as far as I have been
    able to dig up.

    > 13 - take care with random numbers, they can lead to failures that are
    > hard to reproduce. I'd use a pseudo-random sequence that is repeatable
    > with a given seed.
    >
    > 0 and 8 covers 14.


    To some degree, but I'd include them even if #0 was there. I don't
    see "testing first" as a silver bullet, but more as a different
    process aproach.

    > 0 covers 17.


    Not necesserily. #0 states when to write the tests. #17 states that
    the *code* should be written so that the workload of the unit testing
    is minimized.

    > 0 covers 20.


    Yes, assuming everything is tested always. But in that case
    it is covered without #0 as well. What I see in the industry today
    is a major shift in adding unit testing to legacy code. I added
    #20 as a suggestion to start this work at the bottom level.
    Jacob, Mar 20, 2006
    #15
  16. Jacob

    Jacob Guest

    Ian Collins wrote:

    > Another view that tests that require access to private methods are a
    > design smell. Often these can be refactored into objects that can be
    > tested in isolation.
    >
    > In C++, it's very tempting to make the test class a friend of the class
    > under test. I've found that I end up with a better design by resisting
    > this temptation.


    This is my experience as well, and the reason why I added
    recommendation #9 "Test public API".

    That something is technically feasable (private method testing through
    reflection or by other means) doesn't necesserily mean it is a good idea.

    You need to draw the line somewhere, and the public API seems quite
    natural in this case. This is also more robust agains changes, in that it
    will be more stable and require less testing maintainance during code
    refactoring.
    Jacob, Mar 20, 2006
    #16
  17. Jacob

    Jacob Guest

    Timo Stamm wrote:
    > Ian Collins schrieb:
    >
    >> Timo Stamm wrote:
    >>
    >>> It is necessary to have test classes in the same package as the
    >>> tested class in order to test package private methods.
    >>>

    >> Another view that tests that require access to private methods are a
    >> design smell. Often these can be refactored into objects that can be
    >> tested in isolation.

    >
    >
    > Not "private", but "package private".
    >
    > Package private classes are only visible within the same package (same
    > directory). They are useful in large APIs where you have a lot of
    > functionality, but only want to expose a small interface.


    I regard this as "private" in this context. An error in the
    inner logic between classes of the same package (or *friends*
    in C++ syntax) will eventually reveal itself through the public
    API.

    I want to keep test classes close to the class being tested for
    practical reasons rather than technical reasons.

    I understand the objection of "testing too large chunks of code"
    (Ian C.), but test code adds complexity and workload to your system
    afterall, and I really want to keep it to a minimum. That's why I
    reduce the public API of classes as much as possible (by heavy
    use of package private methods for instance) and insist on testing
    public API only.

    But I don't clain that this is the only way, and it might well
    depend on the nature of the project being tested.
    Jacob, Mar 20, 2006
    #17
  18. Jacob

    Jacob Guest

    Adam Maass wrote:

    > I strongly object to number 13. Unit-tests, especially in an automated
    > framework, should be repeatable. (When a test fails, you need to know on
    > what inputs it failed. Once you fix the failure, you should hard-code the
    > inputs it failed on so that subsequent changes do not cause a regression of
    > the error.)


    I understand your objection, but this is actually one of the
    mechanisms that have helped me found some of the hardest to
    trace and most subtle errors in the code. It has proven to be
    extremely helpful. Also, it gives me lots of confidence
    knowing that my test suite of several thousand tests
    are executed every hour with different input each time.
    It is like adding another dimension to unit testing.

    But as tests must be reproducable I agree, I added #15 to ensure
    that when a test fails, the test report will include the input
    parameters if failed with exactly. Then you can add a test with
    this explicit input and debug it from there.
    Jacob, Mar 20, 2006
    #18
  19. Jacob

    Ian Collins Guest

    Jacob wrote:
    > Ian Collins wrote:
    >
    >> I'd add point 0 - write the tests first.

    >
    >
    > Personaly find the XP approach to unit testing a bit too restrictive
    > and therefore left the issue intentionally open. I really like
    > more feedback on it though, as though I have practiced unit testing
    > for years, I never adopted this practice myself.
    >

    TDD is more than an approach to unit testing, it is an approach to the
    full design-test-code cycle.

    >> 8 - names should be more expressive, rather than testSaveAs(), how
    >> about a series of tests, testSaveAsCreatesANewFile(),
    >> testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad
    >> name attempt to test too much ad don't express their intent.

    >
    >
    > Agree. I think this is basically what's in #8 without being to
    > verbose.
    >
    >> Point 0 covers point 11.

    >
    >
    > I am not sure it does, and I wanted to define the two concepts
    > "execution coverage" and "test coverage" anyway. There is a blurred
    > distinction between the two in the literature as far as I have been
    > able to dig up.
    >

    TDD done well will give you 100% execution coverage for free. How good
    your test coverage is depends on how good you are at thinking up edge
    cases to test.

    >> 13 - take care with random numbers, they can lead to failures that are
    >> hard to reproduce. I'd use a pseudo-random sequence that is
    >> repeatable with a given seed.
    >>
    >> 0 and 8 covers 14.

    >
    >
    > To some degree, but I'd include them even if #0 was there. I don't
    > see "testing first" as a silver bullet, but more as a different
    > process aproach.
    >

    Simple, incremental tests are the essence of good TDD.

    >> 0 covers 17.

    >
    >
    > Not necesserily. #0 states when to write the tests. #17 states that
    > the *code* should be written so that the workload of the unit testing
    > is minimized.
    >

    If you start with the tests,the code will have to be written that way.

    >> 0 covers 20.

    >
    >
    > Yes, assuming everything is tested always. But in that case
    > it is covered without #0 as well. What I see in the industry today
    > is a major shift in adding unit testing to legacy code. I added
    > #20 as a suggestion to start this work at the bottom level.


    Very true.

    --
    Ian Collins.
    Ian Collins, Mar 20, 2006
    #19
  20. Ian Collins wrote:
    > Jacob wrote:
    >> Ian Collins wrote:
    >>
    >>> I'd add point 0 - write the tests first.

    >>
    >>
    >> Personaly find the XP approach to unit testing a bit too restrictive
    >> and therefore left the issue intentionally open. I really like
    >> more feedback on it though, as though I have practiced unit testing
    >> for years, I never adopted this practice myself.
    >>

    > TDD is more than an approach to unit testing, it is an approach to the
    > full design-test-code cycle.



    More fundamentally, TDD is Design Methodology, Not a Testing Methodology.

    It just happens to use Unit tests as its means of describing the design,
    much like RUP uses UML.

    Indeed, some TDD practitioners are starting to call it BDD - as in

    http://www.google.co.uk/search?hl=en&q=behaviour driven development&btnG=Google Search&meta=

    >
    >>> 8 - names should be more expressive, rather than testSaveAs(), how
    >>> about a series of tests, testSaveAsCreatesANewFile(),
    >>> testSaveAsSavesCuentdataInNewFile() etc. Often tests with a broad
    >>> name attempt to test too much ad don't express their intent.

    >>
    >>
    >> Agree. I think this is basically what's in #8 without being to
    >> verbose.
    >>
    >>> Point 0 covers point 11.

    >>
    >>
    >> I am not sure it does, and I wanted to define the two concepts
    >> "execution coverage" and "test coverage" anyway. There is a blurred
    >> distinction between the two in the literature as far as I have been
    >> able to dig up.
    >>

    > TDD done well will give you 100% execution coverage for free.


    I'd clarify that with 'TDD done *Correctly will give you 100% execution
    coverage'

    *Correctly = Write 1 failing Testcase,
    Write only enough code to make test Pass,
    Refactor to Remove Duplication,
    Repeat

    More commonly referred to as Red, Green, Refactor.

    > How good your test coverage is depends on how good you are at thinking up edge
    > cases to test.


    Always starting with the test first, only allows for 100%.

    >
    >>> 13 - take care with random numbers, they can lead to failures that
    >>> are hard to reproduce. I'd use a pseudo-random sequence that is
    >>> repeatable with a given seed.
    >>>
    >>> 0 and 8 covers 14.

    >>
    >>
    >> To some degree, but I'd include them even if #0 was there. I don't
    >> see "testing first" as a silver bullet, but more as a different
    >> process aproach.
    >>

    > Simple, incremental tests are the essence of good TDD.
    >


    These kind of tests are unit tests as in the TDD usage - they are stress
    tests that happen to be written in the same framework as the TDD unit
    tests.

    However, looping over a random set of numbers isn't the best approach to
    this style of testing. If the OP wants to do this style, then using one
    of the various Agitating frameworks/products will give a better result.

    These tools tend to use byte code manipulation to random change various
    values which aren't just numbers, but anything: int, long, float,
    Integer, Double, String, Boolean, boolean, introducing Nulls, etc.

    See http://www.agitar.com/
    Andrew McDonagh, Mar 20, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Andrew McDonagh

    Re: unit testing guidelines

    Andrew McDonagh, Mar 20, 2006, in forum: Java
    Replies:
    0
    Views:
    398
    Andrew McDonagh
    Mar 20, 2006
  2. Jacob

    unit testing guidelines

    Jacob, Mar 17, 2006, in forum: C++
    Replies:
    72
    Views:
    1,498
    Adam Maass
    Mar 30, 2006
  3. John Maclean
    Replies:
    1
    Views:
    329
    Martin P. Hellwig
    Apr 13, 2010
  4. Ulrich Eckhardt

    unit-profiling, similar to unit-testing

    Ulrich Eckhardt, Nov 16, 2011, in forum: Python
    Replies:
    6
    Views:
    318
    Roy Smith
    Nov 18, 2011
  5. Bill Mosteller
    Replies:
    0
    Views:
    209
    Bill Mosteller
    Oct 22, 2009
Loading...

Share This Page