C unit testing and regression testing

Discussion in 'C Programming' started by James Harris, Aug 8, 2013.

  1. I have seen you write about this mock generator before and I am
    interested in learning more about it.
    Is this generator available somewhere?

    Bart v Ingen Schenau
     
    Bart van Ingen Schenau, Aug 9, 2013
    #21
    1. Advertisements

  2. James Harris

    Jorgen Grahn Guest

    In my case, not very informative. int n=3; assert_eq(3.14, n) will generate
    something like

    "test Foo: FAIL because the assert_eq(3.14, 3) doesn't hold"

    I can do something like this if I want:

    if(3.14 != n) throw Failure("some descriptive text");

    but in practice I rarely do, unless I'm writing a custom assertion,
    assert_sorted(sequence) or something ...

    Then I also have an option (at runtime) to trigger a core dump
    instead, so I can check it in a debugger. I rarely do that.

    I know unit test frameworks like CPPUNIT typically have assert-like
    macros similar to my assert_eq(), but which also provide file and line
    number information. I *do* miss that sometimes, but in the end it's not
    vital enough for me to implement it.
    That would work. Although I'm personally not so fond of extra
    descriptions: I want as little as possible to distract from writing
    and reading the test cases. Readable messages when a test fails is
    less important to me.
    You're probably right. There's always the convention "a function
    which returns 0 is a failing test case" of course ...

    /Jorgen
     
    Jorgen Grahn, Aug 9, 2013
    #22
    1. Advertisements

  3. James Harris

    Siri Cruise Guest

    Siri Cruise, Aug 9, 2013
    #23
  4. James Harris

    Les Cargill Guest

    That's simply a specialization of the principle "all code is bad." Well,
    Other People's Code is *worse*.

    I am using cost as the factor of optimization here... not purtchase
    cost, I mean uncertainty cost.
    Unless they aren't. Unless it is decided that test is on a different
    stovepipe from development.
    I understand the theory, and have significant quantities of evidence to
    be skeptical of it. Building your own framework doesn't cost much, and
    you know the internals of it that way.

    Bad software isn't a technology problem - it's a culture problem.
     
    Les Cargill, Aug 9, 2013
    #24
  5. James Harris

    Les Cargill Guest


    Just IMO - but the Big Gain here is that it is command line to begin
    with. I can generate permutations and limited range executions
    reasonably easily with any scripting language ( Tcl seems better
    at this because it has lists as a first class object ).

    You're then creating an "engine" that can be run in a pipe by Tcl.
    so perhaps you now wish to make it have multiple "channels"; so
    you run it over sockets. One for commands/responses, perhaps
    another for asynchronous events.

    If a GUI is needed, then make the GUI run the command line "engine".

    This, of course, runs counter to how development is taught, but I
    have much less trouble with this approach than others.
     
    Les Cargill, Aug 9, 2013
    #25
  6. James Harris

    Ian Collins Guest

    Most places I have worked...

    I probably should open source it, but so far time pressures have got in
    the way. I'll have another look at moving it to somewhere like GitHub
    or BitBucket.
     
    Ian Collins, Aug 9, 2013
    #26
  7. You may, (or nor,) find some answers in this book:
    "Test Driven Development for Embedded C"

    http://pragprog.com/book/jgade/test-driven-development-for-embedded-c

    Cannot comment. I have it, but did not read it yet. From a summary
    page leafing it seems to have good material, one you get pass the
    ideology and dogmas of test driven development. (Write test cases
    first, write code only to pass a failing test, etc.)
     
    Roberto Waltman, Aug 12, 2013
    #27
  8. James Harris

    James Harris Guest

    ....
    I used that Expect years ago to automate some terminal operations. I must
    say I hated it! I can't remember the details but suspect it was because of
    the influence of TCL. (I subsequently used the telnetlib expect method in
    Python. That, by contrast, was very easy to use and was well thought out.)

    I was thinking of a check-expect clause. Maybe instead of EXPECT I should
    call the macro CHECK or similar as the former term does sound like a
    terminal control mechanism.

    James
     
    James Harris, Aug 13, 2013
    #28
  9. James Harris

    James Harris Guest

    ....
    I find such things much easier to read if they can be expressed on one line.
    That's particularly true where there is a series of tests one after the
    other without any code in between. That helps to see differences between
    tests and to spot if any are missing.

    BTW, what I have tried so far is OK for testing scalars but sometimes we
    want to test composite objects. For example, if generating an array of
    values the test should be that the array (or part thereof) has the values we
    expect.

    Do you guys have any good way of testing such composites? Or do you just
    check each item individually? Say you had an array which, at some point in
    the tesing, should have two or three elements with specific values. Would
    you just write a check for each element or do you have a more clever way of
    testing such things?

    James
     
    James Harris, Aug 13, 2013
    #29
  10. James Harris

    Jorgen Grahn Guest

    Not /influence/ -- expect is an extension to the Tcl language.
    I'm no friend of expect either, but I doubt I'd like this telnetlib
    thing more. It's an inherently fragile approach, unless you have a
    full spec for the interactive program you're automating.

    Still, when you have to use it you're happy to have it. Kind of like a
    heavy revolver when you're in the Arctic -- you may run into a hungry
    polar bear ...

    /Jorgen
     
    Jorgen Grahn, Aug 13, 2013
    #30
  11. James Harris

    Jorgen Grahn Guest

    [Attributions lost, but this is probably the "what I have tried so
    far" below:]
    As far as I'm concerned, it's normal programming. Do it in whatever
    way that seems maintainable. Use helper functions if it seems that
    would help. That's why I'm suspicious of the EXPECT macro above --
    you cannot generally squeeze normal code into such a template, and you
    can't squeeze unit tests into it either.

    If in a bunch of tests it's often useful to assert that a struct Foo
    has some specific properties, I'd write a helper

    void assert_Foo_has_properties(const struct Foo*, properties ...);

    /Jorgen
     
    Jorgen Grahn, Aug 13, 2013
    #31
  12. James Harris

    Ian Collins Guest

    Having multiple test cases in one test function is generally not a good
    idea. It makes spotting the actual failure harder unless you are very
    careful with your failure reports.
    With CppUnit (and I assume other test frameworks) you can define your
    own equivalence operator for a given type.
     
    Ian Collins, Aug 13, 2013
    #32
  13. James Harris

    James Harris Guest

    I'm not convinced by the idea of such a macro either. It was just put in as
    a possibility, and the implementation has issues with different types. That
    said, there may still be some mileage in it. I'm not sure yet.
    For sure that would be good if you had one such function per struct type and
    you wanted to test all the fields each time.

    Here's a case to illustrate why I asked about testing composites. The unit
    under test is a module which manipulates a list. It supports operations such
    as get, set, insert and delete. The tests might be

    initialise the list
    check that list has length 0
    insert 10
    check that list is just the value 10
    insert 11 at the end
    check that list is 10,11
    insert 9 at the beginning
    check that list is 9,10,11
    etc.

    The last test shown above (to check that the list is 9,10,11) could be

    check that list has length 3
    check that list[0] is 9
    check that list[1] is 10
    check that list[2] is 11

    but that is quite long-winded and the resulting set of tests could become
    hard to maintain as we carry out further operations on the list such as
    deletions and sets. Perhaps more importantly, having to test each item
    individually somewhat obscures what we really want to test for in the "list
    is" clause. It would be helpful to have a way to match the whole list in a
    single test.

    James
     
    James Harris, Aug 14, 2013
    #33
  14. James Harris

    James Harris Guest

    I can't quite see this. If we wanted to test the response of a function to
    certain inputs wouldn't it be best to group those test cases together as in

    check that func(1,2,3) returns 4;
    check that func(9,10,11) returns 8;
    check that func(7,0,7) returns -1;
    etc

    Maybe I should look at CUnit (though I don't think anyone has recommended
    it). Or perhaps I should learn some C++ and try CppUnit. From comments on
    this thread it looks like it could be used to test C code. However, I am
    somewhat reticent to start getting into C++ at the moment.

    James
     
    James Harris, Aug 14, 2013
    #34
  15. James Harris

    Ian Collins Guest

    It sometimes is (especially if you are retrofitting tests) and it isn't
    too bad if you can easily tell which of the conditions failed in the
    test output. One drawback is with the frameworks I've used, if the
    first condition fails, the whole test function fails and you end up with
    more test/fix cycles than you would with smaller tests. When writing
    test first, you tend to build up small incremental tests.
    You don't have to learn that much C++ to get going. I think most C
    programmers would be happy with it. I have used writing unit tests for
    C as a good way of introducing C++ to C programmers. Way back testing C
    was how I first learned C++.

    The IDE I use these days (NetBeans) has good support for working with
    both C and CppUnit tests. So the programmer just writes test cases and
    the IDE does the boilerplate. I haven't tried CUnit.
     
    Ian Collins, Aug 14, 2013
    #35
  16. James Harris

    Jorgen Grahn Guest

    Good with an example, because I think you may be making it harder by
    looking at it from a too general perspective. I don't think you need
    a general pattern for writing assertions than you need for, say, text
    output.
    I'd skip the intermediate checks. If the list is empty at the start
    and 9,10,11 when you're done, you can be pretty damn sure it was ok
    inbetween, too. (And presumably you have another test case for
    initialising empty lists, so you only need the last check.)
    Why not simply define an equality test for lists? That leaves you
    with the problem of creating reference lists and not confusing them
    with each other, but perhaps you can fix your tests so that when the
    manipulations are done, the expected contents are 9,10,11. Then you
    only need a 'static struct list ref_9_10_11'.

    Or you could fix your tests so all expected outcomes are lists on the
    form N, N+1, N+2, ... and have an assertion

    void assert_list_is_sequence(const struct List*, int m, int n);

    That would cover all interesting cases except those where you're
    interested in having duplicates (e.g. you're testing a sort function).
    Yes. Test cases need to be as readable as possible.

    /Jorgen
     
    Jorgen Grahn, Aug 14, 2013
    #36
  17. James Harris

    James Harris Guest

    Maybe. I have wondered if I was making a rod for my own back. My thought is
    that if I can work out a general approach now it will make future testing
    easier.

    In fact I have got to a point I am happy enough with to ask for comments on
    (if that's not too much a mix of trailing prepositions). It combines a
    number of the suggestions people have made in this thread. Comments and
    criticsms would be appreciated. Bear in mind that I'm not as familiar with C
    as I would like so I would welcome suggestions even on small things like
    choice of names, use of the elements of C, style etc. The purpose of the
    approach is to use some fairly simple code (no large libraries etc) while
    making it easy to write clear tests for arbitrary types of data.

    The approach is intended to be generally usable but the main example of an
    awkward case is to test manipulation of a list. As mentioned before, for a
    list of, say, five elements, rather than having to write five tests, one for
    each element and probably another one or two to confirm the limits of the
    list, I felt it would be much clearer to test the list as a whole on a
    single line.

    First, the list module has an addional function which converts the list to a
    string. It is named, naturally enough, list_to_striing(). The numbers that
    make up the list are written in decimal with one space between them. At the
    moment that converter is included only if debugging. I have phrased that
    test as #ifndef NDEBUG so it is present by default. I don't know if that's
    considered a good way to do it or not.

    In practice, list_to_string() is quite short so it may be worth keeping in
    the load module even when not debugging. It is limited but it might be handy
    in other circumstances.

    Second, there is a macro which compares two strings and produces a suitable
    one-line report if they are not identical. The macro is called CHECK_SEQ for
    check-string-equal (there is currently another macro to compare integers).
    The string comparison macro is intended to be called as

    CHECK_SEQ(function_call(args), "expected result", "descriptive text")

    Then if the string returned from the function call does not match the
    expected result a one-line report is printed saying what was expected and
    what was received. The function call has to return a pointer to some
    allocated memory.

    The macro is as follows.

    #define CHECK_SEQ(fcall, expected, note) \
    check_buf = fcall; \
    if (strcmp(check_buf, expected) != 0) { \
    CHECK_PRINT_LINE_INFO \
    fprintf(CHECK_STREAM, " expected %s", expected); \
    fprintf(CHECK_STREAM, ", got %s", check_buf); \
    CHECK_PRINT_NOTE(note) \
    fprintf(CHECK_STREAM, ".\n"); \
    check_errors++; \
    } \
    else { \
    check_passes++; \
    } \
    free(check_buf);

    The two helper macros it uses are

    #define CHECK_PRINT_LINE_INFO \
    /* fprintf(CHECK_STREAM, " file %s", __FILE__); */ \
    fprintf(CHECK_STREAM, " line %i", __LINE__);

    #define CHECK_PRINT_NOTE(note) \
    if (strcmp(note, "") != 0) { \
    fprintf(CHECK_STREAM, ", %s", note); \
    }

    Third, and finally, the above makes it easy to write tests (which is the
    whole point of the exercise) such as the following.

    Here are some tests to check the creation of and the effects of changes to a
    list.

    list_create(list, space, ELEMENTS);
    CHECK_SEQ(list_to_string(list), "", "new list")

    list_insert(list, 0, 0);
    CHECK_SEQ(list_to_string(list), "0", "single-element list")

    list_insert(list, 1, 1);
    CHECK_SEQ(list_to_string(list), "0 1", "two elements")

    list_insert(list, 2, 2);
    list_insert(list, 1, 11);
    list_insert(list, 0, 10);
    CHECK_SEQ(list_to_string(list), "10 0 11 1 2", "many elements")

    To me, the above is a bit wordy because the list needs to be manipulated
    between tests. To try to illustrate why I think the macro part of the
    approach is useful here are some tests which check the results of a
    function. They can be one per line. IMO the brevity makes the tests easier
    to write and easier to read. For the moment, at least, these compare
    integers rather than strings.

    CHECK_IEQ(list_scan_ae(list, 0, 4), 0, "scan from 0 for 4")
    CHECK_IEQ(list_scan_ae(list, 3, 12), 4, "scan from 3 for 12")
    CHECK_IEQ(list_scan_ae(list, 0, 55), 5, "scan from 0 for 55")

    ....
    I can see that it could be done and there is a possiblity that an equality
    test will be needed anyway. One big advantage of strings, though, is that
    although C doesn't support arbitrary use of manifest constants string
    literals can be written directly anywhere they are needed. For the case in
    point I can just compare the results of the call against "9 10 11". There is
    no need to separately define a variable with 9,10,11 as content.

    Also, strings should be immediately usable in other contexts apart from
    lists of numbers so similar comparisons should work for other data types.
    I'm aware that they wouldn't work well for comparisons other than equality
    but testing whether the results of a call are equal to what was expected are
    probably the most useful.
    Agreed. I did think about that when I was trying to work out how best to do
    this ... before strings came to the rescue. :)

    James
     
    James Harris, Aug 15, 2013
    #37
  18. James Harris

    Jorgen Grahn Guest

    Speaking of TDD, does anyone here use it together with C?

    I meet people once in a while who like the idea, and I've read a
    pamphlet by Kent Beck ... but I've never really done it according to
    the dogmas. Or seen anyone else do it for that matter, in any
    language ... but I tend to work in conservative places.

    (I sometimes write tests before the code, but I understand this does
    not qualify as TDD.)

    /Jorgen
     
    Jorgen Grahn, Aug 17, 2013
    #38
  19. James Harris

    Jorgen Grahn Guest

    Maybe. For me, sometimes such things work out. More often I need
    concrete experience first and then maybe I can step back and see a
    general pattern.
    I didn't read all of it, but the basic idea seems to be to convert
    values to strings, and use that in your checks/assertions.

    Doesn't seem too bad. It's an attractive feature that one thing takes
    care of both the assertion and the printing of "this is why your test
    failed".

    Just be prepared to run into cases where this isn't the best approach.

    /Jorgen
     
    Jorgen Grahn, Aug 17, 2013
    #39
  20. James Harris

    Ian Collins Guest

    Yes, I do. With C, C++, JavaScript and PHP. The same techniques apply
    to any language.
    I find it really helps if I'm not sure of the best way to solve a
    problem. I start with the very basics and build a solution. If I'm
    doing a demonstration, I often use curve fitting, starting with the
    gradient of a straight line, then moving on to a generic solution.

    When it's all done, you have a full set of regression tests ready to go.
     
    Ian Collins, Aug 17, 2013
    #40
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.