Re: When is unit-testing bad? [was: Re: does lack of type...]

Discussion in 'Python' started by Cameron Laird, Jul 1, 2003.

  1. In article <>,
    Edvard Majakari <> wrote:
    >
    >I'm rather interested in (and fond of) unit testing as well as TDD, so I
    >though to share my opinions. I once discussed TDD with my friend in here,
    >and the common agreement seemed to be that TDD is not a silver bullet -
    >often there are types of computational modules that are very hard to unit
    >test, eg. think of graphical user interfaces or complex systems, where
    >forking or threading is concerned.

    .
    .
    .
    While we're all being rational enough to look at bundles of
    costs and benefits, this observation hints at a different
    insight: that it's possible to *design* for testability.

    Yes, GUIs and forking and threading are hard to test. SO
    DON'T DO THEM. Or, as I prefer, adopt models of GUI and
    concurrency development that lend themselves to testing.

    At this point, I should explain what I mean by that. I
    haven't yet figured out how to say it in less than a book;
    therefore, all I can contribute for today are the slogans
    above.

    If you're in a situation where unit-testing is bad, you're
    probably in a bad situation. Change your situation.
    --

    Cameron Laird <>
    Business: http://www.Phaseit.net
    Personal: http://phaseit.net/claird/home.html
     
    Cameron Laird, Jul 1, 2003
    #1
    1. Advertising

  2. "Aurélien Géron" <> wrote in message
    news:bdunoh$1pms$...

    > The same goes for GUIs. I'd be interested to know about projects where
    > they automate unit tests for GUIs, but I suspect it would be just as
    > cumbersome.


    I don't think one would necessary need to automate the GUI itself.
    A lot of applications seem to have poor separation of the GUI, something
    that the developers of GUI frameworks seem to encourage (not necessarily
    as part of their evil plan). If the end result of designing a GUI application for
    testing was that it was both easy to script and to port to another toolkit, as a
    side effect, than I can see the possible benefit.
     
    Richard Brodie, Jul 2, 2003
    #2
    1. Advertising

  3. Cameron Laird

    David Bolen Guest

    "Aurélien Géron" <> writes:

    > There are things that are hard to unit test (I'm talking about automated
    > unit tests here) and which you just can't do without in some projects. For
    > instance, the database! I worked in a couple of projects with automatic
    > unit tests for the DB related components, and having lived through it, I'm
    > not sure that it was worth it at all! Manual unit tests would have made
    > much more sense. (...)
    > (...)
    > I guess my point is that though you may be right in some specific contexts,
    > I don't think you can generalize. That's IMHO of course.


    I'd agree that there are elements that are much more difficult than
    others to test in an automated fashion - sometimes impractically so
    given the cost to benefits comparison. Personally, I'd agree that
    databases, GUIs and asynchronous activities such as network or
    multi-threading are areas that fall into that category. But I also
    think that such areas can often be tested far more than some might
    think within a given development effort.

    The idea is to strip any layer that can't be (or for a given project
    can't be justified to be) automatically tested down to its bare
    minimum, because an automated test will almost always be of more value
    over time (with repeated use, different developers, etc...) than a
    manual test.

    This is where using the tests during development (as opposed to
    afterwards) can help ensure a test friendly design, and one that is
    often more modular and structured than a typical developer might have
    otherwise arrived at without the tests during development.

    For example, when writing a DB interface layer or component (whether
    for persistence or direct RDBMS access), if you're trying to construct
    tests as you develop, you will more than likely arrive at a structure
    where the final database access layer is isolated enough that you can
    mock it up for most of your tests of the remainder of your code - if
    only because without it you find yourself not able to construct tests :)

    This might take the form of a specific component used to wrap raw
    database access. Or, if it's a SQL database, you might find yourself
    designing business components such that they serve as producers of the
    actual SQL, for which a separate object consumes it and executes it
    against a database. That interface (the SQL output) provides a test
    point for the business logic without involving the database.

    In the former case, tests would simulate the lowest level database
    layer to function just as is necessary for the test (accepting and
    returning appropriate data), thus being able to rapidly and
    efficiently test the remainder of the code without a live database in
    an automated and controlled fashion. In the latter case, the tests
    would execute the business logic and compare the result to expected
    SQL. A much smaller suite of tests to exercise the database layer
    would use a test database (or equivalent) but would be focused
    strictly on the raw database interface, or execution of known SQL test
    statements, and should be much more containable (in time and
    development effort), and probably still automatable, although perhaps
    on a longer cycle time.

    If the system wasn't designed with testing involved, the actual
    database I/O might very well be embedded in a higher layer (such as
    the persistence mechanism which might be tightly coupled to an object
    layer) and hard to isolate from the rest of the logic. For example,
    you might find business objects directly calling a persistence or
    database I/O layer within their implementation, making it hard to test
    the business logic without a live database.

    Similar ideas apply to GUIs - keep the actual interface layer as
    extremely thin as possible, and containing no business logic or
    application decision processing. Whether you use MVC, MVP or an
    equivalent approach, ensure that you can test all underlying business
    logic without actually needing a GUI. This holds for applications and
    web sites (in many respects, a web site can just be modeled as a
    different form of view).

    For example, in MVC/MVP, your automated test can call
    controller/presenter methods to simulate any aspect of user
    interaction with the UI, but without the need for a UI, and can mock
    up a non-UI view to ensure that the model is notified of appropriate
    changes and that queries to it reflect the new information.

    At some point you need to verify that performing action X against UI
    object Y generates the right requests to the other system components,
    but that's far more constrained testing then trying to test the
    business logic through that UI. And yes, that could probably involve
    external UI testing tools, or some manual interaction with an
    automated test mocking the underlying controller/presenter and model.

    Of course, this is all easiest if you are designing a system with
    testability in mind from the get go. Trying to retrofit unit tests
    onto a GUI for example, where much of the business logic is attached
    to specific UI elements and not reachable without going through the UI
    is just plain old difficult. The idea of making up tests as you go
    (whether TDD or equivalent) and using them as part of the development
    process helps not only ensure that the final product is as tested, in
    as automated a fashion, as possible, but that it ends up being
    designed to be tested.

    -- David
     
    David Bolen, Jul 2, 2003
    #3
  4. Cameron Laird

    Peter Hansen Guest

    "Aurélien Géron" wrote:
    >
    > There are things that are hard to unit test (I'm talking about automated
    > unit tests here) and which you just can't do without in some projects. For
    > instance, the database! I worked in a couple of projects with automatic
    > unit tests for the DB related components, and having lived through it, I'm
    > not sure that it was worth it at all! Manual unit tests would have made
    > much more sense. The same goes for GUIs. I'd be interested to know about
    > projects where they automate unit tests for GUIs, but I suspect it would be
    > just as cumbersome. And live without threads? A lot of projects would be
    > better off not using them, I agree with that, but some projects just can't
    > do without them! Web sites are also horrible to automatically unit test.
    > In all those cases, I much prefer having a human being go out and do the
    > unit tests manually.


    Databases are generally considered much easier to do unit tests for than
    GUIs.

    GUIs are recognized as difficult, but there are already fairly well-known
    approaches for simplifying the process, depending on what you need/expect
    out of the testing. For example, one should generally count on the GUI
    framework to do what it does correctly and not try to test, for example,
    that a call to a drawText() method actually puts pixels on the canvas.
    Instead, one should be testing that a call to drawText() is made, and
    contains the correct text and positioning.

    Web sites are equivalently simple to test, if you aren't trying to test
    the presentation itself. For example, unit-testing web forms is pretty
    trivial. In fact, in most cases for web stuff, you don't need to actually
    run a server, but I believe here I'm getting into XP-specific terminology
    where "unit" test refers to something quite different from "acceptance"
    or "customer" test, which is where you would be running the actual server.

    -Peter
     
    Peter Hansen, Jul 2, 2003
    #4
  5. On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
    > I'd be interested to know about
    > projects where they automate unit tests for GUIs, but I suspect it would be
    > just as cumbersome.


    I'm doing one right now. It's designed from the ground up by me, so it's
    testable. It almost has to be, since it's fairly complicated and Needs To
    Work.

    For the record, I've found Tkinter does *great* with this, because you can
    drive the GUI without ever actually starting the event loop. It doesn't
    seem perfect but largely works, much better then it has any right to.

    wxPython doesn't work at all without an event loop, so I never did get it
    tested, which has contributed to me dropping it. ;-) (I'm gonna guess that
    never entered into anybody's head when they were designing toolkits...)

    > In all those cases, I much prefer having a human being go out and do the
    > unit tests manually.


    The thing is, they won't. We've got about 50 years of collective
    experience backing that up.
     
    Jeremy Bowers, Jul 3, 2003
    #5
  6. "Jeremy Bowers" <> wrote in message
    news:p...
    > On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
    > > I'd be interested to know about
    > > projects where they automate unit tests for GUIs, but I suspect it

    would be
    > > just as cumbersome.

    >
    > I'm doing one right now. It's designed from the ground up by me, so it's
    > testable. It almost has to be, since it's fairly complicated and Needs To
    > Work.
    >
    > For the record, I've found Tkinter does *great* with this, because you

    can
    > drive the GUI without ever actually starting the event loop. It doesn't
    > seem perfect but largely works, much better then it has any right to.
    >
    > wxPython doesn't work at all without an event loop, so I never did get it
    > tested, which has contributed to me dropping it. ;-) (I'm gonna guess

    that
    > never entered into anybody's head when they were designing toolkits...)
    >
    > > In all those cases, I much prefer having a human being go out and do

    the
    > > unit tests manually.

    >
    > The thing is, they won't. We've got about 50 years of collective
    > experience backing that up.


    UnitTesting GUI is not that difficult. UnitTesting multi-threaded programs
    is not that difficult. UnitTesting database programs is not that difficult.
    I have experiences in all of them, of which the degree and depth might
    vary.

    For example, I follow a sequence(as in Christopher Alexander's style) for
    TDDable GUI program:


    Principles:
    1. Test everything that is in P and M. Don't test V.
    2. There is no GUI framework API call in P and M.
    3. In V, there are stuffs that you can have enough confidence even without
    testing.
    4. No duplication.
    5. Keep the cycle.

    Sequence:
    1. Start from one chunk. Think about what should be shown from the
    abstract level, what concepts(M) should be there. Don't use GUI API calls.
    2. You can take M and P apart. How it should be shown goes into P. What
    goes into M.
    3. P uses the interfaces of V. Firstly, test P with mock-V. Put into V
    only those that can be considered as "need not tests, really sure" in GUI
    aspects.
    4. As mock-V getting its form, implement real-V along with it.
    5. If there is anything you feel not confident pass it over to P.

    These are what I get after years of experience in TDDing GUI programs.

    see also(all highly recommended):
    * http://martinfowler.com/articles/separation.pdf
    * http://www.objectmentor.com/resources/articles/TheHumbleDialogBox.pdf
    * http://www.sdmagazine.com/documents/sdm0206b/
    *
    http://www.object-arts.com/EducationCentre/Overviews/ModelViewPresenter.htm

    Best wishes,

    June
     
    Changjune Kim, Jul 3, 2003
    #6
  7. On Thu, 03 Jul 2003 01:14:44 +0000, Jeremy Bowers wrote:

    > On Wed, 02 Jul 2003 15:21:45 +0200, Aurélien Géron wrote:
    >> In all those cases, I much prefer having a human being go out and do the
    >> unit tests manually.

    >
    > The thing is, they won't. We've got about 50 years of collective
    > experience backing that up.


    It occurs I can expand on this. What I'm doing right now at my job (as
    opposed to the posting I'm replying to which is my hobby), I'm writing
    unittests for an existing mod_perl-based application that is several years
    old now. Been an adventure.

    Part of the reason I'm doing this is we're safely past the point where we
    can just poke and stab at the system and get any sort of coverage. We've
    got a role-based system (a user can have several types of roles, and
    technically, it's a permission-based system where roles are bundles of
    permissions...), and of course any number of things the user can do. Add
    up 10-ish roles, any number of actions (lets just call it 50 for
    simplicity, though there's no real way good way to put a number on
    "distinct actions in this system"), and a couple of other dimensions, and
    you've got a lot to cover. Well, actually, it's not addition, it's
    multiplication, so even in this rather simple case, you've got 10*50*8 or
    something things to cover.

    Each of these actions are relatively easy to characterize as unit tests
    (well, in proper XP terms these would be acceptance tests, which is still
    a step up for the existing system and while I hope to see true unit tests
    someday, a whole lotta refactoring would have to happen first), but we've
    recently been getting *hammered* with bugs that only occur on one or two
    cells in that (easily) 4000-cell matrix. (Of course they're the .5% fringe
    cases, important enough to actually occur but not important enough to bang
    on in poke-and-grunt testing.) We had a bug that only occurred with Role A
    in situation B in environment-type C. Unfortunately, while none of us
    ever though to test that exact combination of settings, it was reasonably
    common in the real world. Fully automated testing with absolutely no human
    interaction needed during the testing is really, really important,
    and I can't believe it took me this long to realize that. Not to mention
    the rest of the industry.

    On that topic, I'm starting to develop a multi-level unit testing
    philosophy, since it seems like every program I've ever worked on ends up
    with these multi-dimensional tests, and *yes* the equivalent of (1,4,2,4)
    will sometimes fail, all by itself. Is there any official XP dogma on
    that? Running a complete suite would take too long to be practical, so I'm
    starting to get this image of defining a subset of tests for development
    purposes (each axis, selected diagonals, smattering of others), and
    running the full suite, say, overnight for full coverage... or even
    constantly on a spare machine. Would make a great demo for
    tourists coming into the lab, with suitably verbose output, probably
    impress customers too. ;-)
     
    Jeremy Bowers, Jul 3, 2003
    #7
  8. Cameron Laird

    Jim Hefferon Guest

    Re: When is unit-testing bad?

    "Jeremy Bowers" <> wrote
    > We're using mod_perl, and I'm using Test::Unit, which is a testing
    > framework for Perl deliberately set up to look like JUnit, as Python's
    > unittest is. What I'm doing is implementing an Apache Request faked-up
    > object (which I later found existed in the form of the Perl module
    > Apache::FakeRequest, but I think my own solution is better suited to our
    > needs anyhow), and then going ahead and calling the Apache handlers
    > directly with that object. It's tedious to construct the object if you
    > can't find one pre-made, but it's easy.
    >

    Does anyone know of a place where I could see a sample of doing this
    under Python?

    Thanks in advance
    Jim
     
    Jim Hefferon, Jul 4, 2003
    #8
  9. Re: When is unit-testing bad?

    On Sat, 05 Jul 2003, wrote:

    > Part I
    > http://www-106.ibm.com/developerworks/opensource/library/os-puffin.html
    >
    > Part II
    > http://www-106.ibm.com/developerworks/opensource/library/os-puffin2.html


    Having read part I, I really have have to say it looks quite nice tool. I
    guess it's no use to starting my own - also, I'm more convinced now that I
    should do the testing at lower layers, making presentation layer 'thin'
    enough so that there's no need to test it thoroughly. Thanks for the links.

    --
    #!/usr/bin/perl -w
    $h={23,69,28,'6e',2,64,3,76,7,20,13,61,8,'4d',24,73,10,'6a',12,'6b',21,68,14,
    72,16,'2c',17,20,9,61,11,61,25,74,4,61,1,45,29,20,5,72,18,61,15,69,20,43,26,
    69,19,20,6,64,27,61,22,72};$_=join'',map{chr hex $h->{$_}}sort{$a<=>$b}
    keys%$h;m/(\w).*\s(\w+)/x;$_.=uc substr(crypt(join('',60,28,14,49),join'',
    map{lc}($1,substr $2,4,1)),2,4)."\n"; print;
     
    Edvard Majakari, Jul 8, 2003
    #9
  10. Cameron Laird

    Peter Hansen Guest

    Re: When is unit-testing bad?

    Edvard Majakari wrote:
    >
    > On Sat, 05 Jul 2003, wrote:
    >
    > > Part I
    > > http://www-106.ibm.com/developerworks/opensource/library/os-puffin.html
    > >
    > > Part II
    > > http://www-106.ibm.com/developerworks/opensource/library/os-puffin2.html

    >
    > Having read part I, I really have have to say it looks quite nice tool. I
    > guess it's no use to starting my own - also, I'm more convinced now that I
    > should do the testing at lower layers, making presentation layer 'thin'
    > enough so that there's no need to test it thoroughly. Thanks for the links.


    A refinement of the above statement: make the presentation layer
    "thin" enough that you need to test it thoroughly only *once*. Also
    make it thin enough that you very rarely if ever have to make changes
    to it, thereby making such even a manual test adequate.

    You shouldn't _not_ test something... but it's okay sometimes to avoid
    the writing fully automated tests, and this is a good example of when.

    -Peter
     
    Peter Hansen, Jul 8, 2003
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Christopher Blunck
    Replies:
    0
    Views:
    1,205
    Christopher Blunck
    Jun 24, 2003
  2. Veloso
    Replies:
    3
    Views:
    389
    Christopher Benson-Manica
    Oct 12, 2007
  3. rantingrick
    Replies:
    44
    Views:
    1,303
    Peter Pearson
    Jul 13, 2010
  4. Ulrich Eckhardt

    unit-profiling, similar to unit-testing

    Ulrich Eckhardt, Nov 16, 2011, in forum: Python
    Replies:
    6
    Views:
    359
    Roy Smith
    Nov 18, 2011
  5. Bill Mosteller
    Replies:
    0
    Views:
    250
    Bill Mosteller
    Oct 22, 2009
Loading...

Share This Page