Acceptance test spike example

[Steve Jorgensen]

I'm posting this message for 2 reasons.

First, I'm still pretty new and shakey to the whole Acceptance Testing thing,
and I'm hoping for some feedback on whether I'm on the right track. Second,
although all the Agile literature talks about the importance of doing
Acceptance Testing, there's very little in any of the books or out on the Web
that helps with how to do it. If I am on the right track, this will be one
more helpful item folks can find on a Google search.

The code below is basically a spike I wrote in a few hours last night to prove
to myself and my team that it would be feasible to quickly create a simple,
useful Acceptance Testing harness for our learning project.

First, I wrote an example test script I would hope to be able to execute...

===============
?recipeListCount:0
!new
?name:"New Recipe"
name:"PB&J"
?name:"PB&J"
!save
!close
?recipeListCount:1
!goToListItem 1
!openListItem
?name:"PB&J"
===============

In this script, ? means test a value, ! means execute an action, and a line
that starts with neither ? or ! is a value assignment.

Next, here's the Python code I wrote to try to run the script. This was just
proof of concept, so at this point, it just runs a single test script with the
script file name hard coded. Also, for simplicty, the skeleton of the
hypothetical application model object is, for now, in-line in the AT system
code.

===============
import string


class RecipeOrgModel:
recipeListCount = 0
name = "New Recipe"

def new(self):
pass

def save(self):
pass

def close(self):
pass

def goToListItem(self,itemNum):
pass

def openListItem(self):
pass


class ParsedStatement:
name = None
value = ""

def __init__(self,statementText,nameValuesDelim):
delimPos = string.find(statementText,nameValuesDelim)
if delimPos==-1:
self.name = statementText
else:
self.name = statementText[0:delimPos]
self.value = statementText[delimPos+1:]


class TestSession:
fileName=""
lines = None
model = RecipeOrgModel()

def __init__(self,fileName):
self.fileName = fileName

def run(self):
self.loadLines(self.fileName)

for line in self.lines:
if not self.processLine(line): break

def loadLines(self, fileName):
file = open(fileName)
lines = file.readlines()
file.close

lines = map(string.strip,lines)
self.lines = lines

def processLine(self,line):
print(line)

if line[0]=='?':
return(
self.TestValue(line[1:])
)
elif line[0]=='!':
self.DoAction(line[1:])
return 1
else:
self.AssignValue(line)
return 1

def TestValue(self,statement):
parsed = ParsedStatement(statement,":")
valueExpr = "self.model." + parsed.name
valueIs = eval(valueExpr)
valueExpected = eval(parsed.value)
hasExpectedVal = (valueExpected == valueIs)
if not hasExpectedVal:
print(" - Expected " + str(parsed.value) + ", but got " +
str(valueIs))
return(hasExpectedVal)

def AssignValue(self,statement):
parsed = ParsedStatement(statement,":")
exec("self.model." + parsed.name + "=" + parsed.value)

def DoAction(self,statement):
parsed = ParsedStatement(statement," ")
methodCall = "self.model." + parsed.name +"(" + parsed.value + ")"
exec(methodCall)


session = TestSession("test1.txt")
session.run()
===============

Note how the powerful, context-aware exec() and eval() procedures really help
simplify the code.

Finally, here's the output I get when I run this with the example script above
saved in a file called test1.txt in the current directory.

===============
?recipeListCount:0
!new
?name:"New Recipe"
name:"PB&J"
?name:"PB&J"
!save
!close
?recipeListCount:1
- Expected 1, but got 0
===============

And this result is pretty much what we expect to see - cool.

 

[Andrew McDonagh]

I'm posting this message for 2 reasons.

First, I'm still pretty new and shakey to the whole Acceptance Testing thing,
and I'm hoping for some feedback on whether I'm on the right track. Second,
although all the Agile literature talks about the importance of doing
Acceptance Testing, there's very little in any of the books or out on the Web
that helps with how to do it. If I am on the right track, this will be one
more helpful item folks can find on a Google search.

The code below is basically a spike I wrote in a few hours last night to prove
to myself and my team that it would be feasible to quickly create a simple,
useful Acceptance Testing harness for our learning project.

First, I wrote an example test script I would hope to be able to execute...

===============
?recipeListCount:0
!new
?name:"New Recipe"
name:"PB&J"
?name:"PB&J"
!save
!close
?recipeListCount:1
!goToListItem 1
!openListItem
?name:"PB&J"
===============

Snipped.

Its best to aim the Acceptance Test at your Customers level of
expertise, not at the developers. As a Customer is the person best
placed to tell us What they want. Yes Our team people (Devs or testers)
can help write the actual tests, but the Customer should still be able
to read them. After all, the tests are Executable Requirements.

There's two flavors of Customer oriented styles I've come across.

The first uses a Domain Specific Language...eg using your values above..

---------------------------------
IsTotalNumberOfRecipes 0

ClickNewRecipeButton
RecipeNameToUse "PB&J"
SaveNewRecipe
CloseDialog

IsTotalNumberOfRecipes 1

IsThereOneRecipeCalled "PB&J"
---------------------------------

All commands are the first full word. No need for special tokens to
decide what the command is supposed to do, as each Command is the name
of a class to use.

You can use reflection to dynamically create the Command instances or
the name can be used by a factory as an id.

The Commands themselves know what they are supposed to do, not the
Script parsing engine.

The second approach is using FIT/Fitnesse - google these for exact
description. They take information in Table form. This allows the
customer to use any software to create the test scripts, so long as it
can create tables: Word processors, Spread Sheets, etc.

Andrew

 

[John Roth]

Snipped.

Its best to aim the Acceptance Test at your Customers level of expertise,
not at the developers. As a Customer is the person best placed to tell us
What they want. Yes Our team people (Devs or testers) can help write the
actual tests, but the Customer should still be able to read them. After
all, the tests are Executable Requirements.

There's two flavors of Customer oriented styles I've come across.

The first uses a Domain Specific Language...eg using your values above..

---------------------------------
IsTotalNumberOfRecipes 0

ClickNewRecipeButton
RecipeNameToUse "PB&J"
SaveNewRecipe
CloseDialog

IsTotalNumberOfRecipes 1

IsThereOneRecipeCalled "PB&J"
---------------------------------

All commands are the first full word. No need for special tokens to decide
what the command is supposed to do, as each Command is the name of a class
to use.

You can use reflection to dynamically create the Command instances or the
name can be used by a factory as an id.

The Commands themselves know what they are supposed to do, not the Script
parsing engine.

The second approach is using FIT/Fitnesse - google these for exact
description. They take information in Table form. This allows the customer
to use any software to create the test scripts, so long as it can create
tables: Word processors, Spread Sheets, etc.

The Python version of FIT (currently at 0.7a2) can be found in the
file section of either the FitNesse Yahoo group or the ExtremeProgramming
Yahoo group. One of these days I'm going to get it on PyPi.

It's full featured, works both in batch and with FitNesse, and
includes most of FitLibrary.

There's a book coming out on FIT (FIT for Developing Software)
by Rick Mugridge and Ward Cunningham. It was supposed to be out
a week ago Friday, but B&N says it was delayed.

John Roth
PyFit

 

[Terry Reedy]

Note how the powerful, context-aware exec() and eval() procedures really
help
simplify the code.

A stylistic note: I believe that most or all of your eval/exec uses could
be done with getattr and setattr instead, which are in the language for
precisely those situations in which the name of an attribute is in a
runtime string. To my mind, this would be simpler and better style.

valueExpr = "self.model." + parsed.name
valueIs = eval(valueExpr)

I believe this is valueIs = getattr(self.model, parsed.name)

methodCall = "self.model." + parsed.name +"(" + parsed.value + ")"
exec(methodCall)

I believe this is getattr(self.model, parsed.name)(parsed.value).

exec("self.model." + parsed.name + "=" + parsed.value)

I believe this is setattr(self.model, parsed.name, parsed.value).

and so on.

Terry J. Reedy

 

[Steve Jorgensen]

A stylistic note: I believe that most or all of your eval/exec uses could
be done with getattr and setattr instead, which are in the language for
precisely those situations in which the name of an attribute is in a
runtime string. To my mind, this would be simpler and better style.


I believe this is valueIs = getattr(self.model, parsed.name)



I believe this is getattr(self.model, parsed.name)(parsed.value).



I believe this is setattr(self.model, parsed.name, parsed.value).

and so on.

Thanks. I'm new to Python, so I'll take all the style advice I can get.

- Steve J.

 

[Robert C. Martin]

I'm posting this message for 2 reasons.

First, I'm still pretty new and shakey to the whole Acceptance Testing thing,
and I'm hoping for some feedback on whether I'm on the right track. Second,
although all the Agile literature talks about the importance of doing
Acceptance Testing, there's very little in any of the books or out on the Web
that helps with how to do it. If I am on the right track, this will be one
more helpful item folks can find on a Google search.

Check out "Fit For Software Development". It should be published now.

-----
Robert C. Martin (Uncle Bob) | email: (e-mail address removed)
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716


"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo

 

[Steve Jorgensen]

I'm posting this message for 2 reasons.

First, I'm still pretty new and shakey to the whole Acceptance Testing thing,
and I'm hoping for some feedback on whether I'm on the right track. Second,
although all the Agile literature talks about the importance of doing
Acceptance Testing, there's very little in any of the books or out on the Web
that helps with how to do it. If I am on the right track, this will be one
more helpful item folks can find on a Google search.

The code below is basically a spike I wrote in a few hours last night to prove
to myself and my team that it would be feasible to quickly create a simple,
useful Acceptance Testing harness for our learning project.

....

Here's an updated script and code to process the script. I made the script
syntax more friendly.


New example script...

==========
Check recipeListCount is 0

Do new
Check name is "New Recipe"

Keyin "PB&J" to name
Ch eck name is "PB&J"

Do save
Do close
Check recipeListCount is 1

Do goToListItem 1
Do openListItem
Check name is "PB&J"
==========


New Python code for test runner including skeleton of application model that
can be tested, but can't pass the tests...

==========
import string


class RecipeOrgModel:
recipeListCount = 0
name = "New Recipe"

def new(self):
pass

def save(self):
pass

def close(self):
pass

def goToListItem(self,itemNum):
pass

def openListItem(self):
pass


class Action:
failMessage = ""

def __init__(self, actionArgs):
pass

def tryAction(self, model):
return True


class NoAction (Action):
pass

class ActionDo (Action):
item = ""
args = ""

def __init__(self, actionArgs):
delimPos = string.find(actionArgs, " ")
self.args = ""
if delimPos==-1:
self.item = actionArgs
else:
self.item = actionArgs[0:delimPos]
self.args = string.strip(actionArgs[delimPos+1:])

def tryAction(self, model):
methodCall = "model." + self.item +"(" + self.args + ")"
exec(methodCall)
return True


class ActionKeyin (Action):
item = ""
value = ""

def __init__(self, actionArgs):
delimPos = string.find(actionArgs, " to ")
self.args = ""
if delimPos==-1:
self.item = actionArgs
else:
self.value = eval( actionArgs[0:delimPos] )
self.item = string.strip(actionArgs[delimPos+len(" to "):])

def tryAction(self, model):
setattr(model, self.item, self.value)
return True


class ActionCheck (Action):
item = ""
expectValue = ""

def __init__(self, actionArgs):
delimPos = string.find(actionArgs, " is ")
self.args = ""
if delimPos==-1:
self.item = actionArgs
else:
self.item = actionArgs[0:delimPos]
self.expectValue = eval( string.strip(actionArgs[delimPos+len(" is
"):]) )

def tryAction(self, model):
valueIs = getattr(model, self.item)
if self.expectValue == valueIs:
return True
else:
self.failMessage = ( "Expected " + str(self.expectValue) +
" but got " + str(valueIs) )
return False


class ActionUnknown (Action):
actionArgs = ""

def __init__(self, actionArgs):
self.actionArgs = actionArgs

def tryAction(self, model):
self.failMessage = "Test statement not understood: " + self.actionArgs
return False


def MakeTestAction(lineText):
delimPos = string.find(lineText, " ")
commandArgs = ""
if delimPos==-1:
commandType = lineText
else:
commandType = lineText[0:delimPos]
commandArgs = string.strip(lineText[delimPos+1:])

if commandType == "Do":
return ActionDo(commandArgs)
elif commandType == "Keyin":
return ActionKeyin(commandArgs)
elif commandType == "Check":
return ActionCheck(commandArgs)
elif commandType == "":
return NoAction(commandArgs)
else:
return ActionUnknown(commandType + " " + commandArgs)


class TestSession:
fileName=""
lines = None
model = RecipeOrgModel()

def __init__(self,fileName):
self.fileName = fileName

def run(self):
self.loadLines(self.fileName)

for line in self.lines:
print(line)
action = MakeTestAction(line)
actionOk = action.tryAction(self.model)
if not actionOk:
print(" !!! " + action.failMessage)
break

def loadLines(self, fileName):
file = open(fileName)
lines = file.readlines()
file.close

lines = map(string.strip,lines)
self.lines = lines


session = TestSession("test1.txt")
session.run()
==========


Ouptut of test runner using example script...

==========
Check recipeListCount is 0

Do new
Check name is "New Recipe"

Keyin "PB&J" to name
Check name is "PB&J"

Do save
Do close
Check recipeListCount is 1
!!! Expected 1 but got 0
==========