Jim Ley said:
I was hoping we could get the hundred monkeys to develop the
test-suite too - so there's just a guideline of how to create,
then they're just reviewed for inclusion by the monkey-minders.
I think you can expect people to want to extend the test-suite (they may
want to test aspects dear to their personal interests that may not have
been covered by others) and they would need guidelines if their
contributions were to become part of a consistent whole. But I think
that we cannot duck having to implement at least some of the test-suite.
Guidelines are easier to follow when you have concrete examples to
follow and there is noting like attempting to implement guidelines to
show up their weaknesses.
... , but I also wanted the reports to be able to be editted
in a text editor etc. as some people will prefer that, and
make it easier to break up testing than a live wizard interface.
Anyway I don't really know, which is why the big problem is the
initial planning.
<snip>
As you note at <URL:
http://jibbering.com/discussion/js-doc-system.1 >
"The biggest problem to solve before any real work can begin, is the
format of the test-results, ...". RDF seems to fulfil your text editor
requirement (just about) and to provide the required flexibility. Your
more specific proposal of EARL (Evaluation and Report Language 1.0 <URL:
http://www.w3.org/TR/EARL10/ > (an application of RDF)) strikes me as
suitable for the final stage of asserting the outcome of the tests
(though it might need the extension of describing the "severity" of a
pass [1]). (incidentally, my plans for the early part of this week have
been put back a day so I had the chance to read the EARL documents
today.)
[1] The "severity" of a pass for DOM compliance of, for example -
document.createElement("input"); - might require that the resulting
object implement the DOM core Element interface (and Node by extension)
and the HTML DOM HTMLInputElement interface but the default values of
properties of the HTMLInputElement may not be set consistently. The -
type - property should default to "text" according to the HTML DTD but
would it be grounds for claiming non-conformance with DOM if it
defaulted to "" as it would be usual practice to assign a value to -
type - immediately after creating the element (even if that assignment
was "text") and the browser implementation may decide to hold of
defaulting un-set values until the node was inserted into a document.
Giving practical conformance if not quite 100%.
The EARL working draft depicts its role in the expressing of test
results with a diagram like (best viewed with a fixed width font):-
___________________________
| |
| Reuirements Capture |
|_________________________|
||
\/
___________________________
| |
| Test Specification |
|_________________________|
||
\/
___________________________
| |
| Test |
|_________________________|
||
\/
___________________________
| | /|_ _
| Test Result | < _ EARL _|
|_________________________| \|
||
\/
___________________________
| |
| Collate Results |
|_________________________|
||
\/
___________________________
| |
| Analyze Results (query) |
|_________________________|
||
\/
___________________________
| |
| Present Results |
|_________________________|
- which places it as an expression of the results and facilitating easy
collation and analysis. But I think that there is more to which a formal
description could be applied and maybe more to record during the
testing.
Expanding the middle of the EARL diagram (based on a personal perception
of the situation (possibly misguided)):-
...
||
\/
___________________________
| |
| Test Specification |
|_________________________|
|| ||
\/ \/
___________ ____________
| | | |
| Test | | Test |
|Procedure| | Criteria |
|_________| |__________|
|| ||
\/ ||
___________ ||
| | ||
| Apply | ||
| Test | ||
|Procedure| ||
|_________| ||
|| ||
\/ ||
__________ ||
| | ||
| Raw | ||
|Results | ||
|________| ||
|| ||
\/ \/
___________________________
| |
| Application of Criteria |
|_________________________|
||
\/
___________________________
| | /|_ _
| Test Result | < _ EARL _|
|_________________________| \|
||
\/
...
EARL already expects a specification of "Test Criteria" which I think
could benefit form a formal description that could be referenced in the
test results. A formal specification of test criteria would allow
alternative test methods to be applied, including entirely manual
testing and allow the results produced to be collated with results for
the same criteria generated by other methods.
I also think it might be worth defining a format for raw test result
storage (there is potentially a lot of extra information about the
browser in those results as they represent what actually happened).
Another aspect of the procedure that might need to be formally expressed
is the dependency of one test on browser/JavaScript features that may be
expressed as the results of another test. For example, while testing a
script in the Palm OS Web Browser 2.0 last night I noticed that its ECMA
Script implementation fails to follow the algorithm for the production
LogicalORExpression (ECMA 262 3rd Ed. Section 11.11) in that - var x =
"" || {}; - will assign a boolean value to the x variable instead of the
specified object. That means that a test script for a more involved
aspect of DOM implementation that uses a binary logical OR operator may
produce a false negative result when the apparent failure could result
from a flawed ECMA script implementation.
Knowing that a test procedure (script) was dependent on a set of other
features would allow either a different script to be used or a low
confidence to be assigned to failure in the event that tests for the
features depended upon had already been failed (or were subsequently
failed).
That observation about the binary logical operators makes me think that
checking the ECMA Script implementation should precede the DOM testing.
So:-
var a = "";
var b = {};
var c = a||b;
if(c){
if(c === b){
c = b||a;
if(c && (c === b)){
//full ECMA 262 3rd Ed.
//section 11.11 LogicalORExpression
//algorithm implementation.
}
}
if(typeof c == "boolean"){
//partial implementation
//producing boolean results.
//good enough for most tests.
}
}
// else total failure of binary logical OR operator.
But that test is in turn dependent on:
1. variable creation and assignment.
2. String and Object literals.
3. if statements.
4. The binary logical AND operator.
5. The identity operator.
6. The typeof operator.
7. Automatic type-conversion.
It must be theoretically possible for a script's dependence on ECMA
implementation to be automatically determined but the Web Browser 2.0
failure highlights the main problem with that as an automatic dependence
determining program would have to discriminate between scripts that
require a full implementation and scripts for which an implementation
that erroneously returned a boolean value would be good enough.
Richard.