Fit Specification: Fixtures

 

Fit documents are owned by the customers and domain experts on your team.  The documents contain examples of how the program should work and Fit checks the examples against the actual program.  The examples are placed in tables with a special first row, but other than that, there are no limitations on how the examples are structured.  Customers should write tables in the way that comes most naturally to them.

 

Since there aren’t any limits to the way tables are structured, Fit can’t interpret the tables automatically.  It requires a fixture to interpret the table.  A fixture is a piece of code, written by the team’s programmers, that understands the structure of each type of table and how to check it against the actual program.

 

Contents:

 

Loading Fixtures. 1

Errors. 2

Interpreting Tables. 2

Default Interpretation. 3

Checking Cells. 4

Right and Wrong. 4

Empty Cells. 4

Unexpected Errors. 4

Expected Errors. 5

Expected Blank. 5

Cell HTML Converts to Text 6

Comparison Values Convert to HTML. 6

Type Conversion.. 7

HTML to Text Conversion. 7

Type Conversion. 7

Predefined Fixtures. 7

Run Results. 8

 

Loading Fixtures

 

The first row of a table is always the name of the fixture used to interpret that table.  There’s no special logic applied; if the first cell in the first row is not the correct fixture name, Fit won’t try to guess where it is.  (Note: we use the keyword “(missing)” to indicate a missing fixture name in the table below.)

 

fat.FixtureNameFixture
Table	FixtureName()
[exampleFixture]	exampleFixture
[exampleFixture] [1] [2] [3] [4]	exampleFixture
[] [exampleFixture]	(missing)
[exampleFixture] [foo]	exampleFixture
[] [exampleFixture]	(missing)

 

The fixture name is the full name of the code that will interpret the table.  (In Java, it’s a class.)  The fixture name is case sensitive and must be fully qualified (that is, include any namespace information).  All implementations of Fit use the “dot” notation of Java to qualify names.

 

fat.FixtureLoadFixture
FixtureName	LoadResult()
fat.ExampleFixture	loaded
fat.exampleFixture	error
Fat.ExampleFixture	error
ExampleFixture	error
fat.Example	error

 

Errors

 

When the fixture isn’t valid, Fit annotates the cell with an explanatory error message.  What makes something a fixture or not depends on which implementation of Fit you’re using.  In Java, for example, only classes that extend the “Fixture” class are fixtures.  Other languages might use the presence of specific methods to determine whether something is a fixture or not.

 

This error message may be displayed as part of additional information, such as an exception stack trace.

 

fat.FixtureLoadFixture
FixtureName	LoadResult()	ErrorMessage()
fat.ExampleFixture	loaded	(none)
NoSuchFixture	error	The fixture "NoSuchFixture" was not found.
fat.NotAFixture	error	"fat.NotAFixture” was found, but it’s not a fixture.

 

Case-sensitivity errors use the same error message as other “not found” errors.

 

fat.FixtureLoadFixture
FixtureName	LoadResult()	ErrorMessage()
fat.exampleFixture	error	The fixture "fat.exampleFixture" was not found.

 

 

 

NOTE TO IMPLEMENTORS:

The remainder of this document is incomplete.  The above examples should pass.  Don’t try to make any of the following examples pass, as they will surely change.

 

Interpreting Tables

 

By default, fixtures interpret tables from left to right and top to bottom.

 

fat.InterpretationOrderFixture
Table	InterpretationOrder()
[fat.ConcatenateFixture] [top left] [top middle] [top right] [bottom left] [bottom middle] [bottom right]	top left, top middle, top right, bottom left, bottom middle, bottom right

 

Although left-to-right and top-to-bottom is the default, fixtures may interpret tables in any order they please.  The fixture in the following example interprets tables column by column, from right to left and top to bottom.

 

fat.InterpretationOrderFixture
Table	InterpretationOrder()
[fat.ReverseConcatenateFixture] [top left] [top middle] [top right] [bottom left] [bottom middle] [bottom right]	top right, bottom right, top middle, bottom middle, top left, bottom left

 

Default Interpretation

 

Fit’s default interpretation of a cell is to mark it with the “ignore” standard annotation.  (See the annotation specification for details about standard annotations.)

 

fat.InterpreterFixture
InputTable	InterpretedTable()	RenderedOutput()
[fat.DoNothingFixture] [left] [right]	<table>   <tr><td>fat.DoNothingFixture</td></tr>   <tr><td bgcolor="#efefef">left</td><td bgcolor="#efefef">right</td></tr> </table>

 

If an error occurs while interpreting a cell, Fit marks it with the “error” annotation and includes whatever error information is available.  (In Java, this is a stack trace.)

 

??

 

Fixtures may override the default interpretation and do whatever they wish.  The fixture in the following example marks all cells with the “right” standard annotation unless an error occurs, in which case it is marked with the “ignore” standard annotation.

 

??

 

Checking Cells

 

Fit provides a standard service for checking a table cell’s correctness.  Given a table cell and a value, Fit compares the two and annotates the cell accordingly.  (This section assumes that the comparison value is a string.  See “Type Conversion,” below, for information about what happens when it’s not.)

 

Right and Wrong

 

When the table cell and the comparison value match, Fit marks the cell “right” using a standard annotation.  (See the annotation specification for more about standard annotations.)

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()
<td>Something</td>	Something	right

 

When they don’t match, Fit marks the cell wrong and includes the comparison value.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Expected()	Actual()
<td>Something</td>	Something else	wrong	Something	Something else

 

Empty Cells

 

If the table cell is empty, Fit uses the “info” standard annotation to put the comparison value in the cell.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Info()
<td></td>	Anything	info	Anything

 

Unexpected Errors

 

If an error occurs while comparing the table cell to the fixture’s value, Fit uses the “error” standard annotation to put information about the error into the table.  If a stack trace or similar information is available, it is included.

 

The following examples use the keyword “(causes error)” in the “ComparisonValue” column to indicate that an error occurs.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()
<td>something</td>	(causes error)	error

 

If the table cell is empty, Fit uses the “info” standard annotation to put the word “error” into the table.  No error information (such as a stack trace) is included.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Info()
<td></td>	(causes error)	info	error

 

Expected Errors

 

If the table cell contains the word “error,” Fit assumes that an error was expected.  If an error occurs, Fit marks the cell right.  If it doesn’t, Fit marks the cell wrong.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Expected()	Actual()
<td>error</td>	(causes error)	right	n/a	n/a
<td>error</td>	Anything	wrong	error	Anything

 

Expected Blank

 

Fit does not support a standard “blank” keyword similar to the “error” keyword.  (Although the following example uses a “(blank)” keyword to indicate a blank comparison value, this keyword was programmed as part of fat.CheckFixture and is not a standard part of Fit.)

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Expected()	Actual()	Info()
<td>blank</td>	(blank)	wrong	blank	(blank)	n/a
<td>blank</td>	Anything	wrong	blank	Anything	n/a
<td></td>	(blank)	info	n/a	n/a	(blank)
<td></td>	Anything	info	n/a	n/a	Anything

 

If a fixture wishes to allow blank values, it must handle this on its own.  A common approach is for the fixture to look for blank results and to convert them into a special keyword, such as “(blank)” or “(none)”.  Use this keyword in the table when a blank result is expected.

 

Cell HTML Converts to Text

 

The contents of the cell are converted from HTML to text as described in the parsing specification.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Expected()	Actual()
<td><i>anything</i></td>	anything	right	n/a	n/a
<td><i>anything</i></td>	<i>anything</i>	wrong	anything	<i>anything</i>
<td>&lt;i>anything&lt;/i></td>	<i>anything</i>	right	n/a	n/a
<td>&lt;i>anything&lt;/i></td>	anything	wrong	<i>anything</i>	anything

 

This conversion applies to blank cells and the “error” keyword as well.

 

fat.CheckFixture
Cell	ComparisonValue	Annotation()	Info()
<td>&nbsp;</td>	anything	info	anything
<td><font color="red">error</font></td>	(causes error)	right	n/a

 

Comparison Values Convert to HTML

 

When a comparison value is used in an annotation, it’s converted to HTML as described in the annotations specification.  See that section for examples.

 

Type Conversion

 

(to do)

 

Converting strings to objects

Converting objects to strings

Comparing objects

 

Predefined Fixtures

 

Fit includes several predefined fixtures.  Each fixture is described in a separate document, shown in the table below.

 

fat.ReferenceFixture
Description	Location	Result()
Primitive Fixture	primitive-fixture.html
Column Fixture	column-fixture.html
Row Fixture	row-fixture.html
Action Fixture	action-fixture.html
Summary Fixture	summary-fixture.html

 

Run Results

 

fit.Summary

 

 

 

 

 

(to do)

 

Be sure to include right, wrong, exception, and ignore counts in ‘check’ discussion

 

 

Check

Expected values specified in cells are tested for equality with actual values extracted from the program under test.

fat.Equals
type	x	y	=
boolean	true	TRUE	true
integer	00001	1	true
real	1000	1e3	true
string	abc	ABC	false
string	a b c	a b c	true

A sequence of values can be entered and checked as a single (composite) value.

fat.Equals
type	x	y	=
integers	1, 3, 5	01,03,05	true
integers	1, 3, 5	01,05,03	false
booleans	true, true, false	true, false	false
strings	a , b , c	a,b,c	true

Domain values can be constructed from cell contents. Equality comparisons will be subject to appropriate domain rules (i.e. delegated to the domain objects.)

fat.Equals
type	x	y	=
date	Jan 1, 1995	January 1, 1995	true
money	$10000	$10,000.00	true

Floating point numbers are checked to the precision that is normal in the host language. It is possible for a domain object to infer precision from the string representation of expected values. For example, [?]ScientificDouble checks equality to the precision implied by the number of significant digits in a value.

·         non-zeros -- 123.45 is 5 digits

·         zeros between non-zeros -- 100003 is 6 digits

·         zeros in the fraction part -- 12.20 is 4 digits

·         but not leading zeros -- 0001000 is 4 digits

·         and not exponent digits -- 6.02e23 is 3 digits

·         left argument (receiver) controles precision

fat.Equals
type	x	y	=
real	123.45	123.449	false
scientific	123.45	123.449	true
scientific	100003	100003.1	true
scientific	100003.1	100003	false
scientific	12.20	12.210	false
scientific	12.21	12.210	true

Improperly specified values may throw exceptions which are reported in the cell that contains the invalid number. Improper value detection is only as good as would be expected in the host language. (We are assuming that incorrectly converted values will lead to detected errors elsewhere.)

·         xyz is not an integer

·         10000000000000000000000000 may or may not be an integer

fat.Table
fat.Equals
type	x
integer	200
integer	xyz

 

fat.Color
white	white
white	white
white	white
white	yellow

Some character strings have special meaning outside of type conversion.

·         blank -- omit check, report value (reported in gray letters)

·         error -- expect an error

fat.Table
fat.Divide
x	y	divide()
100	2	50
100	2
100	0	error
100	0

 

fat.Color
white	white	white
white	white	white
white	white	green
white	white	gray/white
white	white	green
white	white	gray/white

The framework looks for parse support in different places.

·         fixture

·         domain object

·         type adapter

The framework will allow fixtures to implement custom parsing.

·         yea = true

·         nay = false

Fixture

By default cells are processed in order, by table, by row within tables, and by cell within rows. Distinct routines handle sequencing (plural) and interpretation (singular).

·         doTables -- sequence tables within a document

·         doTable -- interpret a table

·         doRows -- sequence rows within a table

·         doRow -- interpret a row

·         doCells -- sequence cells within a row

·         doCell -- interpret a row

The default interpretation of a cell is to mark it as ignored.

Cells contain givens or expected results. The fixture is responsible for distinguishing givens from expected results, checking expected results against actuals, and indicating the result with the background annotation as follows.

·         white -- no check wanted (blank input)

·         green -- right: expected equals actual

·         red -- wrong: expected not equal actual

·         yellow -- exception: trouble computing actual or comparing result

·         gray -- ignore: desired check not possible

Fixtures cooperate to count the number of each result.

·         counts accumulate between tables

·         counts are summarized as "# right, # wrong, # exceptions, # ignores".

·         runner fixtures (e.g. AllFiles) can control accumulation of counts

Fixtures cooperate to accumulate state that can be summarized at any point in the processing of a document.

·         state variables

·         state lifetime

·         state summary

PrimitiveFixture

The primitive fixture does not depend on TypeAdapter. Type specific functions handle all checking.

·         checkBoolean

·         checkInteger

·         checkFloat

·         checkString

ColumnFixture

A column fixture distinguishes givens from expected values by the form of the column head.

·         name is setter or field

·         name() is getter or field

·         camel case -- present value ( ) = presentValue()

·         deprecate execute()

RowFixture

A row fixture expects all rows to be expected values. The fixture is assumed to have access to actual rows which it pairs with expected rows.

·         name is field or getter

·         name() is getter

·         camel case

Rows are matched by comparing just enough values from the left most columns to get a unique match.

·         some given, none actual -- missing row

·         none given, some actual -- surplus rows

·         one given, one actual -- check remaining columns

·         many given, many actual -- match on more columns

Missing and surplus rows are marked as such in their first column

·         each counts as one wrong

·         surplus rows added to table in gray

ActionFixture

An action fixture distinguishes given from expected values based on keywords in the first column.

·         enter -- put given in setter or field

·         check -- check value with getter or field

·         press -- call action (no check other than for execptions)

·         start -- new actor

An actor represents some interface state such as the currently active window. The actor gives meaning to many of the words in the second column (field to be entered and buttons to be pressed).

·         actor persists between tables

·         initial actor is action fixture (subclass) itself

·         start with no argument specifies initial actor

Start could have various meanings depending on the nature of the interface being modeled.

·         create a new actor

·         return to an existing actor

Other actions, like press, can change the current actor.

·         replace the current actor with a new actor

·         replace the current actor with another preexisting actor

·         restore the previous actor

Subclasses of action fixture can add new action words.

·         camel case

Runner

A runner creates top-level Fixtures and provides them with parsed documents to be annotated.

·         top-level fixture is instance of Fixture

A runner retrieves and disposes of documents as appropriate to its environment. The FileRunner reads and writes html files.

·         input-file only -- default output

·         input-file output-file -- specific output

·         input-directory only -- default output in directory

·         input-directory output-directory -- default outputs in specific directory

·         index.html result summaries

·         - is standard in/out

A runner signals the presence of errors to its environment so that subsequent processing can be triggered.

·         exit status is sum of wrong and exceptions up to 255

·         main is factored to simplify [?]RunnerFixtures

A runner establishes run specific state that may influence loading and/or configuration of the program under test.

·         [?]AssemblyPath ???