![](bg6.png)
guard evaluates to Unknown, since the guard is only
sufficient.
The reason why we might need to settle for
guards that are sufficient but not necessary is the
potential difficulty of computing the guard g. In
general, g will be a function of both the input
entered at S, and the inputs entered on the subpath p
that led up to S during an execution of P. The steps
on the path p leading up to S provide the context for
the computation step that occurs at S. If that step is
context free, then the guard will only depend on the
input entered at S. If the step is not context free g
will have to evaluate the effects of the inputs along
the path p leading to S. It cannot do this by simply
observing the state of P at S, since then we would be
using information about the state of P in order to
validate the behaviour of P, which is circular
reasoning. There are several possible approaches to
this problem.
For the guards in our test model M, we
constructed simple path pattern recognition routines
that could be used to define simple sufficiency
guards. These routines are capable of examining a
path p leading to a state S to see if a sufficient
pattern of steps has occurred.
For example, the state deleteMember in the
model M has a transition to a next state
noSuchMember. Assume that, during testing, the
program always starts out with an empty
membership data base. Examples of possible
sufficiency guards that could be used on the
transition include the following path patterns: (no
addMember), or (for each addMember(x) there is a
deleteMember(x)). If an execution path p arrives at
S = deleteMember, and one of these simple guards is
satisfied for p and the data entered at S, then we
know that for that execution the program should
transition to the model state noSuchMember. The
transition to a different next state constitutes
incorrect behaviour. There are more complex paths
to S that do not satisfy these simple patterns, for
which the correct next state is also noSuchMember,
and along which the program also fails. For these
tests we would not be able to determine failure using
the guard, since the guard for the transition to
noSuchMember would return Unknown, and not
True. But all we need is one path for which the
guard evaluates to True.
An alternative to the use of guard path patterns is
the "second program" approach. An oracle program
would be used to determine the correct program
states and transitions. This approach may be both
feasible and necessary for some applications, but it
seems that the path pattern approach will often be
simpler.
There are several ways to use test models. One
involves a test harness that traverses paths in the
model, keeping track of path coverage, while
simultaneously executing the program under test on
the input derived from the domain generators in the
model states occurring along a followed path. As it
follows a path, the traverser evaluates program
behaviour and checks guards to see if an observed
transition to an observed screen is valid.
The second way of using a test model is to have
a separate test specifications generator that traverses
the model, generating model paths that are then
handed off to a test harness that works with one path
at a time. In our testing tool we used this approach.
The model traverser generates a test specification in
the form of a FIT test table, which is handed off to a
FIT test runner (Mugridge, 2005). This was done
for several reasons. One was to make use of an
already existing FIT tool. The other was related to
the desirability of the basic approach to systems
testing that is followed in FIT. FIT uses test fixture
classes to map from easily readable FIT test tables to
the underlying application code that has to be
executed for the specified steps in the table. We
used similar test fixtures to map from the test model
to the steps that should appear in a FIT test table. A
more detailed description of this use of the FIT
testing strategy is contained in (Barzin, 2008).
6 CONCLUSIONS
The necessity/sufficiency framework, introduced in
this paper, was found to be useful way to
characterize both individual oracles and the
construction of more general oracles from less
general components. It has been used to analyze
both hybrid and non-hybrid oracles, such as the two
examples included here. The elusive bug hypothesis
(EBH), also introduced here, is consistent with our
analysis of elusive defects and establishes a basis for
further research into the elusive bug problem. The
BET hypothesis seems intuitively reasonable on its
own, but it is the EBH that may be one of the
principal reasons why BET is effective: limited tests
are adequate because they generally contain
instances of elusive bug combinations, and these
combinations cause a failure whenever they appear.
In the case of interactive programs, the combinations
of conditions causing a failure occur on short paths,
which are associated with simple sufficiency guards.
The use of incomplete necessity and sufficiency
oracles in automated testing can be justified per se
because they will be least as effective as robustness
testing, which uses a kind of minimal necessity
oracle. The addition of more general sufficiency
ICSOFT 2008 - International Conference on Software and Data Technologies
120