![](bg5.png)
unit (or integration) tests. To improve the efficiency
of this model-based testing approach even beyond the
classical introduction of abstraction, the principle of
mutation-based testing was applied. Here, the test
models are slightly modified by a mutation algorithm.
Afterwards, the test case generator produces test
sequences which uncover observable deviations in
the behaviour of these mutants. The intention of this
approach is to improve testing efficiency by
increasing the test coverage rate for the generated test
cases in comparison to a more straightforward
mapping of the test model to a test suite. More details
can be found in (Aichernig, et al., 2014), where the
approach as such has been verified successfully. For
instance, the quality and efficiency of several types of
auto-generated test suites has been evaluated and a
more efficient mutation-based test case generation
algorithm with a significantly reduced runtime has
been developed (Jöbstl, 2014), (Aichernig et al.
2015).
Despite its illustrated usefulness, however, the
industrial adoption of the approach remained limited.
An obvious indicator was the low number of
modelled devices during the project and the amount
of time needed to correctly specify a device model.
For instance, the creation of an initial model took
roughly 12 hours, followed by an additional 40 hours
for fine-tuning and bug-fixing. This estimate does not
include the effort of familiarizing oneself with the
overall model-based testing approach, as well as the
modelling tools, which took 6 additional hours. This
was despite the use of a standard modelling language
(UML) and a standard compliant tool (Papyrus).
In the following, we are analysing possible root
causes of the limited adoption by evaluating the
applied methods of the TRUFAL project against
some of the MDE micro injection paradigms and
characteristics. First, we raise the question if there
was a clear separation of the application and the
introduction of the TRUFAL methodologies. One
could argue that this separation was indeed present
for the following reason: a team of modelling
advocates put reasonable effort on the definition of
the right abstraction for device modelling to fit the
requirements of the mutation-based test case
generation approach. Only due to their expertise, an
adequate level of abstraction was found to fulfil the
project’s aims of automated, reproducible test-case
generation with reasonable test coverage. So, the
baseline for a successful application was set.
However, is such a baseline enough for a successful
introduction?
To answer this question, we examine the
desirability of the approach. In case of the TRUFAL
project, this examination remains inconclusive, since
the end-users have neither been interviewed, nor were
otherwise involved in the design phase. Instead, they
were confronted with model structures, which fulfil
the requirements of the test case generator to enable a
successful evaluation of mutation-based test case
generation methodology but not necessarily reflected
the test engineer’s way of thinking. This led to
scepticism against this approach, which is quite the
opposite of desirability. One could argue that
corresponding modelling courses would bridge this
gap, another could argue that it is still advisable to
minimize the gap by a more suitable language design.
However, since this was considered to be outside the
scope of the TRUFAL project, the corresponding
MDE injection has turned out not to be very viral.
If the MDE injection was not viral, did it at least
fulfil the characteristic of small doses to avoid
negative side-effects? In case of the TRUFAL
project, the test models have been based on UML
state machines and class diagrams, which were
implemented in the UML modelling tool Papyrus.
However, it was as well beyond the scope of the
project to analyse, which subset of these UML
diagrams would be sufficient for test case generation
(e.g. by applying a UML profile) or how Papyrus
needs to be tailored (e.g. limiting menu entries to
those needed by the involved diagram types).
Consequently, the end-users were confronted with a
full-fledged UML tool. Due to the feature-richness of
both the tool and the UML language they had to learn
the semantics of UML first and map their intuitive
view of individual device tests to an appropriate UML
representation using the correct tool features.
Additionally, the underlying test case generator
required some additional model features (specifically,
class diagrams) to specify the test interface. The end-
user had to be aware of all these issues in order to
create a test model suitable for test case generation.
Any mistake led to an obscure failure of the test case
generation process, the root cause of which was hard
to evaluate. Due to limited user guidance by the
model editor, considerable efforts were needed to
train the end-users, which contradicts the MDE micro
injection characteristic of a small dosage. Incorrect
models and misinterpretations of model semantics
caused frustration and led to the subjective
impressions among the test engineers that MDE
introduces complexity rather than diminishing it.
Due to the lack of a UML profile, it could be
argued that defining one would have avoided this
situation. However, defining such a profile does not
intrinsically imply that the right elements are selected
and, even if so, that the semantics of the profile
IndTrackMODELSWARD 2018 - MODELSWARD - Industrial Track
646