![](bg7.png)
to be added to the “measure” characteristic, because
in our case this element was addressed by several
sources. It describes an aspect relevant to the
fulfilment of the principle and will be defined as:
“level of acceptance of the principle by all of its
users”. The attribute “preconditions fulfilled”, related
to the “prerequisites” characteristic, is also relevant
to add. We explicitly saw in the case that, when
preconditions were set, it was also relevant to know
whether the preconditions were fulfilled. The
definition of this attribute can be described as “the
level of fulfilment of the preconditions defined.”
For the architecture principle set we will add an
extra characteristic: “prerequisites”. We discovered
in the case study that some prerequisites were not
related to a specific principle, but to a group of
principles. Besides the “precondition” attribute,
“basic assumptions” were described for some sets as
well. Basic assumptions are “relevant criteria for
successful use of the principle”.
In this case we did not find any inconsistencies in
the coherence of the description model. Some of the
relationships as described in the model, e.g. “depends
on” or “level of fulfilment”, were described explicitly
in the documents or way mentioned during the
interviews. The amount of information, though, is
insufficient to make fact-based statements about the
consistency of the coherence. In this case it was clear
that there are interrelationships between attributes,
e.g. the missing of the rationale and therefore a lower
score at quality, as we already described in (Borgers
and Harmsen, 2016). More research data is necessary
to make clear statements about the coherence.
4.3 Evaluation of the Measurement
Method
We evaluate the measurement method by discussing
the reliability and validity of the case study results.
To challenge the reliability of the results, we want to
know to what extent the results would be consistent
when doing the case study research again. In the
measurement method are different protocols defined
to assure the reliability of the outcome: using
different kinds of data collections, working with a
research team, minutes including feedback, etc. All
these mechanisms are important, because this case
demonstrates the subjectivity of facts collected. Two
architects, for example, were working closely
together during the project, but had different opinions
about the fulfilment of some of the architecture
principles. The research team could, based on all
different sources, make an expert judgement about
the fulfilment.
Although we used different ways of collection
data, in this case study we were lacking some in-depth
information about the essential requirements. As a
result, it was difficult to see to what extent
architecture principles were adding value in meeting
the essential requirements. Additional sources related
to the essential requirements, e.g. interviewing extra
business owners, would help to bridge this gap.
In evaluating the validity of the measurement
method, we concluded that the description of the
architecture principles reflects the real situation of
TDi. Although we found some contradictory data,
especially in the interviews, we were able to explain
the differences in the data. In this specific case we
were not always able to go into details of specific
architecture principles. Because the case used quite
some principles, 36 in total, it was difficult to address
all individual principles. So, in following cases we
need mechanisms to get more in-depth information
about the individual principles.
5 LIMITATIONS, CONCLUSIONS
AND NEXT STEPS
The aim of this research was to build and test the
architecture principle measurement instrument. To do
so, we chose the case study approach to juxtaposition
theory with practice.
5.1 Limitations
Although the arguments for using the case study
approach are still valid, there are some limitations
important to address in this case study.
We are aware that one case cannot prove the
completeness of the measurement instrument. As
discussed in section 4, the objective of this research
is to test to what extent the instrument is useful in
practice in the first place. For an extended test on
completeness and coherence of the measurement
instrument, we need more test cases.
A second limitation might be that the researchers,
although all kind of protocols are defined, are biased
in searching for characteristics and attributes. The
moment we are introducing a model as a description
of our research object, we see architecture principles
through this model. We tried to avoid this prejudice
by avoiding naming of attributes during the survey
and interviews. The fact that we identified new
attributes and characteristics, shows that we were
open for new elements as well.
Finally we can also note that there is currently no
ICEIS 2019 - 21st International Conference on Enterprise Information Systems
540