Using Multi-criteria Analysis to Evaluate
Enterprise Architecture Scenarios
Francisco Cansado
1
, André Vasconcelos
1
and Gonçalo Santos
2
1
Instituto Superior Técnico and INESC Inovação, Av. Rovisco Pais, Lisboa, Portugal
2
Deloitte Consultores, S.A, Praça do Duque de Saldanha, Lisboa, Portugal
Keywords: Enterprise Architecture, Evaluation, Multi-criteria Analysis, Weighting Methods, Metric Selection.
Abstract: In this paper we propose a framework based on multi-criteria analysis as a way to conduct an enterprise
architecture scenario evaluation and selection. The proposed solution supports decision making, by
evaluating each of the available scenarios using metrics derived from defined goals, based on a Goal-
Question-Metric methodology. In this approach the overall goals or objectives (the problem’s criteria) are
on the top of the GQM-tree, whereas questions of stakeholders and measurable attributes (also called
“metrics”) are represented as intermediate and leave nodes. According to the multi-criteria analysis each of
the alternatives x is evaluated on each metric i, by means of a value function Vi(x). Then, since every
criterion has a weight which represents its relative importance in the analysis, the final overall result of each
alternative scenario is therefore computed through a weighted sum model.
1 INTRODUCTION
In this paper, we analyse how multi-criteria analysis,
can be used in the context of enterprise architecture
scenarios. When organizations face a change
situation, multiple scenarios may be available to
implement that change, each of them with different
impacts, and opportunities.
Figure 1: Scenario selection problem.
The decision of which scenario to implement,
requires a set of metrics to quantify and compare the
existing scenarios between each other, in a formal
way. This formal evaluation has its advantages over
an informal evaluation, because it will produce
results that are better supported and justified, when
compared to the results obtained informally.
We will show how the problem of selection
enterprise architecture scenarios can be supported by
a framework based on multi-criteria analysis.
1.1 Enterprise Architecture
The enterprise architecture, by definition is a
representation of a real world organization, either in
an “As Is” state, or in a possible “To Be” scenario.
This representation can be compared to a city’s
architecture, although it hides some details about the
real city, it gives us a high level view. This holistic
view on enterprises can be used to extract some
broader indicators about the status of the actual
organization, enabling us to place some
characteristics in evidence.
The creation of an enterprise architecture, is an
essential step in a modern organization. It allows
stakeholders to communicate and facilitates decision
making within the organization, by creating a
common understanding of the organization and its
elements (Johnson et al., 2004).
It is not only a communication tool but it also
allows to understand how the different components
of an organization, actors, processes, applications
interact with each other, and assure that they are
correctly aligned and together contribute to achieve
the organizations goals (Lankhorst, 2009). It is also
232
Cansado F., Vasconcelos A. and Santos G..
Using Multi-criteria Analysis to Evaluate Enterprise Architecture Scenarios.
DOI: 10.5220/0004097502320237
In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS-2012), pages 232-237
ISBN: 978-989-8565-12-9
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
an evaluation tool through the usage of metrics that
allow us to check the organization status regarding
certain qualities (Vasconcelos et al., 2005).
The framework proposed in this work, is one of
such evaluation methods, with the objective of
helping in the scenario selection problem, described
early.
The structure of the paper is as follows, first we
start by defining enterprise architecture in the
context of this work, and why its evaluation is
important. Then we present how multi-criteria
analysis adapts to this particular context and what
steps are needed to conduct the proposed analysis.
Finally future work and conclusions are presented.
2 MULTI-CRITERIA ANALYSIS
PROCESS FOR ENTERPRISE
ARCHITECTURE
2.1 Enterprise Architecture Evaluation
Since enterprise architectures represent the real
organizations in a holistic view, with its elements
connections and rules (Fischer and Winter, 2007), it is
possible to evaluate the organization through its
architecture, using metrics (Vasconcelos et al., 2007).
In our proposal, this evaluation corresponds to
apply a set of metrics, to a set of scenarios. Through
the application of these metrics the different
scenarios will be scored, and their score will be used
to compare them.
These metrics are defined as a way to measure
and evaluate different qualities, and their objective is
to reduce the uncertainty level related with some
reality by quantifying it.
According to (Blackburn and R. Valerdi, 2009)
the metrics must be aligned with some objective, in
order to quantify it.
Our goal is to realize a multi-criteria analysis,
using these metrics as the criteria, and the enterprise
architecture scenarios as the possible alternatives.
2.2 Multi-criteria Analysis
Multi-criteria analysis is a method for selecting an
option given a set of criteria. In other words, it is a
process to discover the most preferred option, given
a set of criteria (Dodgson et al., 2009).
It allows us to structure a complex problem with
multiple options and restrictions. This is possible by
identifying the existing points of view over the
problem, and analyse them one at a time, and then
through the usage of a weighting method compute
the overall result of each alternative.
This method has been widely tested in various
contexts with good results, supporting the option
selection problem in a structured and formal way.
There are some variants of multi-criteria
analysis, depending on the selected method to
realize each of its steps. Nevertheless there is a
common set of structural elements present in all the
multi-criteria analysis methods:
Criteria: they represent a stakeholder’s point of
view and concerns in the problem. It’s possible
to create the problem’s set of criteria using two
approaches, bottom-up or top-down, they are
described in section 2.5;
Alternatives: are the scenarios which we are
evaluating, the possible options to choose
from;
Decision makers: are the stakeholders in the
problem’s context, they must be able to
understand the criteria, in order to give their
preferences;
Uncertainty: since not all factors can be
controlled in a given context, when we build
the alternatives, different possibilities
regarding the uncertainty must be generated;
Environment: is the whole context where the
problem and the analysis are developed. There
are a multitude of factors that can affect the
analysis, and the best solution can be different
depending on the context.
The procedural structure of the analysis presented
over the next sections, is shown in Figure 2. For
each of the steps we present possible methods to
implement the given step, focusing on the enterprise
architecture scenario evaluation problem.
Figure 2: Proposed method steps.
2.3 Context Definition
This is the first step of the analysis. The scope and
constraints of the problem are defined, in order to
have a complete view of the problem. To do so it’s
necessary to first identify the stakeholders, since
UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios
233
they are the decision makers and will be asked for
feedback in future steps of the method. In our
problem the context of the analysis, will be the
change’s scope, this means that the decision makers
will be the stakeholders and the teams responsible
for the implementation of the change process.
2.4 Identify Alternatives
After having the context of the problem analysed
and its scope clarified, we need to identify the
possible solutions being evaluated. Depending on
the nature of the problem, these scenarios, may be
already defined or may be generated in this step. In
our problem the alternatives correspond to the
possible To Be architecture scenarios.
All the scenarios must be described with the
same level of detail and focus. This means using the
same framework and architecture viewpoint for all
scenarios, otherwise the score of the scenarios could
be biased since not all the scenarios would have the
same elements or information represented (Johnson
et al., 2004).
2.5 Identify Criteria
In order to evaluate the different options, we must
also identify the existing criteria. This process
depends on the context where the analysis is being
done and the selected approach (Blackburn and R.
Valerdi, 2009):
Top-Down: in this approach, criteria are
structured in a hierarchical manner. This
structure can be seen as a tree, where the main
objectives are in the top and are successively
detailed into more specific criteria.
Bottom-Up: in this method, criteria are identified
through an elicitation process, and then grouped
in broader categories or objectives.
In our proposal, we will be using a Top-Down
approach, starting by identifying the main broader
scope objectives (Basili et al., 1994). Since our
context is an organizations enterprise architecture,
more specifically during a change situation, one of
the concerns is align the objectives of the change
with the organization’s objectives. One way to
achieve this is to look at the organization strategic
map and scorecards in order to extract the high
levels objectives of the organization.
2.5.1 Strategic Map and Balanced Scorecard
The Balanced Scorecard is a widely used,
organization performance measurement tool, and it
allows managers to keep track of the defined
objectives, as well as measures used to evaluate those
objectives (Kaplan and Norton, 2008). A typical
scorecard will contain various objectives regarding
different organization domains. These objectives can
be from the financial domain, client / market domain
or the organization development and learning domain.
It’s a flexible approach allowing virtually, all types of
organizations to represent their goals.
In order to make the link between the high level
strategy and the objectives in the balanced
scorecard, a strategic map can be used. This type of
artifact shows the link between objectives, allowing
the stakeholders to see the dependencies between
objectives (Kaplan and Norton, 2008).
2.5.2 Metric Selection
Since objectives, are not normally directly
measurable, in order to quantify them we must detail
objectives into metrics, following the Top-Down
approach described early. We will use the Goal
Question Metric process (Basili et al., 1994). This
methodology allows us to select a group of
measurable metrics, that will evaluate the defined set
of goals.
The process starts by identifying the goals, and
in our case these goals correspond to the ones
extracted from the strategic map and scorecard (V.
Basili et al., 2007).
Secondly, stakeholders are asked to define
questions, that when answered would allow them to
be confident about the achievement of each
objective. These questions will indicate what entities
we need to measure, and what metrics should be
used. The next step is to look at the proposed
questions, and define metrics that allow us to give a
quantitative answer to each of the questions. This
process generates a tree, with goals on top that
derive into questions and that in turn are linked to
metrics. These goals and metrics are the criteria we
will be using in our proposed analysis to evaluate the
existing scenarios.
2.6 Evaluate Scenarios
Having identified the available alternatives and the
evaluation criteria, we must evaluate each scenario
versus the identified set of criteria, in order to obtain
the scenarios score in all criteria. Depending on the
nature of the criteria, the evaluation may be
different, going from counting elements in a given
scenario’s architecture to subjective metrics like
preference regarding some service provider. The
ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems
234
usage of subjective metrics although possible must
be limited, and whenever possible is better to choose
a more objective metric. At the end of this step, the
responsible for the analysis, must be in possession of
each scenario’s scores.
2.7 Weight Criteria
Most of the choice problems analysed in real life do
not have a single selection criterion, but multiple
criteria as presented in multi-criteria analysis. But
since not all criteria are equally important, some sort
of compensation, must be applied so that a more
important criteria, contributes more to the overall
score than less important criteria.
To do this compensation there are several
weighting methods available. In the next sections we
describe several weighting methods that can be
integrated in a multi-criteria analysis (Dodgson et
al., 2009).
2.7.1 Trade-off
This method can reveal the indecisions faced by
stakeholders, comparing pairs of criteria. The
process is the following: for each pair of criteria,
two hypothetical alternatives are constructed, one of
them has the best score on criterion A and the worst
on B, the other alternative is the reverse of the first
one. We start by asking the stakeholders which is the
preferred scenario, and after they made their choice,
we ask how much they were willing to sacrifice the
best performing criterion, in order to maximize the
worst. The answer to these questions reveals the
Trade-Off between the two criteria, or on other
words, the weight associated with which criterion
(Daniels et al., 2001).
2.7.2 SWING
The SWING method also requires generation of
hypothetical alternatives, in this case only two, a
Worst alternative (W), where all criteria have the
lowest possible score and a Best alternative (B),
where all criteria have the best possible score
(Mustajoki et al., 2005).
This method starts with the scenario W, and the
stakeholders are asked which criterion they want to
move first from W to B, and a value of 100 points is
attributed to this criterion. Next they are asked
which criterion they wish to move next from W to B
and how much they value this transition comparing
to the 100 points of the first choice. This last step is
repeated for every criterion, and at the end we will
have all the criteria weighted relatively to the most
preferred criterion, in a normalized scale, since all
weights are contained in the [0;100] interval.
2.7.3 Change Resistance
In this approach each criterion is given two different
performance poles, best and worst, assuming that all
criteria are desirable in the final solution. By putting
all criteria in the best performance, and asking to the
stakeholders to compare all the criteria pairwise, and
choose one to be moved from best to worst state,
repeatedly, until all criteria have been compared
with the rest. The number of times a criterion
maintains its best performance, or in other words,
resists change, is the weight of that criterion.
2.7.4 Macbeth
The Macbeth method regards not only the weighting
step of the analysis, but it integrates weighting
criteria as an essential part. It has some swing and
trade-off, elements, like generating hypothetical
scores (good and neutral), for each criterion. The
objective of this method is to build a cardinal scale
of value, regarding the stakeholder’s preferences, or
alternatives attractiveness, like described in (Bana e
Costa et al., 1997).
2.7.5 Holistic
The holistic approach, as the name suggests, takes in
account the complete set of criteria and the
stakeholders are asked to rank the alternatives
regarding the overall score. In order to extract the
individual criterion weights, is necessary to apply
regression statistical methods. This process although
simple for the stakeholders, since they don’t have to
worry about the individual weights, causes other
problems like judgement inconsistencies, because
stakeholders are unaware of certain factors when
thinking over the full criteria set instead of each
criterion at a time. The need for statistical regression
operations, also adds complexity to the work of the
analyst realizing the analysis (Dodgson et al., 2009).
2.7.6 Selected Weighting Method
In our analysis we need each criterion individual
weight, relatively to the rest of the set, in order to
compute a global score combining the determined
weights with the scenarios score obtained in the
previous step, section 2.7.2. Any of the suggested
weighting methods could be used but in our proposal
we will use SWING, due to its simplicity, the
capacity to deal with large criteria number without
UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios
235
adding to much complexity to the analysis and
because it provides all its weights in a normalized
scale that will facilitate computing the overall score.
2.8 Decision and Sensitivity Analysis
Finally with the overall scores of the selected
scenarios, it’s still necessary to test if the variation
of certain weights causes a change in the preference
rank of the scenarios. This is called a sensitivity
analysis, and can help stakeholders to see the impact
of their preferences and revise those same
preferences. As so, it’s possible to return to a
previous step in order to test different weights and
their impact on the alternatives scores (Dodgson et
al., 2009).
In the end of this analysis a consensus about the
chosen scenario must be achieved, and that choice
correctly validated according to the scores and
preferences of the existing stakeholders.
3 CRITICAL ANALYSIS
The multi-criteria analysis is a solid proved method,
for structuring and conducting an evaluation over
multiple alternatives, when there are also multiple
criteria. Our objective is to apply it to the domain of
enterprise architecture evaluation, due to its
capability to adapt to diverse domains and formalize
the evaluation process.
The process allows stakeholders to understand
the impacts of their choices, but also justify those
same choices facilitating the decision making
process and communication among them.
Selection of metrics from goals using Goal
Question Metric, as we propose on this paper, is a
different approach from the one proposed in
(Vasconcelos et al., 2005), where metrics are
associated with quality attributes. The later approach
is derived from software evaluation methods.
Since there is still not so much work done in
defining a set of general enterprise architecture
quality attributes, we use the goal based metric
selection. This approach is more flexible, and
applicable to a broader set of metrics not limited to
quality attributes scope.
The weighting of criteria, is an essential step in
the proposed evaluation, since it can drastically
change the results. Other weighting method could be
used, without consequences to the analysis, but
given the simplicity, versatility and stakeholder
involvement in SWING method, we suggest the
usage of this method over other more complex
approaches.
4 CONCLUSIONS
In this work we proposed an enterprise architecture
evaluation framework that applies to a common
organization situation that is the selection of a future
To-Be scenario, in order to implement some new
function or respond to another change situation. This
framework is still in development, although we have
already applied it to test cases and we are currently
testing its applicability in real world cases.
Our main goal is that this framework when
finished, will contribute to formalize and facilitate
the problem of scenario selection.
5 FUTURE WORK
The framework proposed in this work is currently in
progress, so there are some points where it can be
improved and extended. First regarding the metric
selection, here we propose using Goal Question
Metric, since its ability to select metrics in various
domains, like software and non-software domains
has been proved. It also helps at keeping the metrics
aligned with the objectives.
But other metrics selection methods or even a
pre-defined metrics set could be used. A related
future work would be the creation of a metrics
library, where metrics were associated with some
objective or objective type, and given a particular
objective, one could simple search the library for the
related metrics and apply them.
An alternative to this could be the definition of
quality attributes in enterprise architecture, similar to
the ones found in the software domain, and given a
quality attribute we would have a set of metrics that
measure that attribute.
Other area that could be improved is the
alternative identification step. Since our domain is
enterprise architecture, developing a tool for
scenario generation based on As Is scenario and a set
of parameters would be an important improvement.
REFERENCES
Basili, V. R., G. Caldiera, & H. D. Rombach. (1994). The
goal question metric approach. In Encyclopedia of
Software Engineering. Wiley.
Blackburn, C., & R. Valerdi. (2009). Navigating the
Metrics Landscape: An Introductory Literature Guide
ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems
236
to Metric Selection, Implementation, & Decision
Making. Annual Conference on Systems Engineering
Research.
C.A., B. e., Stewart T.J., & Vansnick J.-C. (1997).
Applications of the macbeth approach in the
framework of an additive aggregation model. Journal
of Multi-Criteria Decision Analysis, 107-114.
Daniels, J., Paul W. Werner, & A. Terry Bahill. (2001).
Quantitative methods for tradeoff analyses. Systems
Engineering.
Dodgson, J., Spackman, M., Pearman, A., & Phillips, L.
(2009). Multi-criteria analysis : a manual. London:
Department for Communities and Local Government.
Fischer, R., & Winter, R. (2007). Essential layers,
artifacts, and dependencies. Journal of Enterprise
Architecture, 7-19.
Johnson, P., Ekstedt, M., Silva, E., & Plazaola, L. (2004).
Using Enterprise Architecture for CIO Decision-
making On the Importance of Theory. Statistics.
Kaplan, R. S., & Norton, D. P. (2008). Mastering the
management system. Harvard Business Review, 62-
77.
Lankhorst, M. (2009). Enterprise Architecture at Work:
Modelling, Communication and. Berlin: Springer.
Mustajoki, J., Hamalainen, R. P., & Salo, A. (2005).
Decision Support by Interval SMART/SWING-
Incorporating Imprecision in the SMART and SWING
Methods. Decision Sciences.
The Open Group : ArchiMate® 2.0 Specification. (2012).
Van Haren Publishing.
V. Basili, J. Heidrich, M. Lindvall, J. Munch, M.
Regardie, & A. Trendowicz. (2007). Gqm+ strategies
{aligning business strategies with software
measurement. First International Symposium on
Empirical Software Engineering and Measurement,
488-490.
Vasconcelos, A., Mendes, R., & Tribolet, J. (2004).
projects, Using organizational modeling to evaluate
health care IS/IT. Proceedings of the 37th Annual
Hawaii International Conference on System Sciences.
Vasconcelos, A., Sousa, P., & Tribolet, J. . (2005).
Information System Architecture Evaluation: From
Software to Enterprise Level Approaches. Evaluation.
12th European Conference on IT Evaluation.
Vasconcelos, A., Sousa, P., & Tribolet, J. (2007).
Information system architecture metrics: an enterprise
engineering evaluation approach. The Electronic
Journal Information Systems Evaluation 10.
Victor Basili, Gianluigi Caldiera, & Dieter H. Rombach.
(1994). The goal question metric approach. In
Encyclopedia of Software Engineering. Wiley.
Vijayalakshmi, S., Zayaraz, G., & Vijayalakshmi, V.
(2010). Multicriteria Decision Analysis Method for
Evaluation of Software Architectures. International
Journal of Computer Applications, 33-38.
UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios
237