Using Multi-criteria Analysis to Evaluate

Enterprise Architecture Scenarios

Francisco Cansado

, André Vasconcelos

and Gonçalo Santos

Instituto Superior Técnico and INESC Inovação, Av. Rovisco Pais, Lisboa, Portugal

Deloitte Consultores, S.A, Praça do Duque de Saldanha, Lisboa, Portugal

Keywords: Enterprise Architecture, Evaluation, Multi-criteria Analysis, Weighting Methods, Metric Selection.

Abstract: In this paper we propose a framework based on multi-criteria analysis as a way to conduct an enterprise

architecture scenario evaluation and selection. The proposed solution supports decision making, by

evaluating each of the available scenarios using metrics derived from defined goals, based on a Goal-

Question-Metric methodology. In this approach the overall goals or objectives (the problem’s criteria) are

on the top of the GQM-tree, whereas questions of stakeholders and measurable attributes (also called

“metrics”) are represented as intermediate and leave nodes. According to the multi-criteria analysis each of

the alternatives x is evaluated on each metric i, by means of a value function Vi(x). Then, since every

criterion has a weight which represents its relative importance in the analysis, the final overall result of each

alternative scenario is therefore computed through a weighted sum model.

1 INTRODUCTION

In this paper, we analyse how multi-criteria analysis,

can be used in the context of enterprise architecture

scenarios. When organizations face a change

situation, multiple scenarios may be available to

implement that change, each of them with different

impacts, and opportunities.

Figure 1: Scenario selection problem.

The decision of which scenario to implement,

requires a set of metrics to quantify and compare the

existing scenarios between each other, in a formal

way. This formal evaluation has its advantages over

an informal evaluation, because it will produce

results that are better supported and justified, when

compared to the results obtained informally.

We will show how the problem of selection

enterprise architecture scenarios can be supported by

a framework based on multi-criteria analysis.

1.1 Enterprise Architecture

The enterprise architecture, by definition is a

representation of a real world organization, either in

an “As Is” state, or in a possible “To Be” scenario.

This representation can be compared to a city’s

architecture, although it hides some details about the

real city, it gives us a high level view. This holistic

view on enterprises can be used to extract some

broader indicators about the status of the actual

organization, enabling us to place some

characteristics in evidence.

The creation of an enterprise architecture, is an

essential step in a modern organization. It allows

stakeholders to communicate and facilitates decision

making within the organization, by creating a

common understanding of the organization and its

elements (Johnson et al., 2004).

It is not only a communication tool but it also

allows to understand how the different components

of an organization, actors, processes, applications

interact with each other, and assure that they are

correctly aligned and together contribute to achieve

the organizations goals (Lankhorst, 2009). It is also

232

Cansado F., Vasconcelos A. and Santos G..

Using Multi-criteria Analysis to Evaluate Enterprise Architecture Scenarios.

DOI: 10.5220/0004097502320237

In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS-2012), pages 232-237

ISBN: 978-989-8565-12-9

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

an evaluation tool through the usage of metrics that

allow us to check the organization status regarding

certain qualities (Vasconcelos et al., 2005).

The framework proposed in this work, is one of

such evaluation methods, with the objective of

helping in the scenario selection problem, described

early.

The structure of the paper is as follows, first we

start by defining enterprise architecture in the

context of this work, and why its evaluation is

important. Then we present how multi-criteria

analysis adapts to this particular context and what

steps are needed to conduct the proposed analysis.

Finally future work and conclusions are presented.

2 MULTI-CRITERIA ANALYSIS

PROCESS FOR ENTERPRISE

ARCHITECTURE

2.1 Enterprise Architecture Evaluation

Since enterprise architectures represent the real

organizations in a holistic view, with its elements

connections and rules (Fischer and Winter, 2007), it is

possible to evaluate the organization through its

architecture, using metrics (Vasconcelos et al., 2007).

In our proposal, this evaluation corresponds to

apply a set of metrics, to a set of scenarios. Through

the application of these metrics the different

scenarios will be scored, and their score will be used

to compare them.

These metrics are defined as a way to measure

and evaluate different qualities, and their objective is

to reduce the uncertainty level related with some

reality by quantifying it.

According to (Blackburn and R. Valerdi, 2009)

the metrics must be aligned with some objective, in

order to quantify it.

Our goal is to realize a multi-criteria analysis,

using these metrics as the criteria, and the enterprise

architecture scenarios as the possible alternatives.

2.2 Multi-criteria Analysis

Multi-criteria analysis is a method for selecting an

option given a set of criteria. In other words, it is a

process to discover the most preferred option, given

a set of criteria (Dodgson et al., 2009).

It allows us to structure a complex problem with

multiple options and restrictions. This is possible by

identifying the existing points of view over the

problem, and analyse them one at a time, and then

through the usage of a weighting method compute

the overall result of each alternative.

This method has been widely tested in various

contexts with good results, supporting the option

selection problem in a structured and formal way.

There are some variants of multi-criteria

analysis, depending on the selected method to

realize each of its steps. Nevertheless there is a

common set of structural elements present in all the

multi-criteria analysis methods:

 Criteria: they represent a stakeholder’s point of

view and concerns in the problem. It’s possible

to create the problem’s set of criteria using two

approaches, bottom-up or top-down, they are

described in section 2.5;

 Alternatives: are the scenarios which we are

evaluating, the possible options to choose

from;

 Decision makers: are the stakeholders in the

problem’s context, they must be able to

understand the criteria, in order to give their

preferences;

 Uncertainty: since not all factors can be

controlled in a given context, when we build

the alternatives, different possibilities

regarding the uncertainty must be generated;

 Environment: is the whole context where the

problem and the analysis are developed. There

are a multitude of factors that can affect the

analysis, and the best solution can be different

depending on the context.

The procedural structure of the analysis presented

over the next sections, is shown in Figure 2. For

each of the steps we present possible methods to

implement the given step, focusing on the enterprise

architecture scenario evaluation problem.

Figure 2: Proposed method steps.

2.3 Context Definition

This is the first step of the analysis. The scope and

constraints of the problem are defined, in order to

have a complete view of the problem. To do so it’s

necessary to first identify the stakeholders, since

UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios

233

they are the decision makers and will be asked for

feedback in future steps of the method. In our

problem the context of the analysis, will be the

change’s scope, this means that the decision makers

will be the stakeholders and the teams responsible

for the implementation of the change process.

2.4 Identify Alternatives

After having the context of the problem analysed

and its scope clarified, we need to identify the

possible solutions being evaluated. Depending on

the nature of the problem, these scenarios, may be

already defined or may be generated in this step. In

our problem the alternatives correspond to the

possible To Be architecture scenarios.

All the scenarios must be described with the

same level of detail and focus. This means using the

same framework and architecture viewpoint for all

scenarios, otherwise the score of the scenarios could

be biased since not all the scenarios would have the

same elements or information represented (Johnson

et al., 2004).

2.5 Identify Criteria

In order to evaluate the different options, we must

also identify the existing criteria. This process

depends on the context where the analysis is being

done and the selected approach (Blackburn and R.

Valerdi, 2009):

 Top-Down: in this approach, criteria are

structured in a hierarchical manner. This

structure can be seen as a tree, where the main

objectives are in the top and are successively

detailed into more specific criteria.

 Bottom-Up: in this method, criteria are identified

through an elicitation process, and then grouped

in broader categories or objectives.

In our proposal, we will be using a Top-Down

approach, starting by identifying the main broader

scope objectives (Basili et al., 1994). Since our

context is an organization’s enterprise architecture,

more specifically during a change situation, one of

the concerns is align the objectives of the change

with the organization’s objectives. One way to

achieve this is to look at the organization strategic

map and scorecards in order to extract the high

levels objectives of the organization.

2.5.1 Strategic Map and Balanced Scorecard

The Balanced Scorecard is a widely used,

organization performance measurement tool, and it

allows managers to keep track of the defined

objectives, as well as measures used to evaluate those

objectives (Kaplan and Norton, 2008). A typical

scorecard will contain various objectives regarding

different organization domains. These objectives can

be from the financial domain, client / market domain

or the organization development and learning domain.

It’s a flexible approach allowing virtually, all types of

organizations to represent their goals.

In order to make the link between the high level

strategy and the objectives in the balanced

scorecard, a strategic map can be used. This type of

artifact shows the link between objectives, allowing

the stakeholders to see the dependencies between

objectives (Kaplan and Norton, 2008).

2.5.2 Metric Selection

Since objectives, are not normally directly

measurable, in order to quantify them we must detail

objectives into metrics, following the Top-Down

approach described early. We will use the Goal

Question Metric process (Basili et al., 1994). This

methodology allows us to select a group of

measurable metrics, that will evaluate the defined set

of goals.

The process starts by identifying the goals, and

in our case these goals correspond to the ones

extracted from the strategic map and scorecard (V.

Basili et al., 2007).

Secondly, stakeholders are asked to define

questions, that when answered would allow them to

be confident about the achievement of each

objective. These questions will indicate what entities

we need to measure, and what metrics should be

used. The next step is to look at the proposed

questions, and define metrics that allow us to give a

quantitative answer to each of the questions. This

process generates a tree, with goals on top that

derive into questions and that in turn are linked to

metrics. These goals and metrics are the criteria we

will be using in our proposed analysis to evaluate the

existing scenarios.

2.6 Evaluate Scenarios

Having identified the available alternatives and the

evaluation criteria, we must evaluate each scenario

versus the identified set of criteria, in order to obtain

the scenarios score in all criteria. Depending on the

nature of the criteria, the evaluation may be

different, going from counting elements in a given

scenario’s architecture to subjective metrics like

preference regarding some service provider. The

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

234

usage of subjective metrics although possible must

be limited, and whenever possible is better to choose

a more objective metric. At the end of this step, the

responsible for the analysis, must be in possession of

each scenario’s scores.

2.7 Weight Criteria

Most of the choice problems analysed in real life do

not have a single selection criterion, but multiple

criteria as presented in multi-criteria analysis. But

since not all criteria are equally important, some sort

of compensation, must be applied so that a more

important criteria, contributes more to the overall

score than less important criteria.

To do this compensation there are several

weighting methods available. In the next sections we

describe several weighting methods that can be

integrated in a multi-criteria analysis (Dodgson et

al., 2009).

2.7.1 Trade-off

This method can reveal the indecisions faced by

stakeholders, comparing pairs of criteria. The

process is the following: for each pair of criteria,

two hypothetical alternatives are constructed, one of

them has the best score on criterion A and the worst

on B, the other alternative is the reverse of the first

one. We start by asking the stakeholders which is the

preferred scenario, and after they made their choice,

we ask how much they were willing to sacrifice the

best performing criterion, in order to maximize the

worst. The answer to these questions reveals the

Trade-Off between the two criteria, or on other

words, the weight associated with which criterion

(Daniels et al., 2001).

2.7.2 SWING

The SWING method also requires generation of

hypothetical alternatives, in this case only two, a

Worst alternative (W), where all criteria have the

lowest possible score and a Best alternative (B),

where all criteria have the best possible score

(Mustajoki et al., 2005).

This method starts with the scenario W, and the

stakeholders are asked which criterion they want to

move first from W to B, and a value of 100 points is

attributed to this criterion. Next they are asked

which criterion they wish to move next from W to B

and how much they value this transition comparing

to the 100 points of the first choice. This last step is

repeated for every criterion, and at the end we will

have all the criteria weighted relatively to the most

preferred criterion, in a normalized scale, since all

weights are contained in the [0;100] interval.

2.7.3 Change Resistance

In this approach each criterion is given two different

performance poles, best and worst, assuming that all

criteria are desirable in the final solution. By putting

all criteria in the best performance, and asking to the

stakeholders to compare all the criteria pairwise, and

choose one to be moved from best to worst state,

repeatedly, until all criteria have been compared

with the rest. The number of times a criterion

maintains its best performance, or in other words,

resists change, is the weight of that criterion.

2.7.4 Macbeth

The Macbeth method regards not only the weighting

step of the analysis, but it integrates weighting

criteria as an essential part. It has some swing and

trade-off, elements, like generating hypothetical

scores (good and neutral), for each criterion. The

objective of this method is to build a cardinal scale

of value, regarding the stakeholder’s preferences, or

alternatives attractiveness, like described in (Bana e

Costa et al., 1997).

2.7.5 Holistic

The holistic approach, as the name suggests, takes in

account the complete set of criteria and the

stakeholders are asked to rank the alternatives

regarding the overall score. In order to extract the

individual criterion weights, is necessary to apply

regression statistical methods. This process although

simple for the stakeholders, since they don’t have to

worry about the individual weights, causes other

problems like judgement inconsistencies, because

stakeholders are unaware of certain factors when

thinking over the full criteria set instead of each

criterion at a time. The need for statistical regression

operations, also adds complexity to the work of the

analyst realizing the analysis (Dodgson et al., 2009).

2.7.6 Selected Weighting Method

In our analysis we need each criterion individual

weight, relatively to the rest of the set, in order to

compute a global score combining the determined

weights with the scenarios score obtained in the

previous step, section 2.7.2. Any of the suggested

weighting methods could be used but in our proposal

we will use SWING, due to its simplicity, the

capacity to deal with large criteria number without

UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios

235

adding to much complexity to the analysis and

because it provides all its weights in a normalized

scale that will facilitate computing the overall score.

2.8 Decision and Sensitivity Analysis

Finally with the overall scores of the selected

scenarios, it’s still necessary to test if the variation

of certain weights causes a change in the preference

rank of the scenarios. This is called a sensitivity

analysis, and can help stakeholders to see the impact

of their preferences and revise those same

preferences. As so, it’s possible to return to a

previous step in order to test different weights and

their impact on the alternatives scores (Dodgson et

al., 2009).

In the end of this analysis a consensus about the

chosen scenario must be achieved, and that choice

correctly validated according to the scores and

preferences of the existing stakeholders.

3 CRITICAL ANALYSIS

The multi-criteria analysis is a solid proved method,

for structuring and conducting an evaluation over

multiple alternatives, when there are also multiple

criteria. Our objective is to apply it to the domain of

enterprise architecture evaluation, due to its

capability to adapt to diverse domains and formalize

the evaluation process.

The process allows stakeholders to understand

the impacts of their choices, but also justify those

same choices facilitating the decision making

process and communication among them.

Selection of metrics from goals using Goal

Question Metric, as we propose on this paper, is a

different approach from the one proposed in

(Vasconcelos et al., 2005), where metrics are

associated with quality attributes. The later approach

is derived from software evaluation methods.

Since there is still not so much work done in

defining a set of general enterprise architecture

quality attributes, we use the goal based metric

selection. This approach is more flexible, and

applicable to a broader set of metrics not limited to

quality attributes scope.

The weighting of criteria, is an essential step in

the proposed evaluation, since it can drastically

change the results. Other weighting method could be

used, without consequences to the analysis, but

given the simplicity, versatility and stakeholder

involvement in SWING method, we suggest the

usage of this method over other more complex

approaches.

4 CONCLUSIONS

In this work we proposed an enterprise architecture

evaluation framework that applies to a common

organization situation that is the selection of a future

To-Be scenario, in order to implement some new

function or respond to another change situation. This

framework is still in development, although we have

already applied it to test cases and we are currently

testing its applicability in real world cases.

Our main goal is that this framework when

finished, will contribute to formalize and facilitate

the problem of scenario selection.

5 FUTURE WORK

The framework proposed in this work is currently in

progress, so there are some points where it can be

improved and extended. First regarding the metric

selection, here we propose using Goal Question

Metric, since its ability to select metrics in various

domains, like software and non-software domains

has been proved. It also helps at keeping the metrics

aligned with the objectives.

But other metrics selection methods or even a

pre-defined metrics set could be used. A related

future work would be the creation of a metrics

library, where metrics were associated with some

objective or objective type, and given a particular

objective, one could simple search the library for the

related metrics and apply them.

An alternative to this could be the definition of

quality attributes in enterprise architecture, similar to

the ones found in the software domain, and given a

quality attribute we would have a set of metrics that

measure that attribute.

Other area that could be improved is the

alternative identification step. Since our domain is

enterprise architecture, developing a tool for

scenario generation based on As Is scenario and a set

of parameters would be an important improvement.

REFERENCES

Basili, V. R., G. Caldiera, & H. D. Rombach. (1994). The

goal question metric approach. In Encyclopedia of

Software Engineering. Wiley.

Blackburn, C., & R. Valerdi. (2009). Navigating the

Metrics Landscape: An Introductory Literature Guide

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

236

to Metric Selection, Implementation, & Decision

Making. Annual Conference on Systems Engineering

Research.

C.A., B. e., Stewart T.J., & Vansnick J.-C. (1997).

Applications of the macbeth approach in the

framework of an additive aggregation model. Journal

of Multi-Criteria Decision Analysis, 107-114.

Daniels, J., Paul W. Werner, & A. Terry Bahill. (2001).

Quantitative methods for tradeoff analyses. Systems

Engineering.

Dodgson, J., Spackman, M., Pearman, A., & Phillips, L.

(2009). Multi-criteria analysis : a manual. London:

Department for Communities and Local Government.

Fischer, R., & Winter, R. (2007). Essential layers,

artifacts, and dependencies. Journal of Enterprise

Architecture, 7-19.

Johnson, P., Ekstedt, M., Silva, E., & Plazaola, L. (2004).

Using Enterprise Architecture for CIO Decision-

making On the Importance of Theory. Statistics.

Kaplan, R. S., & Norton, D. P. (2008). Mastering the

management system. Harvard Business Review, 62-

77.

Lankhorst, M. (2009). Enterprise Architecture at Work:

Modelling, Communication and. Berlin: Springer.

Mustajoki, J., Hamalainen, R. P., & Salo, A. (2005).

Decision Support by Interval SMART/SWING-

Incorporating Imprecision in the SMART and SWING

Methods. Decision Sciences.

The Open Group : ArchiMate® 2.0 Specification. (2012).

Van Haren Publishing.

V. Basili, J. Heidrich, M. Lindvall, J. Munch, M.

Regardie, & A. Trendowicz. (2007). Gqm+ strategies

{aligning business strategies with software

measurement. First International Symposium on

Empirical Software Engineering and Measurement,

488-490.

Vasconcelos, A., Mendes, R., & Tribolet, J. (2004).

projects, Using organizational modeling to evaluate

health care IS/IT. Proceedings of the 37th Annual

Hawaii International Conference on System Sciences.

Vasconcelos, A., Sousa, P., & Tribolet, J. . (2005).

Information System Architecture Evaluation: From

Software to Enterprise Level Approaches. Evaluation.

12th European Conference on IT Evaluation.

Vasconcelos, A., Sousa, P., & Tribolet, J. (2007).

Information system architecture metrics: an enterprise

engineering evaluation approach. The Electronic

Journal Information Systems Evaluation 10.

Victor Basili, Gianluigi Caldiera, & Dieter H. Rombach.

(1994). The goal question metric approach. In

Encyclopedia of Software Engineering. Wiley.

Vijayalakshmi, S., Zayaraz, G., & Vijayalakshmi, V.

(2010). Multicriteria Decision Analysis Method for

Evaluation of Software Architectures. International

Journal of Computer Applications, 33-38.

UsingMulti-criteriaAnalysistoEvaluateEnterpriseArchitectureScenarios

237