A Synthesis of Enterprise Architecture Effectiveness Constructs

Siyanda Nkundla-Mgudlwa and Jan C. Mentz

School of Computing, University of South Africa, Private Bag X6, Florida Park, Johannesburg, 1710, South Africa

Keywords: Enterprise Architecture, Effectiveness Measurement, Enterprise Architecture Implementation, Design

Science Research, Critical Success Factors, Measurement Constructs.

Abstract: Companies throughout the world use Enterprise Architecture (EA) because of benefits such as the alignment

of business to Information Technology (IT), centralisation of decision making and cost reductions due to

standardisation of business processes and business systems. Even though EA offers organisational benefits,

EA projects are reported as being costly, time consuming and require tremendous effort. Companies

therefore seek to ascertain ways to measure the effectiveness of EA implementation because of the money

and time being spent on EA projects. EA Effectiveness refers to the degree in which EA helps to achieve the

collective goals of the organisation and its measurement depends on a list of constructs that can be used to

measure the effectiveness of EA implementation. Currently, there exist no comprehensive list of constructs

that are suitable to measure the effectiveness of EA implementation. The paper reports on the results of a

study that explored the development of a comprehensive list of constructs suitable for measuring the

effectiveness of EA implementation. The artefact developed in this research study is called Enterprise

Architecture Effectiveness Constructs (EAEC). The EAEC consists of 6 constructs namely: - alignment;

communication; governance; scope; top leadership commitment and skilled teams, training and education.

To achieve the purpose of this research study, a design science research (DSR) strategy was followed. The

EAEC was evaluated in two rounds by EA experts from industry and academia.

1 INTRODUCTION

Enterprise Architecture is implemented by

companies worldwide because of the benefits and

the value it promises. The EA benefits stated in the

literature are, improved business-Information

Technology (IT) alignment, better decision making,

increased business performance, reduced IT costs

and improved interoperability (Ross et al., 2006,

Wan et al., 2013). Even though the use and

implementation of EA may lead to a company

experiencing these benefits, an EA implementation

requires time, money and effort. An organization’s

ability to quantify the value of an EA

implementation is therefore very important. The

challenge, though, that organisations face is to

understand how the effectiveness of an EA

implementation can be measured (Lankhorst, 2005,

Schelp and Stutz, 2007). Furthermore, EA teams are

under pressure to demonstrate the value and benefits

of EA to the organisation in terms of the cost and

time spent on EA (Weiss, 2006, Rodrigues and

Amaral, 2013).

Regardless of this need there seems to be no

unified list of constructs suitable for measuring

effectiveness of EA implementation reported in the

literature. This paper aims to fill this gap by creating

an integrated list of measurement constructs. The list

is derived from the various existing published

critical success factors as well as EA effectiveness

models and frameworks.

Since the research approach followed in this

paper is Design Science Research the paper is

structured accordingly. The problem awareness and

solution proposal is presented in Section 2 followed

by a description of the research design in Section 3.

The design of the artefact is discussed in Section 4

followed by the evaluation of the artefact in Section

5. The paper concludes with suggestions for further

research in Section 6. Please note that due to the

nature of the EA topic, the terms business,

organization and enterprise are used interchangeably

in this paper.

282

Nkundla-Mgudlwa, S. and Mentz, J.

A Synthesis of Enterprise Architecture Effectiveness Constructs.

DOI: 10.5220/0006321002820293

In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS 2017) - Volume 3, pages 282-293

ISBN: 978-989-758-249-3

2 ENTERPRISE ARCHITECTURE

BACKGROUND

Enterprise Architecture is known by many

definitions (Saint-Louis and Lapalme, 2016).

Zachman, widely regarded as the originator of the

discipline of EA (Mentz et al., 2012), describes EA

as the ontology of the enterprise (Zachman, 2008).

This ontology consists of a set of descriptive

representations, known as primitive and composite

models, that describes an enterprise in such a way

that it can be used to produce systems to

management’s requirements. In a more standardised

tone EA is defined as the fundamental organisation

of a system, embodied in its components, their

relationships to each other and the environment, and

the principles governing its design and evolution

(IEEE, 2011).

The creation of an enterprise's architecture is

achieved by an Enterprise Architecture Framework

(EAF). According to (Cameron and McMillan,

2013) an EAF is used to implement an EA in terms

of a set of models, method and principles. The Open

Group Architecture Framework (TOGAF Ver 9.1)

(The Open Group, 2011) is a popular EAF currently

in use (Zhang, 2012, Sobczak, 2013, Schmidt et al.,

2014). More specifically, TOGAF Ver 9.1 (The

Open Group, 2011) describes itself as creating

business capability via the use of an architecture

development capability. Despite TOGAF's

popularity as an EAF there are other EAF's such as

for example the Zachman Framework (Zachman,

2008) and the Department of Defense Architecture

Framework (DoDAF V2.0) (Department of Defense,

2010).

The successful implementation of an EA holds

potential benefits to the business. These benefits are

varied and has been researched and proposed by EA

practitioner and researcher alike (Foorthuis et al.,

2015). Niemi (2008) summarizes EA benefits as

improved business to IT alignment, enhanced

communication and collaboration among

stakeholders, improved decision making and

reduced IT costs. Despite these promised benefits,

the implementation of EA in an organization

remains a challenge (Löhe and Legner, 2014)

especially considering that the expected benefits

takes some time to manifest (Schmidt and Buxmann,

2011) after the implementation of an EA practice.

EA implementation refers to the process of

establishing an EA practice in an organization based

on a specified scope (Syynimaa, 2013). An EA

practice produces various EA artefacts such as

models, standards, principles and other descriptive

documentation used to describe the enterprise as

comprehensively as possible (Niemi and Pekkola,

2015). The complex nature of the task of

representing the enterprise in artefactual terms lead

to several implementation issues. Seppanen et al.

(2009), for example, highlights 3 sets of interrelated

challenges facing EA implementation in public

administration. These challenges are the lack of

properly established EA governance, insufficient

support for the development of EA, and inadequate

resources to do EA governance and development.

Furthermore, the implementation of an EA practice

requires project management expertise along with an

IT portfolio management process (Seppanen et al.,

2009). Bonnet (2009) states that EA implementation

is of interest to staff that are responsible for

managing change projects and implementing

operational changes. These members of staff are also

referred to as stakeholders (The Open Group, 2011,

Lankhorst, 2005). Stakeholders are either directly

involved in implementing an EA or has a need that

is satisfied by the implemented EA.

The degree to which a stakeholder is satisfied

with an EA implementation depends on the

effectiveness of an EA practice. In an EA

implementation, the term effectiveness refers to the

outputs of an EA implementation that completely

meet the defined goals of an EA project (Rouhani et

al., 2014). As such EA effectiveness is a measure of

the degree to which organizational objectives are

attained through the outputs of the EA practice (Van

der Raadt et al., 2010) as well as the ability of the

EA practice to aid the achievement of the collective

goals of the organization (Rouhani et al., 2015).

According to Rouhani et al. (2015) enterprise

architects find it challenging to determine the

effectiveness of an EA implementation.

Given the challenging nature of measuring EA

effectiveness (Ylimäki, 2008, Espinosa et al., 2011,

Morganwalp and Sage, 2004, Rodrigues and

Amaral, 2013) efforts have been made to determine

the quality of EA implementations. Since an EA

implementation of EA can be costly as well as time

consuming, it is important to investigate and identify

critical success factors (CSF) that contribute to EA

success. Nikpay et al. (2013) define critical success

factors as the things that must go well to ensure the

overall success of the project.

Bricknall et al. (2006) identified 3 groupings of

CSFs that impacts the modelling and management of

EA, namely, IT and business management buy-in,

manageable EA project scope and relevant EA

artefacts as the contents of the EA. According to

A Synthesis of Enterprise Architecture Effectiveness Constructs

283

Ylimäki (2008) the EA CSFs that is important to

organisations are:

 Scoping and purpose

 Communication and common language

 Top management commitment

 Development methodology and tool support

 Business driven approach

 EA models and artefacts

 EA governance

 IT investment and acquisition strategies

 Skilled team, training and education

 Project and Program management

 Assessment and evaluation

 Organizational culture

Aier and Schelp (2010) identified a more

comprehensive list of CSFs to describe EA

implementation success (see Table 1) in terms of 4

factors groups and nineteen individual factors.

Table 1: EA implementation Critical Success Factors (Aier and Schelp, 2010).

Factor Group Individual Factor Description

Contextual

Factors

Size of Company/architecture Size of company impacts the number and size of resulting

architecture models used

Market orientation cost center or profit center

economic pressure Are there cost cutting exercises?

budget Is there a dedicated (E)A budget?

strategic alignment What kind of business/lT alignment exists?

culture How does the corporate culture influence change?

Structure governance Is there an EA governance and how is it anchored?

architectural power How strong are formal and informal architectural power and the

resulting impact?

skills of architects What skills do architects have?

skills of non-architects What are the architectural knowledge and architectural skills Of

non-architects?

EA visibility outside the EA

department

Are any architectural efforts visible Outside the architecture

department?

tools Is there any EA tool support?

coverage What is covered by EA?

Process project support How are architects involved in projects in general?

impact in projects How do they contribute to projects?

rules and EA processes What are the instruments to enact architecture within projects?

EA over Time training of architects frequency and amount of architectural training and further

education

training of non—architects frequency and amount of architectural training and further

education

EA marketing "marketing" measures to raise architecture attention and

architecture sensibilization

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

284

Finally, Wan et al. (2014) categorizes EA CSFs

according to the following 4 categories:

 EA readiness and preparation addresses the

organizational understanding of EA in terms

of introducing the concept of EA to the

organization as well as the management of

change that results from EA implementation.

 Top Commitment and leadership is critical in

order to have a successful EA implementation

due to the need for top leadership commitment

and support.

 EA domain techniques refers to the

professional EA related techniques or the

skills that enterprise architects should acquire

to do EA work.

 EA governance and program management

concern mainly management-control issues in

relation to incremental EA implementation

and EA lifecycle maintenance.

As can be seen from the preceding summary the

selection and determination of CSFs for successful

or effective EA implementation can lead to complex

results. A preliminary analysis can lead to a potential

set of essential CSFs namely, top leadership

commitment, EA governance, EA scope and skilled

EA teams and training.

The precise measurement of an effective EA

implementation according to CSFs is a challenging

task (Kaisler et al., 2005). There are, however,

approaches reported in the academic literature that

Table 2: EA Effectiveness Measurement Papers.

Title Author (s)

A framework for Enterprise

Architecture effectiveness

Kamogawa and

Okada (2005)

Experience report: Assessing a

global financial services

company on its Enterprise

Architecture effectiveness

using NAOMI

Van Der Raadt et al.

(2007)

Measuring the Effectiveness of

Enterprise Architecture

Implementation

Bonnet (2009)

Measuring Enterprise Architecture

Effectiveness

- A focus on Key Performance

Indicators

Van der Raadt et al.

(2010)

An Effectiveness Model for

Enterprise Architecture

Methodologies

Günther (2014)

address the measurement of the effectiveness of EA

implementation problem. What follows is a brief

overview of 5 notable examples in support of the

research problem addressed by this paper. Table 2

lists the frameworks and models in question.

The selection in Table 2 was based on a

systematic keyword search of the EA literature on

EA implementation, these keywords were:

 EA effectiveness

 EA success

 Measuring EA effectiveness

 Measuring EA success.

The keyword search targeted the paper's abstract,

research findings and conclusion. The databases

used to search literature were:

 Google Scholar

 ACM Digital Library

 IEEE Xplore

 Science Direct

 Elsevier

 Springer Link

 Taylor and Francis

Kamogawa and Okada (2005) developed a

framework that assesses EA effectiveness in the

context of e-business. The critical elements

mentioned in their EA effectiveness framework are

namely; EA development power, governance and

EA cognition. Van Der Raadt et al. (2007)

developed the Normalized Architecture

Organization Maturity Index (NAOMI). NAOMI

has three variables to assess EA effectiveness, these

are: architecture awareness, architecture maturity

and architecture alignment. Measuring the

Effectiveness of Enterprise Architecture

Implementation by Bonnet (2009) and Van der

Raadt et al. (2010) provides two organisational

objectives to measure the effectiveness of EA

implementation. The two organizational objectives

are namely; agility and alignment. Focus Framework

for Enterprise Architecture Measurements (FFEAM)

is developed by Günther (2014) and it considers four

areas namely, the decision-making process, the

decision-making results, programme

implementation, and programme results. Rouhani et

al. (2015) explore the factors that affect

effectiveness of EA Implementation Methodology

(EAIM) and propose the effectiveness model for

EAIMs. There are five factors that affect

effectiveness of EAIM, these are:

 Alignment

 Adaptiveness

 Support

A Synthesis of Enterprise Architecture Effectiveness Constructs

285

 Binding

 innovation.

These 5 effectiveness approaches reveal three

common effectiveness elements namely:

 Communication skills

 EA scope

 Alignment

The above discussion shows the varied

approaches and indicators of effective EA

implementation. The increased use of EA introduces

a problem with regards to selecting an appropriate

set of EA CSFs to measure EA effectiveness. This

problem can be addressed by a synthesis of existing

approaches to construct an integrated CSF list.

This paper report on the results of a research

study that applied Design Science Research (DSR)

to the development of an artefact that represents a

comprehensive list of constructs suitable for

measuring effectiveness of EA implementation.

3 RESEARCH APPROACH

The research approach followed in this research is

Design Science Research (DSR) as described by

Vaishnavi and Kuechler (2011). Figure 1 shows the

Figure 1: Design Science Research steps, adapted from

(Vaishnavi and Kuechler, 2011).

5 main steps of the DSR method.

These steps entail the following:

 Awareness of a Problem – an awareness of

the problem may be supported by multiple

sources including new developments in

industry or in a reference discipline. The

problem addressed in this paper is the absence

of an integrated list of constructs suitable to

measure the effectiveness of EA

implementation. EA is costly, time consuming

and requires effort to implement and it is

therefore important to quantify its value.

 Suggestion – the solution suggestion phase

follows an awareness of the problem has been

established. During the suggestion phase an

artefact is proposed as a solution to the

problem. The solution proposed in this paper

is an artefact that represents an integrated list

of EA effectiveness constructs. The name of

the artefact is Enterprise Architecture

Effectiveness Constructs (EAEC).

 Development – the proposed solution is

designed and implemented in this phase. The

inputs to this phase can be existing artefacts or

theories that is relevant to the problem space.

The EAEC is based on a synthesis of existing

EA measurement frameworks, models as well

as EA CSFs. The first step in the design phase

is to select these EA measurement artefacts by

using a literature review approach. After the

artefacts were identified they were compared

to find commonality in terms of measurement

constructs.

 Evaluation – in this phase the newly

developed artefact is used to demonstrate that

the problem has been addressed. Vaishnavi

and Kuechler (2011) state that an artefact is

evaluated according to criteria that that are

always implicit and frequently made explicit

in the awareness of the problem phase. An

appropriate evaluation method needs to be

selected according to the type of the artefact

being developed.

 Conclusion – conclusion is the final phase of

the DSR process and in this step the results of

the process are communicated to all relevant

stakeholders namely; Information Technology

(IT) stakeholders, business stakeholders and

external stakeholders (suppliers).

The sections that follow provide more detail with

regards to the design and evaluation of the EAEC.

The awareness of the problem as well as the solution

proposal phases were addressed in Section 2 of this

paper.

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

286

4 THE EAEC DESIGN PROCESS

In this section the development of the EAEC is

discussed. The EAEC consists of a comprehensive

list of constructs suitable for measuring

effectiveness of EA implementation. Existing

artefacts and measurement frameworks used in this

phase (see Table 2) produced various lists of

effectiveness constructs. The EAEC was developed

by synthesizing these lists as well as the EA CSFs

derived from existing effectiveness approaches.

The list of existing EA CSFs is called the EA

CSF list and contains three elements namely, top

management support or commitment, scope,

governance and skilled teams, training and

education. The list derived from the EA

effectiveness approaches is called the EA

effectiveness elements list and consist of:

communication skills, scope and alignment.

4.1 The Synthesis of EA CSFs and EA

Effectiveness Elements

The EA CSF list and EA effectiveness elements are

synthesized to produce a list of integrated constructs.

This synthesis yields 6 constructs namely, top

leadership commitment; governance; scope and

skilled teams, training and education;

communication skills and alignment. To ease the

task of reference these 6 elements will be referred to

as measurement constructs in this paper. The

detailed meaning of each constructs are as follows:

 Top management commitment is described

by Wan et al. (2014) as a facet that deals with

commitment from top executives and provides

sufficient power to perform organisational

changes. Bricknall et al. (2006) are of the

view that top IT and business stakeholders

buy-in are critical because without their

support EA programmes can easily fail.

Ylimäki (2008) confirms the involvement of

top management in an EA effort.

 Governance deals with the roles and

responsibilities among different stakeholders

in business, IT and suppliers. Issues that are

covered under EA governance are governance

structure, effective governance processes and

activities, effective change management

environment, effective risk management and

business management process integration

(Ylimäki, 2008). Governance is mentioned by

Aier and Schelp (2010) as an important factor

for EA implementation success. EA

governance and program management

involves management-control issues in

relation to incremental EA implementation

and continuous improvement (Wan et al.,

2014).

 Scope refers to the parts of the organisation

such as IS and IT that should be included in

the initial EA project in order to create an EA

(Bricknall et al., 2006). The scope of EA must

be clear and the benefits of EA should also be

included in the scope documentation.

Furthermore, there must be “as-is”

documentation that shows the current status of

the architecture, an IT strategy, target

architecture (‘to be’ architecture) and a plan of

how to reach the targeted architecture (which

is a “to-be” architecture) (Bricknall et al.,

2006). Ylimäki (2008) states that scoping and

purpose refers to how an organisation

addresses issues such as holistic EA, a clear

mission, goals and direction; value and

benefits of EA and a clearly defined EA.

Holistic EA addresses issues such as the

definition of EA in the organisation and the

documentation of key EA stakeholder groups

(Lankhorst, 2005, Ylimäki, 2008). Scoping in

Wan et al. (2014) study falls under the EA

readiness and purpose facet. One of the

success factors in the EA readiness and

purpose facet is an understanding of the high-

level business formal structure such as

strategy, vision, mission, objective.

 Skilled teams, training and education, this

CSF addresses the establishment of skills

required for architecture team and key

stakeholder in terms of architecture work. The

architecture teams need to have both business

and architecture skills (Boster et al., 2000,

Ylimäki, 2008). The skills of architects and

non-architects are also mentioned as critical

by (Aier and Schelp, 2010). Aier and Schelp

(2010) state that communication and regular

training or education are CSFs for the long-

term success of EA. The training and

education of non-architects fosters the

acceptance of architectural issues and reduces

barriers (Aier and Schelp, 2010). According to

Wan et al. (2014) training is covered under the

EA domain techniques facet. This facet refers

to EA skills and business skills that enterprise

architecture should acquire (Wan et al., 2014).

 Business and IT alignment (B-IT) is defined

as the degree to which the IT missions,

objectives, and plans support and are

supported by the business mission, objectives,

A Synthesis of Enterprise Architecture Effectiveness Constructs

287

and plans (Reich and Benbasat, 1996). An

organisation needs to be aligned internally to

be agile externally (Bonnet, 2009). Alignment

refers to IT supply meeting organisation

demands (Lindström et al., 2006). Günther

(2014) refers to B-IT alignment during the

decision-making process as a subjective

alignment of business and IT stakeholders.

The B-IT alignment in this context refers to

understanding of business by IT and

understanding of IT by business Günther

(2014). Alignment is one of the 5 critical

factors that affect the effectiveness of EA

implementation because of the positive

relationship between alignment and

effectiveness of EA implementation (Rouhani

et al., 2014).

 Communication skills refers to the process of

communicating with stakeholders in terms of

EA projects. Kamogawa and Okada (2005)

state that EA should reflect communication in

which all stakeholders are involved.

According to Van Der Raadt et al. (2007),

communication involves architecture

descriptions and models. Further, the level in

which an architecture function can

communicate with its stakeholders is essential

in determining its ability to be effective (Van

Der Raadt et al., 2007, Lange et al., 2012).

Bonnet (2009) mention communication and

understanding as one of the dimension under

alignment. Communication and understanding

refers to common understanding of business

and IT through knowledge sharing, and insight

in consequences of decision-making. The

latter requires that enterprise architects

communicate to business effectively.

Effective engagement ensures that all key

stakeholders share the risks and

responsibilities (Ross et al., 2006).

4.2 The Relationship between

Constructs

Figure 2 shows an illustration of the 6 measurement

constructs as well as the relationships between them.

The Alignment and Communication constructs are

shaded in a different colour than the other four

constructs because they either have a direct or

indirect relationship with the other four

measurement constructs. Therefore, Alignment and

Communication are 2 measurement constructs that

are common among the measurement constructs.

This means that the other 4 measurement constructs

either have a direct or indirect relationship with

Alignment or Communication or both. The

relationships are labelled as bidirectional arrows in

the diagram. These relationships are discussed next:

Figure 2: Measurement Constructs and Relationships.

 Alignment and Scope (A & S)

Alignment refers to the strategic fit between

strategy and operations, functional integration

of business and IT, external suppliers or other

lines of business within the organisation

(Henderson and Venkatraman, 1993, Bonnet,

2009). Bonnet (2009) is of the view that

alignment is achieved when all components of

an organisation are interrelated coherently.

The organisational scope of architecture

indicates which part of an organisation is

involved in the EA program (Van Der Raadt

et al., 2007). As such Scope is an intrinsic

variable that links with Alignment that means

business strategy and IT strategy must be

aligned for an EA implementation to be

regarded as effective.

 Alignment and Communication (A & C)

Communication refers to the ability of the

architecture function to communicate EA

relevant information to the organisation (Van

der Raadt et al., 2010). For communication to

be effective, architecture alignment needs to

be in place as well. Communication within the

alignment dimension refers to a common

understanding of business and IT through

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

288

knowledge sharing, and insight of the

consequences of decision making (Bonnet,

2009, Van der Raadt et al., 2010). The relation

between alignment and communication means

business to IT alignment needs to be

communicated to all the stakeholders for an

EA implementation to be regarded as

effective.

 Alignment and Governance (A & G)

Governance within the alignment dimension

refers to formal decision making, monitoring,

and control of priorities and budget for both

business and IT (Van der Raadt et al., 2010,

Bonnet, 2009). The governance relation

indicates that an effective EA implementation

requires governance of the alignment

produced by EA.

 Communication and governance (C & G)

Ylimäki (2008) states that effective

communication is critical to share knowledge,

achieving a common understanding,

agreement and a shared view of EA scope,

vision and objectives. The linkage of

communication and scope is also shared by

Wan et al. (2014) and Bricknall et al. (2006)

in that organisational understanding of EA is

important therefore effective communication

plays a critical role in ensuring that an

organisation understands the EA scope. The

link between communication and governance

is stated by Ylimäki (2008). There is indirect

communication between communication and

governance according to Van Der Raadt et al.

(2007) in terms of architecture awareness.

This relation means that the roles and

responsibilities of the EA implementation

need to be clearly communicated to all

stakeholders for the EA implementation to be

regarded as effective.

 Communication and skilled teams (C & ST)

Teams need to have good communication

skills to effectively communicate with

different stakeholders and to provide training

effectively. It is critical for EA architects to

have good communication skills to translate

business requirements into EA. The people

that matter in terms of determining the

relevance and effectiveness of EA is the

business not necessarily the technical staff.

Enterprise architects must communicate with

the business in relevant terms (i.e. non-

technical) so that engagement can be fostered.

This relationship is critical to an effective EA

implementation due to the importance of the

business as a key stakeholder and recipient of

the benefit of EA.

 Communication and Top leadership

commitment (C & TC)

Effectiveness and top leadership commitment

is important for effective EA implementation.

This relationship was identified by Ylimäki

(2008). Communication between stakeholders

and Support & commitment from top

executives are factors that are mentioned

under top commitment and leadership facet,

therefore there is a correlation between these

two factors (Wan et al., 2014).

5 EVALUATING THE EAEC

During the evaluation phase the EAEC was

demonstrated to eleven EA experts from industry

and academia (in the South African context). The

objective of evaluating EAEC was to assess whether

problem of a lack of an integrated EA effectiveness

measurement list was adequately addressed (see

Section 2 for problem awareness discussion). The

eleven EA experts were identified based on their

LinkedIn profiles as well as referrals from the power

utility.

The evaluation was conducted in two rounds that

followed a presentation of the EAEC. During the

first round each expert was required to provide

general information about their experience in EA

and to answer 3 open ended questions about the

EAEC. During the second round the EA experts

were asked to review the responses from the other

experts (presented to each expert without revealing

any identities) to indicate their view in terms of

whether they agree, in neutral or disagree with

claims made during the first round. The evaluation

was done under the protection of an ethical

clearance certificate that guaranteed the anonymity

as well as the right to choose to participate in the

research.

5.1 Analysis of Evaluation

The objective of evaluation was determining the

degree of common understanding of EAEC and in so

doing to establish that the EAEC addressed the

problem (Section 2) it was designed to solve.

5.1.1 Round 1 Results

During the first round all eleven EA acknowledged

that there is a need to have a tool that will enable

A Synthesis of Enterprise Architecture Effectiveness Constructs

289

organisations to measure the effectiveness of EA

implementation. The 6 constructs were confirmed by

all eleven EA experts as being critical and

comprehensive to measure the effectiveness of EA

implementation. 6 out of eleven (55%) EA experts

stated that the EAEC has the potential of being a

useful tool once measuring units are determined for

the six constructs. All eleven EA experts expressed

their concerns regarding the EA definition and their

role as enterprise architects. The EA experts stated

that the EA definition is not fully understood and the

role of the enterprise architects is often confused

with the role of an IT architect.

5.1.2 Round 2 Results

In round 2, the EA experts were asked to review

their collective opinions and to indicate their view in

terms of whether they agree, in neutral or disagree

with claims derived in round 1. Table 3 to 5 shows

an overview of the opinions from round 1. A total

number of 9 EA experts participated in round 2.

The EA experts were asked to confirm or change

their original opinions. In terms of question 1 (Table

3) 5 EA experts agreed with the claims, 3 disagreed

and 1 was in neutral. The confirmation of the claims

for question 1 indicate that 56% of the EA experts

support the claims.

Table 3: Question 1 responses.

Question Responses

Reason for

using

EAEC

The artefact pinpoints the pain points we

have in business in terms of EA project

implementation

It covers all critical elements that one

can use to measure the effectiveness of

EA implementation

That artefact has potential

In its current form, it can be used as

guiding artefact

The artefact can be used as a practice

framework to establish EA

With regards to question 2 (Table 4) 8 EA

experts agreed with the four claims and 1 EA expert

disagreed with the claims.

Table 4: Question 2 responses.

Reason for

not using

EAEC

Artefact is too generic, it is not detailed

enough, expand the artefact

I would not know how to use the

artefact. Where do I start and how?

The artefact does not have measuring

units, it does not have tools or methods

to measure EA effectiveness

Does not take organisational or

stakeholder view into account

For question 3, 6 EA experts agreed with the

claims, 2 disagreed and 1 was neutral. Of the total,

67% EA experts stated that the artefact needs to

have measuring units to be effectively evaluated. 1

out of the 9 EA experts stated that the 6

measurement constructs can be measured at different

levels in the organisation depending on the maturity

level of the EA. 2 of the 9 EA architects are of the

view that the correlations between the measurement

constructs must be represented based on strength and

the order of importance.

Table 5: Question 3 responses.

Suggested

changes

Add tools or methods to measuring all 6

measurement constructs

Add architecture standards and recipes

Include direction setting - this is a

strategic guidance and should not be

confused with governance

Add reference models

Make a relation between skilled teams

and alignment because architects are

assigned to projects based on their skills

Order the constructs according to their

importance in measuring EA

effectiveness

Indicate the impact of relations on EA

effectiveness

Indicate a timeline for each construct in

terms of a measurement at different

maturity level

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

290

5.1.3 Discussion

As per the problem discussed in the first stage of the

DSR process (see Section 2) there is a lack of an

integrated set of constructs to effectively establish

the quality of an EA implementation. The proposed

approach was to synthesise the existing CSFs as well

as measurement frameworks and models to establish

a comprehensive list of measurements contained in

an Enterprise Architecture Effectiveness Constructs

(EAEC) (see discussion in Section 4).

The evaluation of the EAEC set out to learn

whether the task of creating a comprehensive

measurement list was achieved. The feedback and

opinions of the interviewed EA experts was most

instructive. On the one hand the acceptance of the

set of constructs and their relations where well

received and in that respect the problem was solved.

On the other hand, though, the experts pointed out

the inherent limitations of addressing the

measurement of effective EA implementation by

way of constructs alone. What is needed is a detailed

process as well. In conclusion, the problem as

formulated in Section 2 seems to have been solved

but as an artefact by itself only solves a part of the

problem of measuring EA implementation. This

necessitate further exploration on the topic in terms

of addressing the issue of specific metrics as well as

a measurement method. That task is left for further

iterations of the DSR method.

6 CONCLUSION AND FURTHER

WORK

This paper reported on a research study that aimed at

designing and evaluating an artefact that entailed a

comprehensive list of constructs suitable for

measuring the effectiveness of EA implementation.

The intention of the evaluation was to establish

whether the artefact is doing what it is supposed to

be doing and then to identify areas of improvements

(see Section 5.1.3). The proposed artefact was

evaluated by 11 EA experts that were selected from

industry and academia.

The research objectives achieved at the end of

the research project was as follows:

1. To identify existing EA critical success

factors and EA effectiveness approaches.

2. To find commonalities in the identified EA

critical success factors and EA effectiveness

approaches.

3. To synthesize the EA critical success factors

and EA effectiveness approaches to produce

an artefact that consists of a comprehensive

list of constructs suitable for measuring

effectiveness of EA implementation.

4. To evaluate a synthesized EA effectiveness

Constructs (EAEC).

The design science research (DSR) (see Section

3) method was applied to this research. The

proposed solution can be deemed effective when it

achieves what it was intended for which means that

it solves (to some degree at least) the identified

problem.

This research results contributes to the

knowledge in the field of Information systems (IS)

as it relates to EA implementation measurement.

The EAEC consists of six measurement constructs

namely: communication, alignment, governance, top

leadership commitment, scope, skilled teams,

training and education. Good communication skills

and business to IT alignment are considered the

common measurement constructs among other

measurement constructs because they either have a

direct relationship or indirect relationship with other

measurement construct. Measuring the effectiveness

of EA implementation requires that the 6

measurement constructs be present.

In terms of further research the following

problems emerged and should be considered for

further study:

 The performance of the EAEC in an

operational context

 Enhancing the EAEC with detailed

performance metrics in support of the

measurement constructs

 The establishment of a measurement method

to expand the EAEC

Finally, the research results have confirmed the

essential difficulty and inherent complexity involved

in determining how effective an EA implementation

really is. The need and resultant problem will

continue to impact on the total value of EA to the

enterprise and must therefore be addressed an

ultimately solved.

REFERENCES

Aier, S. & Schelp, J. A reassessment of enterprise

architecture implementation. Service-Oriented

Computing. ICSOC/ServiceWave 2009 Workshops,

2010. Springer, 35-47.

A Synthesis of Enterprise Architecture Effectiveness Constructs

291

Bonnet, M. 2009. Measuring the effectiveness of

Enterprise Architecture implementation. Master of

Science in Systems Engineering,

Policy Analysis and Management – Information

Architecture, TU Delft, Delft University of

Technology.

Boster, M., Liu, S. & Thomas, R. 2000. Getting the most

from your enterprise architecture. IT Professional, 2,

43-51.

Bricknall, R., Darrell, G., Nilsson, H. & Pessi, K.

Enterprise architecture: critical factors affecting

modelling and management. ECIS, 2006. 2349-2361.

Cameron, B. H. & Mcmillan, E. 2013. Analyzing the

current trends in enterprise architecture frameworks.

Journal of Enterprise Architecture, 9, 60-71.

Department Of Defense 2010. Version Description

Document for The DoD Architecture Framework

(DoDAF) and DoDAF Meta Model (DM2) Version

2.02. Office of the Assistant Secretary of Defense

(OASD) for Network Infrastructure and Integration

(NII): United States of America Department of

Defense.

Espinosa, J. A., Boh, W. F. & Delone, W. The

organizational impact of enterprise architecture: a

research framework. System Sciences (HICSS), 2011

44th Hawaii International Conference on, 2011. IEEE,

1-10.

Foorthuis, R., Van Steenbergen, M., Brinkkemper, S. &

Bruls, W. A. 2015. A theory building study of

enterprise architecture practices and benefits.

Information Systems Frontiers, 1-24.

Günther, W. A. 2014. Measuring Enterprise Architecture

Effectiveness: A Focus on Key Performance

Indicators. Master of Science (M.Sc.) of ICT in

Business Leiden University.

Henderson, J. C. & Venkatraman, H. 1993. Strategic

alignment: Leveraging information technology for

transforming organizations. IBM systems journal, 32,

472-484.

IEEE 2011. ISO/IEC/IEEE 42010:2011, Systems and

software engineering — Architecture description.

http://www.iso-architecture.org/ieee-1471/index.html.

Kaisler, S. H., Armour, F. & Valivullah, M. 2005.

Enterprise architecting: critical problems. 38th Hawaii

International Conference on System Sciences. Hawaii:

IEEE.

Kamogawa, T. & Okada, H. A framework for enterprise

architecture effectiveness. Proceedings of

ICSSSM'05. 2005 International Conference on

Services Systems and Services Management, 2005.,

2005. IEEE, 740-745.

Lange, M., Mendling, J. & Recker, J. A comprehensive

EA benefit realization model--An exploratory study.

System Science (HICSS), 2012 45th Hawaii

International Conference on, 2012. IEEE, 4230-4239.

Lankhorst, M. 2005. Enterprise Architecture at Work:

Modelling, Communication and Analysis. 1 ed.:

Springer, Berlin, Germany, EU. ISBN 3540243712.

Lindström, Å., Cegrell, T. & Grandell, M. The Top IT

Concerns for Chief Information Officers in European

Electric Power Industries. 41st International

Conference on Large High Voltage Electric Systems

2006, CIGRE 2006; Paris; France; 27 August 2006

through 1 September 2006, 2006.

Löhe, J. & Legner, C. 2014. Overcoming implementation

challenges in enterprise architecture management: a

design theory for architecture-driven IT Management

(ADRIMA). Information Systems and e-Business

Management, 12, 101-137.

Mentz, J., Kotzé, P. & Van Der Merwe, A. 2012. A

comparison of practitioner and researcher definitions

of enterprise architecture using an interpretation

method. Advances in Enterprise Information Systems

II, 11-26.

Morganwalp, J. M. & Sage, A. P. 2004. Enterprise

architecture measures of effectiveness. International

Journal of Technology, Policy and Management, 4,

81-94.

Niemi, E. 2008. Enterprise architecture benefits:

Perceptions from literature and practice. Evaluation of

enterprise and software architectures: critical issues,

metrics and practices:[AISA Project 2005-2008]/Eetu

Niemi, Tanja Ylimäki & Niina Hämäläinen (eds.).

Jyväskylä: University of Jyväskylä, Information

Technology Research Institute, 2008.-(Tietotekniikan

tutkimusinstituutin julkaisuja, ISSN 1236-1615; 18).

ISBN 978-951-39-3108-7 (CD-ROM).

Niemi, E. & Pekkola, S. 2015. Using enterprise

architecture artefacts in an organisation. Enterprise

Information Systems, 1-26.

Nikpay, F., Selamat, H., Rouhani, B. D. & Nikfard, P. A

Review of Critical Success Factors of Enterprise

Architecture Implementation. Informatics and

Creative Multimedia (ICICM), 2013 International

Conference on, 2013. IEEE, 38-42.

Reich, B. H. & Benbasat, I. 1996. Measuring the linkage

between business and information technology

objectives. MIS quarterly, 55-81.

Rodrigues, L. S. & Amaral, L. Key Enterprise

Architecture Value Drivers: Results of a Delphi Study.

Proceedings of the 21th IBIMA Conference, 2013

Vienna, Austria.

Ross, J. W., Weill, P. & Robertson, D. 2006. Enterprise

architecture as strategy: Creating a foundation for

business execution, Harvard Business Press.

Rouhani, B. D., Mahrin, M. N., Nikpay, F. & Rouhani, B.

D. 2014. Current issues on enterprise architecture

implementation methodology. New Perspectives in

Information Systems and Technologies, Volume 2.

Springer.

Rouhani, B. D., Mahrin, M. N. R., Nikpay, F., Ahmad, R.

B. & Nikfard, P. 2015. A systematic literature review

on Enterprise Architecture Implementation

Methodologies. Information and Software Technology,

62, 1-20.

Saint-Louis, P. & Lapalme, J. Investigation of the lack of

common understanding in the discipline of enterprise

architecture: A systematic mapping study. Enterprise

Distributed Object Computing Workshop (EDOCW),

2016 IEEE 20th International, 2016. IEEE, 1-9.

ICEIS 2017 - 19th International Conference on Enterprise Information Systems

292

Schelp, J. & Stutz, M. 2007. A balanced scorecard

approach to measure the value of enterprise

architecture.

Schmidt, C. & Buxmann, P. 2011. Outcomes and success

factors of enterprise IT architecture management:

empirical insight from the international financial

services industry. European Journal of Information

Systems, 20, 168-185.

Schmidt, R., Zimmermann, A., Möhring, M., Jugel, D.,

Bär, F. & Schweda, C. M. Social-software-based

support for enterprise architecture management

processes. International Conference on Business

Process Management, 2014. Springer, 452-462.

Seppanen, V., Heikkila, J. & Liimatainen, K. Key issues in

EA-implementation: case study of two Finnish

government agencies. 2009 IEEE Conference on

Commerce and Enterprise Computing, 2009. IEEE,

114-120.

Sobczak, A. Methods of the Assessment of Enterprise

Architecture Practice Maturity in an Organization.

International Conference on Business Informatics

Research, 2013. Springer, 104-111.

Syynimaa, N. Measuring Enterprise Architecture success:

a tentative model for measuring success. 14th

International Conference on Informatics and Semiotics

in Organisations, ICISO 2013, 2013 Stockholm,

Sweden.

The Open Group 2011. TOGAF Version 9.1, Van Haren

Publishing.

Vaishnavi, V. & Kuechler, B. 2011. Design Science

Research in Information Systems [Online]. [Accessed

9 May 2012 2012].

Van Der Raadt, B., Bonnet, M., Schouten, S. & Van Vliet,

H. 2010. The relation between EA effectiveness and

stakeholder satisfaction. Journal of Systems and

Software, 83, 1954-1969.

Van Der Raadt, B., Slot, R. & Van Vliet, H. Experience

report: assessing a global financial services company

on its enterprise architecture effectiveness using

NAOMI. System Sciences, 2007. HICSS 2007. 40th

Annual Hawaii International Conference on, 2007.

IEEE, 218b-218b.

Wan, H., Johansson, B., Luo, X. & Carlsson, S.

Realization of enterprise architecture (EA) benefits.

Working Conference on Practice-Driven Research on

Enterprise Transformation, 2013. Springer, 92-105.

Wan, H., Luo, A. & Luo, X. How Enterprise Architecture

Formative Critical Success Facets Might Affect

Enterprise Architecture Success: A Literature

Analysis. International Conference on Informatics and

Semiotics in Organisations, 2014. Springer, 197-209.

Weiss, D. 2006. Enterprise Architecture Measurement

Program, Part 1: Scoping. ID.

Ylimäki, T. 2008. Potential critical success factors for

enterprise architecture. Evaluation of enterprise and

software architectures: critical issues, metrics and

practices:[AISA Project 2005-2008]/Eetu Niemi,

Tanja Ylimäki & Niina Hämäläinen (eds.). Jyväskylä:

University of Jyväskylä, Information Technology

Research Institute, 2008.-(Tietotekniikan

tutkimusinstituutin julkaisuja, ISSN 1236-1615; 18).

ISBN 978-951-39-3108-7 (CD-ROM).

Zachman, J. A. 2008. The Zachman Framework: the

official concise definition.

http://www.zachman.com/about-the-zachman-

framework [Online].

Zhang, L.-J. 2012. Editorial: Big services era: Global

trends of cloud computing and big data. IEEE

Transactions on Services Computing, 5, 467-468.

A Synthesis of Enterprise Architecture Effectiveness Constructs

293