Enterprise Architecture Performance Management
A Context based Approach to EA Metrics Definition
Anuj Kumar and Premnath S.
Enterprise Architecture Consulting, Wipro Consulting Services, Wipro Limited, Bangalore, India
Keywords: Enterprise Architecture, Metrics, Maturity Model, Context, Perspectives, Scope, Role, Measurement,
Organization, Enterprise, Domain, Value, Activity, Information Source, Model.
Abstract: Management scholars suggest that if you can’t measure something, you can’t improve or manage it. The
same applies to Enterprise Architecture (EA). Surveys related to EA suggest that organizations find it
difficult to measure EA and in some cases it is not even measured. Although, EA metrics have been
proposed in literature but not all are practical and most do not measure the true essence of EA. The situation
becomes even more complicated in the absence of a structured approach to EA performance management.
Additionally, organizations often limit EA performance management to identifying and measuring the
potential benefits of EA and don’t measure the real value of EA. In this paper we propose that before
establishing EA metrics organizations should clearly understand the context of EA from three perspectives,
i.e. the scope of EA for the organization, its usage in the organization and the purpose of its measurement.
We subsequently present a model for defining EA metrics based on the context of EA.
1 INTRODUCTION
It is said that if you can’t measure something, you
can’t improve or manage it. As resources are
limited, organizations cannot afford to lose sight of
the performance of their investments. So
performance management has gained significant
importance for organizations. This applies to every
action that the organization undertakes.
Enterprise Architecture (EA) is being
increasingly adopted by organizations for various
purposes. Adopting and establishing EA requires
significant investments, and although most
organizations are aware of the benefits of EA they
find it challenging to ascertain the performance of
EA (Espinosa, 2011). Published research in the areas
of EA measurement, EA quality and EA benefits do
not specifically consider the context in which EA is
adopted in an organization. Considering the context
provides ‘the circumstances that form the setting for
an event, statement, or idea, and in terms of which it
can be fully understood’ (dictionary definition).
Thus considering the context of EA adoption in the
organization would provide a better understanding
of the circumstances and conditions that form the
adoption EA in the organization. We believe that
this would provide significant information for
guiding EA performance management.
So before venturing into ascertaining the
performance of EA, organizations should clearly
understand the context of EA in the organization. In
this paper we propose that the context of EA should
be understood from three perspectives, namely, ‘the
perspective of scope’, ‘the perspective of role’ and
‘the perspective of measurement’. Section 2
provides a brief literature review. Section 3 defines
the importance of considering context and what we
mean by the term Enterprise Architecture. Section 4
provides details of the three perspectives. In section
5 we propose a model for identifying EA metrics
based on these three perspectives.
2 LITERATURE REVIEW
Although EA has its roots in information system
management, the concept has evolved from IT
architecture to architecture at the level of the entire
organization (Jonkers, 2006). The understanding of
organization wide scope of EA exists in the industry
yet a lot of existing research relates EA performance
management mainly to information system
management areas such as IT investment
management, change management, agility and
249
Kumar A. and S. P..
Enterprise Architecture Performance Management - A Context based Approach to EA Metrics Definition.
DOI: 10.5220/0004098702490254
In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS-2012), pages 249-254
ISBN: 978-989-8565-12-9
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
business-IT alignment. As mentioned earlier EA is
being adopted by organizations for multiple
purposes, thus we have to define the meaning of the
term ‘Enterprise Architecture’ and what could be the
possible adoptions of EA in an organization. We
refer to these set of adoptions as identifying the
‘context of EA’. A summary of the literature review
is presented below.
EA Benefits: Two studies (Niemi, 2006) and
(Tamm, 2011) provide a comprehensive review of
the literature on Enterprise Architecture benefits and
provide models for categorizing EA benefits. These
studies show that the benefits of EA vary depending
on the definition of EA and how it is used. This
reflects the importance of defining the context of EA
in order to clearly understand and gauge the benefits
of EA in the organization.
EA Quality and Maturity Model: Traditionally EA
research was more focused on the development and
modelling of EA, but over time the quality aspects
of EA have also gained attention. This has primarily
driven the focus towards using EA maturity models
for the evolution and improvement of EA.
(Schekkerman, 2004) also provides an EA scorecard
to measure and monitor the status and quality of
various EA elements. (Reekum, 2006) suggests
metrics for measuring quality of EA. But we contend
that quality is only one set of possible measurements
for EA. Thus, the perspective of measurement also
plays an important role in determining the metrics
for measuring EA.
EA Metrics: Identifying quantifiable measures for
EA is a challenging task as it is not simple to express
the real value of EA in technical oriented metrics
(Schelp, 2007). Although (Schelp, 2007) provides a
comprehensive model for identifying EA metrics, it
does not provide a structured way of establishing the
context of EA. (Velitchkov, 2009) and
(Vasconcelos, 2007) both provide a suggestive list
of EA metrics but only from the scope of IT. We
believe that “empirical approaches are not suitable to
define a general metric” for EA, as the adoption of
EA is context driven.
EA Frameworks: None of the EA frameworks
provide any specific information on EA metrics.
TOGAF 9 only suggests that EA measurement
criterion can be developed much like the Capability
Maturity Model (CMM). (Zachman, 2001) suggests
that organizations should invest in architecture to
enable themselves for alignment, integration, change
and mass customization. Long-standing EA
frameworks such as FEAF and MODAF define their
own EA assessment models but they are very
specific to their individual requirements and thus
very specific to their context.
Based on the literature study it is derived that it
is difficult and unsuitable to define a fixed set of
metrics for EA as its implementation depends on the
organization and stakeholder requirements. Thus,
this reflects the value of understanding and defining
the context of EA for driving EA measurement.
3 WHY IS CONTEXT REQUIRED
Context defines the interrelated conditions in which
something exists or occurs. Organizational
endeavours operate under multiple interrelated
conditions. These conditions may change over time.
Without a thorough understanding of such
conditions and the changes thereof, organizations
would operate sub-optimally. Same is true for
Enterprise Architecture endeavours. Although,
before defining the context for EA, it is important to
define EA itself.
EA as a term consists of two words; Enterprise
and Architecture. Oxford dictionary defines
‘Enterprise’ as ‘a unit of economic organization or
activity’ and ‘Architecture’ as ‘a unifying or
coherent form or structure’.
Thus, ‘Enterprise Architecture’ refers to ‘a
coherent structure of a unit of economic
organization’. Simply put EA refers to the
description of an enterprise in terms of its parts, their
form and their logical structure, where the enterprise
could be the entire organization or a unit of the
organization (later referred to as domain in this
paper). The parts mentioned above are anything and
everything that constitutes the enterprise and their
form represents their essential nature.
By the explanation given above, EA should
ideally deal with anything and everything related to
the enterprise which includes apart from other things
organizational culture, management styles and
people. Industrial application of EA through various
frameworks and methodologies do not specifically
cover these aspects. Thus the current
implementations of EA and its measurement are
limited. Most organizations end up defining EA
metrics based on benefits such as integration,
alignment, agility which are difficult to quantify and
even more difficult to justify.
Additionally, the implementation of EA varies
from one organization to another and largely
depends on some conditions which we refer to as the
context. Thus, understanding and defining the
context provides the right setting for successful
implementation of EA in an organization. This in
ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems
250
turn affects how EA should be measured. We
propose that organizations can better manage the
identification and establishment of EA metrics by
clearly defining the context of EA in terms of the
following three perspectives:
What is the scope of EA in the organization?
(For example: organization-wide, business
unit, functions, domains etc.)
What is the role of EA in the organization?
(For example: as an information source, as a
planning tool, as a practice etc.)
What is the purpose of measuring EA?
(For example: value estimation, activity
metrics, measurement and reporting, maturity
improvement etc.)
4 SETTING THE CONTEXT
OF EA
The three perspectives that define the context of EA
are presented in Figure 1.
Figure 1: Three perspectives for the context of EA.
It is also important to note that these three
perspectives are independent of each other. For
example, the scope of EA might be only the IT
department, its role could be for planning (IT
strategy) or for alignment (business-IT alignment)
and its measurement could be value based or activity
based. The three perspectives are explained in the
subsequent sub-sections.
4.1 The Perspective of Scope
It has generally been observed that most EA
implementations are initiated by the IT department
or report to the IT function. In many organizations
the scope of EA is limited to information system
management or business-IT alignment. While, in
some organizations EA is represented across
functions. The scope perspective identifies the span
of EA in the organization. It can be just one domain,
multiple selective domains or organization wide.
Since EA has traditionally been very IT-centric,
most literature primarily suggests IT-centric metrics,
which may satisfy the requirement of EA if the
context of scope was only IT. Thus, by defining the
scope perspective clearly organizations can gain
significant focus and clarity in defining EA metrics.
4.2 The Perspective of Role
EA is evolving as a concept and so is its role in the
organization. The survey results in (Thomas, 2009)
show that the use of EA outside IT is increasing
where EA is also being used for strategy
implementation and enterprise transformation. Since
EA can play multiple roles in the organization,
defining the perspective of EA’s role becomes
important. We believe that EA can play three kinds
of role in an organization, as defined below.
As an ‘information source’ for multiple purposes
such as planning, decision making, standardization,
impact analysis etc. This role is only limited to
capturing and providing the required information.
As an ‘alignment and improvement’ agent EA
ensures that the scope under consideration of EA i.e.
the domain or business-unit under the scope of EA is
aligned to the overarching organization. This role
deals with and is limited to inter-departmental
alignment and alignment with the corporate body.
Here the outcomes of EA can very well be a roadmap
(limited to the perspective of scope), architecture
compliance (IT or non-IT), process management,
capability improvement, linking strategy to execution
etc. If EA is organization wide then this represents the
alignment of the EA roadmap with the organization’s
goals and the improvement that this roadmap brings
to the organization.
As a ‘planning’ agent EA, either existing as a unit
or a practice in the organization, controls and is
responsible for defining and implementing the
organizational plans. When existing as a unit EA
becomes a proxy to the planning department, and
when existing as a practice, the concepts of EA (i.e.
looking at the wholesome picture of the enterprise)
are followed by the planning department. This role of
EA is not implausible; as noted in (Thomas, 2009)
enterprise architects in 59% of the surveyed
organizations are either passively or actively involved
Context of EA
Role of EA
• Information Source
• Alignment and improvement
• Planning
Scope of EA
• Domains
• Organization
Measurement of EA
• Value estimation
• Activity based
• Maturity Improvement
Context of
Scope
Context of
Role
Context of
Measurement
EnterpriseArchitecturePerformanceManagement-AContextbasedApproachtoEAMetricsDefinition
251
in the strategic planning process. Additionally, since
its emergence in IT around the early 1990s, the
strategic planning of IT i.e. IT strategy has become
one of the primary work areas of EA (Thomas, 2009).
Many organizations are also using the concepts of EA
within their planning organizations to enable strategic
initiatives (Asfaw, 2009).
It should be noted that in the perspective of role,
the three roles of EA are not mutually exclusive or
independent but are rather hierarchical, i.e. as a
‘planning’ agent EA can also act as an ‘alignment
and improvement’ agent or ‘information source’ but
not vice-versa. Similarly, as an ‘alignment and
improvement’ agent EA can also act as an
‘information source’ but not vice-versa.
4.3 The Perspective of Measurement
The third perspective for EA is the perspective of
measurement itself. The concept of organizational
performance has evolved over time, ranging from
total quality management to value based
management. We believe that the measurement of
EA is also dependent on what the organization wants
to measure, is it the performance of EA activities,
quality of EA outcomes or the value that EA brings
to the organization? We define the perspective of
EA measurement by three kinds of measurement.
Value estimation deals with the estimation of the
value that EA contributes to the organization. It can
be measured qualitatively by defining value dials
based on stakeholder needs and requirements (which
would in turn depend on the scope perspective and the
role perspective) or quantitatively through established
methods such as EVA (economic value added).
Activity based measurement is more of a TQM
based approach that can establish measures for EA
activities (ex. EA development, EA governance, EA
compliance, EA communication etc.)
Maturity improvement would involve using an
EA maturity model for gauging the maturity of EA.
Additionally; they act as a step by step guide for the
development and enhancement of EA.
It should be noted that all these measurements
are not analogous i.e. all three can be used together
as they represent different types of measurement and
would typically address different set of stakeholders.
5 A MODEL FOR EA METRICS
Based on these three perspectives we propose a
context based model for establishing EA metrics.
Figure 2 shows this model for EA metrics.
Figure 2: Context based Model for EA Metrics.
Based on the scope and role perspectives, three
kinds of EA metrics can be defined; for estimating
the value of EA, for measuring EA activities and for
improving the maturity of the organization.
5.1 Value Estimation
Table 1 substantiates the EA metrics model given in
Figure 2 for the value estimation type of
measurement.
5.1.1 Perspective of Role - Planning
Table 1: Value estimation measures in the EA metrics
model.
Perspective of Role
Pers
p
ective of Sco
p
e
Domain
Specific
Organization
Wide
Planning
Planning
Effectiveness
Planning
Effectiveness
Alignment and
Improvement
EVA of
Roadmap
projects
Domain
Roadmap
contribution
to
organization
goals
EVA of
Roadmap
projects
Information
Source
Value of the
information
Value of the
information
Planning Effectiveness measures how effective the
plans are in meeting the needs of the domain or
organization. A balanced card approach can be taken
here by measuring the effectiveness of plans in the
financial, customer, internal process and learning &
growth perspectives. Effectiveness refers to
producing a desired result, where the desired results
Domain specific Organization wide
Planning
Alignment and
Improvement
Information Source
Perspective of Measurement
• Value Estimation
• Activity Based
• Maturity Improvement
Pers
p
ective of Sco
p
e
Perspective of Role
ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems
252
are captured through the KPIs and measures.
5.1.2 Perspective of Role - Alignment and
Improvement
The EVA of projects identified in the roadmap for a
domain or organization provide the value of EA
when EA is acting as an ‘alignment and
improvement’ agent. Since roadmap is the value-
adding outcome in this role, EVA provides a true
measure of the value added by EA.
Additionally, in case of a domain the
contribution of the projects identified in the roadmap
to the organizational goals can also provide
measures for domain to organization alignment.
5.1.3 Perspective of Role - Information
Source
The value of information is amount someone would
be willing to pay for the information available.
Currently, we do not have any suggestive methods
or metrics for estimating the value of information,
and it would be taken up as an area of future
research. But the areas of value of information
(VOI) theory and decision theory can be explored
further. Additionally, organizations can develop
qualitative models for ascertaining the value of the
information that EA provides.
5.2 Activity Based
Table 2 substantiates the EA metrics model given in
Figure 2 for the activity based type of measurement.
Table 2: Activity based measures in the EA metrics model.
Perspective of Scope
Domain
Specific
Organization
Wide
Perspective of Role
Planning
Planning
Efficiency
Planning
Efficiency
Alignment
and
Improvement
Architecture
Compliance
Level of
standardization
Process metrics
Architecture
Compliance
Process Metrics
Information
Source
Information
quality
Information
completeness
Information
quality
Information
completeness
5.2.1 Perspective of Role - Planning
Planning Efficiency measures how efficiently the
plans are prepared for the domain or organization.
Efficiency refers to the degree of economy and thus
relates to the resource based view. The metrics that
can be measured here are the cycle time of planning,
the cost of planning, planning response to change
(agility in planning) etc.
5.2.2 Perspective of Role - Alignment
and Improvement
The metrics presented in Table 2 are only example
metrics and are not necessarily comprehensive. The
activity based measurement can provide metrics for
process improvement, standardization etc. Level of
architecture compliance also ensures that the
organizational activities or changes are aligned with
enterprise architecture, whether it is in a domain or
organization wide.
5.2.3 Perspective of Role - Information
Source
The primary activity in EA an information source
role is the gathering and maintenance of information
in the EA repository. Thus the metrics are related to
the quality of EA information, its frequency of
updating and its completeness.
5.3 Maturity Improvement
As mentioned earlier the maturity model can be
defined independently. But the EA scope and role
perspectives inform the maturity model and vice
versa. Maturity models guide by providing means to
develop the practice fully and can thus influence
organizations to study and expand the perspectives
of scope and role for attaining higher maturity
levels. How these perspectives apply to the maturity
model would largely depend on how the maturity
model is defined. For ex. a CMM kind of maturity
model (initial, managed, defined, optimized) can be
developed and it can be applied in a similar way
whether the scope of EA is a domain or the entire
organization. Whereas maturity levels could also be
defined based on how EA is used in the organization
or where EA is represented in the organization
(Peter, 2009), and in such a case the scope and role
perspectives would have a bearing on the maturity
model and vice-versa.
EnterpriseArchitecturePerformanceManagement-AContextbasedApproachtoEAMetricsDefinition
253
6 CONCLUSIONS
For continuous improvement and evolution of EA in
the organization it is critical to measure it. In the
past many EA metrics have been proposed but no
fixed or well accepted set of metrics exist. Surveys
have also revealed that many organizations find it
difficult to measure EA and many organizations
don’t even measure EA.
Since EA is an evolving concept and different
organizations adopt EA differently, we believe that
before establishing the metrics for EA organizations
should define the context of EA. In this paper we
propose that organizations should define the context
of EA from three perspectives, namely, the
perspective of scope, the perspective of role and the
perspective of measurement. Using these three
perspectives the paper provides a comprehensive
model for organizations to define and establish
proper and focused measures for EA.
The paper also provides scope for further
research to identify valuable methods or
measurement techniques for estimating the value of
information provided by EA.
REFERENCES
Asfaw T., Bada A., Allario F; 2009. Enablers and
Challenges in Using Enterprise Architecture Concepts
to Drive Transformation: Perspectives from Private
Organizations and Federal Government Agencies.
Journal of Enterprise Architecture, Volume 5,
Number 3.
Espinosa, J.A; Boh, W.F; and DeLone, W.H; 2011. The
Organizational Impact of Enterprise Architecture: A
Research Framework. Proceedings of the 44th Hawaii
International Conference on System Sciences.
Jonkers H., Lankhorst M., Doest H. W. L., Arbab F., Bosma
H., Wieringa R. J; 2006. Enterprise Architecture:
Management tool and blueprint for the organisation.
Information Systems Frontier, Number 8.
Niemi E; 2006. Enterprise Architecture Benefits:
Perceptions from Literature and Practice. Proceedings
of the 7th IBIMA Conference Internet & Information
Systems in the Digital Age.
Peter B; Michael N; Daniel N; Tim P; 2009. Building
Value through Enterprise Architecture: A Global
Study.
Reekum, E. V., Steenbergen, M. van, Berg, M. van den,
Bos, R., Brinkkemper, S; 2006. An Instrument for
Measuring the Quality of Enterprise Architecture
Products. Proceedings Landelijk Architectuur Congres
(LAC). Via Nova Architectura.
Schekkerman, J; 2004. Enterprise Architecture
Scorecard, Version 2.1.
Schelp, J., Stutz, M; 2007. A Balanced Scorecard
Approach to Measure the Value of Enterprise
Architecture. Journal of Enterprise Architecture,
Vloume 3, Number 4.
Tamm T; 2011. How Does Enterprise Architecture Add
Value to Organisations? Communications of the
Association for Information Systems, Volume 29,
Number 1.
Thomas O; Mohan B; 2009. Infosys Enterprise
Architecture Survey 2008/2009 Report.
Vasconcelos A., Sousa P., Tribolet J; 2007. Information
System Architecture Metrics: an Enterprise
Engineering Evaluation Approach. Electronic Journal
of Information Systems Evaluation, Volume 10,
Number 1.
Velitchkov I.; 2009. Enterprise Architecture Metrics in the
Balanced Scorecard for IT. ISACA Journal, Volume 3.
Ylimaki T; 2007. Towards a Generic Evaluation Model
for Enterprise Architecture. Journal of Enterprise
Architecture, Volume 3, Number 3.
Zachman J. A; 2001. You Can't “Cost-Justify”
Architecture. (www.brcommunity.com).
ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems
254
