ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS
A Systematic Literature Review
Belen Blasco, Marcela Genero and Mario Piattini
ALARCOS Research Group, Department of Information Technologies and Languages
University of Castilla-la Mancha, Paseo de la Universidad 4, Ciudad Real, Spain
Keywords: Measurement, Business goals, Systematic literature review.
Abstract: Organizations are currently concerned about the importance of establishing software measurement
programs. They do not, however, obtain all the benefit expected from them. This is, in some cases, owing to
the lack of alignment between these measurement programs and organizations’ business goals. The
objective of this paper is to attempt to identify all existing works concerning the alignment of measurement
programs and business goals, with the aim of identifying future work lines. This has been done by carrying
out a systematic literature review that provides 26 primary studies, found in six digital libraries until January
2010. These studies were classified according to: the technique or techniques used in them, whether they
propose a measurement method or also include a list of measures, the existence of support tools, and the
validation of the proposal. After analyzing these papers, we discovered that the techniques most frequently
used are GQM and BSC, and also that the majority of the papers propose a measurement method and are
validated with real experiences.
1 INTRODUCTION
The motivation of this paper is to study the current
situation with regard to the alignment of
measurement and business goals in software
development organizations, with the aim of
identifying future research lines in this field. We
therefore decided to carry out a systematic literature
review (SLR) (Kitchenham and Charters, 2007),
which would allow us to identify all the papers
published in a systematic and reproducible manner.
The SLR-Tool (Fernandez-Saez et al., 2010) was
used to support the different stages of the review
process.
This paper is structured as follows: Section 2
describes the activities carried out in the planning
and execution of the SLR, and also presents the
results obtained. Finally, Section 3 presents our
conclusions and future work.
2 DESCRIPTION OF SLR
The following sub-sections describe the different
steps performed to carry out the SLR, considering
the guidelines proposed in (Kitchenham and
Charters, 2007).
2.1 Planning the Review
This SLR was developed with the aim of
discovering all the relevant information concerning
the alignment of measurement programs with
business goals. This objective led us to develop a
series of questions that we hope answer with the
results of this research (see Table 1).
Table 1: Research questions.
Research question
RQ1. Which techniques or methods to align measurement and
business goals are used?
RQ2. What are the proposals?
RQ3. Are the proposals validated in any way?
RQ4. Are the proposed techniques supported with a tool?
The searches were made in the following
electronic sources: IEEE, ACM, Scopus, Science
Direct, Springer Link, and Wiley Interscience.
Other papers were provided by experts (the last
two authors of this article). These papers were
considered as “grey literature”, as is suggested in
(Kitchenham and Charters, 2007).
The next step was to identify the search string.
This was done by identifying the major terms and
their synonyms or related words and combining
them using the logic operators “OR” and “AND”.
281
Blasco B., Genero M. and Piattini M..
ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review.
DOI: 10.5220/0003492202810285
In Proceedings of the 6th International Conference on Software and Database Technologies (ICSOFT-2011), pages 281-285
ISBN: 978-989-8425-77-5
Copyright
c
2011 SCITEPRESS (Science and Technology Publications, Lda.)
The string obtained is as follows:
(measure or measurement or metric) and
(business or organization or company) and (goal or
need or requirement or strategy)) and software.
The search was carried out in the title, abstract
and keywords, when the sources had this facility.
Otherwise, the search was carried out in the full text.
The following inclusion and exclusion criteria
were defined in order to facilitate the selection of
primary studies:
¾ Papers can only be written in English or
Spanish.
¾ Papers cannot belong to other research fields
such as robotics or systems control.
¾ Abstracts, conference summaries, or
documents that are not accessible are not
included.
¾ Those papers that do not match the search
string in the title key words, abstract, or full
text, cannot be included as primary studies.
We also defined some guidelines were used to
avoid problems concerning the repetition of papers.
In order to answer the research questions
identified in Table 1 we have considered four
dimensions: technique used, type of proposal,
validation method, and existence of a tool.
The values for each dimension are listed below:
¾ Technique used (a paper may use several
techniques):
BSC (Balanced Score Card), (Kaplan
and Norton, 1996)
GQM (Goal, Question, Metric), (Basili et
al., 1994), (Basili and Weiss, 1984)
AMI (Assess, Analyze, Metricate,
Improve), (Pulford and Kuntzmann-
Combelles, 1996)
KPI (Key Performance Indicators),
(Parmenter, 2007)
Cobit (Control Objectives for
Information and related Technology),
(Information Systems Audit and Control
Association 2007)
CMM (Capability Maturity Model)
(Paulk et al., 1995)
GDM (Goal Driven Measurement),
(Park, 1996)
No determined technique
¾ Type of proposal: measurement method, list
of metrics and related goals.
¾ Validation method: only proposal, example,
and real experience.
¾ Existence of a tool: Yes, No.
2.2 Execution of the SLR
This systematic literature review was carried out
until January 2010. Table 2 shows statistics
concerning those papers that were found and
accepted through the different steps of the review
execution.
The primary studies obtained have been
organized by the search source in which they were
found. The full list of papers is presented in
Appendix 1. There is also included a short view of
the classification.
Table 2: Distribution of papers per source.
Source Found Repeated Ex. 1
s
t
Ex. 2
nd
Total
Grey
literature
7 4 0 0 3
SCOPUS 95 21 70 4 0
ACM 8 0 8 0 0
IEEE
Computer
98 1 56 25 16
Science
Direct
69 0 36 28 5
SpringerLink
6 0 3 2 1
Wiley
InterScience
69 0 67 1 1
Total 352 26 240 60 26
2.3 Results Obtained
The results are structured on the basis of the research
questions stated above. The data extracted from the
papers reviewed was analyzed both quantitatively
and qualitatively to answer the research questions.
2.3.1 RQ1. Techniques Used
The first research question consisted of identifying
which existent techniques are used or adapted to
relate both fields. We also wished to identify any
new technique or method.
We found that most of the papers use or adapt
BSC, GQM or both together. This is because BSC is
a tool with which to identify business goals that is
very well known in business management, and
GQM allows software measurement to be planned
and implemented. The alignment of software
measurement and business goals can thus be
achieved by combining these two techniques.
In “No determined technique” we have included
those primary studies that do not use an existing
technique and do not propose a new one, but
nevertheless provide guidelines to define a new
measurement program or to define business goals.
The most valuable proposal for us is GQM+
[EP-1, EP-2 and EP-3]. This is an adaptation of
ICSOFT 2011 - 6th International Conference on Software and Data Technologies
282
GMQ designed to take into account business goals.
However, from our point of view this technique is
not yet complete, since it is centered on the passage
from business goals to measurement goals but does
not explain issues such as measurement execution
and the presentation of results.
2.3.2 RQ2. Type of Proposal
The next step was to discover what they proposed.
In this case, we distinguished between papers that
centered on explaining a measurement process (16
studies, 62%), and those that are centered on the
measures instead of the process or proposed a
predefined list of goals and their related measures
(10 studies, 38%).
2.3.3 RQ3. Validation Method
The number of primary studies that present case
studies is greater than those that present imaginary
examples, and practically double the number of
simple proposals.
2.3.4 RQ4. Existing Tools
The last research question made reference to the
existence of support tools for the techniques
mentioned in the primary studies. There is only a
15% of studies (4 studies) that mention a support
tool.
2.3.5 Other Results
This review has allowed us to obtain other
conclusions, and is not limited solely to the
knowledge provided by the primary studies. An
example is the maturity of the research filed
according to the kind of publications found. The
percentage of papers published in conferences is
greater than the number of papers published in
journals (50% of conference papers, 42% of journal
papers), and only 8% are presented in workshops.
We did not find any other kinds of document, such
as books or book sections.
3 CONCLUSIONS AND FUTURE
WORK
This paper presents an analysis of the state of the art
in the alignment of measurement programs and
business goals that was obtained by carrying out an
SLR. Between October 2009 and January 2010 we
carried out a systematic literature review, obtaining
26 primary studies. These papers were classified
according to four different dimensions. In the first
we considered the techniques used to carry out the
measurement. Most of the studies used GQM, BSC
or a mixture of both. With regard to the second
dimension (the kind of proposal presented in the
study) it was observed that 62% of the studies
presented a measurement method, in comparison to
32% of metric-centered studies. With regard to the
validation of proposes, 58% were supported by real
cases. Finally, only 15% of the methods mentioned
have a support tool. We have reached the conclusion
that this research is extremely important to
companies, there being a high percentage (58%) that
presented case studies and real experiences.
Moreover, the technique that best aligns the
measurement with business goals is GQM +,
proposed in studies [EP-1, EP-2 and EP-3].
However, other papers also suggest interesting
aspects to consider. One possible line for future
work might therefore be to attempt to unify all the
proposals into a single technique for measuring
programs aligned with business goals. Having
defined this new technique, the following step would
be the development of a tool to support it.
ACKNOWLEDGEMENTS
This research has been funded by the following
projects: MEDUSAS (CDTI-MICINN and FEDER
IDI-20090557), ORIGIN (CDTI-MICINN and
FEDER IDI-2010043 (1-5)), PEGASO/MAGO
(MICINN and FEDER, TIN2009-13718-C02-01),
EECCOO (MICINN TRA2009_0074) MECCA
(JCMM PII2I09-0075-8394) and IMPACTUM (PEII
11-0330-4414).
REFERENCES
Basili, V., Caldiera, G., and Rombach, D. (1994). The
Goal Question Metrics Approach. Wiley.
Basili, V., and Weiss, D. (1984). A Methodology for
Collecting Valid Software Engineering Data. IEEE
Transactions on Software Engineering, 10(3), 728-
738.
Fernandez-Saez, A. M., Genero, M., and Romero, F. P.
(2010). SLR-Tool: A Tool for performing systematic
literature reviews. The 5th International Conference
on Software and Data Technologies (pp. 157-166).
Information Systems Audit and Control Association.
(2007). Control Objectives for Information and
Related Technology, 4.1 th Edition. IT Governance
Institute.
ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review
283
Kaplan, S., and Norton, P. (1996). Using the Balanced
Scorecard as a Strategic Management System.
Harvard Business Review, 76.
Kitchenham, B., and Charters, S. (2007). Guidelines for
Performing Systematic Literature Reviews in Software
Engineering. EBSE-2007-01, Keele University
Park, R. E. (1996). Goal-Driven Software Measurement: A
Guidebook. CMU/SEI-96-HB-002
Parmenter, D. (2007). Key Performance Indicators. John
Wiley and Sons.
Paulk, M. C., Weber, C. V., Curtis, B., and Chrissis, M. B.
(1995). The Capability Maturity Model: Guidelines for
Improving the Software Process. Addison Wesley.
Pulford, K., and Kuntzmann-Combelles, A. (1996). A
Quantitative Approach to Software management: The
AMI handbook.
APPENDIX
1. List and Classification of primary studies
Table 3 shows the classification of each primary study, with a brief comment on each proposal. Classification
column is codified as follow: Techniques; Propose (M=Method, L=List); Validation (P=Proposal, E=Example,
R = Real experience); Tool (Y=yes, N=no).
Table 3: Classification of the primary studies.
Id Reference Classif. Comments
EP
1
Basili, V., Heidrich, J., Lindvall, M., Munch, J., Regardie, M., Trendowicz,
A.: Brinding the Gap Between Business Strategy and Software
Development. International Conference on Information Systems (2007)
GQM; M;
E; N
Very detailed example of a new method
called GQM+
EP
2
Basili, V., Heidrich, J., Lindvall, M., Munch, J., Regardie, M., Trendowicz,
A.: Determining the Impact of Business Strategies Using Principles from
Goal-Oriented Measurement. Wirtschaftsinformatik (2009)
GQM; M;
E; N
Detailed description of GQM+. This is a
method for defining measurement
programs taking into account business
goals.
EP
3
Barthel, H., Heidrich, J., Munch, J., Trendowicz, A.: GQM+ Strategies:
Experiences from Industrial Case Studies and Visualization Needs.
International Software Engineering Research Network (2009)
GQM; M;
R; Y
Real application of GQM+ and description
of its support tool.
EP
4
Xiaodong Guo, Li Meng: Organization Application Oriented Software
Process Measurement Model. International Symposium on Computer
Science and Computational Technology (2008)
GDM; M;
P; N
Integrated measurement process model
and application algorithm.
EP
5
Kulic, P.: A Practical Approach to Software Metrics. IT Pro Software
Development, Volume 2 (1), pp. 38-42 (2000)
None; L;
P; N
List the steps needed to define a good
measurement program. It doesn’t explain
how to link business goals and measures.
EP
6
Patton, J.: Ambiguous Business Value Harms Software Products. IEEE
Software. Volume 25 (1), pp. 50-51 (2008)
None; M;
P; N
Gives some guidelines for identify
measures from business goals.
EP
7
Becker, S. A., Bostelman, M. L.: Aligning Strategic and Project
Measurement Systems. IEEE Software, Volume 16 (3), pp. 46-51 (1999)
BSC,
GQM; M;
R; N
Applies GQM, but separating goals
depending on the perspective of BSC they
belong.
EP
8
Offen, R., J.; Jeffery, R.: Establishing Software Measurement Programs.
IEEE Software, Volume 14 (2), pp. 45-53 (1997)
GQM; M;
R; N
Uses the definition of business goals by
expert meetings, and after defines GQM
measurement goals.
EP
9
Vardangalos, G; Pantelis, A.: A Performance System Based on the
Balanced Scorecard Approach for Measuring Performance in a Business
Environment. International Symposium on Computers and
Communications (2000)
BSC,
QGM,
KPI; M;
P; Y
Combines BSC, KPI and PSM, but the
paper doesn’t specify how to change from
one to another.
EP
10
Van Grembergen, W.: Aligning Business and Information Technology
through the Balanced Scorecard at a Major Canadian Financial Group: its
Status Measured with an IT BSC Maturity Model. Hawaii International
Conference on System Sciences (2001)
BSC; L;
R; N
Adaptation of BSC for IT. Each
perspective is translated into predefined
measures. It can be breaking down in
levels, and it also includes maturity levels.
EP
11
Briand, L. C., Morasca, S., Basili, V. R.: An Operational Process for Goal-
Driven Definition of Measures. IEEE Transactions on Software
Engineering, Volume 28 (12), pp. 1106-1125 (2002)
GQM; M;
R; N
The pass from business goals to
measurement goals is done thanks to the
knowledge of company experts
EP
12
Ki-won Song, Soo-Hwan Lee, Young-Gyun Jang, Il-Seok Suh, Jin-Soo
Kim: Framework for Quantitative S/W Development Performance
Measurement and Analysis in Semiconductor Industry. International
Conference on Convergence and Hybrid Information Technology (2008)
GQM,
KPI; L; R;
N
List of measures.
EP
13
Yan Xu, Chung-Hsing Yeh: Evaluating Critical Strategies for Enterprise
Resource Planning Systems Implementation. International Symposium on
Electronic Commerce and Security (2009)
BSC; L;
R; N
Proposes a list of goals and strategies and
an algorithm to calculate the aggregated
value.
EP
14
Jahankhani, H., Ekeigwe, J.: Adaptation of the balanced scorecard model to
the IT functions. International Conference on Information Technology and
Applications (2005)
BSC; M;
P; N
Explain how to apply BSC to IT. It does
not explicitly how to align business goals
and measurement.
ICSOFT 2011 - 6th International Conference on Software and Data Technologies
284
Table 3: Classification of the primary studies (cont.).
EP
15
Van Grembergen, W., Amelinckx, I.: Measuring and Managing E-business
Projects through the Balanced Scorecard. Hawaii International Conference
on System Sciences (2002)
BSC; L;
P; N
Proposes a generic BSC for e-business. It
includes the list of metrics that results of
the application of the technique.
EP
16
List, B., Bruckner, R., Kapaun, J.: Holistic Software Process Performance
Measurement From the Stakeholders’ Perspective. International Workshop
on Database and expert Systems Applications (2005)
KPI; L; R;
N
Framework for defining measures for
process measurement that considers the
perspective of each stakeholder.
EP
17
Oinas, A.: Defining Goal-driven Fault Management Metrics in a Real
World Environment: A Case-Study from Nokia. Conference on Software
Maintenance and Reengineering (2000)
GQM,
AMI;
M; R; N
Application of a combination of GQM and
AMI at Nokia.
EP
18
Kopanas, V., Sylaidis, V., Nanakis, I.: GQM-based Improvement of
Embedded, Real-time Software Development Practices. International
Workshop on Software Technology and Engineering Practice (1997)
GQM,
AMI;
M; R; N
Application of a combination of GQM and
AMI at a company.
EP
19
Van Grembergen, W., De Haes, S., Van Brempt, H.: Prioritising and
Linking Business and IT Goals in the Financial Sector. Hawaii International
Conference on System Sciences (2007)
Cobit; L;
P; N
Proposes a guide with a cascade of
business goals, IT goals and IT processes,
based on Cobit.
EP
20
Martinsons, M., Davison, R., Tse, D.: The balanced scorecard: a foundation
for the strategic management of information systems. Decision Support
Systems, Volume 25, pp. 71-88(1999)
BSC; L;
R; N
Framework for the evaluation of IT and IS
base don BSC.
EP
21
Roche, J., Jackson, M.: Software measurement methods: recipes for
success? Information and Software Technology Volume 36 (3), pp. 173-
189 (1994)
None; M;
P; N
Review of several measurement methods.
EP
22
Aversano, L., Bodhuin, T., Canfora, G., Tortorella, M.: Technology-driven
business evolution. Journal of Systems and Software, Volume 79, pp. 314-
338 (2006)
BSC,
GQM;
L; R; Y
In the measurement section it applies
GQM using goals identified with BSC.
EP
23
Trienekens, J., Kusters, R., Rendering, B., Stokla, K.: Business-oriented
process improvement: practices and experiences at Thales Naval The
Netherlands (TNNL). Information and Software Technology, Volume 45,
pp. 67-79 (2005)
GQM; M;
R; Y
Proposes a framework for process
improvement, where a very important part
is the alignment of business goals.
EP
24
Lawrence Pfleeger, S.: Maturity, Models, and Goals: How to Build a
Metrics Plan. Journal of Systems and Software, Volume 31, pp- 143-155
(1995)
GQM; M;
R; N
Presents an example of how to apply
GQM taking into account business goals.
EP
25
Kuntzmann-Combelles, A.: Quantitative Approach to Software Process
Improvement. Symposium on Software Quality Techniques and
Acquisition Criteria on Software Quality Techniques and Acquisition
Criteria (1995)
AMI,
CMM;
M; R; N
Describes AMI and how it has been
applied together with CMM in an
industrial environment.
EP
26
Savioja, E., Tukiainen, M.: Measurement Practices in Financial Software
Industry. Software Process Improvement and Practice, Volume 12 (6), pp.
585-595 (2007)
BSC; L;
R; N
Example of how to apply a measurement
program. It shows the obtained measures.
ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review
285