ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS

A Systematic Literature Review

Belen Blasco, Marcela Genero and Mario Piattini

ALARCOS Research Group, Department of Information Technologies and Languages

University of Castilla-la Mancha, Paseo de la Universidad 4, Ciudad Real, Spain

Keywords: Measurement, Business goals, Systematic literature review.

Abstract: Organizations are currently concerned about the importance of establishing software measurement

programs. They do not, however, obtain all the benefit expected from them. This is, in some cases, owing to

the lack of alignment between these measurement programs and organizations’ business goals. The

objective of this paper is to attempt to identify all existing works concerning the alignment of measurement

programs and business goals, with the aim of identifying future work lines. This has been done by carrying

out a systematic literature review that provides 26 primary studies, found in six digital libraries until January

2010. These studies were classified according to: the technique or techniques used in them, whether they

propose a measurement method or also include a list of measures, the existence of support tools, and the

validation of the proposal. After analyzing these papers, we discovered that the techniques most frequently

used are GQM and BSC, and also that the majority of the papers propose a measurement method and are

validated with real experiences.

1 INTRODUCTION

The motivation of this paper is to study the current

situation with regard to the alignment of

measurement and business goals in software

development organizations, with the aim of

identifying future research lines in this field. We

therefore decided to carry out a systematic literature

review (SLR) (Kitchenham and Charters, 2007),

which would allow us to identify all the papers

published in a systematic and reproducible manner.

The SLR-Tool (Fernandez-Saez et al., 2010) was

used to support the different stages of the review

process.

This paper is structured as follows: Section 2

describes the activities carried out in the planning

and execution of the SLR, and also presents the

results obtained. Finally, Section 3 presents our

conclusions and future work.

2 DESCRIPTION OF SLR

The following sub-sections describe the different

steps performed to carry out the SLR, considering

the guidelines proposed in (Kitchenham and

Charters, 2007).

2.1 Planning the Review

This SLR was developed with the aim of

discovering all the relevant information concerning

the alignment of measurement programs with

business goals. This objective led us to develop a

series of questions that we hope answer with the

results of this research (see Table 1).

Table 1: Research questions.

Research question

RQ1. Which techniques or methods to align measurement and

business goals are used?

RQ2. What are the proposals?

RQ3. Are the proposals validated in any way?

RQ4. Are the proposed techniques supported with a tool?

The searches were made in the following

electronic sources: IEEE, ACM, Scopus, Science

Direct, Springer Link, and Wiley Interscience.

Other papers were provided by experts (the last

two authors of this article). These papers were

considered as “grey literature”, as is suggested in

(Kitchenham and Charters, 2007).

The next step was to identify the search string.

This was done by identifying the major terms and

their synonyms or related words and combining

them using the logic operators “OR” and “AND”.

281

Blasco B., Genero M. and Piattini M..

ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review.

DOI: 10.5220/0003492202810285

In Proceedings of the 6th International Conference on Software and Database Technologies (ICSOFT-2011), pages 281-285

ISBN: 978-989-8425-77-5

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

The string obtained is as follows:

(measure or measurement or metric) and

(business or organization or company) and (goal or

need or requirement or strategy)) and software.

The search was carried out in the title, abstract

and keywords, when the sources had this facility.

Otherwise, the search was carried out in the full text.

The following inclusion and exclusion criteria

were defined in order to facilitate the selection of

primary studies:

¾ Papers can only be written in English or

Spanish.

¾ Papers cannot belong to other research fields

such as robotics or systems control.

¾ Abstracts, conference summaries, or

documents that are not accessible are not

included.

¾ Those papers that do not match the search

string in the title key words, abstract, or full

text, cannot be included as primary studies.

We also defined some guidelines were used to

avoid problems concerning the repetition of papers.

In order to answer the research questions

identified in Table 1 we have considered four

dimensions: technique used, type of proposal,

validation method, and existence of a tool.

The values for each dimension are listed below:

¾ Technique used (a paper may use several

techniques):

 BSC (Balanced Score Card), (Kaplan

and Norton, 1996)

 GQM (Goal, Question, Metric), (Basili et

al., 1994), (Basili and Weiss, 1984)

 AMI (Assess, Analyze, Metricate,

Improve), (Pulford and Kuntzmann-

Combelles, 1996)

 KPI (Key Performance Indicators),

(Parmenter, 2007)

 Cobit (Control Objectives for

Information and related Technology),

(Information Systems Audit and Control

Association 2007)

 CMM (Capability Maturity Model)

(Paulk et al., 1995)

 GDM (Goal Driven Measurement),

(Park, 1996)

 No determined technique

¾ Type of proposal: measurement method, list

of metrics and related goals.

¾ Validation method: only proposal, example,

and real experience.

¾ Existence of a tool: Yes, No.

2.2 Execution of the SLR

This systematic literature review was carried out

until January 2010. Table 2 shows statistics

concerning those papers that were found and

accepted through the different steps of the review

execution.

The primary studies obtained have been

organized by the search source in which they were

found. The full list of papers is presented in

Appendix 1. There is also included a short view of

the classification.

Table 2: Distribution of papers per source.

Source Found Repeated Ex. 1

Ex. 2

Total

Grey

literature

7 4 0 0 3

SCOPUS 95 21 70 4 0

ACM 8 0 8 0 0

IEEE

Computer

98 1 56 25 16

Science

Direct

69 0 36 28 5

SpringerLink

6 0 3 2 1

Wiley

InterScience

69 0 67 1 1

Total 352 26 240 60 26

2.3 Results Obtained

The results are structured on the basis of the research

questions stated above. The data extracted from the

papers reviewed was analyzed both quantitatively

and qualitatively to answer the research questions.

2.3.1 RQ1. Techniques Used

The first research question consisted of identifying

which existent techniques are used or adapted to

relate both fields. We also wished to identify any

new technique or method.

We found that most of the papers use or adapt

BSC, GQM or both together. This is because BSC is

a tool with which to identify business goals that is

very well known in business management, and

GQM allows software measurement to be planned

and implemented. The alignment of software

measurement and business goals can thus be

achieved by combining these two techniques.

In “No determined technique” we have included

those primary studies that do not use an existing

technique and do not propose a new one, but

nevertheless provide guidelines to define a new

measurement program or to define business goals.

The most valuable proposal for us is GQM+

[EP-1, EP-2 and EP-3]. This is an adaptation of

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

282

GMQ designed to take into account business goals.

However, from our point of view this technique is

not yet complete, since it is centered on the passage

from business goals to measurement goals but does

not explain issues such as measurement execution

and the presentation of results.

2.3.2 RQ2. Type of Proposal

The next step was to discover what they proposed.

In this case, we distinguished between papers that

centered on explaining a measurement process (16

studies, 62%), and those that are centered on the

measures instead of the process or proposed a

predefined list of goals and their related measures

(10 studies, 38%).

2.3.3 RQ3. Validation Method

The number of primary studies that present case

studies is greater than those that present imaginary

examples, and practically double the number of

simple proposals.

2.3.4 RQ4. Existing Tools

The last research question made reference to the

existence of support tools for the techniques

mentioned in the primary studies. There is only a

15% of studies (4 studies) that mention a support

tool.

2.3.5 Other Results

This review has allowed us to obtain other

conclusions, and is not limited solely to the

knowledge provided by the primary studies. An

example is the maturity of the research filed

according to the kind of publications found. The

percentage of papers published in conferences is

greater than the number of papers published in

journals (50% of conference papers, 42% of journal

papers), and only 8% are presented in workshops.

We did not find any other kinds of document, such

as books or book sections.

3 CONCLUSIONS AND FUTURE

WORK

This paper presents an analysis of the state of the art

in the alignment of measurement programs and

business goals that was obtained by carrying out an

SLR. Between October 2009 and January 2010 we

carried out a systematic literature review, obtaining

26 primary studies. These papers were classified

according to four different dimensions. In the first

we considered the techniques used to carry out the

measurement. Most of the studies used GQM, BSC

or a mixture of both. With regard to the second

dimension (the kind of proposal presented in the

study) it was observed that 62% of the studies

presented a measurement method, in comparison to

32% of metric-centered studies. With regard to the

validation of proposes, 58% were supported by real

cases. Finally, only 15% of the methods mentioned

have a support tool. We have reached the conclusion

that this research is extremely important to

companies, there being a high percentage (58%) that

presented case studies and real experiences.

Moreover, the technique that best aligns the

measurement with business goals is GQM +,

proposed in studies [EP-1, EP-2 and EP-3].

However, other papers also suggest interesting

aspects to consider. One possible line for future

work might therefore be to attempt to unify all the

proposals into a single technique for measuring

programs aligned with business goals. Having

defined this new technique, the following step would

be the development of a tool to support it.

ACKNOWLEDGEMENTS

This research has been funded by the following

projects: MEDUSAS (CDTI-MICINN and FEDER

IDI-20090557), ORIGIN (CDTI-MICINN and

FEDER IDI-2010043 (1-5)), PEGASO/MAGO

(MICINN and FEDER, TIN2009-13718-C02-01),

EECCOO (MICINN TRA2009_0074) MECCA

(JCMM PII2I09-0075-8394) and IMPACTUM (PEII

11-0330-4414).

REFERENCES

Basili, V., Caldiera, G., and Rombach, D. (1994). The

Goal Question Metrics Approach. Wiley.

Basili, V., and Weiss, D. (1984). A Methodology for

Collecting Valid Software Engineering Data. IEEE

Transactions on Software Engineering, 10(3), 728-

738.

Fernandez-Saez, A. M., Genero, M., and Romero, F. P.

(2010). SLR-Tool: A Tool for performing systematic

literature reviews. The 5th International Conference

on Software and Data Technologies (pp. 157-166).

Information Systems Audit and Control Association.

(2007). Control Objectives for Information and

Related Technology, 4.1 th Edition. IT Governance

Institute.

ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review

283

Kaplan, S., and Norton, P. (1996). Using the Balanced

Scorecard as a Strategic Management System.

Harvard Business Review, 76.

Kitchenham, B., and Charters, S. (2007). Guidelines for

Performing Systematic Literature Reviews in Software

Engineering. EBSE-2007-01, Keele University

Park, R. E. (1996). Goal-Driven Software Measurement: A

Guidebook. CMU/SEI-96-HB-002

Parmenter, D. (2007). Key Performance Indicators. John

Wiley and Sons.

Paulk, M. C., Weber, C. V., Curtis, B., and Chrissis, M. B.

(1995). The Capability Maturity Model: Guidelines for

Improving the Software Process. Addison Wesley.

Pulford, K., and Kuntzmann-Combelles, A. (1996). A

Quantitative Approach to Software management: The

AMI handbook.

APPENDIX

1. List and Classification of primary studies

Table 3 shows the classification of each primary study, with a brief comment on each proposal. Classification

column is codified as follow: Techniques; Propose (M=Method, L=List); Validation (P=Proposal, E=Example,

R = Real experience); Tool (Y=yes, N=no).

Table 3: Classification of the primary studies.

Id Reference Classif. Comments

Basili, V., Heidrich, J., Lindvall, M., Munch, J., Regardie, M., Trendowicz,

A.: Brinding the Gap Between Business Strategy and Software

Development. International Conference on Information Systems (2007)

GQM; M;

E; N

Very detailed example of a new method

called GQM+

Basili, V., Heidrich, J., Lindvall, M., Munch, J., Regardie, M., Trendowicz,

A.: Determining the Impact of Business Strategies Using Principles from

Goal-Oriented Measurement. Wirtschaftsinformatik (2009)

GQM; M;

E; N

Detailed description of GQM+. This is a

method for defining measurement

programs taking into account business

goals.

Barthel, H., Heidrich, J., Munch, J., Trendowicz, A.: GQM+ Strategies:

Experiences from Industrial Case Studies and Visualization Needs.

International Software Engineering Research Network (2009)

GQM; M;

R; Y

Real application of GQM+ and description

of its support tool.

Xiaodong Guo, Li Meng: Organization Application Oriented Software

Process Measurement Model. International Symposium on Computer

Science and Computational Technology (2008)

GDM; M;

P; N

Integrated measurement process model

and application algorithm.

Kulic, P.: A Practical Approach to Software Metrics. IT Pro Software

Development, Volume 2 (1), pp. 38-42 (2000)

None; L;

P; N

List the steps needed to define a good

measurement program. It doesn’t explain

how to link business goals and measures.

Patton, J.: Ambiguous Business Value Harms Software Products. IEEE

Software. Volume 25 (1), pp. 50-51 (2008)

None; M;

P; N

Gives some guidelines for identify

measures from business goals.

Becker, S. A., Bostelman, M. L.: Aligning Strategic and Project

Measurement Systems. IEEE Software, Volume 16 (3), pp. 46-51 (1999)

BSC,

GQM; M;

R; N

Applies GQM, but separating goals

depending on the perspective of BSC they

belong.

Offen, R., J.; Jeffery, R.: Establishing Software Measurement Programs.

IEEE Software, Volume 14 (2), pp. 45-53 (1997)

GQM; M;

R; N

Uses the definition of business goals by

expert meetings, and after defines GQM

measurement goals.

Vardangalos, G; Pantelis, A.: A Performance System Based on the

Balanced Scorecard Approach for Measuring Performance in a Business

Environment. International Symposium on Computers and

Communications (2000)

BSC,

QGM,

KPI; M;

P; Y

Combines BSC, KPI and PSM, but the

paper doesn’t specify how to change from

one to another.

Van Grembergen, W.: Aligning Business and Information Technology

through the Balanced Scorecard at a Major Canadian Financial Group: its

Status Measured with an IT BSC Maturity Model. Hawaii International

Conference on System Sciences (2001)

BSC; L;

R; N

Adaptation of BSC for IT. Each

perspective is translated into predefined

measures. It can be breaking down in

levels, and it also includes maturity levels.

Briand, L. C., Morasca, S., Basili, V. R.: An Operational Process for Goal-

Driven Definition of Measures. IEEE Transactions on Software

Engineering, Volume 28 (12), pp. 1106-1125 (2002)

GQM; M;

R; N

The pass from business goals to

measurement goals is done thanks to the

knowledge of company experts

Ki-won Song, Soo-Hwan Lee, Young-Gyun Jang, Il-Seok Suh, Jin-Soo

Kim: Framework for Quantitative S/W Development Performance

Measurement and Analysis in Semiconductor Industry. International

Conference on Convergence and Hybrid Information Technology (2008)

GQM,

KPI; L; R;

List of measures.

Yan Xu, Chung-Hsing Yeh: Evaluating Critical Strategies for Enterprise

Resource Planning Systems Implementation. International Symposium on

Electronic Commerce and Security (2009)

BSC; L;

R; N

Proposes a list of goals and strategies and

an algorithm to calculate the aggregated

value.

Jahankhani, H., Ekeigwe, J.: Adaptation of the balanced scorecard model to

the IT functions. International Conference on Information Technology and

Applications (2005)

BSC; M;

P; N

Explain how to apply BSC to IT. It does

not explicitly how to align business goals

and measurement.

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

284

Table 3: Classification of the primary studies (cont.).

Van Grembergen, W., Amelinckx, I.: Measuring and Managing E-business

Projects through the Balanced Scorecard. Hawaii International Conference

on System Sciences (2002)

BSC; L;

P; N

Proposes a generic BSC for e-business. It

includes the list of metrics that results of

the application of the technique.

List, B., Bruckner, R., Kapaun, J.: Holistic Software Process Performance

Measurement From the Stakeholders’ Perspective. International Workshop

on Database and expert Systems Applications (2005)

KPI; L; R;

Framework for defining measures for

process measurement that considers the

perspective of each stakeholder.

Oinas, A.: Defining Goal-driven Fault Management Metrics in a Real

World Environment: A Case-Study from Nokia. Conference on Software

Maintenance and Reengineering (2000)

GQM,

AMI;

M; R; N

Application of a combination of GQM and

AMI at Nokia.

Kopanas, V., Sylaidis, V., Nanakis, I.: GQM-based Improvement of

Embedded, Real-time Software Development Practices. International

Workshop on Software Technology and Engineering Practice (1997)

GQM,

AMI;

M; R; N

Application of a combination of GQM and

AMI at a company.

Van Grembergen, W., De Haes, S., Van Brempt, H.: Prioritising and

Linking Business and IT Goals in the Financial Sector. Hawaii International

Conference on System Sciences (2007)

Cobit; L;

P; N

Proposes a guide with a cascade of

business goals, IT goals and IT processes,

based on Cobit.

Martinsons, M., Davison, R., Tse, D.: The balanced scorecard: a foundation

for the strategic management of information systems. Decision Support

Systems, Volume 25, pp. 71-88(1999)

BSC; L;

R; N

Framework for the evaluation of IT and IS

base don BSC.

Roche, J., Jackson, M.: Software measurement methods: recipes for

success? Information and Software Technology Volume 36 (3), pp. 173-

189 (1994)

None; M;

P; N

Review of several measurement methods.

Aversano, L., Bodhuin, T., Canfora, G., Tortorella, M.: Technology-driven

business evolution. Journal of Systems and Software, Volume 79, pp. 314-

338 (2006)

BSC,

GQM;

L; R; Y

In the measurement section it applies

GQM using goals identified with BSC.

Trienekens, J., Kusters, R., Rendering, B., Stokla, K.: Business-oriented

process improvement: practices and experiences at Thales Naval The

Netherlands (TNNL). Information and Software Technology, Volume 45,

pp. 67-79 (2005)

GQM; M;

R; Y

Proposes a framework for process

improvement, where a very important part

is the alignment of business goals.

Lawrence Pfleeger, S.: Maturity, Models, and Goals: How to Build a

Metrics Plan. Journal of Systems and Software, Volume 31, pp- 143-155

(1995)

GQM; M;

R; N

Presents an example of how to apply

GQM taking into account business goals.

Kuntzmann-Combelles, A.: Quantitative Approach to Software Process

Improvement. Symposium on Software Quality Techniques and

Acquisition Criteria on Software Quality Techniques and Acquisition

Criteria (1995)

AMI,

CMM;

M; R; N

Describes AMI and how it has been

applied together with CMM in an

industrial environment.

Savioja, E., Tukiainen, M.: Measurement Practices in Financial Software

Industry. Software Process Improvement and Practice, Volume 12 (6), pp.

585-595 (2007)

BSC; L;

R; N

Example of how to apply a measurement

program. It shows the obtained measures.

ALIGNMENT OF MEASUREMENT AND BUSINESS GOALS - A Systematic Literature Review

285