![](bg8.png)
engineers, to improve the analysis of the deviations.
Discussion on productivity related deviations lead
also to (preliminary) conclusions that most of the
activities that needed more actual effort than
estimated, were activities that were carried out in an
external partner organisation, to which part of the
software engineering activities had been outsourced.
It was suggested that these activities were performed
with a lower productivity because of the fact that the
external software developers required assistance
from the more experienced own software
developers. It was planned to monitor these
productivity related deviations on a continuous
basis, to see whether they would decrease as a
consequence of external development teams gaining
more experience.
6 CONCLUSIONS
First, based on interviews with project managers and
organisational managers, problems regarding
measurement in the case study environment have
been identified. From a literature study guidelines
could be derived to define improvement directions.
Using these guidelines three feedback loops have
been distinguished, respectively on the project level,
the multi-project level and the organisational level.
These feedback loops enable control on the
performance of the software development projects.
They also enable the exchange of information on
projects between the different management levels,
an act as backbone for a learning process. Secondly,
the implementation of the feedback loops has been
validated by carrying out small-scale experimental
measurements. The previous rigid approach of
‘product oriented’ metric definition, and the
dictation of their usage on the organisational level
which lead to a large set of metrics, has been
abandoned. Based on the Goal Question Metric
approach, and in collaboration with project
managers and senior managers, new ‘process
oriented’ effort deviation metrics have been defined,
respectively with respect to estimation-related effort
deviations and productivity-related effort deviations.
. From the experimental measurements it became
clear that on both management levels the same type
of ‘effort deviation’ information is of interest, and
can be provided. The metrics applied, i.e. were
relatively simple and easy to use. The visualisation
of the data from these metrics lead to interesting
discussions and more insight in the estimations and
the (resource) productivity of the software
development projects.
Thirdly, it also became clear that this information
had to be provided on a continuous basis,
respectively on the organisational level on a mid-
term (month-to-month) basis and on the project level
on a (short-term) day-to-day or week-to-week basis.
ACKNOWLEDGEMENTS
The authors would like to thank Kees-Jochem
Wehrmeijer for his contributions to this project.
REFERENCES
Balla, K., Bemelmans, T., Kusters, R., Trienekens, J.
2001, "Quality through Managed Improvement and
Measurement (QMIM)", Software Quality Journal,
vol. 9, no. 3, pp. 177-193.
Basili, V. R. 1996, "The Role of Experimentation in
Software Engineering: Past, Current, and Future",
Proceedings of the 18th International Conference on
Software engineering pp. 442-449.
Basili, V. R., McGarry, F. E., Pajerski, R., and Zelkowitz,
M. V. 2002, "Lessons learned from 25 years of
process improvement: the rise and fall of the NASA
software engineering laboratory", Proceedings of the
24
th
International Conference on Software
Engineering pp. 69-79.
Basili, V. R. & Rombach, H. D. 1988, "The TAME
project: towards improvement-oriented software
environments", Software Engineering, IEEE
Transactions on, vol. 14, no. 6, pp. 758-773.
Basili, Victor R., et al. 2010. "Linking software
development and business strategy through
measurement." Computer 43.4: 57-65.
CMMI Product Team, CMMI for Development Version
1.3, Software Engineering Institute, USA, 2010.
Houdek, F., Schneider, K., & Wieser, E. 1998,
"Establishing experience factories at Daimler-Benz an
experiencereport", Software Engineering, 1998. Proc.
of the 1998 (20th) Intern. Conf. on pp. 443-447.
Jäntti, Marko, Antti Lahtela, and Jukka Kaukola. 2011,
Establishing a Measurement System for IT Service
Management Processes: A Case Study. International
Journal on Advances in Systems and Measurements
3.3 and 4, pp. 125-136.
Oman, P. W. & Pfleeger, S. L. 1997, Applying software
metrics IEEE Computer Society Press Los Alamitos,
Calif.
Solingen, R. & Berghout, E. 1999, The
Goal/Question/Metric Method - A Practical Guide for
Quality Improvement of Software Development,
McGraw-Hill.
Unterkalmsteiner, Michael, et al. 2012, Evaluation and
Measurement of Software Process Improvement—A
Systematic Literature Review. Software Engineering,
IEEE Transactions on 38.2: 398-424.
ICEIS2014-16thInternationalConferenceonEnterpriseInformationSystems
136