Why Do We Not Learn from Defects? - Towards Defect-Driven Software Process Improvement

Niklas Mellegård, Miroslaw Staron, Fredrik Törner

2013

Abstract

In this paper, we put forth the thesis that state-of-the-art defect classification schemes – such as ODC and IEEE Std. 1044 – have failed to meet their target; limited industrial adoption is taken as part of the evidence combined with published studies on model driven software development. Notwithstanding, a number of publications show that defect reports can provide valuable information about common, important, or dangerous problems with software products. In this paper, we present the synthesis of two industrial case studies that illustrate that even expert judgement can be deceptive; demonstrating the need for more objective evidence to allow project stakeholder to make informed decisions, and that defect classification is one effective means to that end. Finally, we propose a roadmap that will contribute to improving the defect classification approach, which in consequence will lead to a wider industrial adoption.

References

  1. Buse, R. P. L., Thomas Zimmermann, 2011. Information Needs for Software Development Analytics (Microsoft Technical report No. MSR-TR-2011-8)
  2. Butcher, M., Munro, H., Kratschmer, T., 2002. Improving software testing via ODC: Three case studies. IBM Systems Journal 41, 31 -44.
  3. Chillarege, R., Bhandari, I. S., Chaar, J. K., Halliday, M. J., Moebus, D. S., Ray, B. K., Wong, M.-Y., 1992. Orthogonal defect classification-a concept for inprocess measurements. IEEE Trans. Softw. Eng. 18.
  4. Chillarege, R., Ram Prasad, K., 2002. Test and development process retrospective - a case study using ODC triggers, Int'l Conf Dependable Syst. and Netw.
  5. Freimut, B., 2001. Developing and using defect classification schemes (IESE- Report No. 072.01/E). Fraunhofer IESE.
  6. Freimut, B., Denger, C., Ketterer, M., 2005. An industrial case study of implementing and validating defect classification for process improvement and quality management, 11th IEEE Int'l Symp. Softw. Metrics.
  7. Grady, R. B., 1992. Practical Software Metrics for Project Management and Process Improvement. Prentice Hall.
  8. Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S., 2011. A Systematic Review of Fault Prediction Performance in Software Engineering. IEEE Trans. Softw. Eng.
  9. Henningsson, K., Wohlin, C., 2004. Assuring fault classification agreement - an empirical evaluation, Int'l Symposium on Empirical Softw. Eng. (ISESE).
  10. Huang, L., Ng, V., Persing, I., Geng, R., Bai, X., Tian, J., 2011. AutoODC: Automated generation of Orthogonal Defect Classifications, 26th IEEE/ACM Int'l Conf. on Automated Softw. Eng. (ASE).
  11. IEEE, 2010. IEEE Std. 1044-2009. Standard Classification for Software Anomalies.
  12. ISO/IEC, 2007. ISO/IEC 15939 - Systems and Software Engineering - Measurement Process.
  13. Jin, A., Jiang, J., 2009. Fault Injection Scheme for Embedded Systems at Machine Code Level and Verification, 15th IEEE Pacific Rim Int'l Symp. on Dependable Comput. (PRDC).
  14. Li, J., Stalhane, T., Conradi, R., Kristiansen, J. M. W., 2012. Enhancing Defect Tracking Systems to Facilitate Software Quality Improvement. IEEE Softw. 29, 59 -66.
  15. Li, N., Li, Z., Sun, X., 2010. Classification of Software Defect Detected by Black-Box Testing: An Empirical Study, 2nd World Congress Softw. Eng. (WCSE).
  16. Li, N., Li, Z., Zhang, L., 2010. Mining Frequent Patterns from Software Defect Repositories for Black-Box Testing, 2nd Int'l Worksh. Intell. Syst. and Appl. (ISA).
  17. Mellegård, N., Staron, M., 2010. Characterizing Model Usage in Embedded Software Engineering: A Case Study, 8th Nordic Workshop on Model Driven Softw. Eng. (NW-MoDE). Copenhagen, Denmark.
  18. Mellegård, N., Staron, M., Törner, F., 2012a. A Lightweight Defect Classification Scheme for Embedded Automotive Software and its Initial Evaluation, IEEE Int'l Symp. Softw. Rel. Eng. (ISSRE), Dallas Tx USA.
  19. Mellegård, N., Staron, M., Törner, F., 2012b. A LightWeight Defect Classification Scheme for Embedded Automotive Software (Technical Report No. 2012:0x, ISSN: 1654-4870), Research Reports in Software Eng. and Management. Chalmers Univ. of Tech., Göteborg.
  20. Nagappan, N., Williams, L., Hudepohl, J., Snipes, W., Vouk, M., 2004. Preliminary results on using static analysis tools for software inspection, 15th Int'l Symp. Softw. Rel. Eng. (ISSRE).
  21. Pareto, L., Sandberg, A. B., Eriksson, P., Ehnebom, S., 2012. Collaborative prioritization of architectural concerns. Journal of Syst. and Softw. 85.
  22. Paul, R. A., Bastani, F., I-Ling Yen, Challagulla, V. U. 2000. Defect-based reliability analysis for missioncritical software, Comp. SW and Applications Conf. (COMPSAC).
  23. Paul, R. A., Bastani, F. B., Challagulla, V. U. B., Yen, I.- L., 2002. Software measurement data analysis using memory-based reasoning, 14th IEEE Int'l Conf. Tools with AI, (ICTAI).
  24. Wagner, S., 2008. Defect classification and defect types revisited, Workshop on Defects in Large Softw. Syst., DEFECTS.
  25. Wang He, Wang Hao, Lin Zhiqing, 2009. Improving Classification Efficiency of Orthogonal Defect Classification via a Bayesian Network Approach, Comp. Intell. and Softw. Eng. (CiSE).
  26. Vetro', A., Zazworka, N., Seaman, C., Shull, F., 2012. Using the ISO/IEC 9126 product quality model to classify defects: A controlled experiment, 16th Int'l Conf. Eval. Assessment in Softw. Eng. (EASE).
  27. Zheng, J., Williams, L., Nagappan, N., Snipes, W., Hudepohl, J. P., Vouk, M. A., 2006. On the value of static analysis for fault detection in software. IEEE Trans. Softw. Eng.
Download


Paper Citation


in Harvard Style

Mellegård N., Staron M. and Törner F. (2013). Why Do We Not Learn from Defects? - Towards Defect-Driven Software Process Improvement . In Proceedings of the 1st International Conference on Model-Driven Engineering and Software Development - Volume 1: MODELSWARD, ISBN 978-989-8565-42-6, pages 297-303. DOI: 10.5220/0004345602970303


in Bibtex Style

@conference{modelsward13,
author={Niklas Mellegård and Miroslaw Staron and Fredrik Törner},
title={Why Do We Not Learn from Defects? - Towards Defect-Driven Software Process Improvement},
booktitle={Proceedings of the 1st International Conference on Model-Driven Engineering and Software Development - Volume 1: MODELSWARD,},
year={2013},
pages={297-303},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004345602970303},
isbn={978-989-8565-42-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Model-Driven Engineering and Software Development - Volume 1: MODELSWARD,
TI - Why Do We Not Learn from Defects? - Towards Defect-Driven Software Process Improvement
SN - 978-989-8565-42-6
AU - Mellegård N.
AU - Staron M.
AU - Törner F.
PY - 2013
SP - 297
EP - 303
DO - 10.5220/0004345602970303