EXPLORING FEASIBILITY OF SOFTWARE DEFECTS ORTHOGONAL CLASSIFICATION

Davide Falessi, Giovanni Cantone

2006

Abstract

Defect categorization is the basis of many works that relate to software defect detection. The assumption is that different subjects assign the same category to the same defect. Because this assumption was questioned, our following decision was to study the phenomenon, in the aim of providing empirical evidence. Because defects can be categorized by using different criteria, and the experience of the involved professionals in using such a criterion could affect the results, our further decisions were: (i) to focus on the IBM Orthogonal Defect Classification (ODC); (ii) to involve professionals after having stabilized process and materials with students. This paper is concerned with our basic experiment. We analyze a benchmark including two thousand and more data that we achieved through twenty-four segments of code, each segment seeded with one defect, and by one hundred twelve sophomores, trained for six hours, and then assigned to classify those defects in a controlled environment for three continual hours. The focus is on: Discrepancy among categorizers, and orthogonality, affinity, effectiveness, and efficiency of categorizations. Results show: (i) training is necessary to achieve orthogonal and effective classifications, and obtain agreement between subjects, (ii) efficiency is five minutes per defect classification in the average, (iii) there is affinity between some categories.

References

  1. Abdelnabi Z., G. Cantone, M. Ciolkowski, D. Rombach: “Comparing Code Reading Techniques Applied to Object-oriented Software Frameworks with regard to Effectiveness and Defect Detection Rate”, Proceedings of the 2004 International Symposium on Empirical Software Engineering, pp. Redondo Beach (CA), 2004.
  2. Basili V.R, G. Caldiera, H.D. Rombach: “Goal Question Metric Paradigm”, in Encyclopaedia of Software Engineering, J.J. Marciniak Edr., Vol. 1, pp. 528-532, John Wiley & Sons, 1994.
  3. Basili V.R., and R. Selby: “Comparing the Effectiveness of Software Testing Strategies”, IEEE Transactions on Software Engineering, CS Press, December, 1987, pp. 1278 -1296.
  4. Cantone G., Z. A. Abdulnabi, A. Lomartire, G. Calavaro: “Effectiveness of Code Reading and Functional Testing with Event-Driven Object-Oriented Software”, Empirical Methods and Studies in Software Engineering, R. Conradi and A. I. Wang Eds., LNCS 2765, pp. 166-193, Springer, 2003.
  5. Cohen J.: "A Coefficient of Agreement for Nominal Scales". In Educational and Psychological Measurement, 20:37-46, 1960.
  6. Durães J. and Madeira H., "Definition of Software Fault Emulation Operators: a Field Data Study", In Proc. of 2003 International Conference on Dependable Systems and Networks", (2003)
  7. El Emam K. and I. Wieczorek: “The Repeatability of Code Defect Classifications”, Proceedings of International Symposium on Software Reliability Engineering, pp. 322-333, 1998.
  8. Henningsson K. and C. Wohlin: “Assuring Fault Classification Agreement - An Empirical Evaluation” Proceedings of the 2004 International Symposium on Empirical Software Engineering, 2004.
  9. Juristo N. and S. Vegas: “Functional Testing, Structural Testing, and Code Reading: What Fault Type Do They Each Detect?”, Empirical Methods and Studies in Software Engineering, R. Conradi and A. I. Wang Eds., LNCS 2765, pp. 208-232, Springer, 2003.
  10. Myers G.J.: “A Controlled Experiment in Program Testing and Code Walkthroughs/Reviews”, Communications of ACM, Vol. 21 (9), pp. 760-768, 1978.
  11. Wohlin C., P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell, A. Wesslén: “Experimentation in Software Engineering: An Introduction”, The Kluwer International Series in Software Engineering, 2000.
  12. IBM a, “Details of ODC v 5.11”, www.research.ibm.com/softeng/ODC/DETODC.HTM , last access: 02/05/2006.
  13. IBM b, “ODC Frequently Asked Questions”, www.research.ibm.com/softeng/ODC/FAQ.HTM, last access: 02/05/2006.
Download


Paper Citation


in Harvard Style

Falessi D. and Cantone G. (2006). EXPLORING FEASIBILITY OF SOFTWARE DEFECTS ORTHOGONAL CLASSIFICATION . In Proceedings of the First International Conference on Software and Data Technologies - Volume 1: ICSOFT, ISBN 978-972-8865-69-6, pages 107-117. DOI: 10.5220/0001311201070117


in Bibtex Style

@conference{icsoft06,
author={Davide Falessi and Giovanni Cantone},
title={EXPLORING FEASIBILITY OF SOFTWARE DEFECTS ORTHOGONAL CLASSIFICATION},
booktitle={Proceedings of the First International Conference on Software and Data Technologies - Volume 1: ICSOFT,},
year={2006},
pages={107-117},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001311201070117},
isbn={978-972-8865-69-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the First International Conference on Software and Data Technologies - Volume 1: ICSOFT,
TI - EXPLORING FEASIBILITY OF SOFTWARE DEFECTS ORTHOGONAL CLASSIFICATION
SN - 978-972-8865-69-6
AU - Falessi D.
AU - Cantone G.
PY - 2006
SP - 107
EP - 117
DO - 10.5220/0001311201070117