ON THE PERFORMANCE OF FAULT SCREENERS IN SOFTWARE DEVELOPMENT AND DEPLOYMENT

Rui Abreu, Alberto González, Peter Zoeteweij, Arjan J. C. van Gemund

2008

Abstract

Fault screeners are simple software (or hardware) constructs that detect variable value errors based on unary invariant checking. In this paper we evaluate and compare the performance of two low-cost screeners (Bloom filter, and range screener) that can be automatically integrated within a program, while being automatically trained during the testing phase. While the Bloom filter has the capacity of retaining virtually all variable values associated with proper program execution, this property comes with a much higher false positive rate per unit training effort, compared to the more simple range screener, that compresses all value information in terms of a single lower and upper bound. We present a novel analytic model that predicts the false positive and false negative rate for both type of screeners. We show that the model agrees with our empirical findings. Furthermore, we describe the application of both screeners, where the screener output is used as input to a fault localization process that provides automatic feedback on the location of residual program defects during deployment in the field.

References

  1. Abreu, R., González, A., Zoeteweij, P., and van Gemund, A. (2008). Automatic software fault localization using generic program invariants. In Proc. SAC'08, Fortaleza, Brazil. ACM Press. accepted for publication.
  2. Abreu, R., Zoeteweij, P., and van Gemund, A. (2006). An evaluation of similarity coefficients for software fault localization. In Proccedings of PRDC'06.
  3. Abreu, R., Zoeteweij, P., and van Gemund, A. (2007). On the accuracy of spectrum-based fault localization. In Proc. TAIC PART'07.
  4. Bloom, B. (1970). Space/time trade-offs in hash coding with allowable errors. Commun. ACM, 13(7):422- 426.
  5. Cleve, H. and Zeller, A. (2005). Locating causes of program failures. In Proc. ICSE'05, Missouri, USA.
  6. Dallmeier, V., Lindig, C., and Zeller, A. (2005). Lightweight defect localization for Java. In Black, A. P., editor, Proc. ECOOP 2005, volume 3586 of LNCS, pages 528-550. Springer-Verlag.
  7. David, H. A. (1970). Order Statistics. John Wiley & Sons.
  8. Ernst, M., Cockrell, J., Griswold, W., and Notkin, D. (1999). Dynamically discovering likely program invariants to support program evolution. In Proc. ICSE'99, pages 213-224.
  9. Ernst, M., Perkins, J., Guo, P., McCamant, S., Pacheco, C., Tschantz, M., , and Xiao, C. (2007). The Daikon system for dynamic detection of likely invariants. In Science of Computer Programming.
  10. Gautama, H. and van Gemund, A. (2006). Low-cost static performance prediction of parallel stochastic task compositions. IEEE Trans. Parallel Distrib. Syst., 17(1):78-91.
  11. González, A. (2007). Automatic error detection techniques based on dynamic invariants. Master's thesis.
  12. Gumbel, E. (1962). Statistical theory of extreme values (main results). In Sarhan, A. and Greenberg, B., editors, Contributions to Order Statistics, pages 56-93. John Wiley & Sons.
  13. Hangal, S., Chandra, N., Narayanan, S., and Chakravorty, S. (2005). IODINE: A tool to automatically infer dynamic invariants for hardware designs. In DAC'05, San Diego, California, USA.
  14. Hangal, S. and Lam, M. (2002). Tracking down software bugs using automatic anomaly detection. In Proc. ICSE'02.
  15. Harrold, M., Rothermel, G., Wu, R., and Yi, L. (1998). An empirical investigation of program spectra. ACM SIGPLAN Notices, 33(7):83-90.
  16. Hutchins, M., Foster, H., Goradia, T., and Ostrand, T. (1994). Experiments of the effectiveness of dataflowand controlflow-based test adequacy criteria. In Proc. ICSE'94, Sorrento, Italy. IEEE CS.
  17. Jain, A. and Dubes, R. (1988). Algorithms for clustering data. Prentice-Hall, Inc.
  18. Jones, J. and Harrold, M. (2005). Empirical evaluation of the tarantula automatic fault-localization technique. In Proc. ASE'05, pages 273-282, NY, USA.
  19. Kephart, J. and Chess, D. (2003). The vision of autonomic computing. Computer, 36(1):41-50.
  20. Lattner, C. and Adve, V. (2004). LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proc. CGO'04, Palo Alto, California.
  21. Liu, C., Fei, L., Yan, X., Han, J., and Midkiff, S. (2006). Statistical debugging: A hypothesis testing-based approach. IEEE TSE, 32(10):831-848.
  22. Patterson, D., Brown, A., Broadwell, P., Candea, G., Chen, M., Cutler, J., Enriquez, P., Fox, A., Kiciman, E., Merzbacher, M., Oppenheimer, D., Sastry, N., Tetzlaff, W., Traupman, J., and Treuhaft, N. (2002). Recovery Oriented Computing (ROC): Motivation, definition, techniques, and case studies. Technical Report UCB/CSD-02-1175, U.C. Berkeley.
  23. Pytlik, B., Renieris, M., Krishnamurthi, S., and Reiss, S. (2003). Automated fault localization using potential invariants. In Proc. AADEBUG'03.
  24. Racunas, P., Constantinides, K., Manne, S., and Mukherjee, S. (2007). Perturbation-based fault screening. In Proc. HPCA'2007, pages 169-180.
  25. Renieris, M. and Reiss, S. (2003). Fault localization with nearest neighbor queries. In Proc. ASE'03, Montreal, Canada. IEEE CS.
  26. Yang, J. and Evans, D. (2004). Automatically inferring temporal properties for program evolution. In Proc. ISSRE'04, pages 340-351, Washington, DC, USA. IEEE CS.
  27. Zhang, X., He, H., Gupta, N., and Gupta, R. (2005). Experimental evaluation of using dynamic slices for fault location. In Proc. AADEBUG'05, pages 33-42, Monterey, California, USA. ACM Press.
Download


Paper Citation


in Harvard Style

Abreu R., González A., Zoeteweij P. and J. C. van Gemund A. (2008). ON THE PERFORMANCE OF FAULT SCREENERS IN SOFTWARE DEVELOPMENT AND DEPLOYMENT . In Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-8111-28-9, pages 123-130. DOI: 10.5220/0001764601230130


in Bibtex Style

@conference{enase08,
author={Rui Abreu and Alberto González and Peter Zoeteweij and Arjan J. C. van Gemund},
title={ON THE PERFORMANCE OF FAULT SCREENERS IN SOFTWARE DEVELOPMENT AND DEPLOYMENT},
booktitle={Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2008},
pages={123-130},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001764601230130},
isbn={978-989-8111-28-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - ON THE PERFORMANCE OF FAULT SCREENERS IN SOFTWARE DEVELOPMENT AND DEPLOYMENT
SN - 978-989-8111-28-9
AU - Abreu R.
AU - González A.
AU - Zoeteweij P.
AU - J. C. van Gemund A.
PY - 2008
SP - 123
EP - 130
DO - 10.5220/0001764601230130