Mutation Selection: Some Could be Better than All

Zhiyi Zhang, Dongjiang You, Zhenyu Chen, Yuming Zhou, Baowen Xu

2011

Abstract

In previous research, many mutation selection techniques have been proposed to reduce the cost of mutation analysis. After a mutant subset is selected, researchers could obtain a test suite which can detect all mutants in the mutant subset. Then they run all mutants over this test suite, and the detection ratio to all mutants is used to evaluate the effectiveness of mutation selection techniques. The higher the ratio is, the better this selection technique is. Obviously, this measurement has a presumption that the set of all mutants is the best to evaluate test cases. However, there is no clearly evidence to support this presumption. So we conducted an experiment to answer the question whether the set of all mutants is the best to evaluate test cases. In this paper, our experiment results show that a subset of mutants may be more similar to faults than all the mutants. Two evaluation metrics were used to measure the similarity – rank and distance. This finding reveals that it may be more appropriate to use a subset rather than all the mutants at hand to evaluate the fault detection capability of test cases.

References

  1. Andrews, J. H., Briand, L. C., and Labiche, Y. (2005). Is mutation an appropriate tool for testing experiments? ICSE 2005: 402-411.
  2. Delamaro, M. E. and Maldonado, J. C. (1996) Proteum - A Tool for the Assessment of Test Adequacy for C Programs - User's guide.
  3. DeMillo, R. A., Lipton, R. J., and Sayward, F. G. (1978). Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11(4): 34-41 (1978).
  4. Do, H., Elbaum, S. G., Rothermel, G. (2005). Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Software Engineering 10(4): 405-435 (2005).
  5. Elbaum, S. G., Malishevsky, A. G., and Rothermel, G., (2000). Prioritizing test cases for regression testing. ISSTA 2000: 102-112.
  6. Hamlet, R. G., (1977). Testing Programs with the Aid of a Compiler. IEEE Transactions on Software Engineering 3(4): 279-290 (1977).
  7. Hussain, S., (2008). Mutation Clustering. Master's Thesis, King's College London, Strand, London, 2008.
  8. Mathur, A. P., (1991). Performance, Effectiveness, and Reliability Issues in Software Testing. COMPSAC 1991: 604-605.
  9. Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., and Zapf, C., (1996). An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering and Methodology 5(2): 99-118 (1996).
  10. Offutt, A. J. and Pan, J. (1977). Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability 7(3): 165-192 (1997).
  11. Rothermel, G., Harrold, M. J., Ostrin, J., and Hong, C., (1998). An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites. ICSM 1998: 34- 43.
  12. Wong, W. E. and Mathur, A. P., (1995). Reducing the cost of mutation testing: An empirical study. Journal of Systems and Software 31(3): 185-196 (1995).
Download


Paper Citation


in Harvard Style

Zhang Z., You D., Chen Z., Zhou Y. and Xu B. (2011). Mutation Selection: Some Could be Better than All . In Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011) ISBN 978-989-8425-58-4, pages 10-17


in Bibtex Style

@conference{east11,
author={Zhiyi Zhang and Dongjiang You and Zhenyu Chen and Yuming Zhou and Baowen Xu},
title={Mutation Selection: Some Could be Better than All},
booktitle={Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011)},
year={2011},
pages={10-17},
publisher={SciTePress},
organization={INSTICC},
doi={},
isbn={978-989-8425-58-4},
}


in EndNote Style

TY - CONF
JO - Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011)
TI - Mutation Selection: Some Could be Better than All
SN - 978-989-8425-58-4
AU - Zhang Z.
AU - You D.
AU - Chen Z.
AU - Zhou Y.
AU - Xu B.
PY - 2011
SP - 10
EP - 17
DO -