A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing

Arnaud Gotlieb, Mats Carlsson, Dusica Marijan, Alexandre Pétillon

2016

Abstract

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) distinct criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage. In this paper, we revisit this problem with a new approach based on constraint optimization with a special constraint called GLOBAL CARDINALITY and a sophisticated search heuristic. This constraint enforces the coverage of all features through the computation of max flows in a network flow representing the coverage relation. The computed max flows represent possible solutions which are further processed in order to determine the solution that optimizes the given objective function, e.g., the lowest test execution costs. Our approach was implemented in a tool called Flower/C and experimentally evaluated on both randomly generated instances and industrial case instances. Comparing Flower/C with MINTS (Minimizer for Test Suites), the State-Of-the-Art tool based on an integer linear formulation for performing similar test suite optimization, we show that our approach either outperforms MINTS or has comparable performance on random instances. On industrial instances, we compared three distinct models of Flower/C (using distinct global constraints) and the one mixing distinct constraints showed excellent performances with high reduction rates. These results opens door to an industrial adoption of the proposed technology.

References

  1. Agrawal, H. (1999). Efficient coverage testing using global dominator graphs. In Workshop on Program Analysis for Software Tools and Eng. (PASTE'99).
  2. Black, J., Melachrinoudis, E., and Kaeli, D. (2004). Bicriteria models for all-uses test suite reduction. In 26th Int. Conf. on Software Eng., pages 106-115.
  3. Campos, J., Riboira, A., Perez, A., and Abreu, R. (2012). Gzoltar: an eclipse plug-in for testing and debugging. In IEEE/ACM International Conference on Automated Software Engineering, ASE'12, Essen, Germany, September 3-7, 2012, pages 378-381.
  4. Chen, Z., Zhang, X., and Xu, B. (2008). A degraded ILP approach for test suite reduction. In 20th Int. Conf. on Soft. Eng. and Know. Eng.
  5. Cohen, D. M., Dalal, S. R., Fredman, M. L., and Patton, G. C. (1997). The aetg system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437 - 444.
  6. Ferrer, J., Kruse, P. M., Chicano, F., and Alba, E. (2015). Search based algorithms for test sequence generation in functional testing. Information and Software Technology, 58(0):419 - 432.
  7. Gotlieb, A., Carlsson, M., Liaeen, M., Marijan, D., and Petillon, A. (2016). Automated regression testing using constraint programming. In Proc. of Innovative Applications of Artificial Intelligence (IAAI'16), Phoenix, AZ, USA.
  8. Gotlieb, A. and Marijan, D. (2014). Flower: Optimal test suite reduction as a network maximum flow. In Proc. of Int. Symp. on Soft. Testing and Analysis (ISSTA'14), San José, CA, USA.
  9. Hao, D., Zhang, L., Wu, X., Mei, H., and Rothermel, G. (2012). On-demand test suite reduction. In Int. Conference on Software Engineerig, pages 738-748.
  10. Harrold, M. J., Gupta, R., and Soffa, M. L. (1993). A methodology for controlling the size of a test suite. ACM TOSEM, 2(3):270-285.
  11. Henard, C., Papadakis, M., Perrouin, G., Klein, J., and Traon, Y. L. (2013). Multi-objective test generation for software product lines. In 17th International Software Product Line Conference, SPLC 2013, Tokyo, Japan - August 26 - 30, 2013, pages 62-71.
  12. Hsu, H.-Y. and Orso, A. (2009). MINTS: A general framework and tool for supporting test-suite minimization. In 31st Int. Conf. on Soft. Eng. (ICSE'09), pages 419- 429.
  13. Jeffrey, D. and Gupta, N. (2005). Test suite reduction with selective redundancy. In 21st Int. Conf. on Soft. Maintenance, pages 549-558.
  14. Johansen, M. F., Haugen, O., and Fleurey, F. (2012). An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference - Volume 1, SPLC 7812, pages 46-55, New York, NY, USA. ACM.
  15. Kuhn, D. R., Wallace, D. R., and Gallo, Jr., A. M. (2004). Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng., 30(6):418-421.
  16. Li, D., Jin, Y., Sahin, C., Clause, J., and Halfond, W. G. J. (2014). Integrated energy-directed test suite optimization. In International Symposium on Software Testing and Analysis, ISSTA 7814, San Jose, CA, USA - July 21 - 26, 2014, pages 339-350.
  17. Martin Fagereng Johansen, O. H. and Fleurey, F. (2011). Properties of realistic feature models make combinatorial testing of product lines feasible. In Conference on Model Driven Engineering Languages and Systems (MODELS'11), pages 638-652.
  18. Mendonca, M., Wkasowski, A., and Czarnecki, K. (2009). SAT-based analysis of feature models is easy. In Proceedings of the 13th International Software Product Line Conference, SPLC 7809, pages 231-240, Pittsburgh, PA, USA. Carnegie Mellon University.
  19. Mouthuy, S., Deville, Y., and Dooms, G. (2007). Global constraint for the set covering problem. In Journées Francophones de Programmation par Contraintes, pages 183-192.
  20. Offutt, A. J., Pan, J., and Voas, J. M. (1995). Procedures for reducing the size of coverage-based test sets. In 12th Int. Conf. on Testing Computer Soft.
  21. Oster, S., Markert, F., and Ritter, P. (2010). Automated incremental pairwise testing of software product lines. In Software Product Lines: Going Beyond, volume 6287 of Lecture Notes in Computer Science, pages 196-210. Springer Berlin Heidelberg.
  22. Pachet, F. and Roy, P. (1999). Automatic generation of music programs. In Principles and Practice of Constraint Prog., volume 1713 of LNCS.
  23. Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., and Traon, Y. L. (2012). Pairwise testing for software product lines: comparison of two approaches. Software Quality Journal, 20(3-4):605-643.
  24. Régin, J.-C. (1996). Generalized arc consistency for global cardinality constraint. In 13th Int. Conf. on Artificial Intelligence (AAAI'96), pages 209-215.
  25. Régin, J.-C. (2011). Global constraints: a survey. In In Hybrid Optimization, pages 63-134. Springer.
  26. Rossi, F., Beek, P. v., and Walsh, T. (2006). Handbook of Constraint Programming (Foundations of Artificial Intelligence). Elsevier Science Inc., New York, NY, USA.
  27. Rothermel, G., Harrold, M. J., Ronne, J., and Hong, C. (2002). Empirical studies of test-suite reduction. Soft. Testing, Verif. and Reliability, 12:219-249.
  28. Stolberg, S. (2009). Enabling agile testing through continuous integration. In Agile Conference, 2009. AGILE'09., pages 369-374. IEEE.
  29. Tallam, S. and Gupta, N. (2005). A concept analysis inspired greedy algorithm for test suite minimization. In 6th Workshop on Program Analysis for Software Tools and Eng. (PASTE'05), pages 35-42.
  30. Uzuncaova, E., Khurshid, S., and Batory, D. (2010). Incremental test generation for software product lines. IEEE Trans. on Soft. Eng., 36(3):309-322.
  31. Wang, S., Ali, S., and Gotlieb, A. (2013). Minimizing test suites in software product lines using weight-based genetic algorithms. In Genetic and Evolutionary Computation Conference (GECCO'13), Amsterdam, The Netherlands.
  32. Wang, S., Ali, S., and Gotlieb, A. (2015). Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software, 103:370- 391.
Download


Paper Citation


in Harvard Style

Gotlieb A., Carlsson M., Marijan D. and Pétillon A. (2016). A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing . In Proceedings of the 11th International Joint Conference on Software Technologies - Volume 1: ICSOFT-EA, (ICSOFT 2016) ISBN 978-989-758-194-6, pages 48-58. DOI: 10.5220/0005983400480058


in Bibtex Style

@conference{icsoft-ea16,
author={Arnaud Gotlieb and Mats Carlsson and Dusica Marijan and Alexandre Pétillon},
title={A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing},
booktitle={Proceedings of the 11th International Joint Conference on Software Technologies - Volume 1: ICSOFT-EA, (ICSOFT 2016)},
year={2016},
pages={48-58},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005983400480058},
isbn={978-989-758-194-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 11th International Joint Conference on Software Technologies - Volume 1: ICSOFT-EA, (ICSOFT 2016)
TI - A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing
SN - 978-989-758-194-6
AU - Gotlieb A.
AU - Carlsson M.
AU - Marijan D.
AU - Pétillon A.
PY - 2016
SP - 48
EP - 58
DO - 10.5220/0005983400480058