Formal Test-Driven Development with Verified Test Cases

Bernhard K. Aichernig, Florian Lorber, Stefan Tiran

2014

Abstract

In this paper we propose the combination of several techniques into an agile formal development process: model-based testing, formal models, refinement of models, model checking, and test-driven development. The motivation is a smooth integration of formal techniques into an existing development cycle. Formal models are used to generate abstract test cases. These abstract tests are verified against requirement properties by means of model checking. The motivation for verifying the tests and not the model is two-fold: (1) in a typical safety-certification process the test cases are essential, not the models, (2) many common modelling tools do not provide a model checker. We refine the models, check refinement, and generate additional test cases capturing the newly added details. The final refinement step from a model to code is done with classical test-driven development. Hence, a developer implements one generated and formally verified test case after another, until all tests pass. The process is scalable to actual needs. Emphasis can be shifted between formal refinement of models and test-driven development. A car alarm system serves as a demonstrating case-study. We use Back’s Action Systems as modelling language and mutation analysis for test case generation. We define refinement as input-output conformance (ioco). Model checking is done with the CADP toolbox.

References

  1. Abrial, J.-R. (2010). Modelling in Event-B: System and software design. Cambridge University Press.
  2. Aichernig, B. K., Brandl, H., Jöbstl, E., and Krenn, W. (2010). Model-based mutation testing of hybrid systems. In de Boer, F. S., Bonsangue, M. M., Hallerstede, S., and Leuschel, M., editors, Formal Methods for Components and Objects - 8th International Symposium, FMCO 2009, Eindhoven, The Netherlands, November 4-6, 2009. Revised Selected Papers, volume 6286 of Lecture Notes in Computer Science, pages 228-249. Springer-Verlag.
  3. Aichernig, B. K., Brandl, H., Jöbstl, E., and Krenn, W. (2011). Efficient mutation killers in action. In IEEE Fourth International Conference on Software Testing, Verification and Validation, ICST 2011, Berlin, Germany, March 21-25 , 2011, pages 120-129. IEEE Computer Society.
  4. Back, R.-J. and Kurki-Suonio, R. (1983). Decentralization of process nets with centralized control. In 2nd ACM SIGACT-SIGOPS Symp. on Principles of Distributed Computing, pages 131-142. ACM.
  5. Baumeister, H. (2004). Combining formal specifications with test driven development. In Extreme Programming and Agile Methods - XP/Agile Universe 2004, 4th Conference on Extreme Programming and Agile Methods, Calgary, Canada, August 15-18, 2004, Proceedings, pages 1-12.
  6. Baumeister, H., Knapp, A., and Wirsing, M. (2004). Property-driven development. In 2nd International Conference on Software Engineering and Formal Methods (SEFM 2004), 28-30 September 2004, Beijing, China, pages 96-102.
  7. Beck, K. (2003). Test Driven Development: By Example. The Addison-Wesley Signature Series. AddisonWesley.
  8. Bonsangue, M. M., Kok, J. N., and Sere, K. (1998). An approach to object-orientation in action systems. In Mathematics of Program Construction, LNCS 1422, pages 68-95. Springer.
  9. Brandl, H., Weiglhofer, M., and Aichernig, B. K. (2010). Automated conformance verification of hybrid systems. In 10th Int. Conf. on Quality Software (QSIC 2010), pages 3-12. IEEE Computer Society.
  10. DeMillo, R., Lipton, R., and Sayward, F. (1978). Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4):34-41.
  11. Hamlet, R. G. (1977). Testing programs with the aid of a compiler. IEEE Transactions on Software Engineering, 3(4):279-290.
  12. Hierons, R. M. (2006). Applying adaptive test cases to nondeterministic implementations. Inf. Process. Lett., 98(2):56-60.
  13. Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649-678.
  14. Krenn, W., Schlick, R., and Aichernig, B. K. (2009). Mapping UML to labeled transition systems for test-case generation - a translation via object-oriented action systems. In Formal Methods for Components and Objects (FMCO), pages 186-207.
  15. Mateescu, R. and Sighireanu, M. (2003). Efficient onthe-fly model-checking for regular alternation-free mu-calculus. Science of Computer Programming, 46(3):255 - 281. Special issue on Formal Methods for Industrial Critical Systems.
  16. Maximilien, E. and Williams, L. (2003). Assessing testdriven development at IBM. In Software Engineering, 2003. Proceedings. 25th International Conference on, pages 564 - 569.
  17. Niese, O., Steffen, B., Margaria, T., Hagerer, A., Brune, G., and Ide, H.-D. (2001). Library-based design and consistency checking of system-level industrial test cases. In Fundamental Approaches to Software Engineering, 4th International Conference, FASE 2001, Genova, Italy, April 2-6, 2001, volume 2029 of Lecture Notes in Computer Science, pages 233-248. SpringerVerlag.
  18. Sanchez, J. C., Williams, L., and Maximilien, E. M. (2007). On the sustained use of a test-driven development practice at IBM. In Proceedings of the AGILE 2007, pages 5-14, Washington, DC, USA. IEEE Computer Society.
  19. Shahbaz, M. and Groz, R. (2009). Inferring mealy machines. In Proceedings of the 2nd World Congress on Formal Methods, FM'09, LNCS, pages 207-222. Springer-Verlag.
  20. Tretmans, J. (1996). Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools, 17(3):103-120.
  21. Utting, M. and Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers.
  22. Utting, M., Pretschner, A., and Legeard, B. (2011). A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability.
Download


Paper Citation


in Harvard Style

Aichernig B., Lorber F. and Tiran S. (2014). Formal Test-Driven Development with Verified Test Cases . In Proceedings of the 2nd International Conference on Model-Driven Engineering and Software Development - Volume 1: MBAT, (MODELSWARD 2014) ISBN 978-989-758-007-9, pages 626-635. DOI: 10.5220/0004874406260635


in Bibtex Style

@conference{mbat14,
author={Bernhard K. Aichernig and Florian Lorber and Stefan Tiran},
title={Formal Test-Driven Development with Verified Test Cases},
booktitle={Proceedings of the 2nd International Conference on Model-Driven Engineering and Software Development - Volume 1: MBAT, (MODELSWARD 2014)},
year={2014},
pages={626-635},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004874406260635},
isbn={978-989-758-007-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 2nd International Conference on Model-Driven Engineering and Software Development - Volume 1: MBAT, (MODELSWARD 2014)
TI - Formal Test-Driven Development with Verified Test Cases
SN - 978-989-758-007-9
AU - Aichernig B.
AU - Lorber F.
AU - Tiran S.
PY - 2014
SP - 626
EP - 635
DO - 10.5220/0004874406260635