RANDOM VS. SCENARIO-BASED VS. FAULT-BASED TESTING - An Industrial Evaluation of Formal Black-Box Testing Methods

Martin Weiglhofer, Franz Wotawa

Abstract

Given a formal model of a system under test there are different strategies for deriving test cases from such a model systematically. These strategies are based on different underlying testing objectives and concepts. Obviously, their usage has impact on the generated test cases. In this paper we evaluate random, scenario-based and fault-based test case generation strategies in the context of an industrial application and assess the advantages and disadvantages of these three strategies. The derived test cases are evaluated in terms of coverage and in terms of the detected errors on a commercial and on an open source implementation of the Voice-Over-IP Session Initiation Protocol.

References

  1. Aichernig, B. K. and Delgado, C. C. (2006). From faults via test purposes to test cases: On the fault-based testing of concurrent systems. In Proceedings of the 9th International Conference on Fundamental Approaches to Software Engineering, volume 3922 of LNCS, pages 324-338. Springer.
  2. Aichernig, B. K., Peischl, B., Weiglhofer, M., and Wotawa, F. (2007a). Protocol conformance testing a SIP registrar: An industrial application of formal methods. In Hinchey, M. and Margaria, T., editors, Proceedings of the 5th IEEE International Conference on Software Engineering and Formal Methods, pages 215- 224, London, UK. IEEE.
  3. Aichernig, B. K., Peischl, B., Weiglhofer, M., and Wotawa, F. (2007b). Test purpose generation in an industrial application. In Proceedings of the 3rd International Workshop on Advances in Model-Based Testing, pages 115-125, London, UK.
  4. Black, P. E., Okun, V., and Yesha, Y. (2000). Mutation operators for specifications. In Proceedings of the 15th IEEE International Conference on Automated Software Engineering, pages 81-88, Grenoble, France. IEEE.
  5. du Bousquet, L., Ramangalahy, S., Simon, S., Viho, C., Belinfante, A., and de Vries, R. G. (2000). Formal test automation: The conference protocol with TGV/- TORX. In Proceedings of 13th International Conference on Testing Communicating Systems: Tools and Techniques, volume 176 of IFIP Conference Proceedings, pages 221-228, Dordrecht. Kluwer Academic Publishers.
  6. Fernandez, J.-C., Jard, C., Jéron, T., and Viho, C. (1997). An experiment in automatic generation of test suites for protocols with verification technology. Science of Computer Programming, 29(1-2):123-146.
  7. Franks, J., Hallam-Baker, P., Hostetler, J., Lawrence, S., Leach, P., Luotonen, A., and Stewart, L. (1999). HTTP authentication: Basic and digest access authentication. RCF 2617, IETF.
  8. Fraser, G., Wotawa, F., and Ammann, P. (2007). Testing with model checkers: A survey. Technical Report SNA-TR-2007-P2-04, Competence Network Softnet Austria.
  9. Garavel, H., Lang, F., and Mateescu, R. (2002). An overview of CADP 2001. European Association for Software Science and Technology Newsletter, 4:13- 24.
  10. Jard, C. and Jéron, T. (2005). TGV: theory, principles and algorithms. International Journal on Software Tools for Technology Transfer, 7(4):297-315.
  11. Kahlouche, H., Viho, C., and Zendri, M. (1998). An industrial experiment in automatic generation of executable test suites for a cache coherency protocol. In 11th International Workshop on Testing Communicating Systems, IFIP Conference Proceedings, pages 211-226. Kluwer.
  12. Kahlouche, H., Viho, C., and Zendri, M. (1999). Hardware testing using a communication protocol conformance testing tool. In Proceedings of the 5th International Conference Tools and Algorithms for Construction and Analysis of Systems, volume 1579 of LNCS, pages 315-329. Springer.
  13. Kovács, G., Pap, Z., Viet, D. L., Wu-Hen-Chang, A., and Csopaki, G. (2003). Applying mutation analysis to sdl specifications. In Proceedings of the 11th International SDL Forum, LNCS, pages 269-284, Stuttgart, Germany. Springer.
  14. Laurencot, P. and Salva, S. (2005). Testing mobile and distributed systems: Method and experimentation. In Higashino, T., editor, Proceedings of the 8th International Conference on Principles of Distributed Systems, volume 3544 of LNCS, pages 37-51. Springer.
  15. Lee, D. and Yannakakis, M. (1996). Principles and methods of testing finite state machines - a survey. Proceedings of the IEEE, 84(8):1090-1123.
  16. Peischl, B., Weiglhofer, M., and Wotawa, F. (2007). Executing abstract test cases. In Model-based Testing Workshop in conjunction with the 37th Annual Congress of the Gesellschaft fuer Informatik, pages 421-426, Bremen, Germany. GI.
  17. Philipps, J., Pretschner, A., Slotosch, O., Aiglstorfer, E., Kriebel, S., and Scholl, K. (2003). Model-based test case generation for smart cards. Electronic Notes in Theoretical Computer Science, 80:1-15.
  18. Pretschner, A., Prenninger, W., Wagner, S., Kühnel, C., Baumgartner, M., Sostawa, B., Zölch, R., and Stauner, T. (2005). One evaluation of model-based testing and its automation. In Proceedings of the 27th International Conference on Software Engineering, pages 392 - 401, St. Louis, Missouri, USA. ACM.
  19. Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and Schooler, E. (2002). SIP: Session initiation protocol. RFC 3261, IETF.
  20. Srivatanakul, T., Clark, J. A., Stepney, S., and Polack, F. (2003). Challenging formal specifications by mutation: a csp security example. In Proceedings of the 10th Asia-Pacific Software Engineering Conference, pages 340-350. IEEE.
  21. Tretmans, J. (1996). Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools, 17(3):103-120.
  22. Tretmans, J. and Brinksma, E. (2003). TorX: Automated model based testing. In Hartman, A. and DussaZieger, K., editors, Proceedings of the 1st European Conference on Model-Driven Software Engineering, pages 13-25, Nurnburg, Germany.
  23. Weiglhofer, M. (2006). A LOTOS formalization of SIP. Technical Report SNA-TR-2006-1P1, Competence Network Softnet Austria, Graz, Austria.
Download


Paper Citation


in Harvard Style

Weiglhofer M. and Wotawa F. (2008). RANDOM VS. SCENARIO-BASED VS. FAULT-BASED TESTING - An Industrial Evaluation of Formal Black-Box Testing Methods . In Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-8111-28-9, pages 115-122. DOI: 10.5220/0001764501150122


in Bibtex Style

@conference{enase08,
author={Martin Weiglhofer and Franz Wotawa},
title={RANDOM VS. SCENARIO-BASED VS. FAULT-BASED TESTING - An Industrial Evaluation of Formal Black-Box Testing Methods},
booktitle={Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2008},
pages={115-122},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001764501150122},
isbn={978-989-8111-28-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the Third International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - RANDOM VS. SCENARIO-BASED VS. FAULT-BASED TESTING - An Industrial Evaluation of Formal Black-Box Testing Methods
SN - 978-989-8111-28-9
AU - Weiglhofer M.
AU - Wotawa F.
PY - 2008
SP - 115
EP - 122
DO - 10.5220/0001764501150122