An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software

Muhammad Zafar, Wasif Afzal, Eduard Enoiu

2023

Abstract

Combinatorial Testing (CT) and Model-Based Testing (MBT) are two recognized test generation techniques. The evidence of their fault detection effectiveness and comparison with industrial state-of-the-practice is still scarce, more so at the system level for safety-critical systems, such as those found in trains. We use mutation analysis to perform a comparative evaluation of CT, MBT, and industrial manual testing in terms of their fault detection effectiveness using an industrial case study of the safety-critical train control management system. We examine the fault detection rate per mutant and relationship between the mutation scores and structural coverage using Modified Condition Decision Coverage (MC/DC). Our results show that CT 3-ways, CT 4-ways, and MBT provide higher mutation scores. MBT did not perform better in detecting ‘Logic Replacement Operator-Improved’ mutants when compared with the other techniques, while manual testing struggled to find ‘Logic Block Replacement Operator’ mutants. None of the test suites were able to find ‘Time Block Replacement Operator’ mutants. CT 2-ways was found to be the least effective test technique. MBT-generated test suite achieved the highest MC/DC coverage. We also found a generally consistent positive relationship between MC/DC coverage and mutation scores for all test suites.

Paper Citation

in Harvard Style

Zafar M., Afzal W. and Enoiu E. (2023). An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software. In Proceedings of the 18th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-758-647-7, SciTePress, pages 293-305. DOI: 10.5220/0011756800003464

in Bibtex Style

@conference{enase23,
author={Muhammad Zafar and Wasif Afzal and Eduard Enoiu},
title={An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software},
booktitle={Proceedings of the 18th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2023},
pages={293-305},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011756800003464},
isbn={978-989-758-647-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - An Empirical Evaluation of System-Level Test Effectiveness for Safety-Critical Software
SN - 978-989-758-647-7
AU - Zafar M.
AU - Afzal W.
AU - Enoiu E.
PY - 2023
SP - 293
EP - 305
DO - 10.5220/0011756800003464
PB - SciTePress