Comparing Testing and Runtime Verification of IoT Systems: A Preliminary Evaluation based on a Case Study

Maurizio Leotta, Diego Clerissi, Luca Franceschini, Dario Olianas, Davide Ancona, Filippo Ricca, Marina Ribaudo

2019

Abstract

Assuring the quality of Internet of Things (IoT) systems is of paramount importance, and guaranteeing their reliability and compliance with the requirements is mandatory, but few attempts have been made so far. In previous works, we proposed two approaches for acceptance testing and runtime verification of IoT systems. Both works rely on a UML state machine to specify the system expected behaviour. In the acceptance testing approach, the interesting paths to exercise are identified and translated into executable test scripts. In the runtime verification approach, the relevant events during the system execution are monitored and compared against a formal specification derived from the UML state machine. In this paper, we compare the effectiveness of our two approaches, by applying them to a mobile health IoT system for the management of diabetic patients, employing over 100 mutated versions of the original system and analysing more than 1000 different executions. Results show that both approaches are effective in different ways in detecting bugs. While the acceptance testing approach is more effective to detect the bugs affecting the user interface, the runtime verification approach tracks better the subtle deviations from the system expected behaviour, in particular those concerning network issues.

Paper Citation

in Harvard Style

Leotta M., Clerissi D., Franceschini L., Olianas D., Ancona D., Ricca F. and Ribaudo M. (2019). Comparing Testing and Runtime Verification of IoT Systems: A Preliminary Evaluation based on a Case Study.In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-758-375-9, pages 434-441. DOI: 10.5220/0007745604340441

in Bibtex Style

@conference{enase19,
author={Maurizio Leotta and Diego Clerissi and Luca Franceschini and Dario Olianas and Davide Ancona and Filippo Ricca and Marina Ribaudo},
title={Comparing Testing and Runtime Verification of IoT Systems: A Preliminary Evaluation based on a Case Study},
booktitle={Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2019},
pages={434-441},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007745604340441},
isbn={978-989-758-375-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - Comparing Testing and Runtime Verification of IoT Systems: A Preliminary Evaluation based on a Case Study
SN - 978-989-758-375-9
AU - Leotta M.
AU - Clerissi D.
AU - Franceschini L.
AU - Olianas D.
AU - Ancona D.
AU - Ricca F.
AU - Ribaudo M.
PY - 2019
SP - 434
EP - 441
DO - 10.5220/0007745604340441