Semantic Coverage: Measuring Test Suite Effectiveness

Samia Al Blwi, Amani Ayad, Besma Khaireddine, Imen Marsit, Ali Mili

2023

Abstract

Several syntactic measures have been defined in the past to assess the effectiveness of a test suite: statement coverage, condition coverage, branch coverage, path coverage, etc. There is ample analytical and empirical evidence to the effect that these are imperfect measures: exercising all of a program’s syntactic features is neither necessary nor sufficient to ensure test suite adequacy; not to mention that it may be impossible to exercise all the syntactic features of a program (re: unreachable code). Mutation scores are often used as reliable measures of test suite effectiveness, but they have issues of their own: some mutants may survive because they are equivalent to the base program not because the test suite is inadequate; the same mutation score may mean vastly different things depending on whether the killed mutants are distinct from each other or equivalent; the same test suite and the same program may yield different mutation scores depending on the mutation operators that we use. Fundamentally, whether a test suite T is adequate for a program P depends on the semantics of the program, the specification that the program is tested against, and the property of correctness that the program is tested for (total correctness, partial correctness). In this paper we present a formula for the effectiveness of a test suite T which depends exactly on the semantics of P, the correctness property that we are testing P for, and the specification against which this correctness property is tested; it does not depend on the syntax of P, nor on any mutation experiment we may run. We refer to this formula as the semantic coverage of the test suite, and we investigate its properties.

Paper Citation

in Harvard Style

Al Blwi S., Ayad A., Khaireddine B., Marsit I. and Mili A. (2023). Semantic Coverage: Measuring Test Suite Effectiveness. In Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT; ISBN 978-989-758-665-1, SciTePress, pages 287-294. DOI: 10.5220/0012063900003538

in Bibtex Style

@conference{icsoft23,
author={Samia Al Blwi and Amani Ayad and Besma Khaireddine and Imen Marsit and Ali Mili},
title={Semantic Coverage: Measuring Test Suite Effectiveness},
booktitle={Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT},
year={2023},
pages={287-294},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012063900003538},
isbn={978-989-758-665-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Conference on Software Technologies - Volume 1: ICSOFT
TI - Semantic Coverage: Measuring Test Suite Effectiveness
SN - 978-989-758-665-1
AU - Al Blwi S.
AU - Ayad A.
AU - Khaireddine B.
AU - Marsit I.
AU - Mili A.
PY - 2023
SP - 287
EP - 294
DO - 10.5220/0012063900003538
PB - SciTePress