Reproducibility Practices of Software Engineering Controlled Experiments: Survey and Prospective Actions

André F. R. Cordeiro, Edson Oliveira Jr

2025

Abstract

Reproducibility can be described as a characteristic that contributes to expanding knowledge in science. This paper investigates the reproducibility of experiments in Software Engineering (SE) in a context where the literature points to challenges in verifying experimental results. The central problem addressed is the difficulty in reproducing experiments in SE due to the different factors, such as sharing and artifact management. We then aimed to identify the factors necessary to achieve reproducibility in SE experiments, characterizing these factors in terms of the reproducibility crisis, experimental workflows, research practices, FAIR principles application, and reproducibility improvements. We planned and conducted one survey with 16 participants who answered a questionnaire with 33 questions. The results show that most participants perceive a reproducibility crisis in the field and point to factors such as lack of public data and incomplete information on methods and experimental setups as the main causes. Furthermore, the results highlight the importance of sharing data, metadata, and information about research teams. We also provide points to possible actions to improve reproducibility in SE experiments. The contributions include a detailed analysis of the challenges to reproducibility in SE, as well as the identification of practices and measures that can improve reproducibility.

Paper Citation

in Harvard Style

Cordeiro A. and Oliveira Jr E. (2025). Reproducibility Practices of Software Engineering Controlled Experiments: Survey and Prospective Actions. In Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS; ISBN 978-989-758-749-8, SciTePress, pages 372-379. DOI: 10.5220/0013475600003929

in Bibtex Style

@conference{iceis25,
author={André Cordeiro and Edson Oliveira Jr},
title={Reproducibility Practices of Software Engineering Controlled Experiments: Survey and Prospective Actions},
booktitle={Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS},
year={2025},
pages={372-379},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013475600003929},
isbn={978-989-758-749-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS
TI - Reproducibility Practices of Software Engineering Controlled Experiments: Survey and Prospective Actions
SN - 978-989-758-749-8
AU - Cordeiro A.
AU - Oliveira Jr E.
PY - 2025
SP - 372
EP - 379
DO - 10.5220/0013475600003929
PB - SciTePress