Reviewing Reproducibility in Software Engineering Research

André F. R. Cordeiro, Edson Oliveira Jr

2025

Abstract

This paper presents a Systematic Mapping Study (SMS) on reproducibility in Software Engineering (SE), analyzing definitions, procedures, investigations, solutions, artifacts, and evaluation assessments. The research explores how reproducibility is defined, applied, and investigated, identifying several approaches and solutions. The final set of studies considered 25 primary studies, grouping the definitions of reproducibility into categories such as method, repeatability, probability, and ability. The application procedures are categorized into method, architecture, container, technique, framework, environment, notebook, toolkit, and benchmark. The investigation of reproducibility is analyzed through workflows, approaches, prototypes, methods, measures, methodologies, technologies, case studies, and frameworks. Solutions to reproducibility problems include environments, tools, benchmarks, initiatives, methodologies, notebooks, and containers. The artifacts considered include tools, environments, scenarios, datasets, models, diagrams, notebooks, algorithms, codes, representation structures, methodologies, containers, repositories, sequences, and workflows. Reproducibility assessment is performed using methods, experiments, measurements, processes, and factors. We also discuss future research opportunities. The results aim to benefit SE researchers and practitioners by providing an overview of organizing and providing reproducible research projects and artifacts, in addition to pointing out research opportunities related to reproducibility in the area.

Paper Citation

in Harvard Style

Cordeiro A. and Oliveira Jr E. (2025). Reviewing Reproducibility in Software Engineering Research. In Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS; ISBN 978-989-758-749-8, SciTePress, pages 364-371. DOI: 10.5220/0013456000003929

in Bibtex Style

@conference{iceis25,
author={André Cordeiro and Edson Oliveira Jr},
title={Reviewing Reproducibility in Software Engineering Research},
booktitle={Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS},
year={2025},
pages={364-371},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013456000003929},
isbn={978-989-758-749-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 2: ICEIS
TI - Reviewing Reproducibility in Software Engineering Research
SN - 978-989-758-749-8
AU - Cordeiro A.
AU - Oliveira Jr E.
PY - 2025
SP - 364
EP - 371
DO - 10.5220/0013456000003929
PB - SciTePress