Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education

Kai Petersen, Kai Petersen, Jefferson Molléri

2021

Abstract

Background: In order to judge evidence it is important to be able to assess study quality. Checklists are means to objectify the assessment. In an earlier study we proposed and evaluated a checklist for surveys, which was assessed by experts. Objective: (1) To assess whether the use of the checklist enables students with limited experience in research to consistently and accurately assess the quality of a research paper. (2) To elicit qualitative feedback to identify improvements to the checklist. Method: The students reviewed a survey in a one-group posttest-only quasi-experiment using the checklist. In total 13 students participated in the context of the course Evidence-based software engineering as part of the study program Information Systems at Flensburg University of Applied Sciences. Results: In total the students achieved 74% percent of agreement among each other. However, the Kappa values indicated mostly a poor level of agreement considering the classification by Fleiss. In addition, the students were quite inaccurate assessing the questions. Though, they performed well on questions for research objectives and the identification of population. Conclusion: Findings indicate that students do not assess reliably. However, further investigations are needed to substantiate the findings.

Paper Citation

in Harvard Style

Petersen K. and Molléri J. (2021). Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education. In Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-758-508-1, pages 437-444. DOI: 10.5220/0010496204370444

in Bibtex Style

@conference{enase21,
author={Kai Petersen and Jefferson Molléri},
title={Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education},
booktitle={Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2021},
pages={437-444},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010496204370444},
isbn={978-989-758-508-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education
SN - 978-989-758-508-1
AU - Petersen K.
AU - Molléri J.
PY - 2021
SP - 437
EP - 444
DO - 10.5220/0010496204370444