Investigating the Quality of AI-Generated Distractors for a Multiple-Choice Vocabulary Test

Wojciech Malec

2024

Abstract

This paper reports the findings of a study into the effectiveness of distractors generated for a multiple-choice vocabulary test. The distractors were created by OpenAI’s ChatGPT (the free version 3.5) and used for the construction of a vocabulary test administered to 142 students learning English as a foreign language at the advanced level. Quantitative analysis revealed that the test had relatively low reliability, and some of its items had very ineffective distractors. When examined qualitatively, certain items were likewise found to have an ill-matched set of options. Moreover, follow-up queries failed to correct the original errors and produce more appropriate distractors. The results of this study indicate that although the use of artificial intelligence has an unquestionably positive impact on test practicality, ChatGPT-generated multiple-choice items cannot yet be used in operational settings without human moderation.

Paper Citation

in Harvard Style

Malec W. (2024). Investigating the Quality of AI-Generated Distractors for a Multiple-Choice Vocabulary Test. In Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG; ISBN 978-989-758-697-2, SciTePress, pages 836-843. DOI: 10.5220/0012762400003693

in Bibtex Style

@conference{aig24,
author={Wojciech Malec},
title={Investigating the Quality of AI-Generated Distractors for a Multiple-Choice Vocabulary Test},
booktitle={Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG},
year={2024},
pages={836-843},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012762400003693},
isbn={978-989-758-697-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG
TI - Investigating the Quality of AI-Generated Distractors for a Multiple-Choice Vocabulary Test
SN - 978-989-758-697-2
AU - Malec W.
PY - 2024
SP - 836
EP - 843
DO - 10.5220/0012762400003693
PB - SciTePress