Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination

Yûsei Kido, Hiroaki Yamada, Takenobu Tokunaga, Rika Kimura, Yuriko Miura, Yumi Sakyo, Naoko Hayashi

2025

Abstract

This paper reports the evaluation results in the usefulness of distractors generated by large language models (LLMs) in creating multiple-choice questions for the Japanese National Nursing Examination. Our research questions are: “(RQ1) Do question writers adopt LLM-generated distractor candidates in question writing?” and “(RQ2) Does providing LLM-generated distractor candidates reduce the time for writing questions?”. We selected ten questions from the proprietary mockup examinations of the National Nursing Examination administered by a prep school, considering the analysis of the last ten-year questions of the National Nursing Examination. Distractors are generated by seven different LLMs, given a stem and a key for each question of the above ten, and they are compiled into the distractor candidate sets. Given a stem and a key for each question, 15 domain experts completed questions by filling in three distractors. Eight experts are provided with the LLM-generated distractor candidates; the other seven are not. The results of comparing the two groups provided us with affirmative answers to both RQs. The current evaluation remains subjective from the viewpoint of the question writers; it is necessary to evaluate whether questions generated with the assistance of LLM work in a real examination setting. Our future plan includes administering a large-scale mockup examination using both human-made and LLM-assisted questions and analysing the differences in the responses to both types of questions.

Paper Citation

in Harvard Style

Kido Y., Yamada H., Tokunaga T., Kimura R., Miura Y., Sakyo Y. and Hayashi N. (2025). Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG; ISBN 978-989-758-746-7, SciTePress, pages 754-764. DOI: 10.5220/0013460300003932

in Bibtex Style

@conference{aig25,
author={Yûsei Kido and Hiroaki Yamada and Takenobu Tokunaga and Rika Kimura and Yuriko Miura and Yumi Sakyo and Naoko Hayashi},
title={Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG},
year={2025},
pages={754-764},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013460300003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG
TI - Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination
SN - 978-989-758-746-7
AU - Kido Y.
AU - Yamada H.
AU - Tokunaga T.
AU - Kimura R.
AU - Miura Y.
AU - Sakyo Y.
AU - Hayashi N.
PY - 2025
SP - 754
EP - 764
DO - 10.5220/0013460300003932
PB - SciTePress