Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination

Yûsei Kido; Hiroaki Yamada; Takenobu Tokunaga; Rika Kimura; Yuriko Miura; Yumi Sakyo; Naoko Hayashi

Research.Publish.Connect.

*Please fill out at least one Field. *Value must be an number!

Title:
ISBN:
Year:
Acronym:
Subject:

Advanced Search Proceedings Search

If you're looking for an exact phrase use quotation marks on text fields.

*Please fill out at least one Field.

Title:
Author:
Affiliation:
Subject:

Advanced Search Papers Search

If you're looking for an exact phrase use quotation marks on text fields.

*Please fill out at least one Field.

Name:
Affiliation:
Country:
Conference:
Subject:

Advanced Search Authors Search

If you're looking for an exact phrase use quotation marks on text fields.

*Please fill out at least one Field.

Name:
Country:
Subject:

Advanced Search Affiliations Search

If you're looking for an exact phrase use quotation marks on text fields.

Proceedings

Proceedings Search *Please fill out at least one Field. *Value must be an number!

Title:
ISBN:
Year:
Acronym:
Subject:

Advanced Search Proceedings Search

If you're looking for an exact phrase use quotation marks on text fields.

Papers

Papers Search *Please fill out at least one Field.

Title:
Author:
Affiliation:
Subject:

Advanced Search Papers Search

If you're looking for an exact phrase use quotation marks on text fields.

Authors

Authors Search *Please fill out at least one Field.

Name:
Affiliation:
Country:
Conference:
Subject:

Advanced Search Authors Search

If you're looking for an exact phrase use quotation marks on text fields.

Advanced Search

Paper

Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination

Topics: AI Support for Item Creation

In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG, 754-764, 2025 , Porto, Portugal

Authors: Yûsei Kido ¹ ; Hiroaki Yamada ¹ ; Takenobu Tokunaga ¹ ; Rika Kimura ² ; Yuriko Miura ² ; Yumi Sakyo ² and Naoko Hayashi ²

Affiliations: ¹ School of Computing, Institute of Science Tokyo, Japan ; ² Graduate School of Nursing Science, St. Luke’s International University, Japan

Keyword(s): Large Language Models, Japanese National Nursing Examination, Distractor Generation, Multiple-Choice Questions, Automatic Question Generation.

Abstract: This paper reports the evaluation results in the usefulness of distractors generated by large language models (LLMs) in creating multiple-choice questions for the Japanese National Nursing Examination. Our research questions are: “(RQ1) Do question writers adopt LLM-generated distractor candidates in question writing?” and “(RQ2) Does providing LLM-generated distractor candidates reduce the time for writing questions?”. We selected ten questions from the proprietary mockup examinations of the National Nursing Examination administered by a prep school, considering the analysis of the last ten-year questions of the National Nursing Examination. Distractors are generated by seven different LLMs, given a stem and a key for each question of the above ten, and they are compiled into the distractor candidate sets. Given a stem and a key for each question, 15 domain experts completed questions by filling in three distractors. Eight experts are provided with the LLM-generated distractor candi dates; the other seven are not. The results of comparing the two groups provided us with affirmative answers to both RQs. The current evaluation remains subjective from the viewpoint of the question writers; it is necessary to evaluate whether questions generated with the assistance of LLM work in a real examination setting. Our future plan includes administering a large-scale mockup examination using both human-made and LLM-assisted questions and analysing the differences in the responses to both types of questions. (More)

CC BY-NC-ND 4.0

Guest: Register as new SciTePress user now for free.

SciTePress user: please login.

My Papers

You are not signed in, therefore limits apply to your IP address 18.224.66.220

In the current month:

Recent papers: 100 available of 100 total

2⁺ years older papers: 200 available of 200 total

Paper citation in several formats:

Kido, Y., Yamada, H., Tokunaga, T., Kimura, R., Miura, Y., Sakyo, Y. and Hayashi, N. (2025). Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG; ISBN 978-989-758-746-7; ISSN 2184-5026, SciTePress, pages 754-764. DOI: 10.5220/0013460300003932

@conference{aig25,
author={Yûsei Kido and Hiroaki Yamada and Takenobu Tokunaga and Rika Kimura and Yuriko Miura and Yumi Sakyo and Naoko Hayashi},
title={Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG},
year={2025},
pages={754-764},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013460300003932},
isbn={978-989-758-746-7},
issn={2184-5026},
}

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: AIG
TI - Evaluation of LLM-Generated Distractors of Multiple-Choice Questions for the Japanese National Nursing Examination
SN - 978-989-758-746-7
IS - 2184-5026
AU - Kido, Y.
AU - Yamada, H.
AU - Tokunaga, T.
AU - Kimura, R.
AU - Miura, Y.
AU - Sakyo, Y.
AU - Hayashi, N.
PY - 2025
SP - 754
EP - 764
DO - 10.5220/0013460300003932
PB - SciTePress