Automatic Question Generation for the Japanese National Nursing Examination Using Large Language Models

Yûsei Kido, Hiroaki Yamada, Takenobu Tokunaga, Rika Kimura, Yuriko Miura, Yumi Sakyo, Naoko Hayashi

2024

Abstract

This paper introduces our ongoing research project that aims to generate multiple-choice questions for the Japanese National Nursing Examination using large language models (LLMs). We report the progress and prospects of our project. A preliminary experiment assessing the LLMs’ potential for question generation in the nursing domain led us to focus on distractor generation, which is a difficult part of the entire questiongeneration process. Therefore, our problem is generating distractors given a question stem and key (correct choice). We prepare a question dataset from the past National Nursing Examination for the training and evaluation of LLMs. The generated distractors are evaluated with compared to the reference distractors in the test set. We propose reference-based evaluation metrics for distractor generation by extending recall and precision, which is popular in information retrieval. However, as the reference is not the only acceptable answer, we also conduct human evaluation. We evaluate four LLMs: GPT-4 with few-shot learning, ChatGPT with few-shot learning, ChatGPT with fine-tuning and JSLM with fine-tuning. Our future plan includes improving the LLMs’ performance by integrating question writing guidelines into the prompts to LLMs and conducting a large-scale administration of automatically generated questions.

Paper Citation

in Harvard Style

Kido Y., Yamada H., Tokunaga T., Kimura R., Miura Y., Sakyo Y. and Hayashi N. (2024). Automatic Question Generation for the Japanese National Nursing Examination Using Large Language Models. In Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG; ISBN 978-989-758-697-2, SciTePress, pages 821-829. DOI: 10.5220/0012729200003693

in Bibtex Style

@conference{aig24,
author={Yûsei Kido and Hiroaki Yamada and Takenobu Tokunaga and Rika Kimura and Yuriko Miura and Yumi Sakyo and Naoko Hayashi},
title={Automatic Question Generation for the Japanese National Nursing Examination Using Large Language Models},
booktitle={Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG},
year={2024},
pages={821-829},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012729200003693},
isbn={978-989-758-697-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG
TI - Automatic Question Generation for the Japanese National Nursing Examination Using Large Language Models
SN - 978-989-758-697-2
AU - Kido Y.
AU - Yamada H.
AU - Tokunaga T.
AU - Kimura R.
AU - Miura Y.
AU - Sakyo Y.
AU - Hayashi N.
PY - 2024
SP - 821
EP - 829
DO - 10.5220/0012729200003693
PB - SciTePress