Evaluating the Potential of LLMs for Better Short Answer Scoring

Aleksandar Todorov, Elisa Klunder, Julia Eva Belloni

2025

Abstract

Automated Short Answer Grading (ASAG) has emerged as a promising tool for the challenge of assessing open student responses in an efficient and scalable manner as manual grading of such open short answers is labor-intensive and time-consuming. In this study, we present several ways of refining LLMs to fit the task of grading student short-answer responses robustly, fairly, and consistently, including a task-specific approach and a combined variant, being able to assess different tasks within the same model. In this regard, we explore two key questions: (1) Are transformer-based models suitable for short-answer grading? (2) Can a single transformer-based model effectively generalize across diverse tasks? The experimental results showed the significant potential of fine-tuned LLMs in ASAG. We further compared different fine-tuning strategies and the experimental results showed that full-fine-tuned models outperformed other fine-tuning approaches.

Paper Citation

in Harvard Style

Todorov A., Klunder E. and Belloni J. (2025). Evaluating the Potential of LLMs for Better Short Answer Scoring. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 108-119. DOI: 10.5220/0013291700003932

in Bibtex Style

@conference{csedu25,
author={Aleksandar Todorov and Elisa Klunder and Julia Belloni},
title={Evaluating the Potential of LLMs for Better Short Answer Scoring},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={108-119},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013291700003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - Evaluating the Potential of LLMs for Better Short Answer Scoring
SN - 978-989-758-746-7
AU - Todorov A.
AU - Klunder E.
AU - Belloni J.
PY - 2025
SP - 108
EP - 119
DO - 10.5220/0013291700003932
PB - SciTePress