Towards LLM-Based Autograding for Short Textual Answers

Johannes Schneider, Bernd Schenk, Christina Niklaus

2024

Abstract

Grading exams is an important, labor-intensive, subjective, repetitive, and frequently challenging task. The feasibility of autograding textual responses has greatly increased thanks to the availability of large language models (LLMs) such as ChatGPT and because of the substantial influx of data brought about by digitalization. However, entrusting AI models with decision-making roles raises ethical considerations, mainly stemming from potential biases and issues related to generating false information. Thus, in this manuscript we provide an evaluation of a large language model for the purpose of autograding, while also highlighting how LLMs can support educators in validating their grading procedures. Our evaluation is targeted towards automatic short textual answers grading (ASAG), spanning various languages and examinations from two distinct courses. Our findings suggest that while “out-of-the-box” LLMs provide a valuable tool to provide a complementary perspective, their readiness for independent automated grading remains a work in progress, necessitating human oversight.

Paper Citation

in Harvard Style

Schneider J., Schenk B. and Niklaus C. (2024). Towards LLM-Based Autograding for Short Textual Answers. In Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: CSEDU; ISBN 978-989-758-697-2, SciTePress, pages 280-288. DOI: 10.5220/0012552200003693

in Bibtex Style

@conference{csedu24,
author={Johannes Schneider and Bernd Schenk and Christina Niklaus},
title={Towards LLM-Based Autograding for Short Textual Answers},
booktitle={Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: CSEDU},
year={2024},
pages={280-288},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012552200003693},
isbn={978-989-758-697-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: CSEDU
TI - Towards LLM-Based Autograding for Short Textual Answers
SN - 978-989-758-697-2
AU - Schneider J.
AU - Schenk B.
AU - Niklaus C.
PY - 2024
SP - 280
EP - 288
DO - 10.5220/0012552200003693
PB - SciTePress