Toward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants

Nacir Bouali, Marcus Gerhold, Tosif Ul Rehman, Faizan Ahmed

2025

Abstract

This paper investigates the feasibility of using Large Language Models (LLMs) to automate the grading of Unified Modeling Language (UML) class diagrams in a software design course. Our method involves carefully designing case studies with constraints that guide students’ design choices, converting visual diagrams to textual descriptions, and leveraging LLMs’ natural language processing capabilities to evaluate submissions. We evaluated our approach using 92 student submissions, comparing grades assigned by three teaching assistants with those generated by three LLMs (Llama, GPT o1-mini, and Claude). Our results show that GPT o1-mini and Claude Sonnet achieved strong alignment with human graders, reaching correlation coefficients above 0.76 and Mean Absolute Errors below 4 points on a 40-point scale. The findings suggest that LLM-based grading can provide consistent, scalable assessment of UML diagrams while matching the grading quality of human assessors. This approach offers a promising solution for managing growing student numbers while ensuring fair and timely feedback.

Paper Citation

in Harvard Style

Bouali N., Gerhold M., Rehman T. and Ahmed F. (2025). Toward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 158-169. DOI: 10.5220/0013481900003932

in Bibtex Style

@conference{csedu25,
author={Nacir Bouali and Marcus Gerhold and Tosif Rehman and Faizan Ahmed},
title={Toward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU},
year={2025},
pages={158-169},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013481900003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 1: CSEDU
TI - Toward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants
SN - 978-989-758-746-7
AU - Bouali N.
AU - Gerhold M.
AU - Rehman T.
AU - Ahmed F.
PY - 2025
SP - 158
EP - 169
DO - 10.5220/0013481900003932
PB - SciTePress