Toward Consistency in Writing Proficiency Assessment: Mitigating Classification Variability in Developmental Education

Miguel Da Corte, Miguel Da Corte, Jorge Baptista, Jorge Baptista

2025

Abstract

This study investigates the adequacy of Machine Learning (ML)-based systems, specifically ACCUPLACER, compared to human rater classifications within U.S. Developmental Education. A corpus of 100 essays was assessed by human raters using 6 linguistic descriptors, with each essay receiving a skill-level classification. These classifications were compared to those automatically generated by ACCUPLACER. Disagreements among raters were analyzed and resolved, producing a gold standard used as a benchmark for modeling ACCUPLACER’S classification task. A comparison of skill levels assigned by ACCUPLACER and humans revealed a “weak” Pearson correlation (ρ = 0.22), indicating a significant misplacement rate and raising important pedagogical and institutional concerns. Several ML algorithms were tested to replicate ACCUPLACER’S classification approach. Using the Chi-square (χ2) method to rank the most predictive linguistic descriptors, Na¨ıve Bayes achieved 81.1% accuracy with the top-four ranked features. These findings emphasize the importance of refining descriptors and incorporating human input into the training of automated ML systems. Additionally, the gold standard developed for the 6 linguistic descriptors and overall skill levels can be used to (i) assess and classify students’ English (L1) writing proficiency more holistically and equitably; (ii) support future ML modeling tasks; and (iii) enhance both student outcomes and higher education efficiency.

Paper Citation

in Harvard Style

Da Corte M. and Baptista J. (2025). Toward Consistency in Writing Proficiency Assessment: Mitigating Classification Variability in Developmental Education. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 139-150. DOI: 10.5220/0013353900003932

in Bibtex Style

@conference{csedu25,
author={Miguel Da Corte and Jorge Baptista},
title={Toward Consistency in Writing Proficiency Assessment: Mitigating Classification Variability in Developmental Education},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={139-150},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013353900003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - Toward Consistency in Writing Proficiency Assessment: Mitigating Classification Variability in Developmental Education
SN - 978-989-758-746-7
AU - Da Corte M.
AU - Baptista J.
PY - 2025
SP - 139
EP - 150
DO - 10.5220/0013353900003932
PB - SciTePress