Authors:
Alisa Barkar
1
;
Mathieu Chollet
2
;
3
;
Matthieu Labeau
1
;
Beatrice Biancardi
4
and
Chloe Clavel
5
Affiliations:
1
LTCI, Institut Polytechnique de Paris, Telecom-Paris, 19 Place Marguerite Perey, 91120 Palaiseau, France
;
2
School of Computing Science, University of Glasgow, G12 8RZ Glasgow, U.K.
;
3
IMT Atlantique, LS2N, UMR CNRS 6004, 44307 Nantes, France
;
4
CESI LINEACT, Nanterre, France
;
5
ALMAnaCH, INRIA, Paris, France
Keyword(s):
Public Speaking Assessment, Large Language Models (LLMs), Persuasiveness Prediction, Interpretable Features, Textual Modality, Automatic Speech Evaluation, Open-Source Models.
Abstract:
The increasing importance of public speaking (PS) skills has fueled the development of automated assessment systems, yet the integration of large language models (LLMs) in this domain remains underexplored. This study investigates the application of LLMs for assessing PS by predicting persuasiveness. We propose a novel framework where LLMs evaluate criteria derived from educational literature and feedback from PS coaches, offering new interpretable textual features. We demonstrate that persuasiveness predictions of a regression model with the new features achieve a Root Mean Squared Error (RMSE) of 0.6, underperforming approach with hand-crafted lexical features (RMSE 0.51) and outperforming direct zero-shot LLM persuasiveness predictions (RMSE of 0.8). Furthermore, we find that only LLM-evaluated criteria of language level is predictable from lexical features (F1-score of 0.56), disapproving relations between these features. Based on our findings, we criticise the abilities of LLMs
to analyze PS accurately. To ensure reproducibility and adaptability to emerging models, all source code and materials are publicly available on GitHub.
(More)