Feasibility of NLP Metrics in Automatic Paraphrase Evaluation for EFL Learners

Minkyung Kim

2025

Abstract

This study investigates the feasibility of evaluation metrics for English as Foreign Language (EFL) learners’ paraphrases. Paraphrasing can effectively measure learners’ writing skills, yet much attention has not been oriented to automated systems in this area. While considerable efforts have been made to reduce burdens for teachers by developing automatic essay scoring system, there is little research on bridging the automatic assessment and paraphrasing in terms of language testing. Thus, this study explores three evaluation metrics in natural language processing (NLP) – dependency distance, cosine similarity, and Jaccard distance – mainly designed for machine translation to assess syntactic and word change as well as semantic congruence. A total of 1,000 paraphrases from Korean EFL undergraduate and graduate students were evaluated via target metrics with the results compared to human rating. Pearson correlation coefficient turned out to be moderate and high in semantic equivalency and lexical diversity, but in syntactic change, there were few significant correlations. Finding appropriate alternative metrics for syntactic complexity and developing automatic evaluation could be crucial steps for future research.

Paper Citation

in Harvard Style

Kim M. (2025). Feasibility of NLP Metrics in Automatic Paraphrase Evaluation for EFL Learners. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU; ISBN 978-989-758-746-7, SciTePress, pages 760-767. DOI: 10.5220/0013297400003932

in Bibtex Style

@conference{csedu25,
author={Minkyung Kim},
title={Feasibility of NLP Metrics in Automatic Paraphrase Evaluation for EFL Learners},
booktitle={Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU},
year={2025},
pages={760-767},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013297400003932},
isbn={978-989-758-746-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU
TI - Feasibility of NLP Metrics in Automatic Paraphrase Evaluation for EFL Learners
SN - 978-989-758-746-7
AU - Kim M.
PY - 2025
SP - 760
EP - 767
DO - 10.5220/0013297400003932
PB - SciTePress