Evaluating Text Summarization Generated by Popular AI Tools

Zhuoran Lin, Sifan Chen, Ning Wang, Hongjun Li

2023

Abstract

Automatic summarization is a crucial component of Natural Language Processing and has long been a prominent area of research. This study focuses on evaluating the performance of several well-known AI tools, namely ChatGPT, Claude, Bart, Pegasus, and T5-Base_GNAD, in the field of text summarization. To conduct the evaluation, we assembled a corpus comprising fifty abstracts from various subject fields. The Jensen-Shannon divergence (DJS) metric was employed to assess the accuracy of these models. The findings indicate the following: a) Bart outperforms other AI models in the task of summarization, with ChatGPT3.5 and Pegasus following closely behind, b) ChatGPT3.5 demonstrates proficiency in Agricultural Science. Bart’s summarization capabilities are more evenly distributed. Notably, in the domain of physics, all AI tools yield relatively higher DJS scores, while performing well in Arts & Humanities and Interdisciplinary subjects. c) Statistical significance tests conducted between the models reveal substantial differences, and both ChatGPT3.5 and Bart exhibit significant performance variations across subject fields.

Paper Citation

in Harvard Style

Lin Z., Chen S., Wang N. and Li H. (2023). Evaluating Text Summarization Generated by Popular AI Tools. In Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT; ISBN 978-989-758-677-4, SciTePress, pages 350-354. DOI: 10.5220/0012283800003807

in Bibtex Style

@conference{anit23,
author={Zhuoran Lin and Sifan Chen and Ning Wang and Hongjun Li},
title={Evaluating Text Summarization Generated by Popular AI Tools},
booktitle={Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT},
year={2023},
pages={350-354},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012283800003807},
isbn={978-989-758-677-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 2nd International Seminar on Artificial Intelligence, Networking and Information Technology - Volume 1: ANIT
TI - Evaluating Text Summarization Generated by Popular AI Tools
SN - 978-989-758-677-4
AU - Lin Z.
AU - Chen S.
AU - Wang N.
AU - Li H.
PY - 2023
SP - 350
EP - 354
DO - 10.5220/0012283800003807
PB - SciTePress