A Comparison of Methods for the Evaluation of Text Summarization Techniques

Marcello Barbella, Michele Risi, Genoveffa Tortora

2021

Abstract

Automatic Text Summarization techniques aim to extract key information from one or more input texts automatically, producing summaries and preserving the meaning of content. These techniques are divided into two main families: Extractive and Abstractive, which differ for their operating mode. The former picks up sentences directly from the document text, whilst the latter produces a summary by interpreting the text and rephrases sentences by incorporating information. Therefore, there is the need to evaluate and verify how close a summary is to original text. The research question is: how to evaluate the quality of the summaries produced by these techniques? Different metrics and scores have been proposed in the literature (e.g., ROUGE) for the evaluation of text summarization. Thus, the main purpose of this paper is to deeply estimate the behaviour of the ROUGE metric. In particular, we performed a first experiment to compare the metric efficiency for the evaluation of the Abstractive versus Extractive Text Summarization algorithms while, in a second one, we compared the obtained score for two different summary approaches: the simple execution of a summarization algorithm versus the multiple execution of different algorithms on the same text. The conclusions lead to the following interesting results: ROUGE does not achieve excellent results, because it has similar performance on both the Abstractive and Extractive algorithms; multiple execution works better than single one most of the time.

Paper Citation

in Harvard Style

Barbella M., Risi M. and Tortora G. (2021). A Comparison of Methods for the Evaluation of Text Summarization Techniques. In Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA, ISBN 978-989-758-521-0, pages 200-207. DOI: 10.5220/0010523002000207

in Bibtex Style

@conference{data21,
author={Marcello Barbella and Michele Risi and Genoveffa Tortora},
title={A Comparison of Methods for the Evaluation of Text Summarization Techniques},
booktitle={Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA,},
year={2021},
pages={200-207},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010523002000207},
isbn={978-989-758-521-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Data Science, Technology and Applications - Volume 1: DATA,
TI - A Comparison of Methods for the Evaluation of Text Summarization Techniques
SN - 978-989-758-521-0
AU - Barbella M.
AU - Risi M.
AU - Tortora G.
PY - 2021
SP - 200
EP - 207
DO - 10.5220/0010523002000207