Authors:
Marcello Barbella
;
Michele Risi
and
Genoveffa Tortora
Affiliation:
Department of Computer Science, University of Salerno, Fisciano (SA), Italy
Keyword(s):
Automatic Text Summarization Algorithms, Extractive, Abstractive, ROUGE Metric, Bert.
Abstract:
Automatic Text Summarization techniques aim to extract key information from one or more input texts automatically, producing summaries and preserving the meaning of content. These techniques are divided into two main families: Extractive and Abstractive, which differ for their operating mode. The former picks up sentences directly from the document text, whilst the latter produces a summary by interpreting the text and rephrases sentences by incorporating information. Therefore, there is the need to evaluate and verify how close a summary is to original text. The research question is: how to evaluate the quality of the summaries produced by these techniques? Different metrics and scores have been proposed in the literature (e.g., ROUGE) for the evaluation of text summarization. Thus, the main purpose of this paper is to deeply estimate the behaviour of the ROUGE metric. In particular, we performed a first experiment to compare the metric efficiency for the evaluation of the Abstracti
ve versus Extractive Text Summarization algorithms while, in a second one, we compared the obtained score for two different summary approaches: the simple execution of a summarization algorithm versus the multiple execution of different algorithms on the same text. The conclusions lead to the following interesting results: ROUGE does not achieve excellent results, because it has similar performance on both the Abstractive and Extractive algorithms; multiple execution works better than single one most of the time.
(More)