Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques

Victor Margallo

2022

Abstract

In the task of providing extracted summaries, the assessment of quality evaluation has been traditionally tackled with n-gram, word sequences, and word pairs overlapping metrics with human annotated summaries for theoretical benchmarking. This approach does not provide an end solution for extractive summarising algorithms as output summaries are not evaluated for new texts. Our solution proposes the expansion of a graph extraction method together with an understanding layer before delivering the final summary. With this technique we strive to achieve a categorisation of acceptable output summaries. Our understanding layer judges correct summaries with 91% accuracy and is in line with experts’ labelling providing a strong inter-rater reliability (0.73 Kappa statistic).

Paper Citation

in Harvard Style

Margallo V. (2022). Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-547-0, pages 605-611. DOI: 10.5220/0010954300003116

in Bibtex Style

@conference{icaart22,
author={Victor Margallo},
title={Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques},
booktitle={Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2022},
pages={605-611},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010954300003116},
isbn={978-989-758-547-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Understanding Summaries: Modelling Evaluation in Extractive Summarisation Techniques
SN - 978-989-758-547-0
AU - Margallo V.
PY - 2022
SP - 605
EP - 611
DO - 10.5220/0010954300003116