UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization

Lucas Liebe, Jannis Baum, Tilman Schütze, Tim Cech, Willy Scheibel, Jürgen Döllner

2023

Abstract

Text synthesis tools are becoming increasingly popular and better at mimicking human language. In trust-sensitive decisions, such as plagiarism and fraud detection, identifying AI-generated texts poses larger difficulties: decisions need to be made explainable to ensure trust and accountability. To support users in identifying AI-generated texts, we propose the tool UNCOVER. The tool analyses texts through three explainable linguistic approaches: Stylometric writing style analysis, topic modeling, and entity recognition. The result of the tool is a prediction and visualization of the analysis. We evaluate the tool on news articles by means of accuracy of the prediction and an expert study with 13 participants. The final prediction is based on classification of stylometric and evolving topic analysis. It achieved an accuracy of 70.4% and a weighted F1-score of 85.6%. The participants preferred to base their assessment on the prediction and the topic graph. In contrast, they found the entity recognition to be an ineffective indicator. Moreover, five participants highlighted the explainable aspects of UNCOVER and overall the participants achieved 69% accuracy. Eight participants expressed interest to continue using UNCOVER for identifying AI-generated texts.

Paper Citation

in Harvard Style

Liebe L., Baum J., Schütze T., Cech T., Scheibel W. and Döllner J. (2023). UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization. In Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-671-2, SciTePress, pages 39-50. DOI: 10.5220/0012163300003598

in Bibtex Style

@conference{kdir23,
author={Lucas Liebe and Jannis Baum and Tilman Schütze and Tim Cech and Willy Scheibel and Jürgen Döllner},
title={UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization},
booktitle={Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2023},
pages={39-50},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012163300003598},
isbn={978-989-758-671-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization
SN - 978-989-758-671-2
AU - Liebe L.
AU - Baum J.
AU - Schütze T.
AU - Cech T.
AU - Scheibel W.
AU - Döllner J.
PY - 2023
SP - 39
EP - 50
DO - 10.5220/0012163300003598
PB - SciTePress