Comparing Human and Machine Generated Text for Sentiment

WingYin Ha, Diarmuid O’Donoghue

2024

Abstract

This paper compares human and machine generated texts, focusing on a comparison of their sentiment. We use two corpora; the first being the HC3 question and answer texts. We present a second corpus focused on human written text-materials sourced from psychology experiments and we used a language model to generate stories analogous to the presented information. Two sentiment analysis tools generated sentiment results, showing that there was a frequent occurrence of statistically significant differences between the sentiment scores on the individual sub-collections within these corpora. Generally speaking, machine generated text tended to have a slightly more positive sentiment than the human authored equivalent. However, we also found low levels of agreement between the Vader and TextBlob sentiment-analysis systems used. Any proposed use of LLM generated content in the place of retrieved information needs to carefully consider subtle differences between the two – and the implications these differences may have on down-stream tasks.

Paper Citation

in Harvard Style

Ha W. and O’Donoghue D. (2024). Comparing Human and Machine Generated Text for Sentiment. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-716-0, SciTePress, pages 335-342. DOI: 10.5220/0012986100003838

in Bibtex Style

@conference{kdir24,
author={WingYin Ha and Diarmuid O’Donoghue},
title={Comparing Human and Machine Generated Text for Sentiment},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2024},
pages={335-342},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012986100003838},
isbn={978-989-758-716-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Comparing Human and Machine Generated Text for Sentiment
SN - 978-989-758-716-0
AU - Ha W.
AU - O’Donoghue D.
PY - 2024
SP - 335
EP - 342
DO - 10.5220/0012986100003838
PB - SciTePress