Using Machine Learning to Distinguish Human-Written from Machine-Generated Creative Fiction

Andrea Cristina McGlinchey, Peter J. Barclay

2025

Abstract

Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and “fake news”. However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers’ output to generate “sham books” in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Na¨ıve Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.

Paper Citation

in Harvard Style

McGlinchey A. and Barclay P. (2025). Using Machine Learning to Distinguish Human-Written from Machine-Generated Creative Fiction. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 79-90. DOI: 10.5220/0013110100003890

in Bibtex Style

@conference{icaart25,
author={Andrea Cristina McGlinchey and Peter Barclay},
title={Using Machine Learning to Distinguish Human-Written from Machine-Generated Creative Fiction},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2025},
pages={79-90},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013110100003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - Using Machine Learning to Distinguish Human-Written from Machine-Generated Creative Fiction
SN - 978-989-758-737-5
AU - McGlinchey A.
AU - Barclay P.
PY - 2025
SP - 79
EP - 90
DO - 10.5220/0013110100003890
PB - SciTePress