Collaborative Emotion Annotation: Assessing the Intersection of Human and AI Performance with GPT Models

Hande Aka Uymaz, Senem Kumova Metin

2023

Abstract

In this study, we explore emotion detection in text, a complex yet vital aspect of human communication. Our focus is on the formation of an annotated dataset, a task that often presents difficulties due to factors such as reliability, time, and consistency. We propose an alternative approach by employing artificial intelligence (AI) models as potential annotators, or as augmentations to human annotators. Specifically, we utilize ChatGPT, an AI language model developed by OpenAI. We use its latest versions, GPT3.5 and GPT4, to label a Turkish dataset having 8290 terms according to Plutchik’s emotion categories, alongside three human annotators. We conduct experiments to assess the AI’s annotation capabilities both independently and in conjunction with human annotators. We measure inter-rater agreement using Cohen’s Kappa, Fleiss Kappa, and percent agreement metrics across varying emotion categorizations- eight, four, and binary. Particularly, when we filtered out the terms where the AI models were indecisive, it was found that including AI models in the annotation process was successful in increasing inter-annotator agreement. Our findings suggest that, the integration of AI models in the emotion annotation process holds the potential to enhance efficiency, reduce the time of lexicon development and thereby advance the field of emotion/sentiment analysis.

Paper Citation

in Harvard Style

Aka Uymaz H. and Kumova Metin S. (2023). Collaborative Emotion Annotation: Assessing the Intersection of Human and AI Performance with GPT Models. In Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-671-2, SciTePress, pages 298-305. DOI: 10.5220/0012183200003598

in Bibtex Style

@conference{kdir23,
author={Hande Aka Uymaz and Senem Kumova Metin},
title={Collaborative Emotion Annotation: Assessing the Intersection of Human and AI Performance with GPT Models},
booktitle={Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2023},
pages={298-305},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012183200003598},
isbn={978-989-758-671-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Collaborative Emotion Annotation: Assessing the Intersection of Human and AI Performance with GPT Models
SN - 978-989-758-671-2
AU - Aka Uymaz H.
AU - Kumova Metin S.
PY - 2023
SP - 298
EP - 305
DO - 10.5220/0012183200003598
PB - SciTePress