Benchmarking the Ability of Large Language Models to Reason About Event Sets

Svenja Kenneweg, Jörg Deigmöller, Philipp Cimiano, Julian Eggert

2024

Abstract

The ability to reason about events and their temporal relations is a key aspect in Natural Language Understanding. In this paper, we investigate the ability of Large Language Models to resolve temporal references with respect to longer event sets. Given that events rarely occur in isolation, it is crucial to determine the extent to which Large Language Models can reason about longer sets of events. Towards this goal, we introduce a novel synthetic benchmark dataset comprising of 2,200 questions to test the abilities of LLMs to reason about events using a Question Answering task as proxy. We compare the performance of 4 state of the art LLMs on the benchmark, analyzing their performance in dependence of the length of the event set considered as well as of the explicitness of the temporal reference. Our results show that, while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorates with larger event set length and when temporal references get less explicit. The Benchmark is available at https://gitlab.ub.uni-bielefeld.de/s.kenneweg/bamer.

Paper Citation

in Harvard Style

Kenneweg S., Deigmöller J., Cimiano P. and Eggert J. (2024). Benchmarking the Ability of Large Language Models to Reason About Event Sets. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD; ISBN 978-989-758-716-0, SciTePress, pages 74-82. DOI: 10.5220/0013046100003838

in Bibtex Style

@conference{keod24,
author={Svenja Kenneweg and Jörg Deigmöller and Philipp Cimiano and Julian Eggert},
title={Benchmarking the Ability of Large Language Models to Reason About Event Sets},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD},
year={2024},
pages={74-82},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013046100003838},
isbn={978-989-758-716-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 2: KEOD
TI - Benchmarking the Ability of Large Language Models to Reason About Event Sets
SN - 978-989-758-716-0
AU - Kenneweg S.
AU - Deigmöller J.
AU - Cimiano P.
AU - Eggert J.
PY - 2024
SP - 74
EP - 82
DO - 10.5220/0013046100003838
PB - SciTePress