loading
Papers Papers/2022 Papers Papers/2022

Research.Publish.Connect.

Paper

Authors: Svenja Kenneweg 1 ; Jörg Deigmöller 2 ; Philipp Cimiano 1 and Julian Eggert 2

Affiliations: 1 Bielefeld University, Germany ; 2 Honda Research Institute Europe, Germany

Keyword(s): Temporal Question Answering, Events, Synthetic Benchmark.

Abstract: The ability to reason about events and their temporal relations is a key aspect in Natural Language Understanding. In this paper, we investigate the ability of Large Language Models to resolve temporal references with respect to longer event sets. Given that events rarely occur in isolation, it is crucial to determine the extent to which Large Language Models can reason about longer sets of events. Towards this goal, we introduce a novel synthetic benchmark dataset comprising of 2,200 questions to test the abilities of LLMs to reason about events using a Question Answering task as proxy. We compare the performance of 4 state of the art LLMs on the benchmark, analyzing their performance in dependence of the length of the event set considered as well as of the explicitness of the temporal reference. Our results show that, while the benchmarked LLMs can answer questions over event sets with a handful of events and explicit temporal references successfully, performance clearly deteriorat es with larger event set length and when temporal references get less explicit. The Benchmark is available at https://gitlab.ub.uni-bielefeld.de/s.kenneweg/bamer. (More)

CC BY-NC-ND 4.0

Sign In Guest: Register as new SciTePress user now for free.

Sign In SciTePress user: please login.

PDF ImageMy Papers

You are not signed in, therefore limits apply to your IP address 3.137.159.17

In the current month:
Recent papers: 100 available of 100 total
2+ years older papers: 200 available of 200 total

Paper citation in several formats:
Kenneweg, S.; Deigmöller, J.; Cimiano, P. and Eggert, J. (2024). Benchmarking the Ability of Large Language Models to Reason About Event Sets. In Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KEOD; ISBN 978-989-758-716-0; ISSN 2184-3228, SciTePress, pages 74-82. DOI: 10.5220/0013046100003838

@conference{keod24,
author={Svenja Kenneweg. and Jörg Deigmöller. and Philipp Cimiano. and Julian Eggert.},
title={Benchmarking the Ability of Large Language Models to Reason About Event Sets},
booktitle={Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KEOD},
year={2024},
pages={74-82},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013046100003838},
isbn={978-989-758-716-0},
issn={2184-3228},
}

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - KEOD
TI - Benchmarking the Ability of Large Language Models to Reason About Event Sets
SN - 978-989-758-716-0
IS - 2184-3228
AU - Kenneweg, S.
AU - Deigmöller, J.
AU - Cimiano, P.
AU - Eggert, J.
PY - 2024
SP - 74
EP - 82
DO - 10.5220/0013046100003838
PB - SciTePress

- Science and Technology Publications, Lda.
RESOURCES

Proceedings

Papers

Authors

Ontology

CONTACTS

Science and Technology Publications, Lda
Avenida de S. Francisco Xavier, Lote 7 Cv. C,
2900-616 Setúbal, Portugal.

Phone: +351 265 520 185 (National fixed network call)
Fax: +351 265 520 186
Email: info@scitepress.org

EXTERNAL LINKS

PRIMORIS

INSTICC

SCITEVENTS

CROSSREF

PROCEEDINGS SUBMITTED FOR INDEXATION BY:

dblp

Ei Compendex

SCOPUS

Semantic Scholar

Google Scholar

Microsoft Academic