Fine-Tuning and Aligning Question Answering Models for Complex Information Extraction Tasks

Matthias Engelbach, Dennis Klau, Felix Scheerer, Jens Drawehn, Maximilien Kintz

2023

Abstract

The emergence of Large Language Models (LLMs) has boosted performance and possibilities in various NLP tasks. While the usage of generative AI models like ChatGPT opens up new opportunities for several business use cases, their current tendency to hallucinate fake content strongly limits their applicability to document analysis, such as information retrieval from documents. In contrast, extractive language models like question answering (QA) or passage retrieval models guarantee query results to be found within the boundaries of an according context document, which makes them candidates for more reliable information extraction in productive environments of companies. In this work we propose an approach that uses and integrates extractive QA models for improved feature extraction of German business documents such as insurance reports or medical leaflets into a document analysis solution. We further show that fine-tuning existing German QA models boosts performance for tailored extraction tasks of complex linguistic features like damage cause explanations or descriptions of medication appearance, even with using only a small set of annotated data. Finally, we discuss the relevance of scoring metrics for evaluating information extraction tasks and deduce a combined metric from Levenshtein distance, F1-Score, Exact Match and ROUGE-L to mimic the assessment criteria from human experts.

Paper Citation

in Harvard Style

Engelbach M., Klau D., Scheerer F., Drawehn J. and Kintz M. (2023). Fine-Tuning and Aligning Question Answering Models for Complex Information Extraction Tasks. In Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR; ISBN 978-989-758-671-2, SciTePress, pages 196-205. DOI: 10.5220/0012159000003598

in Bibtex Style

@conference{kdir23,
author={Matthias Engelbach and Dennis Klau and Felix Scheerer and Jens Drawehn and Maximilien Kintz},
title={Fine-Tuning and Aligning Question Answering Models for Complex Information Extraction Tasks},
booktitle={Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR},
year={2023},
pages={196-205},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012159000003598},
isbn={978-989-758-671-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management - Volume 1: KDIR
TI - Fine-Tuning and Aligning Question Answering Models for Complex Information Extraction Tasks
SN - 978-989-758-671-2
AU - Engelbach M.
AU - Klau D.
AU - Scheerer F.
AU - Drawehn J.
AU - Kintz M.
PY - 2023
SP - 196
EP - 205
DO - 10.5220/0012159000003598
PB - SciTePress