Authors:
Desiree Heim
1
;
2
;
Christian Jilek
1
;
Adrian Ulges
3
and
Andreas Dengel
1
;
2
Affiliations:
1
Smart Data and Knowledge Services Department, German Research Center for Artificial Intelligence (DFKI), Germany
;
2
Department of Computer Science, University of Kaiserslautern-Landau (RPTU), Germany
;
3
Department DCSM, RheinMain University of Applied Sciences, Germany
Keyword(s):
Knowledge Work Dataset Generator, Large Language Model, Configurability.
Abstract:
The evaluation of support tools designed for knowledge workers is challenging due to the lack of publicly available, extensive, and complete data collections. Existing data collections have inherent problems such as incompleteness due to privacy-preserving methods and lack of contextual information. Hence, generating datasets can represent a good alternative, in particular, Large Language Models (LLM) enable a simple possibility of generating textual artifacts. Just recently, we therefore proposed a knowledge work dataset generator, called KnoWoGen. So far, the adherence of generated knowledge work documents to parameters such as document type, involved persons, or topics has not been examined. However, this aspect is crucial to examine since generated documents should reflect given parameters properly as they could serve as highly relevant ground truth information for training or evaluation purposes. In this paper, we address this missing evaluation aspect by conducting respective u
ser studies. These studies assess the documents’ adherence to multiple parameters and specifically to a given domain parameter as an important, representative. We base our experiments on documents generated with KnoWoGen and use the Mistral-7B-Instruct model as LLM. We observe that in the given setting, the generated documents showed a high quality regarding the adherence to parameters in general and specifically to the parameter specifying the document’s domain. Hence, 75% of the given ratings in the parameter-related experiments received the highest or second-highest quality score which is a promising outcome for the feasibility of generating high-qualitative knowledge work documents based on given configurations.
(More)