Interpreting Workflow Architectures by LLMs

Michal Töpfer, Tomáš Bureš, František Plášil, Petr Hnětynka

2025

Abstract

In this paper, we focus on how reliably can a Large Lanuage Model (LLM) interpret a software architecture, namely a workflow architecture (WA). Even though our initial experiments show that an LLM can answer specific questions about a WA, it is unclear how correct its answers are. To this end, we propose a methodology to assess whether an LLM can correctly interpret a WA specification. Based on the conjecture that the LLM needs to correctly answer low-abstraction level questions to answer questions at a higher abstraction level properly, we define a set of test patterns, each of them providing a template for low-abstraction level questions, together with a metric for evaluating the correctness of LLM’s answers. We posit that having this metric will allow us not only to establish which LLM model works the best with WAs, but also to determine what their concrete syntax and concepts are suitable to strengthen the correctness of LLM’s interpretability of WA specifications. We demonstrate the methodology on the workflow specification language developed for a currently running Horizon Europe project.

Paper Citation

in Harvard Style

Töpfer M., Bureš T., Plášil F. and Hnětynka P. (2025). Interpreting Workflow Architectures by LLMs. In Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE; ISBN 978-989-758-742-9, SciTePress, pages 608-617. DOI: 10.5220/0013358000003928

in Bibtex Style

@conference{enase25,
author={Michal Töpfer and Tomáš Bureš and František Plášil and Petr Hnětynka},
title={Interpreting Workflow Architectures by LLMs},
booktitle={Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE},
year={2025},
pages={608-617},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013358000003928},
isbn={978-989-758-742-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE
TI - Interpreting Workflow Architectures by LLMs
SN - 978-989-758-742-9
AU - Töpfer M.
AU - Bureš T.
AU - Plášil F.
AU - Hnětynka P.
PY - 2025
SP - 608
EP - 617
DO - 10.5220/0013358000003928
PB - SciTePress