
milestone in the field of natural language processing
by introducing the attention mechanism into the trans-
former model, a new and promising approach that de-
parts from traditional recurrent or convolutional neu-
ral networks. The attention mechanism allows the
encoders and decoders to focus on different parts of
the input sequence by assigning them more weight.
Based on the task of translating a text from English
to German, this transformer-based model was supe-
rior in quality and also required a significantly smaller
amount of training time by being more parallelizable
(Vaswani et al., 2017). Consequently, these advance-
ments enabled the GPTs to train on huge datasets to
create LLMs with more general knowledge and un-
derstanding of the concepts in the real world.
2.2.3 Training GPT Models
GPT models belong to the family of LLMs and have
an incredible power of generating natural text due
to their training performance that enabled them to
gain knowledge through huge datasets. This abil-
ity can be further utilized to effectively train pre-
trained models on a specific topic and transfer the
already-known concepts and patterns to new domains
(Amaratunga, 2023). While the process of fine-tuning
LLMs requires an understanding of algorithms like
back-propagation and gradient descent, vendors like
OpenAI effectively simplify this complexity with an
additional abstraction layer, making the technology
more accessible. By tailoring such models through
fine-tuning, we achieve a higher level of precision
and reliability, by greatly improving their applicabil-
ity and effectiveness on specific tasks.
2.3 OpenAI’s GPT API
Since its introduction by OpenAI in 2018, the GPT se-
ries of models has evolved, with each iteration labeled
as GPT-n, where n indicates the version number. The
GPT-3.5 model, released in 2022, marked a signifi-
cant advancement with its extensive API supporting
fine-tuning, a key feature in this research. In fine-
tuning, the model receives prompts specifying de-
sired output characteristics, and each interaction con-
tributes to a conversational context, influencing sub-
sequent responses. A critical aspect of GPT models is
their context size limitation, with the latest GPT-3.5-
turbo-1106 model handling up to 16K tokens.
The interaction architecture between a human and
the LLM is structured so that the model receives a
prompt detailing specific requirements that its output
must meet. Each prompt and subsequent response
generated by the LLM contributes to a conversational
context that is continuously utilized to produce the
next segment of the response. A notable distinction
among GPT versions is their maximum comprehen-
sible context size, managed internally by a context
window that automatically drops the oldest context
information if the tokens exceed the limit (OpenAI,
2023a). Currently, GPT-3.5-turbo-1106 is the most
recent available model for fine-tuning jobs, and is ca-
pable of handling a context with the size of 16K tokes
(OpenAI, 2023b).
GPT models are fine-tuned with a collection of
JavaScript Object Notation (JSON) Objects in the
JSON Lines format. Each line of this format maps
a system description and a full conversation between
the two personas ”user” and ”assistant”. The dataset
represents example conversations between a user and
with the anticipated response of the LLM (OpenAI,
2023a). During the fine-tuning process, this data set
is used to adjust the parameters of the model so that
the expected responses are reproduced as closely to
the training set as possible.
3 APPROACH
As outlined in Section 1 a user-centered approach to
MBSE could be supported by the utilization of ar-
tificial intelligence. To achieve the results for this
research, this section deals extensively with the ap-
proach of creating the training dataset, generating
XMI with the trained GPT model and the approach
for evaluating the results.
3.1 Dataset Generation
To conduct the research, UML component diagram el-
ements are utilized. These are structured and straight-
forward elements describing significant components
and their interconnections within a system. Repre-
sentative for all UML elements, their XMI representa-
tion is used to generate the fine-tuning dataset for the
GPT model. The dataset consists of 10 different con-
versations between a user and an assistant, where the
user is requesting XMI code from the LLM. The re-
quested code examples are UML component diagram
elements with increasing complexity, organized into
three complexity classes. These classes are designed
to progressively demonstrate more intricate compo-
nent interactions and system architectures based on
the reference implementation in Section 2.1.1:
1. Standalone components and interfaces: The
simplest class involves standalone components,
which may have a port with exposed interfaces
and their realizing interfaces. These examples fo-
cus on individual elements in isolation, providing
Facilitating User-Centric Model-Based Systems Engineering Using Generative AI
373