A Framework for Intelligent Virtual Reality Tutoring System Using

Semantic Web Technology

Victor Häfner

, Tengyu Li

∗ b

, Felix Longge Michels

, Polina Häfner

, Haoran Yu

and Jivka Ovtcharova

Institute for Information Management in Engineering (IMI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

Keywords:

Virtual Reality Training, Personalized Tutoring, Semantic Web Technology, Intelligent Tutoring System,

Engineering Education, Virtual Flow Rig Training.

Abstract:

Adaptive and personalized immersive virtual reality (VR) training is a current research focus, aiming to make

training more efﬁcient and let users have a better learning experience. But there is currently an unmet need for

ﬂexible VR training that can adapt to the user’s abilities and performance to provide a personalized learning

process. The main problem is the high cost of setting up and maintaining virtual environments, resulting in

a lack of ﬂexibility to easily update training content. To solve this problem, we incorporate Semantic Web

technology into VR training. Using the heterogeneous knowledge integration and reasoning capabilities of the

Semantic Web, VR training content can be easily created and updated. In addition, due to the advantages of

the intelligent tutoring system (ITS) in tracking the student performance and providing automated and person-

alized tutoring, we explore the integration of the ITS into the VR learning environment. Therefore, this paper

proposes a framework for a Semantic Web-Enabled Intelligent VR Tutoring System, especially for engineer-

ing education, and demonstrates its feasibility by a proof of concept application. The system is exempliﬁed

using a virtual ﬂow rig application for training, facilitating the education of students and professionals on ﬂuid

ﬂow principles, control strategies, and measurement techniques.

1 INTRODUCTION

With the emergence of consumer hardware of im-

mersive virtual reality (VR), the application range

of VR technology continues to expand, involving

many ﬁelds such as education, training and enter-

tainment (Häfner, 2020; Zahabi and Abdul Razak,

2020). Among them, the application of VR in the

ﬁeld of training has attracted much attention. It can

simulate expensive or dangerous scenarios, thereby

reducing the cost of training, eliminating risks for

trainees, and greatly improving the ﬂexibility of train-

ing in time and location (Ruthenbeck and Reynolds,

2015). In addition, immersive and contextual experi-

ences provide the best basis for understanding knowl-

edge and skills (Dale, 1969). Due to the signiﬁcant

advantages of VR training, it has been widely used

in industry (Richard et al., 2021b), medical (Vaughan

https://orcid.org/0000-0001-8682-1122

https://orcid.org/0009-0008-8174-1009

https://orcid.org/0000-0001-6533-4886

https://orcid.org/0000-0003-4534-351X

∗

Corresponding author

et al., 2016), military (Bhagat et al., 2016), construc-

tion (Schiavi et al., 2021) and other ﬁelds. In order to

make VR training more effective, current research fo-

cuses on adaptation and personalization in VR train-

ing, and many achievements have been made (Zahabi

and Abdul Razak, 2020). The U.S. National Academy

of Engineering (U.S. National Academy of Engineer-

ing, 2017) has also listed “Enhancing Virtual Real-

ity” and “Advancing Personalized Learning" among

the major challenges in Engineering in the 21st cen-

tury. Nonetheless, there is an unmet need for ﬂexi-

ble VR training that can adapt to the user’s abilities

and performance to provide a personalized learning

process. This is because most studies concentrate on

the adaptation of features or their combinations in a

given training scenario (such as display features, con-

trolled elements, difﬁculty level determined by other

features, etc.) (Zahabi and Abdul Razak, 2020), and

pay less attention to the adaptation of training con-

tent. The main problem is the high cost of setting

up and maintaining virtual environments, resulting in

a lack of ﬂexibility to easily update training content.

To address this problem, it is promising to utilize Se-

Häfner, V., Li, T., Michels, F., Häfner, P., Yu, H. and Ovtcharova, J.

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology.

DOI: 10.5220/0012746400003693

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024) - Volume 2, pages 141-152

ISBN: 978-989-758-697-2; ISSN: 2184-5026

141

mantic Web technology to facilitate ontology-driven

VR training. Ontologies can support the integration

and management of heterogeneous knowledge such

as virtual three-dimensional (3D) scenes, interactions

and tasks (Benferdia et al., 2021). Once implemented,

the training content can be easily created and updated

with the support of the reasoner. However, there are

few literatures on this aspect at present. An Intelli-

gent Tutoring System (ITS) is a computer-based tutor-

ing agent engineered to track and analyze a “student

model", which encompasses the student’s progress,

performance, and learning patterns. The primary ob-

jective of an ITS is to provide automated and per-

sonalized tutoring, comprising task selection, error

detection and rectiﬁcation, step guidance, and other

adaptive learning approaches (Häfner, 2021; Herbert

et al., 2018). Integrating Intelligent Tutoring Systems

(ITSs) into VR learning environments represents an

emerging research ﬁeld, yet the extent of exploration

to date remains limited (Laine et al., 2022). As a re-

sponse to this gap, we propose the concept of a Se-

mantic Web-Enabled Intelligent VR Tutoring System,

a fusion of Semantic Web technology, ITS, and VR

training applications. Semantic Web technology, an

artiﬁcial intelligence (AI) approach based on knowl-

edge representation, offers an alternative pathway to

implement an ITS. Hence, the integration of these

three components is viable, and currently, there exists

no established framework to describe such a compre-

hensive system.

In view of the above development trends and chal-

lenges of VR training, this paper proposes a frame-

work of Semantic Web-Enabled Intelligent VR Tutor-

ing System. To illustrate its practical application, we

speciﬁcally focus on the development of an intelligent

VR tutoring system for a ﬂow rig. The ﬂow rig is a

valuable tool utilized extensively in research, experi-

mentation, analysis, testing, and validation within en-

gineering, ﬂuid dynamics, thermodynamics, and re-

lated domains. While the ﬂow rig serves a wide

array of applications, our particular emphasis is on

its pivotal role in education and the creation of VR

training application to showcase its effectiveness in

teaching control principles, methods, and operational

procedures for fundamental process parameters such

as ﬂow rate, level, volume, and differential pressure

within piping systems. The application caters to a di-

verse range of users including students in educational

settings, vocational training programs, sales and tech-

nical support personnel, as well as professionals seek-

ing continuing education and research and develop-

ment teams in engineering and manufacturing indus-

tries. The ﬂow rig use case serves as an example train-

ing application and a proof of concept for the pro-

posed framework. The implementation methods of

the three core modules within the framework are de-

tailed, aiming to provide readers with a clear under-

standing of the system’s functionality and potential.

Among them, in the implementation of immersive VR

learning environment, we focus on the high-ﬁdelity

simulation of complex machines in the ﬁeld of me-

chanical engineering. Furthermore, we point out the

need and importance of automating the virtualization

process of machines, aiming to facilitate further the

process of creation of immersive VR trainings.

The subsequent sections of this paper are outlined

as follows. In Section 2, we present a comprehensive

review of related research. Section 3 elaborates on

the framework of the proposed ITS system. Follow-

ing that, Section 4 provides an introduction to the im-

plementation approaches for the three core modules.

Finally, in Section 5, we summarize the paper and of-

fer insights into potential avenues for future research.

2 RELATED WORK

Here, we mainly review the prior work on the appli-

cation of Semantic Web technology in virtual train-

ing and ITSs. Due to the advantages of the Semantic

Web in knowledge representation, integration, shar-

ing, reuse, reasoning, etc., there is growing interest in

using the Semantic Web in virtual training. Häfner

(Häfner, 2017) pointed out that connecting the virtual

world with semantic models to build a smart virtual

environment has great prospects for training applica-

tions, especially in the engineering ﬁeld. Walczak

et al. (Walczak et al., 2020) proposed an ontology-

based semantic modeling method for VR training sce-

narios, and developed a scenario editor based on this

method, which supports users without advanced pro-

gramming and modeling skills to easily create train-

ing scenarios. Elenius et al. (Elenius et al., 2016)

proposed a framework for users to interact with vir-

tual environments based on ontologies and rules, and

applied it to the training of weapon skills. Filho et

al. (Filho and Vieira, 2014; Filho et al., 2015) used

an ontology approach to facilitate the development of

virtual scenarios in a training simulator for the op-

eration of electrical systems. Havard et al. (Havard

et al., 2017) proposed an industrial ontology describ-

ing operational guidelines for integration with infor-

mation systems. Gorecky et al. (Gorecky et al., 2014;

Gorecky et al., 2017) used an ontology-based seman-

tic modeling method to solve the integration problem

of heterogeneous data in order to promote the existing

enterprise data (such as production line structure, pro-

cess description, etc.) to be used in the setting of vir-

CSEDU 2024 - 16th International Conference on Computer Supported Education

142

Figure 1: Framework of the Semantic Web-Enabled Intelligent VR Tutoring System (adapted from (Häfner, 2021)).

tual training scenarios. It can be seen that most studies

use Semantic Web technology to solve the problems

of scenario creation and heterogeneous knowledge in-

tegration in virtual training, while few studies focus

on how to use Semantic Web to drive the VR training

process, including user interaction, training content

update, and evaluation.

Some achievements have also been made in the

application of Semantic Web in ITS. Chang et al.

(Chang et al., 2020) used ontology to build a tutor-

ing model in an ITS, and explored the use of data

mining technology to accelerate this process. Jac-

into and Oliveira (Jacinto and Oliveira, 2008) pre-

sented an architecture of an ITS based on multiple on-

tologies. Muñoz Merino and Kloos (Muñoz Merino

and Kloos, 2008) combined the ITS with Semantic

Web technology to provide students with personal-

ized prompts adaptively. Vesin et al. (Vesin et al.,

2012) used semantic reasoning to identify students’

learning styles and recommend personalized courses

for students. Wen et al. (Wen et al., 2022) devel-

oped an ontology-driven dialogue-based ITS. How-

ever, few studies have explored the combination of

Semantic Web, ITS and VR training.

3 FRAMEWORK OVERVIEW

The framework of the Semantic Web-enabled intelli-

gent VR tutoring system is shown in Figure 1. The

whole is divided into three core modules, namely the

semantic layer, the immersive VR learning environ-

ment and the tutoring system.

The semantic layer is a module that provides in-

telligence based on Semantic Web technology. It

includes an ontology-based knowledge base and a

reasoner. The ontology deﬁnes the data model of

the knowledge base (Häfner et al., 2013; Wicaksono

et al., 2013), which includes several small ontologies

for different domains describing the virtual world,

user interactions and tutoring situations, such as the

ontology characterizing the state of the visual interac-

tive object like machine virtual twin ontology, student

behavior ontology, tutoring task ontology, student

skill ontology, etc. These ontologies are comprised

of domain-related concept (class) taxonomies, prop-

erties, reasoning rules, concept instances and their

property assertions. Concepts from different ontolo-

gies are linked to each other through rules. Virtual

world assets are linked to the corresponding concepts.

In the stage of system initialization, relevant object

instances are generated according to the initial train-

ing requirements and populate the knowledge base,

such as speciﬁc virtual scene instances, initial state

parameter instances for both the user and the sys-

tem, tutoring task instances, among other relevant el-

ements. During the tutoring phase, the system con-

tinuously adds or updates instances in the knowledge

base in reaction to changes within the VR environ-

ment and tutoring system, like populating user behav-

ior instances, updating tasks or virtual scenes, and so

on. The semantic layer tracks the knowledge of the

virtual learning environment and the user in real time,

which is very important for the dynamic update of the

visual scene and the function implementation of the

tutoring system.

Reasoner is a module in the semantic layer that

implements intelligence. It engages in automatic rea-

soning and logical deduction using ontology rules

to derive new knowledge from the existing knowl-

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology

143

edge within the knowledge base, ultimately facilitat-

ing decision-making and query responses. Scenar-

ios that trigger the reasoner include user interactions

with the virtual scene and queries from the application

logic in the VR environment or the tutoring system.

The reasoner uses predeﬁned ontology rules to infer

the changes that should be made in the virtual world

based on the user interactions and their interpreted be-

haviors, or to execute prescribed tutoring actions. The

inference results are reﬂected as modiﬁcations to the

instance information in the knowledge base. The rea-

soner cannot directly change the virtual world, which

requires the application logic to achieve it. Therefore,

the application logic should send queries to the rea-

soner to learn about changes in the knowledge base.

In addition, the tutoring system can also issue queries

to the reasoner to check the progress of task execu-

tion, the user’s wrong behavior, or the improvement

of the user’s skills, etc. The reasoner will traverse all

the knowledge of the virtual world and the user in the

knowledge base, and make corresponding feedback

based on all relevant rules.

The primary roles of the tutoring system are to

generate tutoring tasks, evaluate tasks in real time,

and generate user feedback. These three functions are

implemented via interacting with the semantic layer,

which will be described in detail in Section 4.3. The

results produced by the tutoring system will change

the virtual environment, for example, changes in tasks

may affect changes in the virtual scene, and user feed-

back is visually presented in real time on the graph-

ical interface. In addition, in order to give users the

option to control their own learning, it is also neces-

sary to design an interactive interface for the tutoring

system in the virtual environment.

The immersive VR learning environment consists

of scene management, simulations, user interaction

and application logic. A virtual scene encompasses

assets such as 3D models, materials, textures and

sounds, as well as artefacts like virtual cameras, lights

and virtual sound sources. The arrangement of these

virtual assets and their topological relationships can

be managed using a hierarchical structure diagram.

In each frame, the hierarchical structure diagram is

traversed to compute the transformation of the vir-

tual world. Simulations are approximations of var-

ious behaviors in the real world, and are also the

core of building an immersive VR learning environ-

ment. Usually, VR engines have been embedded with

physics engines to simulate the basic physical behav-

ior of objects, such as collisions, gravity, vehicle dy-

namics, and so on. User interaction, a key character-

istic of immersive VR applications, depends on the

user behavior and the choice of interactive devices.

Users can virtually select, grab, and move virtual ob-

jects, as well as implement different navigation modes

as needed. Additionally, the realistic manipulation of

virtual objects can be realized through the simulation

of physical reactions of these objects like deforma-

tions and the implementation of force feedback. Fi-

nally, developers should add application logic to the

virtual assets as needed to implement the necessary

actions in the virtual environment. The application

logic is programmatically created using scripting lan-

guages. Within this ITS system framework, an impor-

tant function that the application logic should perform

is to populate various instance data into the seman-

tic layer during system initialization and run-time. In

addition, the application logic is also responsible for

dynamically generating some ontology rules, issuing

queries, and activating the reasoner.

In the following sections, we will provide a de-

tailed description of the implementation of the three

core modules of this system.

4 IMPLEMENTATION

4.1 Concept of Automating the

Virtualization to Build up Machine

Virtual Twins

The core and difﬁculty of constructing an immersive

VR learning environment is the simulation of the real

world. Since our long-term work focuses on VR ap-

plications in the ﬁeld of mechanical engineering, this

section mainly discusses the simulation of real ma-

chines in the VR environment, which will be used for

virtual training of machines.

The virtual twin of a machine is its virtual rep-

resentation, which includes all dynamic and func-

tional aspects (Häfner, 2019). The creation of the

virtual twin involves knowledge from multiple do-

mains, such as mechanical, electrical, and machine

programming. Integrating planning data from vari-

ous domains to generate an interactive virtual twin

with realistic machine behavior is highly complex,

time-consuming, and resource-consuming. Auto-

matic aggregation of overall knowledge from plan-

ning data is desired, in particular automatic asso-

ciation of component data in Electrical Computer-

Aided Design (ECAD) and Mechanical Computer-

Aided Design (MCAD) (Michels and Häfner, 2022;

Häfner et al., 2020). On the other hand, if the seman-

tic information required for the simulation of kine-

matics, dynamics, hydrodynamics, etc. is lacking in

the MCAD data, then this information should be de-

CSEDU 2024 - 16th International Conference on Computer Supported Education

144

Figure 2: Architecture of ﬂow rig virtual twin.

termined elsewhere. The development of automatic

analysis algorithms for geometric objects helps to

quickly capture as much inherent knowledge as pos-

sible from MCAD data to support parameter settings

for various simulations. Speciﬁcally, the choice of

the ﬁrst data source in the virtualization process often

comes to MCAD data as the entry point. These ge-

ometries are often exported from native engineering

software tools in the exchange format STEP, which

is the status quo due to its general compatibility with

other Computer-Aided Design (CAD) software ven-

dors. The exports lack further semantic information,

which either is stored in the native design tool and

cannot be exchanged digitally or rests in the non-

digital knowledge of the engineers and domain ex-

perts. This semantic data can be for example me-

chanical relevant data such as rotation axis, radius and

pitch of gears for the calculation of ratios in gearboxes

to virtually generate a kinematic chain for the kine-

matic simulation (Michels and Häfner, 2022) or pipe

parameters for generating a ﬂow simulation, as shown

in the example of ﬂow rig below. Another regular

difﬁculty with MCAD data exports is the inconsis-

tent data quality, depending on the CAD system and

the modelling methods used by the construction en-

gineer. It may be that the product structure is bad,

lacking meaningful groups of parts into assemblies.

For complex machines, it is more urgent to realize the

automation of MCAD data structure optimization.

The automated virtualization process forms the

foundation for expediting the implementation of di-

verse virtual engineering applications, including but

not limited to machine training, virtual commission-

ing, material ﬂow simulation, and more (Michels and

Häfner, 2022; Häfner et al., 2020). This approach

holds signiﬁcant promise and potential to streamline

the authoring process of immersive learning applica-

tions. Furthermore, embedding the semantic layer in

the virtual environment not only helps integrate the

heterogeneous data needed to build a virtual twin, but

also paves the way for the automatic parameteriza-

tion of the simulation model by linking semantic in-

formation to virtual 3D assets, as well as expanding

the range of possibilities for machine behavior simu-

lation and user interaction options.

Let’s take the VR tutoring system for a speciﬁc

ﬂow rig, which we are currently working on, as an

illustrative example to further elucidate our concept.

During the creation of a ﬂow rig virtual twin, the pre-

existing MCAD data was directly imported and de-

ployed, while the system simulation for the hydrody-

namics was coupled through the externally running

simulation tool. In order to save the time and cost of

creating the virtual twin, the existing SIMIT simula-

tion was combined with the MCAD model to obtain

an interactive virtual twin of the ﬂow rig, as shown

in Figure 2. The software SIMIT from SIEMENS is

a simulation environment used for simulating the be-

havior of the ﬂow rig, as shown on the right side of

Figure 2. The virtual scene of the ﬂow rig was gen-

erated in PolyVR, an open-source VR authoring soft-

ware (Häfner, 2019), as shown on the left side of Fig-

ure 2. The communication between the virtual scene

running and the hydrodynamic simulation was facili-

tated via MQTT (OASIS Standard, 2015), a well es-

tablished protocol for Industry 4.0 applications. Over

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology

145

this MQTT interface we sent sensor data such as ﬂow

rate, or tank level towards the virtual twin as well as

action commands input from the user for individual

valves and pumps towards SIMIT. One of the typical

issues we encountered was that in the MCAD data,

the scene graph generated from the product tree ex-

hibited a ﬂat structure without any hierarchical group

concepts and the names of components were missing.

A lack of semantic information to identify the com-

ponents makes it difﬁcult to map MCAD components

to the SIMIT simulation model, which is necessary

for visualizing and interacting with the virtual ﬂow

rig. To solve this problem, it is necessary to clas-

sify the components of the piping system and identify

their topological position to map them to the simula-

tion model. Moreover, for pipes, in addition to their

type, semantic information such as length and inner

radius are particularly important for their parametric

simulation and analysis of ﬂow characteristics such

as throughput or pressure loss due to friction. Finally,

we adopted a geometric analysis method to realize the

classiﬁcation of the components and the extraction of

the basic pipe parameters, and further implemented

the automatic generation of the piping system topol-

ogy. Since this part is not the focus of this paper, the

speciﬁc classiﬁcation implementation method will not

be covered in detail here.

4.2 Implementation of the Semantic

Layer

Here, we give the general procedure for implementing

the semantic layer, as shown in Figure 3. Ontology

design is subjective, which means that the ontology

tions of concepts and properties. A good ontology

taxonomy should be intuitive, and its internal rela-

tionships should be easy to grasp and explore (Häfner,

2019). In order to standardize the ontology design as

much as possible to facilitate the subsequent merging

and mapping of different ontologies, the guidelines

proposed by Häfner (Häfner, 2021) can be referred to.

The Web Ontology Language (OWL) is a prominent

ontology description language, widely utilized in var-

ious domains. Speciﬁcally, OWL ontologies can be

effectively modeled using tools like Protégé, an open-

source graphical ontology development tool that sig-

niﬁcantly simpliﬁes the process, enhancing accessi-

bility for researchers and practitioners (Musen, 2015).

In addition, the tool has a built-in reasoner, whose

main function is to infer the hierarchical structure of

classes, i.e. concepts, and to check the consistency of

ontologies in the design phase.

Analyzing and determining all the domains

Figure 3: General procedure for implementing the semantic

layer (adapted from (Häfner, 2021)).

needed to implement the tutoring system is the

premise of constructing ontologies in the semantic

layer, and it is also the ﬁrst step in constructing the

semantic layer. This paper takes the intelligent VR

tutoring system of the ﬂow rig mentioned in the previ-

ous section as an example, and brieﬂy analyzes which

domain ontologies are required for its semantic layer.

First, the content and objectives of the tutoring should

be clearly deﬁned. While the ﬂow rig is a versatile

tool with applications in research, experimentation,

analysis, testing, and validation across engineering,

ﬂuid dynamics, thermodynamics, and more, its pri-

mary function remains in education. It helps teach

students the control principles and methods of basic

process parameters such as ﬂow rate, level, volume

and differential pressure in the piping system, as well

as the operation methods of common control com-

ponents, equipment and measuring instruments, such

as the starting steps of the pump, the measurement

method of the ﬂow meter, etc. More complexly, it can

be used to teach the setting of a single closed-loop

level/ﬂow control system, the test method of the step

response characteristics of the tank level, and the test

method of the ﬂow characteristics of the valve, etc.

This type of learning is called skill training. The vast

majority of skill training tasks for machines are pro-

cedural, which means that the task can be represented

as an ordered sequence of behaviors or actions.

Therefore, based on the above analysis and the

content of the preceding sections, we preliminarily

concluded that the semantic layer of the ITS ought

CSEDU 2024 - 16th International Conference on Computer Supported Education

146

Figure 4: Flow rig virtual twin ontology.

to comprise four key domain ontologies. First among

these is the virtual twin ontology, serving to elucidate

information about each component within the ﬂow rig

and the interrelationships between these components.

The second is the behavior ontology, describing the

interactions available to students, along with the cor-

responding execution rules for these interactions on

the virtual twin components. The third, the tutor-

ing task ontology, is used to describe the procedural

task content, task evaluation and user feedback con-

tent. Lastly, there is a skill ontology, responsible for

recording the learning intensity and mastery of vari-

ous skills, which is used to support the adaptive gen-

eration of tutoring tasks.

The component class hierarchy deﬁned in the ﬂow

rig virtual twin ontology is shown in Figure 4. Deﬁne

a set of properties for each component class, which

can be divided into static properties to record basic

component parameters such as the volume of the tank,

the oriﬁce diameter of the valve, etc., and dynamic

properties to capture the current state of the compo-

nent such as the opening degree of the valve, the level

of the tank, etc. In the system initialization stage, the

component instances of the ﬂow rig virtual twin es-

tablished in the previous section and their initial se-

mantic information are populated into the virtual twin

ontology knowledge base, where the basic structure

of representing facts is a triple based on Resource De-

scription Framework (RDF), that is, the format of sub-

ject (instance of class) - predicate (property of class)

- object (property value) (Pan, 2009). Take the on-off

valve class as an example, which includes the oriﬁce

diameter property and the switch position property to

indicate the state of the on-off valve. Their range

classes are Decimal class and Boolean class respec-

tively. After system initialization, several fact asser-

tions about speciﬁc on-off valves will be generated in

the knowledge base. Take one of them as an example

and use the RDF graph to represent it as shown in Fig-

ure 5. Each entity in the ontology (including classes,

properties, and instances) has a unique International-

ized Resource Identiﬁer (IRI) used for reference pur-

poses. After mapping the ﬂow rig virtual twin to the

ontology, an automatic parametric simulation model

is formed. Driven by the reasoner and behavior ontol-

ogy rules, the ﬂow rig virtual twin enables automatic

responses to user interactions.

Figure 5: RDF graph describing the on-off valve instance

OnOffValve.1.

The purpose of constructing the behavior ontology

is, on the one hand, to endow the basic interface inter-

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology

147

action between users and objects (such as click, drag

and drop, etc.) with speciﬁc semantics, which helps

the tutoring system meaningfully understand user be-

havior and make evaluations, and on the other hand, to

realize the impact of user behavior on the virtual en-

vironment by using ontology rules. In addition, there

may be more than one semantic operation that can be

performed on an object, which should be explicitly

presented and chosen by the user when manipulating

the object. For example, when operating the pump of

the ﬂow rig, in addition to the operation to start the

pump, it should also include the operation to check

the submergence degree of the pump, which should

be performed before starting the pump. By classify-

ing various object operations, we consider construct-

ing a behavior ontology composed of general behav-

ior classes and object-speciﬁc behavior classes. The

deﬁnition of general behavior classes helps the be-

havior ontology to be reused in other machine train-

ing, as well as to keep the ontology simple. Gen-

eral behaviors include: Open, Close, Adjust, Press,

Check, Record, etc. The object-speciﬁc behavior can

be adding water to the tank, checking the submer-

gence depth of the pump, etc. Based on the needs

of rule reasoning, deﬁne a set of properties for the

behavior class to capture information associated with

the behavior instance. For example, the Open class

has an openedObject property, whose value can be a

certain on-off valve or the pump; in addition to having

an adjustedObject property, the Adjust class should

also have a value property that records the ﬁnal value

of the adjustment, such as the opening adjustment of

the control valve. When the user performs a behav-

ior on a component object of the virtual twin, a new

behavior instance and its property assertions will be

created, which will be populated into the knowledge

base in the form of RDF triples. Finally, the ontology

rules describe the effects of different behaviors on the

virtual twin. Ontology rules are also a kind of for-

malized knowledge in essence, which is represented

based on if-then logic. Semantic Web Rule Language

(SWRL) (Horrocks et al., 2004) is used to model on-

tology rules. Taking the adjustment of the control

valve as an example, the behavior rule is modeled as

follows:

• adjustedObject(adjustInstance, ?cv)^

value(adjustInstance, ?val)^

openingDegree(?cv, ?opd)^

swrlb:notEqual(?val, ?opd)->

openingDegree(?cv, ?val)

The prerequisite for the above rule to be true is

that the expression on the left side of the arrow “->”,

that is, the if statement, is true. The part on the right

side of the arrow represents the result of the rule, that

is, the then statement. While generating an instance

of the Adjust class (in this case replacing the instance

name with the parameter adjustInstance), the above

rule is created dynamically and the reasoner is trig-

gered to reason based on the rule. The symbol “^”

represents the logical operation “and”, linking mul-

tiple subexpressions. The expressions adjustedOb-

ject, value, and openingDegree are all property ex-

pressions. Their ﬁrst parameter represents the indi-

vidual being tested, and the second parameter repre-

sents the property value of the individual. The SWRL

syntax uses the form of a question mark plus a char-

acter to represent a wildcard, which is used to dy-

namically bind each property value of the individual.

The swrlb:notEqual is a built-in function of SWRL

to determine whether two parameters are unequal.

On the result side of the rule, the property expres-

sion openingDegree is used as a property assertion

to set the openingDegree property value of the indi-

vidual bound to the ﬁrst argument (that is, the control

valve) to the value bound to the second argument (that

is, the adjusted value of the Adjust behavior). Due

to the monotonic reasoning characteristics of SWRL,

SWRL rules can only add new reasoning informa-

tion to the ontology but cannot modify the existing

information of the ontology. Therefore, SPARQL,

short for “SPARQL Protocol and RDF Query Lan-

guage”(World Wide Web Consortium, 2013) can be

used to delete old axioms. The overall meaning of this

behavior rule is that if the user’s adjustment behavior

does change the opening degree of the control valve,

then update the value of the opening degree property

of the control valve in the virtual twin ontology. Once

the application logic learns through queries that the

information in the virtual twin ontology has changed,

it will re-run the ﬂow rig parametric simulation model

to update the virtual environment.

The details and functions of the other two ontolo-

gies will be explained in conjunction with the content

of the next section.

4.3 Intelligent Tutoring System

This section summarizes speciﬁc requirements for

setting up an ITS and delivers some insights into their

realization. As mentioned in Section 3, in our pro-

posed framework, the functions of an ITS mainly in-

clude generating tutoring tasks, evaluating tasks in

real time, and generating user feedback. These three

functional modules are described in turn as follows.

First, for the tutoring task generation module, it

should be able to adaptively generate the most suit-

able tutoring task considering the student’s skill level,

proﬁciency in skills and forgetting curve. In order

to evaluate the skill level of the students, a simple

CSEDU 2024 - 16th International Conference on Computer Supported Education

148

method is proposed, which is to divide the skills re-

quired by students into different levels through a top-

down analysis. The learning of a higher-level skill is

based on the mastery of all lower-level skills. Taking

the training of the ﬂow rig as an example, the premise

for students to learn process parameter control strate-

gies is that they have been able to skillfully operate

various control components, equipment and measur-

ing instruments. With this approach, a student’s skill

level can be obtained by analyzing his learning situ-

ation of all tasks at each level, which is recorded in

the user proﬁle. As long as there are tasks at a certain

level that are not fully mastered, meaning that the stu-

dent is still at that skill level, new tasks generated by

the tutoring system should also be at that level. The

tasks generated based on the student’s skill level pre-

vent students from being frustrated by the excessive

burden during the training process or losing interest

in learning due to tasks that are too easy.

For the evaluation of skill proﬁciency, it can be

realized by formulating a user performance-based

penalty rule, and taking the task score obtained based

on the rule as the user proﬁciency of the skill cor-

responding to the task. The user’s score for each

task will be updated in the user proﬁle. The tutor-

ing system learns the user’s proﬁciency in each skill

by checking the latest scores for each task. A full

score means complete mastery. In addition, the tutor-

ing system should also consider the forgetting curve

of knowledge when generating tasks, that is, con-

sider the time factor. Periodically repeating tasks

with perfect proﬁciency and re-evaluating the user’s

skill performance can effectively combat the forget-

ting curve. Therefore, the learning intensity of each

skill expressed as the learning interval should also be

recorded in the user proﬁle and updated regularly.

It should be noted that all the tasks that the tu-

toring system can provide, and their corresponding

scenes, should be deﬁned in advance in the underlying

data model of the task generation module. A task se-

lection strategy that takes into account the user’s skill

level, proﬁciency, and timely spaced repetition needs

to be designed and integrated. Before each task is

generated, the module will read the relevant informa-

tion of each skill from the skill ontology for use by

the task selection algorithm. This is exactly why we

created the skill ontology in the semantic layer of the

tutoring system of the ﬂow rig, as described in Sec-

tion 4.2. A preliminary and simple skill ontology for

the ﬂow rig training can be constructed as shown in

Figure 6. Each skill is endowed with proﬁciency and

learning intensity properties. During system initial-

ization, the proﬁciency and learning intensity infor-

mation of the skills from the user proﬁle will populate

the skill ontology. In fact, in addition to the properties

directly related to task selection, there are some prop-

erties in the ontology that are deﬁned due to the need

to build rule reasoning, which are ﬂexibly added dur-

ing the rule design process. The new tasks are inferred

based on the designed ontology rules. The informa-

tion of the newly generated tasks will also be updated

in time to the tutoring task ontology. A preliminary

tutoring task ontology for the ﬂow rig is presented in

Figure 7. All tutoring tasks for the ﬂow rig are per-

formed in the same scene.

Figure 6: Skill ontology for ﬂow rig training.

Figure 7: Tutoring task ontology.

Secondly, for the task real-time evaluation mod-

ule, its main goal is to segment the user’s interaction

with the immersive environment in real time, espe-

cially for continuous interactions, identify the user’s

misbehavior, and ultimately evaluate the proﬁciency

of the skill. For example, in a driving simulation, the

computer should segment the user’s driving behavior

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology

149

in real time into a sequence of discrete atomic behav-

iors or events (e.g. acceleration, shifting, etc.) based

on the time series of the user’s hardware input data

(e.g. the simulator’s pedal signals, steering wheel sig-

nals, etc.) and the driving simulation output data. For

the ﬂow rig example above, the user’s interaction with

the ﬂow rig virtual twin is inherently discrete, so it

does not need to be segmented. And since each user’s

behavior will be recorded in the behavior ontology,

the user’s behavior can be easily traced. Since the task

is also typically represented as a sequence of behav-

iors or events and is pre-stored in the semantic layer,

the reasoner can detect whether the user is misbehav-

ing based on predeﬁned ontology rules. Detections

can be event-triggered or periodically triggered. Here,

based on the example of the control valve adjustment

behavior in the previous section, the following sim-

ple example is given to show how the semantic layer

implements the evaluation of user behavior, which is

implemented based on SWRL and SPARQL.

• order(adjustInstance, ?ord1)^

type(adjustInstance, ?t1)^

adjustedObject(adjustInstance, ?obj1)^

value(adjustInstance, ?val)^

TaskBehavior(?tb)^order(?tb, ?ord1)^

type(?tb, ?t1)^adjustedObject(?tb, ?obj1)^

value(?tb, ?val)->

isCorrect(adjustInstance, true)

• SELECT (COUNT(?x) AS ?c) WHERE

{adjustInstance isCorrect ?x}

The ﬁrst one is the behavior evaluation rule based

on SWRL, which is triggered immediately after the

user performs the behavior. As shown in Figure 7, the

TaskBehavior class is deﬁned in the tutoring task on-

tology, which is a subclass of the behavior class. Each

expected behavior in the task behavior sequence is an

instance of this class, and each behavior instance has

an order property value that represents the position of

the behavior instance in the behavior sequence. The

type property is used to indicate the type of the be-

havior. The isCorrect property indicates whether the

user is performing the correct behavior, which is not

asserted by default. The evaluation logic of this rule

is to compare the behavior performed by the user with

the expected behavior at the same position in the task

behavior sequence. If the two are completely consis-

tent, the reasoner asserts that the user’s current behav-

ior is correct.

The second one is the SPARQL query statement,

which is used to output the reasoning results of the

reasoner. It can also be implemented using Seman-

tic Query-Enhanced Web Rule Language (SQWRL)

(O’Connor and Das, 2009). The WHERE part de-

ﬁnes the pattern to be matched in the query, which

is based on the RDF triple structure. The wildcard ?x

dynamically binds the value obtained through pattern

matching. For brevity, the preﬁxes of the behavior in-

stance adjustInstance and property isCorrect, which

represent the namespace, are omitted here. The SE-

LECT part determines what data to output. Here, we

use the COUNT function to calculate the number of

?x that satisﬁes the pattern, and assign it to the wild-

card ?c for output. Since the isCorrect property of a

behavior instance is not asserted by default, when the

output value is 0, it indicates that the current behavior

is wrong; when the output value is 1, it indicates that

the behavior is correct. SPARQL queries can also be

used to implement prompts for task information.

To be sure, different developers deﬁne the rules

differently. The evaluation results will also popu-

late the tutoring task ontology for use by the user

feedback generation module. It should be noted that

tasks should be deﬁned with consideration that there

may be more than one correct sequence of user be-

haviors, which can support students in exploratory

learning and thus promote their understanding and

mastery of skills (Aleven et al., 2009). Moreover,

in order to simplify the task authoring process, it is

promising to arrange the task behavior sequence by

directly letting tutors to demonstrate in the virtual en-

vironment, called authoring-by-doing (Richard et al.,

2021a). Based on the user’s performance, this mod-

ule will ultimately evaluate the user’s proﬁciency in

the currently trained skill and update it with the skill

learning intensity to the skill ontology. Once the tu-

toring concludes, all skill-related information will be

saved in the user’s proﬁle for future reference.

Finally, the main goal of the user feedback gen-

eration module is to present the prompts of task in-

formation and the real-time evaluation results of the

evaluation module to the user. The presentation can

be visual or audio output. During the ﬁrst training of

the task, the system should show the user the recom-

mended and detailed task solution, and provide the

most detailed error feedback in real time, including

the wrong behavior and the corrective solution. In or-

der to promote students’ mastery of skills, the sys-

tem should adaptively increase the difﬁculty of the

task with the improvement of users’ skill proﬁciency,

which can be achieved by reducing the level of de-

tail of prompts and feedback. To achieve this, a va-

riety of feedback forms with different levels of detail

can be pre-designed, and difﬁculty levels deﬁned for

them. When a new task is generated, its difﬁculty

level is set, which will be recorded in the tutoring

task ontology. Based on the difﬁculty property value

of the task and real-time evaluation results, the feed-

back generation module generates feedback content

with the corresponding level of detail with the help of

CSEDU 2024 - 16th International Conference on Computer Supported Education

150

the reasoner, which will also be populated in the tutor-

ing task ontology for visual or audio output module to

read. There should also be the possibility for users to

manage their training themselves, including task se-

lection and difﬁculty settings.

5 CONCLUSION AND FUTURE

WORK

This paper proposed a framework of Semantic Web-

Enabled Intelligent VR Tutoring System, and ex-

pounded the implementation methods of the three

core modules of the framework in combination with

the ﬂow rig use case. With regard to the development

of an immersive VR learning environment, the prob-

lems and expected solutions in implementing the vir-

tual twin for training were discussed, and it was em-

phasized that automating the virtualization process is

helpful to accelerate the development of training ap-

plications in the ﬁeld of mechanical engineering.

An intelligent VR tutoring system based on this

framework can quickly deploy personalized training

content according to the user’s context and provide

real-time evaluation and feedback on user behavior,

which will help improve the effectiveness of VR train-

ing. The application of the ontology-driven method-

ology for updating training content is expected to

lower the cost of setting up virtual environments for

all training scenarios, which will be evaluated experi-

mentally in the future. Indeed, the proposed frame-

work can be extended to the development of skills

training applications in other ﬁelds beyond engineer-

ing education, such as driving, construction, emer-

gency scenarios, and more.

In the future work, the details of functional imple-

mentation of each module in the framework will be

studied, such as the design method of each domain

ontology in the semantic layer, the method of gen-

erating task behavior sequence through authoring-by-

doing (Richard et al., 2021a), the task selection al-

gorithm based on user context, and the adaptive user

feedback, etc. The proposed framework and method-

ology will be utilized for the ﬂow rig immersive train-

ing application, along with more complex use cases.

Finally, in-depth user tests will be conducted to fully

evaluate our research results.

ACKNOWLEDGEMENTS

This work is funded by the China Scholarship Coun-

cil (CSC) under grant No. 202206890036, and the

Ministry of Science, Research and Arts of the Federal

State of Baden-Württemberg, Germany under Ko-

labBW project within the InnovationCampus Future

Mobility (ICM).

REFERENCES

Aleven, V., Mclaren, B. M., Sewall, J., and Koedinger, K. R.

(2009). A new paradigm for intelligent tutoring sys-

tems: Example-tracing tutors. International Journal

of Artiﬁcial Intelligence in Education, 19(2):105–154.

Benferdia, Y., Ahmad, M. N., Mustafa, M., and Ali, M.

A. M. (2021). The role of ontologies through the life-

cycle of virtual reality based training (VRT) develop-

ment process: A review study. International Jour-

nal of Advanced Computer Science and Applications,

12(9).

Bhagat, K. K., Liou, W.-K., and Chang, C.-Y. (2016). A

cost-effective interactive 3D virtual reality system ap-

plied to military live ﬁring training. Virtual Reality,

20:127–140.

Chang, M., D’Aniello, G., Gaeta, M., Orciuoli, F., Samp-

son, D., and Simonelli, C. (2020). Building ontology-

driven tutoring models for intelligent tutoring systems

using data mining. IEEE Access, 8:48151–48162.

Dale, E. (1969). Audiovisual Methods in Teaching. Dryden

Press, New York, 3

edition.

Elenius, D., Denker, G., and Kim, M. (2016). Seman-

tically enhanced virtual learning environments using

sunﬂower. In Proc. MTSR, pages 81–93. Springer,

Cham.

Filho, F. T., Aguiar, Y. P. C., and Vieira, M. F. Q.

(2015). Ontology based modelling of operator train-

ing simulator scenarios from human error reports. In

Proc. SIMULTECH, pages 261–270, Setúbal, Portu-

gal. SciTePress.

Filho, F. T. and Vieira, M. F. Q. (2014). An ontology-driven

framework to support scenario representation in a 3D

operator training simulator. In Proc. SIMULTECH,

pages 298–303, Setúbal, Portugal. SciTePress.

Gorecky, D., Khamis, M., and Mura, K. (2017). Introduc-

tion and establishment of virtual training in the factory

of the future. International Journal of Computer Inte-

grated Manufacturing, 30(1):182–190.

Gorecky, D., Loskyll, M., and Stahl, C. (2014). Semantic

digital factory–using engineering knowledge to create

ontologies for virtual training. IFAC Proceedings Vol-

umes, 47(3):7825–7830.

Häfner, P. (2020). Categorisation of the beneﬁts and lim-

itations of immersive technologies for education. In

Proc. MAS, pages 154–159, Rende, Italy. Cal-Tek.

Häfner, P. (2021). Holistic Approach for Authoring Immer-

sive and Smart Environments for the Integration in En-

gineering Education. PhD thesis, Karlsruhe Institute

of Technology, Germany.

Häfner, V. (2019). PolyVR - A Virtual Reality Authoring

Framework for Engineering Applications. PhD thesis,

Karlsruhe Institute of Technology, Germany.

A Framework for Intelligent Virtual Reality Tutoring System Using Semantic Web Technology

151

Havard, V., Jeanne, B., Savatier, X., and Baudry, D. (2017).

Inoovas-Industrial ontology for operation in virtual

and augmented scene: The architecture. In Proc.

CoDIT, pages 0300–0305, New York. IEEE.

Herbert, B., Ens, B., Weerasinghe, A., Billinghurst, M., and

Wigley, G. (2018). Design considerations for combin-

ing augmented reality with intelligent tutors. Comput-

ers & Graphics, 77:166–182.

Horrocks, I., Patel-Schneider, P. F., Boley, H., Tabet, S.,

Grosof, B., and Dean, M. (2004). SWRL: A seman-

tic web rule language combining OWL and RuleML.

https://www.w3.org/submissions/SWRL/. Accessed:

2023-09-01.

Häfner, P., Häfner, V., Wicaksono, H., and Ovtcharova, J.

(2013). Semi-automated ontology population from

building construction drawings. In Proc. KEOD,

pages 379–386, Setúbal, Portugal. SciTePress.

Häfner, V. (2017). Modelling smart virtual en-

vironments. In 4. Fachkonferenz zu VR/AR-

Technologien in Anwendung und Forschung an der

Professur Werkzeugmaschinen und Umformtechnik,

pages 151–162, Chemnitz, Deutschland. Technische

Universität Chemnitz. https://var2.org/downloads/

Tagungsband-VAR2-2017_digital.pdf.

Häfner, V., Benedix, A.-C., and Häfner, P. (2020). Au-

tomatisierung des virtualisierungsprozesses im anla-

genbau. Zeitschrift für wirtschaftlichen Fabrikbetrieb,

115(3):148–152.

Jacinto, A. S. and Oliveira, J. M. P. (2008). An

ontology-based architecture for intelligent tutoring

systems. Interdisciplinary Studies in Computer Sci-

ence, 19(1):25–35.

Laine, J., Lindqvist, T., Korhonen, T., and Hakkarainen, K.

(2022). Systematic review of intelligent tutoring sys-

tems for hard skills training in virtual reality environ-

ments. International Journal of Technology in Educa-

tion and Science, 6(2):178–203.

Michels, F. L. and Häfner, V. (2022). Automating virtu-

alization of machinery for enabling efﬁcient virtual

engineering methods. Frontiers in Virtual Reality,

3:1034431.

Muñoz Merino, P. J. and Kloos, C. D. (2008). An archi-

tecture for combining semantic web techniques with

intelligent tutoring systems. In Proc. ITS, pages 540–

550. Springer, Berlin, Heidelberg.

Musen, M. A. (2015). The protégé project: A look back and

a look forward. AI Matters, 1(4):4–12.

OASIS Standard (2015). MQTT Version 3.1.1. https://docs.

oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html.

Accessed: 2023-07-03.

O’Connor, M. and Das, A. (2009). SQWRL: a query lan-

guage for OWL. In Proc. OWLED, page 208–215,

Aachen, Germany. CEUR-WS.org.

Pan, J. Z. (2009). Resource Description Framework, pages

71–90. Springer, Berlin, Heidelberg.

Richard, K., Havard, V., and Baudry, D. (2021a).

Authoring-by-doing: An event-based interaction

module for virtual reality scenario authoring frame-

work. In Proc. AVR, pages 519–527, Cham. Springer.

Richard, K., Havard, V., His, J., and Baudry, D. (2021b).

INTERVALES: INTERactive Virtual and Augmented

framework for industriaL Environment and Scenarios.

Advanced Engineering Informatics, 50:101425.

Ruthenbeck, G. S. and Reynolds, K. J. (2015). Virtual real-

ity for medical training: the state-of-the-art. Journal

of Simulation, 9(1):16–26.

Schiavi, B., Havard, V., Beddiar, K., and Baudry, D. (2021).

Semi-automatic generation of virtual reality procedu-

ral scenarios for operation in construction based on

4D building information models. In Proc. CONVR,

pages 104–111, Middlesbrough, UK. Teesside Uni-

versity Press.

U.S. National Academy of Engineering (2017).

NAE grand challenges for engineering.

http://www.engineeringchallenges.org/File.aspx?

id=11574&v=34765dff. Accessed: 2023-09-14.

Vaughan, N., Dubey, V. N., Wainwright, T. W., and Middle-

ton, R. G. (2016). A review of virtual reality based

training simulators for orthopaedic surgery. Medical

engineering & physics, 38(2):59–71.

Vesin, B., Ivanovi

c, M., Klašnja-Mili

cevi

c, A., and Budi-

mac, Z. (2012). Protus 2.0: Ontology-based seman-

tic recommendation in programming tutoring system.

Expert Systems with Applications, 39(15):12229–

12246.

Walczak, K., Floty

nski, J., Strugała, D., Strykowski, S.,

Soboci

nski, P., Gał ˛azkiewicz, A., Górski, F., Bu

n, P.,

Zawadzki, P., Wielgus, M., and Wojciechowski, R.

(2020). Semantic modeling of virtual reality training

scenarios. In Proc. EuroVR, pages 128–148. Springer,

Cham.

Wen, Y., Zhu, X., and Zhang, L. (2022). CQACD: A

concept question-answering system for intelligent tu-

toring using a domain ontology with rich semantics.

IEEE Access, 10:67247–67261.

Wicaksono, H., Dobreva, P., Häfner, P., and Rogalski, S.

(2013). Ontology development towards expressive

and reasoning-enabled building information model for

an intelligent energy management system. In Proc.

KEOD, pages 38–47, Setúbal, Portugal. SciTePress.

World Wide Web Consortium (2013). SPARQL 1.1 Query

Language. https://www.w3.org/TR/sparql11-query/.

Accessed: 2023-08-25.

Zahabi, M. and Abdul Razak, A. M. (2020). Adaptive vir-

tual reality-based training: a systematic literature re-

view and framework. Virtual Reality, 24:725–752.

CSEDU 2024 - 16th International Conference on Computer Supported Education

152