Integration of Virtual Reality with Intelligent Tutoring for High

Fidelity Air Traffic Control Training

Alvin T. S. Chan

, Peng Cheng Wang

, Frank Guan

, Saw Han Soo and Haris Lim Hao Li

Singapore Institute of Technology, 10 Dover Dr, Singapore 138683, Singapore

Keywords: Virtual Reality, Air Traffic Control, Speech Recognition, Intelligent Tutoring System.

Abstract: Air traffic control plays a significant role and service by ground-based air traffic controllers (ATC) in

providing specific and clear advisory guidance to pilots at every stage of a flight. Specifically, the air traffic

control purpose is to ensure safety procedures and protocols are adhered to avoid collisions, and to ensure

organized and systematic flow of air traffic on the ground and in the air. In this paper, we present the design

and implementation of an immersive and collaborative Virtual Reality (VR) training platform that is scalable

and cost-effective compared to traditional method of training ATCs in a physical mock-up of a 360-degree

air control tower simulator. The use of immersive VR technology through Head Mounted Display (HMD)

would not only solve the space constraints but also immerse users in their tasks while supporting better

management and analysis of the complex data produced during training. Through the integration of intelligent

tutoring that actively tracks the training progress of the trainee, the system facilitates personalized training

that has been shown to significantly improve the learning experience.

1 INTRODUCTION

Virtual reality (VR) is increasingly being adopted as

a technology choice to enable users to experience the

immersion of a virtual world through 3D near-

displays. The advance in technology allows VR to be

embedded into education curriculum, to train trainees

with varying competency levels (from novice to

experts) on complex and multi-faced scenarios within

a controlled and safe environment. There are many

advantages that come with a VR-enabled training

environment, such as enhancing learning engagement

and interactivity. providing diversified multi-modal

training (off site and on site) resulting in smaller

physical footprints and space use, and allowing

trainees multiple repeats of the training/procedures at

their own pace and time. The use of VR can also

circumvent the need of a large physical space for

training, or to reduce or eliminate the inherent risk to

the training, especially in the area of aviation training.

However, many of the VR training systems lack

architectural support for personalized training which

has been shown to significantly improve user’s

https://orcid.org/0000-0003-1196-4879

https://orcid.org/0000-0002-1796-1946

https://orcid.org/0000-0003-0549-142X

learning experience. In this paper, we describe the

development of a VR platform that reproduces an air

traffic control tower in an interactive and immersive

environment. It aims to seamlessly integrate the

benefits of intelligent tutoring to enable the system to

automatically track, guide and diagnose the trainee’s

progress in terms of his/her learning experience.

Based on the performance of the learners, the system

will progressively adapt the instructional activities to

ensure that proficient level of competencies is

achieved at every stage of the training.

2 BACKGROUND AND

CHALLENGES

This section highlights some past works that were

done in air traffic control training. Specifically, it

discusses the use of multiple fidelity simulations that

help orientate trainees in understanding air traffic

control processes and activities, including voice

phrases to use in guiding pilots when landing and

Chan, A., Wang, P., Guan, F., Soo, S. and Li, H.

Integration of Virtual Reality with Intelligent Tutoring for High Fidelity Air Trafﬁc Control Training.

DOI: 10.5220/0011732200003470

In Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023) - Volume 2, pages 199-206

ISBN: 978-989-758-641-5; ISSN: 2184-5026

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

199

taking-off. It highlights some challenges of existing

simulation methods and how VR presents an

attractive and cost-effective solution for personalized

training through the integration of intelligent tutoring

system.

2.1 Air Traffic Control Training and

Simulations

Air traffic control training covers a wide spectrum of

concepts and operations that collectively equip the

controllers to handle how air traffic flows are directed

and orchestrated. In addition to learning the academic

aspect of the work, they are required to perform on-

site facility training. Therefore, the use of simulation

in training becomes an important bridge to fill the gap

between academic theory and on-site facility training

in which real-time traffic data will be used and

procedures exercised. The simulation training in

tower traffic control needs to be highly visual and

immersive that closely matches on-site facility

training.

The training of ATCs goes through a four-stage

process. The air traffic academic course uses

classroom instruction to introduce the basic concepts

of aviation and air traffic control. The part-task

training consists of lectures and basic laboratory

activities. It introduces more complex aspects of air

traffic control with some hands-on activities using

low to medium fidelity simulations. In the next phase

of skill building training, trainees are exposed to high

fidelity simulation environment that closely

replicates the control tower in the form of 360-degree

air traffic control tower simulator similar. The final

stage of the training is to expose the trainees to the

on-site facility with close monitoring and

supervision.

2.2 Challenges

To train the air tower traffic controllers, the training

centre is equipped with an air traffic control training

tower setup that provides 360 degrees simulation of a

traffic control room (Aerospace Operations Division.

2018). The physical room is installed with 360

degrees projected screen of the airfield and its

surrounding. However, there are four main challenges

with these kinds of simulation-based training

systems:

• The size of the control room often limits the

number of air traffic control tower trainees that

can be trained at any point in time. As a result,

the trainees have limited hands-on learning in

such a highly skilled and detail-oriented role.

• The training is often restricted to allocated

pockets of slots, while the remaining time is

spent on academic aspects of the training.

• In the training control room, at least one

instructor needs to be present to coordinate the

training procedures and operations. He or she is

required to orchestrate complex scenarios of

plane landing, taxiing, and departing.

Additionally, unexpected scenes such as change

of weather or accidents may be injected to the

scenario to simulate unexpected turn of events.

All these translate to the need for the instructor

to be present which may form a bottleneck in the

training process which does not facilitate a

trainee to practice independently.

While the control room provides realistic and

high-fidelity simulation of the air traffic and control

in a classroom setting, it does not track or monitor the

learning progress of trainees. As such, personalized

instructional quality of the training may be lacking.

3 IMMERSIVE SOLUTION FOR

AIR TRAFFIC CONTROL

TRAINING

Air traffic control is a demanding work requiring

intensive training in an immersive environment that

simulates the actual on-site facility. Although the

360-degree air traffic control tower provides an

immersive environment and experience to train the

controllers, the prohibitive cost and limited

availability of such facilities make it not readily

accessible to the trainees. As such, there is a need to

find new techniques that bridge the technological gap

of providing immersive experience in training air

traffic control tower trainees and making the solution

scalable and accessible.

3.1 Virtual Reality Platform

In this project, we aim to build a VR platform that

reproduces an air traffic control tower in an

interactive and immersive environment. Importantly,

the platform aims to enable anytime, anywhere and

anyplace access to hands-on training in a distributed

and multisensory operating environment. Equally

important, the platform aims to replicate the actual

learning environment as much as possible to ensure

learning is not compromised.

Air traffic control is a highly skilled and spatial

work. In addition to equipping Singapore with state-

of-the-art research in this area, we believe it is equally

CSEDU 2023 - 15th International Conference on Computer Supported Education

200

important for us to beef up on the training of the

workforce to cope with the increased number of

flights and traffic as Singapore continues to position

itself as the transport hub in the region.

4 BENEFITS

On-the-job-training alone is one model that does not

fit well with air traffic control training. Instead, the

use of advanced simulation devices and setup in a

controlled environment is required and utilized in

every phase of air traffic control training programs.

Specifically, the following are the benefits:

• Scalability and Low Cost. The number of

available simulators will eventually affect the

number of air traffic controllers that can be

trained. In order to meet the demand, current

simulation practices and equipment need to be

evaluated.

• Sustainable High-Fidelity Simulations and

Training. Training like the 360-degree air control

tower requires high setup cost and space which

can create a bottleneck to high-fidelity training.

In addition, these systems are often expensive to

run and maintain which will make maintenance

and simulation updates more challenging.

• Self-paced and Independent Learning. Projected

increases in air traffic controllers will require

innovative ways to provide quality training that

supports intelligent and personalized tutoring.

The VR system may employ speech recognition

and synthesis to provide guided instruction in

training complex scenarios without the physical

presence of the instructors.

5 SYSTEM ARCHITECTURE

Shown in Figure 1 is the system architecture of the

implementation setup that encompasses the

intelligent tutoring system and the virtual reality

simulation engine. The simulation engine is

comprised of the standalone VR headset that is worn

by the trainee. Instead of using the built-in speakers

and microphone of the Oculus Quest VR headset, the

trainee is required to wear a Bluetooth-based headset

to ensure high fidelity reception of sound and pickup

of voice signals. Audio signals from the headset are

streamed directly to and from the desktop server via

Figure 1: System Architecture.

Integration of Virtual Reality with Intelligent Tutoring for High Fidelity Air Trafﬁc Control Training

201

Bluetooth protocol. To facilitate communication

between the traffic controller and pilot, the volume up

button of the VR headset is used as the push-to-talk

button to emulate switching from voice mode to

transmit mode. The VR headset is connected to the

desktop server via the Wi-Fi router to provide cross-

platform device communications through TCP/IP

communication protocols. AI software agents are

proxy agents that are used to generate responsive,

adaptive and behaviours of pilots as non-player

characters with human-like intelligence. These agents

that are hosted within the VR headset are controlled

by the intelligent tutoring system that is residing in the

desktop server. The limited computing capacity of the

VR headset is unable to handle speech recognition and

AI voice generation (Bakhturina et al., 2021). As such,

the digitized audio signals (streamed to and from the

Bluetooth headset) are processed within the desktop

server that is equipped with GeForce RTX 3080 GPU

processor to deliver high performance and near real-

time speech and voice processing. The programs

within the VR headset are created with C++ and

running in an Unreal engine environment while the

programs within the desktop server are created with

Python in a Linux environment. The setup allows a

single desktop server to serve multiple clients.

6 INTELLIGENT TUTORING

SYSTEM

Figure 2 above shows the Intelligent Tutoring

System, as a core component of the development that

drives the training processes and learning pedagogy

of the air traffic. This system can be viewed as a

‘human tutor’, backed up with a complex virtual

architecture (with data analytics) to address to the

needs of the learners. The system is comprised of

three essential modules that closely interact with one

another to enable personalized, incremental, and

adaptive learning of the training contents. These

modules aim to capture three types of knowledge that

include 1) the expert knowledge of the content

structure and its atomic content inter-dependencies 2)

knowledge about the trainee’s competencies and

progression, and 3) knowledge of the teaching

environment and learning pedagogy of the training.

In addition, the ITS is furnished with the user

interface module that agglomerates various user

interfaces to track and monitor trainee’s interaction

with the system through hand gesture tracking, gaze

tracking, voice interaction and text prompt, among

others.

(Yuce et al., 2019). The system is comprised of

three essential modules that closely interact with one

another to enable personalized, incremental, and

adaptive learning of the training contents.

6.1 Expert Knowledge Module

The expert knowledge module captures the lessons’

flow and contents from the perspective of a domain

expert. The domain expert in this case is the air traffic

controller training model that represents the lesson

content, facts, concepts, and training rules that would

progressively equip the trainee to acquire knowledge

and skills in the domain. The knowledge captured is

derived from a domain expert that is comprised of not

only the lesson content on Air Traffic Control but also

the possible phrases that are valid ATC phraseology.

Figure 2: Intelligent Tutoring System.

CSEDU 2023 - 15th International Conference on Computer Supported Education

202

Figure 3: Expert Knowledge Module.

Figure 3 shows how the lessons’ contents are

organized in the domain expert module. The logical

lesson progression is based on the concept of Zone of

Proximal Development in which trainees familiarize

themselves with basic taxi procedures, console

controls and the aerodrome environment to handling

difficult scenarios that involve conflicts and

accidents. The use of text prompts guides the trainee

on what phrases to use to respond to pilot. Each

scenario comprises of lesson steps which in turn

comprises of necessary information for the ITS to

facilitate users’ learning experience.

6.2 Trainee Model Module

An instance of the trainee’s profile is retrieved

whenever the trainee logs on to the system and starts

a lesson. If the trainee is newly recognized, a template

of the trainee model is created, and its environment

data is populated from that of the attributes from the

expert data together with the session data. The

environment data in the trainee’s profile helps the

expert knowledge module to understand the context

of the training and the trainee’s learning experience

and progress. For example, an important attribute

being tracked is the location of the aircraft that is used

as a signal to initiate pilot requests. This is because

each location in the aerodrome environment has a

specified phrase associated with the lesson type. The

session data stores all performance assessment data of

the trainee to be processed at the end of the session

6.3 Tutoring Module

Figure 4 shows how the tutoring module helps to

disseminate information from the expert knowledge

module to the user interface module and vice versa.

Specifically, it provides personalized tutoring to the

trainee based on the trainee profile and learning

progress. Essentially, it emulates a human-like tutor

to decide how to teach and what to teach as it

progressively equips the trainee with the necessary

knowledge and skills to become a competent air

traffic controller. There are three levels as below

where the trainee may fall under their skill sets.

• Trainee cannot accomplish the tasks with

assistance. This is when the tasks are outside the

zonal proximal development of the trainee in

which he/she will not be able to complete the

tasks even with guidance.

• Trainee can accomplish tasks with assistance.

With additional guidance, the trainee is close to

completing the tasks assigned and master a skill

set.

• Trainee can accomplish tasks without

assistance. Here, the trainee can complete tasks

by themselves and has mastered the skill set

required.

Figure 5 shows how trainee’s voice commands are

processed. Voice is transcribed into text by the speech

recognition engine residing in the desktop server. The

text is then corrected for any spelling errors and each

Integration of Virtual Reality with Intelligent Tutoring for High Fidelity Air Trafﬁc Control Training

203

Figure 4: Tutoring Module.

Figure 5: Trainee Solution.

CSEDU 2023 - 15th International Conference on Computer Supported Education

204

Figure 6: User Interface Module.

word is matched to the closest word that exist in our

vocabulary and validated as an ATC phrase (ICAO

Doc, 2016). If it is validated as a standard ATC

phrase, it is considered a trainee potential solution to

be then compared to the expert model solution

together with the ATC console settings. If the

solutions matched, the performance is recorded and

the lesson progresses. To contextualize and boost the

transcription accuracy of rare and domain-specific

words or phrases, the speech repository is in-house

trained with over 7-hours of voice audio data

collected.

6.4 User Interface Module

Figure 6 shows the user interface module that is

embedded with the necessary components for the

trainee to visually study the environment and

understand the situation with the ability to

communicate and interact with the server via hand

gesture tracking and push-to-talk voice. The user's

gaze is also tracked via the VR headset headtracking.

Since gazes are a natural and intuitive interaction

modality for human beings, it allows assessment of

trainee’s cognitive and visual orientation of the

aerodrome as aircraft lands or takes off. This would

allow the ITS to understand trainee’s level of

confidence, immersiveness and learning experience

as he/she navigates the air traffic through visual

tracking and communication with the pilot. We

believe that gaze-based interaction is one dimension

in user interface that could enhance trainee’s

experience by having the ITS track the visual

attention of trainee in air space from the control

tower. As the trainee progresses until all lesson steps

are completed, a report of the trainee’s performances

is generated for the trainee and ATC instructor to

review. The simulated environment includes a

panorama view of the airport runway, sight lines and

views to all parts of the airfield from the perspective

of the control tower. Within the control room, the

trainee is able to gain access to various dashboards

and control buttons to facilitate proactive monitoring

of aircrafts’ positions and control status. In addition,

it features an application control interface to enable

trainee to control, initiate and select the lesson for

training (Alkhatlan & Kalita, 2018). To begin a

lesson, the trainee is required to enter a login ID for

identification via the application control interface

which is routed to the desktop server for verification.

This allows the ITS to create a profile of the trainee

for performance assessment and recording. The

domain expert module selects a lesson scenario from

its lesson library database and sets up the

environment in the VR headset A pilot agent requests

a command from the trainee serving as the air traffic

controller. A problem is presented as audio request

from the ITS controlled pilot. In which case, the

trainee is required to appropriately respond to the

request by issuing a voice reply through the

microphone. After the trainee responds with a voice

command, the solution from the domain expert is

compared to that of the trainee. Text transcribed

through speech recognition engine is compared with

the expected user response to validate the voice

command from the trainee. If the voice command

given is valid, the pilot agent will respond

appropriately otherwise, the trainee is informed of

his/her error through the user interface.

Integration of Virtual Reality with Intelligent Tutoring for High Fidelity Air Trafﬁc Control Training

205

7 CONCLUSION

Simulation-based training has advanced rapidly over

the last decade as computing resources become more

available and powerful. This has accelerated the

advancement in immersive solutions to provide high

fidelity and real-time interactive simulations. Air

traffic control is a highly skilled and spatial-oriented

work that requires intensive training and hand-on

experience in which simulation is used extensively.

This project is innovative in bringing air traffic

control training to individual trainees in the form of

personalized training that can be self-paced and

directed towards independent learning. The

innovation applied in this project lies in the

aggregation of the following approaches:

• The first approach is to apply the concept of

composability in the design of the VR simulation

engine that supports flexible and configurable

objects. This would form the platform for the

system to support dynamic configuration of

training procedures and scenarios that leads to

individualized learning.

• The second approach builds on the first to

support multi-model training schemes. The first

scheme supports traditional approach of

classroom-based and instructor guided training,

while the second scheme automates the process

through computer guided training by integrating

domain expert knowledge and intelligent speech

processing.

• The third approach is to apply the concept of

intelligent tutoring system to automate

monitoring of trainee’s learning progress to adapt

instruction and contents to facilitate self-paced

and independent learning.

In our future publications, we plan to present details

of the system implementation and evaluation of

various modules, including user experience

assessment of the system.

ACKNOWLEDGEMENTS

The authors gratefully acknowledge the funding

support received from the Ministry of Education of

Singapore through the Translational R&D and

Innovation Fund, MOE-MOE2019-TIF-0009.

REFERENCES

Aerospace Operations Division. (2018). Augmented

Reality for Tower Control. Augmented Reality for

Tower Control. https://www.nlr.org/flyers/en/f337-

augmented-reality-for-tower-control.pdf

Alkhatlan, A., & Kalita, J. (2018). Intelligent tutoring

systems: A comprehensive historical survey with recent

developments. ArXiv Preprint ArXiv:1812.09628.

Bakhturina, E., Lavrukhin, V., Ginsburg, B., & Zhang, Y.

(2021). Hi-fi multi-speaker english tts dataset. ArXiv

Preprint ArXiv:2104.01497.

ICAO Doc. (2016). 10056 Manual on Air Traffic

Controller Competency-based Training and

Assessment First Edition—2017. In 2016. International

Civil Aviation Organization.

Yuce, A., Abubakar, A. M., & Ilkan, M. (2019). Intelligent

tutoring systems and learning.

CSEDU 2023 - 15th International Conference on Computer Supported Education

206