Symbolic AI for Crew Assistance: Using Ontologies in the Cockpit

Dargent Lauren and Girod Hervé

Dassault Aviation, 78 quai Marcel Dassault, Saint-Cloud, France

Keywords: Ontologies, Cockpit, Symbolic AI, Aircraft.

Abstract: This paper presents the use of knowledge-based technologies, ontologies, as an interesting way to create a

reasoning framework for the machine. Dassault Aviation is convinced that, for system automation, this

technique is complementary with data-driven approaches and enhances performances: while deep learning

algorithms and other machine learning techniques can provide “sensory services”, such as understanding

aural messages, understanding images, texts, interpreting low-level signals, etc., knowledge-based

technologies can provide the system a framework to ensure “cognitive services”, such as manipulating

concepts and reasoning. From Dassault Aviation’s perspective, both approaches are necessary to team the

system and the crew in tomorrow’s missions.

1 INTRODUCTION

Crew cockpits look more intuitive than before, for

instance with large screens replacing several cockpit

instruments in a « glass cockpit », but on the other

hand they display more information than ever. The

aircraft missions are becoming more complex too,

with the crew assuming more activities in a dynamic

environment, including many interconnected assets

(Le Gleut, R., Conway-Mouret, H., 2020). In

parallel, the automation of cockpits has led to a

significant reduction of the amount of hazards and

accidents within the past decades (Ministère chargé

des transports, 2019). However, it created more

complex design issues such as faulty human-system

interactions due to human errors, and more generally

human factors issues (Kharoufah, H. et al, 2018).

The complexity of the missions and systems

highlights the need for a human-centered approach

in cockpit design, and the need to switch to a new

paradigm for Human Machine Interaction: Human

Machine Teaming.

The concept of Human Machine Teaming

defines the relation between the crew and the system

as a collaboration paradigm, instead of supervisory

paradigm where the crew would be the only decision

maker (Walliser et al., 2019). Within this

framework, the crew and the system collaborate

towards a common objective and are able to jointly

allocate between them the tasks to be realized: the

system is able to understand the situation and decide

as a virtual team member (Madni et al., 2018). This

approach is particularly suited for complex and

dynamic environments with potentially high

workload, such as aircraft, and represents the next

step for future cockpits. In order to perform Human

Machine Teaming, we need to give the machine the

ability to understand the aircraft’s data flow and

reason on the associated concepts.

This paper presents the use of knowledge-based

technologies, ontologies, as an interesting way to

create a reasoning framework for the machine.

Dassault Aviation is convinced that, for system

automation, this technique is complementary with

data-driven approaches and enhances performances:

while deep learning algorithms and other machine

learning techniques can provide “sensory services”,

such as understanding aural messages,

understanding images, texts, interpreting low-level

signals, etc., knowledge-based technologies can

provide the system a framework to ensure “cognitive

services”, such as manipulating concepts and

reasoning. From Dassault Aviation’s perspective,

both approaches are necessary to team the system

and the crew in tomorrow’s missions.

This paper presents three use cases for ontology

technologies to assist the crew during a mission.

This paper outlines the problematics and benefits for

such technologies, identifies the incoming

challenges and provides recommendations for future

researches from Dassault Aviation’s point of view..

Lauren, D. and Hervé, G.

Symbolic AI for Crew Assistance: Using Ontologies in the Cockpit.

DOI: 10.5220/0011964000003622

In Proceedings of the 1st International Conference on Cognitive Aircraft Systems (ICCAS 2022), pages 88-91

ISBN: 978-989-758-657-6

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

2 USE CASES

2.1 Using Ontologies to Enable

Machine Reasoning

Ontologies are a computing concept, which model

concepts and their relationships, therefore modeling

the knowledge, which derive from these relations.

Web Ontology Language (OWL) (RDF 1.1 XML

Syntax - W3C Recommendation, 2014) is a family

of languages, which formalize ontologies, also

allowing to execute queries about the concepts that

use the SPARQL language (SPARQL Query

Language for RDF - W3C Recommendation, 2008).

Ontologies may look like structured databases like

SQL, but rather than monolithic structured database,

they encourage and facilitate the segregation of the

knowledge. This segregation simplifies how you

must evolve the modelling of the structure of the

ontology (the TBox) if you want to implement new

concepts. For instance, OWL ontologies encourage

to reuse Upper Ontologies in association with

domain ontologies about a specific domain, because

Semantic Web ontologies are built to easily allow

interoperability between ontologies (Doan, A.H. and

Halevy, A.Y., 2005). For example, if you use the

concept of time in your ontology, you will be able to

reuse the OWL Time ontology rather than

implement time concepts in your own Ontology.

All these characteristics allow very easily

reasoning using several ontologies « databases »

with loose coupling. For example, it should be

possible to associate an ontology about airports and

runways, another dealing with waypoints, another

dealing with ATM, and a specific ontology about the

aircraft itself (Best project, 2016). Semantic

reasoners engines (Bienvenu et al., 2020) can also be

used in an ontology engine to infer logical

consequences about facts, using first-order logics.

Building a general knowledge base that uses

several loosely coupled ontologies should also allow

to simplify the communication between the systems

and the crew (Ferrer, B. R. et al, 2021). Suppose for

example that the crew desires to find the nearest

airport on which the aircraft could land, with a

suitable weather condition. It would be possible to

interrogate the airports ontology to detect the nearest

airport, adding filters in the request to consider only

airports that have a suitable landing runway. For

example, the following diagram presents an OWL

ontology with concepts about aircrafts, waypoints

and weather information on a waypoint using the

METAR format.

Figure 1: Example of a domain ontology in the context of

an aircraft.

2.2 Using Ontologies to Enable Crew

Machine Dialogue

Another usage of ontology technologies appears to

be interesting for cockpit applications, in order to

support the crew: the natural language

understanding.

Future aircrafts will be integrated into more

complex combined air operations or civilian air

operations, including heterogeneous assets

(unmanned aircrafts, heterogeneous manned

aircrafts, aircraft controls, etc.), and linked to more

networks and data (radio, transponders, satellite

communications, datalink, etc.). We consider

enabling natural language dialogue between the

system and the crew as a key enabler to navigate and

“dig” within these flows of data, and to allow the

crew interacting in a more complex manner with

their system.

Natural language processing technologies are

also widespread for everyday usages, with the

expansion of personal assistants. As these

applications rely on everyday usages, these

products’ developers were able to gather huge

labelled databases to train data-driven algorithms.

However, for cockpit conversational assistants, the

“natural” language relies on specific operational

vocabulary and specific syntaxes, and fewer data is

available for the training. Moreover, this vocabulary

is dynamically updated during the different

operational missions (for instance, waypoint names,

cities, aircraft labels, mission code-names, etc.).

Symbolic AI for Crew Assistance: Using Ontologies in the Cockpit

Knowledge based technologies, and especially

ontologies, appear to be an interesting alternative to

data-driven techniques for natural language

understanding in aircraft operations:

• As expert-domain models, they require less data

to train: using an ontology allows to quickly

generate a knowledge base for the

conversational engine without having to gather

and label thousands of sentences.

• They are more versatile than other technologies

(neural-network, decision trees, etc.): it is easy

to update the elements of the ontology (intents,

concepts, vocabulary) and thus to extend the

dialogue perimeter of the embedded assistant

without re-training the module.

• They are more robust to specific syntaxes used

in aeronautic operations.

Figure 2: Crew requests examples using “aeronautical”

language.

For instance, if a crew wants to address the

following requests to their system: “Where is the

closest runway?”, then we would need to model the

concepts: “airport”, the parameter “closest” and the

intents “retrieve”.

Figure 3: Extract of the mapping of one sentence on a

dialogue ontology.

The ontology technologies are thus a very

promising technology to assist the crew and enable

crew-system dialogue, using natural language.

2.3 Mapping Heterogeneous Ontologies

for Databases Interoperability

Ontology matching is a key subject for future

aircraft implementation.

For instance, even if the ontology used for the

crew dialog represents the same concepts as the

domain ontology used for reasoning, these two

ontologies might not be identical.

In this case, we therefore need to be able to

convert the request issued from the dialog ontology

to a request applied on the domain ontology.

Figure 4: Mapping between a dialog ontology and a

domain ontology.

This is a use case for ontology matching (also

called ontology alignment): semantic integration

research in the database community (Shvaiko and

Euzenat, 2013). The general case for Ontology

matching is complex, but in this case, the Dialog

Ontology represents part of the concepts of the

Domain Ontology, sometimes with a simplified

relationships graph. Therefore, it is simpler to

convert a query made on the first ontology to a

query applied on the latter.

Furthermore, it can be possible to populate an

ontology with the result of the communication

between the system and the crew. For example, in

the CAB project, the system should learn from the

interactions with the user (CAB project – Cockpit et

Assistant Bidirectionnel, 2021): it will both assist

the user if he asks questions about the situation, but

also will update its internal database depending on

its interaction with the user.

3 DIRECTION FOR RESEARCH

Ontology, and all technologies related to “expert

systems” and knowledge modeling are less

“trending” these last years in regards of the

exponential expansion of research on data-driven

technologies. However, Dassault Aviation strongly

believes that they are essential to the next generation

of cockpits, where the machine will team with the

crew (analyse and interprete the data, understand,

reason and advise the crew, manage the tasks…) and

not only execute the crew commands. The use cases

described above are three examples of ontologies

applications to assist the crew. They were studied

during the Man Machine Teaming project (Direction

Générale de l’Armement, 2019), and resulted in

functional prototypes.

ICCAS 2022 - International Conference on Cognitive Aircraft Systems

Many challenges remain:

• Increasing the technological maturity of these

technologies for these applications, by

prototyping these applications into more

significant environments. Testing the entire loop

in a representative environment is a key element

in the future :

o Enable the crew-system dialogue in

natural language using a dialogue

ontology

o Enable machine reasoning on system

data using an system ontology

o Create the mechanisms to update these

knowledge bases, by creating feedback

loops with the user

• Generating ontologies that use existing

databases (textual documentation, etc.): the

processes and concepts manipulated during the

operational missions are well documented. To

harvest this huge data source could be an

interesting way of creating or expanding the

domain ontologies.

• Applying more robust and state-of-the-art

techniques to match the dialogue and domain

ontologies for aeronautical applications

• Creating a framework to modify manually the

concepts and reasoning rules of the ontologies is

also a key challenge, especially if we want to

enable the end-user to update the dialogue

ontologies.

More generally, one main challenge is to develop a

hybrid system to assist the crew: couple data-driven

technologies, enable “sensory” services for the

system, with knowledge-based technologies, enable

“cognitive” services for the system. The

combination of these two types of technologies, as

well as the ability to quickly orient and modify

them, is an important step to creat a machine that

can team with the crew during aeronautical

missions.

REFERENCES

Best – Achieving the benefits of SWIM by making smart

use of semantic technologies (2016), https://project-

best.eu/project.html

Bienvenu, M. et al, Reasoning with ontologies (2020), A

Guided Tour of Artificial Intelligence Research,

pp.185-215, 2020, Volume I: Knowledge

Representation, Reasoning and Learning

CAB – Cockpit et Assistant Bidirectionnel (2021),

https://www.irt-systemx.fr/projets/cab/

Direction Générale de l’Armement Lancement du 2ème

cycle d’appel du projet “Man Machine Teaming »

(2019), https://www.defense.gouv.fr/dga/actualite/lan

cement-du-2e-cycle-d-appel-du-projet-man-machine-

teaming

Doan, A.H., Halevy, A.Y., Semantic integration research

in the database community: A brief survey (2005). AI

magazine, 26(1)

Ferrer, B. R. et al, Comparing ontologies and databases: a

critical review of lifecycle engineering models in

manufacturing (2021), Knowledge and Information

Systems, 1-34

Kharoufah, H. et al, A review of human factor causations

in commercial air transport accidents and incidents:

from to 2000-2016, (2018), Progress in Aerospace

Sciences Sci, 99, 1-3

Le Gleut, R., Conway-Mouret, H., Rapport d’information

fait au nom de la commission des affaires étrangères,

de la défense et des forces armées, sur le système de

combat aérien du futur (SCAF), (2020), Sénat –

Session extraordinaire de 2019-2020,

http://www.senat.fr/rap/r19-642/r19-6421.pdf

Madni A. M., Madni, C. C., Architectural Framework for

Exploring Adaptive Human-Machine Teaming

Options in Simulated Dynamic Environment (2018)

Systems, 6 (4), 44

Ministère chargé des Transports, Rapport sur la sécurité

aérienne (2019), https://www.ecologie.gouv.fr/sites/

default/files/rapport_securite_aerienne_2019_0.pdf

RDF 1.1 XML Syntax - W3C Recommendation 25

February 2014, https://www.w3.org/TR/rdf-syntax-

grammar/

Shvaiko, P., Euzenar, J., Ontology matching: state of the

art and future challenges (2013), IEEE Transactions on

Knowledge and Data Engineering, 25, 158-176

SPARQL Query Language for RDF - W3C

Recommendation 15 January 2008,

https://www.w3.org/TR/rdf-sparql-query/

Walliser, J.C. et al, Team Structure and Team Building

Improve Human–Machine Teaming with Autonomous

Agents (2019), Journal of Cognitive Engineering and

Decision Making, 13, 258-278

Symbolic AI for Crew Assistance: Using Ontologies in the Cockpit