Exploring Empathetic and Cognitive Interfaces

for Autonomous Vehicles

Benjamin Chateau

, Hélène Unrein

and Jean-Marc André

Catie, 1 avenue Dr Albert Schweitzer, 33400 Talence, France

IM UMR 5218, ENSC-BDX INP, 33400 Talence, France

Keywords: Autonomous Vehicle, Acceptability, Trust, Situational Awareness, Empathetic Interface, Mental

Representation.

Abstract: The work presented is carried out within the framework of the European SUaaVE project, whose objective is

to study different factors of acceptability of the Autonomous Vehicle (AV) and to test different solutions on

a simulator. Among these solutions is an interface capable of informing the user at any time about the road

situation and reassuring him/her about the information processed by the vehicle. To do this, it benefits from

an empathic function to estimate the cognitive and emotional state of the user and offer answers in terms of

comfort and information. This capability is based on a cognitive model of the passenger developed in

conjunction with the interface. A comfort module (driving dynamics and ambient comfort) and an emotional

module (participates in empathy functions) are developed in parallel by project partners. These modules will

be integrated once the prototype presented here has been tested by users and adjusted.

1 STUDY CONTEXT

The Autonomous Vehicle (AV) seems to be emerging

as a solution for the future. Some manufacturers are

already proposing to take advantage of a certain level

of autonomy (Endsley, 2017) and some States are

opening their roads to these technologies. This is not

really the case in Europe yet. Although road users are

increasingly open to AV, there is still a long way to

go to convince the population (Bel, Coeugnet &

Watteau, 2019) and build a real transition. To achieve

this, European countries are funding various projects

around the AV (e.g. Autopilot, BRAVE). These

projects examine everything that impacts its

development, and everything that will be impacted by

its deployment. An identified study question is

acceptability, the intention of use (Nielsen, 1993)0,

which is a predictor of the adoption of technologies

0Davis, 1989) such as AV. The SUaaVE project

(SUpporting acceptance of automated VEhicle)

brings together 10 European partners on this issue. In

particular, SUaaVE questions new uses and features

that can be offered by a “level 4+ AV”, i.e. 100%

autonomous but still with manual controls. For

example, a user freed from the driving task would be

https://orcid.org/0000-0003-2570-4381

https://orcid.org/0000-0001-9844-4694

able to perform other activities that were previously

impossible, such as sleeping, working or playing.

However, these uses could generate inconveniences

(e.g. motion sickness, fear) if certain factors are not

taken into account, such as dynamics, comfort,

emotions ...

To integrate these uses into VA development, the

SUaaVE project focuses on 5 axes of innovation

presented in Figure 1. Each axis involves the

development of a concept that is refined iteratively,

using user tests on a driving simulator.

Figure 1: ALFRED, a travel assistant from 5 axis of study.

(Ethical Module, Empathetic Module, Cognitive Assistant,

Conduit Comfort, Ambiant and Postural Comfort).

ALFRED = Automation Level Four - Reliable Empathic

Chateau, B., Unrein, H. and André, J.

Exploring Empathetic and Cognitive Interfaces for Autonomous Vehicles.

DOI: 10.5220/0010130701390144

In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 139-144

ISBN: 978-989-758-480-0

139

Driver ; EmY = EmpathY unit ; ACE = Adaptive,

Cognitive, Emotional.

The development axis presented here is based on

a "cognitive" information system (Smart Cognitive

Assistant), that is, capable of helping the user to

construct a mental representation rich enough to

understand the driving situation. The presentation of

this work is based on the notion of mental

representation, to understand the principles (1) of the

assessment of acceptability, (2) of building trust and

(3) of the design of a cognitive interface.

2 FROM MENTAL

REPRESENTATION TO

SITUATIONAL AWARENESS.

Mental representation is the key concept of this work.

It is used here to define the measure of acceptability,

to understand trust, to describe situational awareness

and finally to propose a cognitive and empathetic

interface. The theoretical concepts presented will be

illustrated with the case of the use of a taxi, which has

analogies with the use of an AV.

The mental representation of a device (e.g. a taxi)

is the activation in memory of concepts specific to the

device (e.g. car, driver, taximeter, yellow) or related

to the use of this device (e.g. airport, luggage, travel,

reserved parking). In other words, representation is

based on elements from previous experiences.

Among these elements there are also event schemata

(Hard, Becchia and Tversky, 2011) (e.g. the way the

driver welcomes the customer and handles luggage).

Such a schema is a structure that encodes in memory

an action (goal) and the intermediate actions (sub-

goal, steps) necessary for its realization (Zacks and

Tversky, 2001).

To construct or update his representation, an

observer is able to rely on an empathetic mechanism

by spontaneously taking the perspective of an

observed actor. The observer then mentally simulates

the actor's point of view and actions (Hard et al.,

2011). It is also an important process in social

interactions, for example it allows two interlocutors

to activate a set of shared representations on which

dialogue can be based (Knutsen and Lebigot, 2015).

This empathic capacity would rely on nervous

structures called mirror neurons (Sinigaglia &

Rizzolatti, 2011 ; Lamm, Batson, & Decety, 2007)

that activate in memory known patterns of action

from observed behaviors. This allows the observer to

understand what is the actor perceiving, how is he

drawing his goals and how is he operating his actions

(Davis, 1983).

These theories about representation provide a

relevant perspective on the empathic mechanism at

work to assess, for example, the abilities of a driver.

They also show the limitations faced by the AV user:

how does he know what the virtual driver is doing?

However, these theories are also applicable in the

other direction: the driver is able to simulate the

mental states of his passenger and adapt his driving.

This is a track that is explored in the project to support

the user's situational awareness.

Situational awareness is an applied approach of

mental representation. The model proposed by

Endsley (Endsley and Jones, 2012) describes a

continuous process in decision-making and

evaluation of actions. This process is structured by

three successive steps: (1) The perception of the

elements of the situation, (2) the understanding of the

situation, (3) the projection of the future status.

Mental representation and situational awareness

provide an understanding of the process of assessing

acceptability and building trust, and outline the

principles of a cognitive interface for AV.

3 TESTING THE

ACCEPTABILITY OF AV

FROM SIMULATIONS

Designers have a variety of methods at their disposal

to improve the user experience of their future

products (Lallemand and Gronier, 2018). In

particular, it is possible to apply an iterative design

approach, alternating design phases and testing

phases, to gradually adjust the product. During the

testing phases, the product is evaluated by measuring

the user's attitude using questionnaires such as the

TAM3 (Technology Acceptance Model, version 3)

(Politis, et al., 2018) or UTAUT2 (Unified Theory of

Acceptance and Use of Technology) (Venkatesh,

Thong, and Xu, 2012). This type of tool offers a

prediction of the acceptance of the future product

from a real or simulated experiment performed by

testers (potential users). Testers are immersed in a

physical or virtual situation to construct a mental

representation of the object as accurate as possible,

and then they answer the questionnaire. Each item

group in the questionnaire is comparable to a probe

that extracts a specific fragment from that

representation. For example, the first group of items

in TAM3 extract the representation of the

performance gain offered by the product; another

group extracts the perceived ease of use, etc. TAM3

thus offers a look at different dimensions of product

acceptability on practical (e.g. Perceived Usefulness),

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

140

hedonic (e.g. Perceived Enjoyment) or social aspects

(e.g. Subjective Norm), etc.

However, the acceptability of technologies based

on forms of Artificial Intelligence (AI), such as AV,

seems to encounter the issue of trust (Wintersberger

et al., 2019) Trust is not tackled frontally in

acceptability reference models such as the TAM

(Bastien & Scapin, 1993 ; Politis et al., 2019),

UTAUT (Venkatesh et al., 2012) or model of Nielsen

(1993). However, the link between trust and

acceptability has been studied for a long time and the

emergence of AI has led to the enrichment of models

(Hegner, Beldad, & Brunswick, 2019). Some

determinants of trust are close to those of

acceptability. Starting with the attitude that is

associated with both acceptability (Davis &

Venkatesh, 1996) and trust (Politis et al., 2018 ;

Wintersberger et al., 2019).

4 A POINT OF VIEW ON TRUST

ATTRIBUTION

On the same principle as acceptability, trust in a

system is determined by different factors. There are

many definitions of trust in a person (Rajaonah,

2006). A fairly general description would be to

associate trust with "expectations, assumptions or

beliefs about the likelihood that another person's

future actions will be beneficial, favourable or at

least not detrimental to his or her interests."

(Robinson, 1996). This prognosis is based on clues of

attributes, such as competence (Degenne, 2009 ;

Karsenty, 2015) or reliability (Payre, Cestac &

Delhomme, 2014).

These attributes can be found in the questionnaire

proposed by Jian, Bisantz and Drury (2000) to

measure trust in a system. In their model, they

differentiated non-confidence factors (e.g.

misleading, lack of transparency...) and confidence

factors (e.g. reliable, understandable). The

determinants of trust in a system (Henger et al., 2019

; Rajaonah, 2006) are close to those of trust in a third

party, especially for reliability that is fundamental for

AV (Payre et al., 2014). Reliability can be assessed

over the long term, which introduces a notion of

familiarity that is favourable to trust (Rajaonah,

2006). In this case it is possible to assign a level of

confidence in a target (person, group of persons,

object or type of object) based on observations made

over the course of the experiences with it. However,

some situations do not have the support of recurrent

experiences. This is the case, for example aboard a

taxi in a foreign country, it is necessary to obtain

quickly clues on the driver’s abilities to provide the

desired result. For this it is possible to use action

schemata constructed on the basis of road

experiences, and which allow the user to check

whether the actions observed are compliant. If this is

the case, trust can be established. These schemes are

bricks of mental representation, but also of situational

awareness.

5 PRINCIPLES OF AN

EMPATHETIC AND

COGNITIVE INTERFACE

The activities that can be carried out on board an AV

will distract the user's attention from the road. The

user's cognitive support covers two important aspects

of the driving situation: road situation and

autonomous driving. The road situation corresponds

to the flow of information available in the

environment that allows the user to understand the

vehicle's behaviours (e.g. traffic, pedestrian presence,

signage, weather, etc.). Autonomous driving refers to

driving actions developed from information taken by

the AV in the environment. The treatments operated

by the AV are not very visible to the user given their

speed and complexity. On the other hand, it is

possible to make visible certain "goals" (e.g. increase

speed, anticipate a traffic jam) and share some of the

environmental information processed by the AV. This

information is useful for passengers to understand

how the AV works but also to support their

representation of the situation. It is possible to

communicate information symbolically or verbally

through different sensory channels: visual, audio,

haptic...

There is a lot of information available about the

AV and the road situation. Design must respect a

certain minimalism (Bastien & Scapin, 1993 ; Maeda,

2006) to avoid cognitive overload. The choice of

information and sensory channels is an important

issue, which should not compete too much with the

activities of the passenger, at the risk of questioning

the interest of the AV. This is where the empathetic

nature of the interface comes into play. This empathy

is ideally bi-directional in the same way as a

communication situation: the user needs to

understand how the AV works; the AV needs to

"understand" the user's status to adjust their level of

information. According to the iterative design

principle, the first version of the interface provides a

standard level of information. This level of

information will be optimised in a second phase,

based on user feedback and future measurement of

the passenger's cognitive and emotional states. (see

Figure 1 : Empathic Module).

Exploring Empathetic and Cognitive Interfaces for Autonomous Vehicles

141

6 INTERFACE DESCRIPTION

Two categories of information are displayed on the

interface presented in Figure 2 : AV information, and

road situation information.

To begin with, a first information group presents

the state of the AV-passenger system in order to

check his ability to travel. Three thematic sub-groups

have been distinguished:

 Traveling, with information about speed,

autonomy (battery) and distance remaining.

Arrows above and below the speed indicate the

acceleration or braking process.

 The AV, with a general status icon (mechanical

and computer), and an icon related to the

current driving dynamics (calm, normal,

sporty).

 The passenger, with an icon for the state of the

monitoring (operational or not), and an icon for

the activity detected (e.g. attentive, rest,

daydreaming/reflection, oral communication,

reading/screen). The emotional state is not

displayed so as not to accentuate a possible

negative emotion (e.g. fear, sadness, anger)

A second group of information provides feedback

on the road situation in order to feed the user's

situational awareness on the one hand, and on the

other hand to enable him to check that the VA has

relevant information to drive safely. Three subgroups

are presented:

 The signage flow that impacts driving only

(e.g., speed limit, pedestrian crossing). Other

signs are ignored (e.g. parking entrance,

direction, etc.). Each item disappears when it

becomes obsolete.

 The contextual flow related for example to the

presence of an intersection, the state of the

road, etc. The type of road (e.g. urban,

motorway) and the weather are permanent, the

other information disappears when its becomes

obsolete.

 The Radar indicates on a grid the presence of

other users around “my car” (blue dot). Other

vehicles are shown with a colored dot

according to the risk of collision (low=green ;

medium=orange ; high=red).

Each cell corresponds to a time distance related

to the safe distances. For example, a vehicle

travelling in the same direction is displayed in

green (peripheral cells of the radar) if the safety

distances are respected. If this vehicle is too

close, it turns orange and flashes to alert of a

risk. If a vehicle follows a different trajectory

(perpendicular or face) it is displayed in red.

The cells in which pedestrians or bicycles are

present are highlighted (see the cell in the top

right corner of the radar).

By comparing the radar to real situation

presented in

Figure 3

, we can presume that a

glance at the radar captures more information

on other road users than a glance at the real

environment.

Figure 2: Interface overview. The coded information

corresponds to the driving situation presented in Figure3.

Figure 3 : Baseline driving condition. Three minutes of real

driving were filmed using glasses equipped with a camera.

The path has been coded to dynamically generate the

interface display.

The interface is presented on a touch screen to

access additional information or settings for each item

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

142

described above (e.g. User monitoring icon, general

AV status icon, radar, etc.). This information or

settings are displayed in a dedicated window (the grey

frame in the top left corner). Information related to

the trip is displayed by default. The content is

automatically changed in the event of an alert (e.g. if

a change in passenger activity has resulted in a change

in vehicle dynamics). Finally, some information will

be coupled with an audible signal (e.g., light clicks to

indicate the presence of a vehicle that is too close) or

a voice message (e.g., to reassure the passenger if

road event has generated fear or anger).

7 CONCLUSION AND

PERSPECTIVES

The design of this interface has raised many

theoretical (e.g. cognitive impact) and practical

questions (e.g. choice and cohabitation of

technologies involved). First of all, each piece of

information or interaction that makes up the interface

involves a more or less important technical

development. Therefore, choices were made in order

to respect the schedule of the European project

phases. Also, according to the principle of an iterative

development, it was decided to validate the concept

from a simplified version before engaging more

technical and aesthetic developments (e.g. add audio

or voice functions, refine visual code, etc.). As a

result, some questions remain unsolvable, but

preliminary tests have already made it possible to

identify opportunities for improvement (e.g. too

much visual presence of cars in the opposite way

which are displayed in red on the radar). To go

further, a test with 30 drivers is to be carried out on

board a virtual simulator. Preliminary results will be

presented in the final communication. These tests

have the following objectives:

 Check whether functions and interface

elements are correctly understood,

 Gathering the opinion of users on the

contribution of this interface in an AV,

 Evaluate the gain in acceptability and

confidence of an AV equipped with this

interface compared to an AV without this

interface,

 Collect passengers' needs for information about

the situation according to their activity and

emotional state.

In summary, the first user-test will have to

provide leads to improve the interface's ability to help

the user to mentally represent the environment and

the functioning of the AV. In addition to this main

objective, this test will also enable the concept to be

validated and refined from a technical point of view,

i.e. the system's ability to collect and process

information of the vehicle in real time. The validation

of this principle is important to prepare the integration

of the modules developed by the partners (ethical,

cognitive and emotional modules, comfort modules).

A new version of the interface enriched with these

modules and new functionalities will then be

developed and tested in a second and third iteration.

The final results will be integrated into a virtual

demonstrator in order to present technological

opportunities to the automotive industry. It is

important to note that virtual environments offer

increasingly richer representations, however the

absence of validation in real conditions will be the

main limitation of the developments carried out.

REFERENCES

Bastien, J. M. C., and Scapin, D. L. (1993). Ergonomic

criteria for the evaluation of human-computer

interfaces. Technical report, INRIA, (0156).

Bel., M., Coeugnet, S. and Watteau, P. (2019). Monograph:

Acceptability of the autonomous vehicle. Manuscript

delivered by Vedecom on March 25, 2019 to the

General Directorate of Infrastructure, Transport and the

Sea (DGITM).

Davis, F.D. (1989). Perceived usefulness, perceived ease of

use, and user acceptance of information technology.

MIS Quarterly, 13, 319-339.

Davis, M. H. (1983). Measuring individual differences in

empathy: Evidence for a multidimensional approach.

Journal of Personality and Social Psychology, 44, 113-

126. doi:10.1037/0022-3514.44.1.113

Davis, F.D., Venkatesh, V. (1996). A critical assessment of

potential measurement biases in the technology

acceptance model: Three experiments. International

Journal of Human-Computer Studies, 45(1), 19-45.

https://doi.org/10.1006/ijhc.1996.0040

Degenne, A. (2009). Types of interactions, forms of trust

and relationships. 30.

Endsley, M. R. (2017). Autonomous Driving Systems: A

Preliminary Naturalistic Study of the Tesla Model S.

Journal of Cognitive Engineering and Decision

Making, 11(3), 225-238.

https://doi.org/10.1177/1555343417695197

Endsley, M. R., and Jones, D. G. (2012). Designing for

situation awareness: an approach to user-centered

design (Second Edition). London: Taylor and Francis.

Hard, B., Recchia, G., and Tversky, B. (2011). The shape

of action. Journal of Experimental Psychology:

General, 140, 586-604.

http://dx.doi.org/10.1037/a0024310

Exploring Empathetic and Cognitive Interfaces for Autonomous Vehicles

143

Hegner, S. M., Beldad, A. D., and Brunswick, G. J. (2019).

In Automatic We Trust: Investigating the Impact of

Trust, Control, Personality Characteristics, and

Extrinsic and Intrinsic Motivations on the Acceptance

of Autonomous Vehicles. International Journal of

Human-Computer Interaction, 1-12.

https://doi.org/10.1080/10447318.2019.1572353

Jian, J.-Y., Bisantz, A. M., Drury, C. G. (2000).

Foundations for an empirically determined scale of trust

in automated systems. International Journal of

Cognitive Ergonomics, 4(1), 53-71.

https://doi.org/10.1207/S15327566IJCE0401_04.

Rajaonah B. (2006). Rôle de la confiance de l'opérateur

dans son interaction avec une machine autonome sur la

coopération homme-machine. Thèse de doctorat de

l'Université Paris 8. 27 février 2006. 234.

Karsenty, L. (2015). How can we maintain relationships of

trust and build meaning in the face of a crisis? Human

work, 78(2), 141. https://doi.org/10.3917/th.782.0141

Knutsen, D., and Le Bigot, L. (2015). The influence of

reference acceptance and reuse on conversational

memory traces. Journal of Experimental Psychology:

Learning, Memory, and Cognition, 41(2), 574-585.

https://doi.org/10.1037/xlm0000036

Lallemand, C., Gronier, G. (2018). Ux Design Methods: 30

basic methods to design optimal experiences. Paris:

Eyrolles.

Lamm, C., Batson, C.D., and Decety, J. (2007). The Neural

Substrate OF Human Empathy: Effects OF Perspective-

taking and Cognitive Appraisal. Journal of Cognitive

Neuroscience, 19(1), 4258.

https://doi.org/10.1162/jocn.2007.19.1.42

Maeda, J. (2006). The Laws of Simplicity. Cambridge, MA:

MIT Press

Nielsen, J. (1993). Usability Engineering. Boston:

Academic Press.

Payre, W., Cestac, J., Delhomme, P. (2014). Intention to use

a fully automated car: Attitudes and a priori

acceptability. Transportation Research Part F: Traffic

Psychology and Behaviour, 27, 252263.

https://doi.org/10.1016/j.trf.2014.04.009

Politis, I., Langdon, P., Bradley, M., Skrypchuk, L.,

Mouzakitis, A., and Clarkson, P. J. (2018). Designing

Autonomy in Cars: A Survey and Two Focus Groups

on Driving Habits of an Inclusive User Group, and

Group Attitudes Towards Autonomous Cars. In G. Di

Bucchianico and P. F. Kercher (Eds.), Advances in

Design for Inclusion (Vol. 587, 161-173). Springer

International Publishing. https://doi.org/10.1007/978-

3-319-60597-5_15

Robinson, S.L. (1996). Trust and breach of the

psychological contract. Administrative Science

Quaterly, 41, 574-599.

Sinigaglia, C., And Rizzolatti, G. (2011). Through the

looking glass: Self and others. Consciousness and

Cognition, 20(1), 6474.

https://doi.org/10.1016/j.concog.2010.11.012

Venkatesh, V., Bala, H. (2008). Technology Acceptance

Model 3 and a Research Agenda on Interventions.

Decision Sciences, 39, 273-315.

Venkatesh, V., Thong, J. Y. L., Xu, X. (2012). Consumer

acceptance and use of information technology:

Extending the unified theory of acceptance and use of

technology. MIS Quarterly, 36, 157-178.

Wintersberger, P., Frison, A.-K., Riener, A., & Sawitzky,

T. von. (2019). Fostering User Acceptance and Trust in

Fully Automated Vehicles: Evaluating the Potential of

Augmented Reality. PRESENCE: Virtual and

Augmented Reality, 27(1), 46–62.

https://doi.org/10.1162/pres_a_00320

Zacks, J., and Tversky, B. (2001). Event structure in

perception and design. Psychological Bulletin, 127,3-

21.

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

144