REAL-TIME BIOMETRIC EMOTION ASSESSMENT

IN AN IMMERSIVE ENVIRONMENT

Vasco Vinhas, Daniel Castro Silva, Luís Paulo Reis and Eugénio Oliveira

Department of Informatics Engineering, Faculty of Engineering, University of Porto

Artificial Intelligence and Computer Science Laboratory, Rua Dr. Roberto Frias s/n 4200-465, Porto, Portugal

Keywords: Emotion Assessment, Biometric Readings, Immersive Digital Environments, Aeronautical Simulation.

Abstract: Both the academic and industrial worlds have increased investment and dedication to the affective

computing area in the past years. At the same time, immersive environments have become more and more a

reliable domain, with progressively cheaper hardware and software solutions. With this in mind, the authors

used biometric readings to perform real-time user emotion assessment in an immersive environment. In the

example used in this paper, the environment consisted in a flight simulation, and biometric readings were

based on galvanic skin response, respiration rate and amplitude, and phalanx temperature. The detected user

emotional states were also used to modify some simulation variables, such as flight plan, weather and

maneuver smoothness. The emotion assessment results were consistent with user-described emotions,

achieving an overall success rate of 78%.

1 INTRODUCTION

The presence of sensors, actuators and processing

units in unconventional contexts is becoming

consistently inevitable. This fact brings to both

academic and industrial stages the term of

Ubiquitous Computing as a regular one. In a

parallel, yet complementary line, Affective

Computing has recently gained the attention of

researchers and business organizations worldwide.

As a common denominator for these two concepts

resides Emotion Assessment. Although this topic is

no novelty by itself, it has been rediscovered in light

of the mentioned knowledge areas breakthroughs, as

it became theoretically possible to perform real-time

minimal-invasive user emotion assessment based on

live biosignals at economically feasible levels.

Having all that in mind, the authors envisioned

an integrated interactive multimedia system where

internal parameters are changed according to the

user’s emotional response. As the application

example in this paper, an aviation environment was

considered. The main reasons behind this decision

are related to the human fascination for everything

related to flying. Still, and as with most things, this

attraction co-exists with the fear of flying, usually

referred to as pterygophobia. According to a poll by

CNN and Gallup for the USA Today in March 2006,

27% of U.S. adults would be at least somewhat

fearful of getting on an airplane (Stoller, 2006).

The conducted experimental protocol was

carried out in a quiet controlled environment where

subjects assumed the pilot’s seat for roughly 25

minutes. Internal variables were unconscientiously

affected by the online assessed user emotions.

The project achieved rather transversal goals as it

was possible to use it as a fully functional testbed for

online biometric emotion assessment through

galvanic skin response, respiration rate and

amplitude and phalanx temperature readings fusion

and its incorporation with Russell’s Circumplex

Model of Affect (Russell, 1980) with success rates

of around 78%. Considering the aeronautical

simulation, an immersive realistic environment was

achieved, with the use of 3D video eyewear.

It was found that those without fear of flying

found the experience rather amusing, as virtual

entertainment, while the others considered the

simulation realistic enough to trigger an emotional

response – verified by biometric readings.

The present document is organized as follows: in the

next section a broad, detailed revision of related

work is depicted; in section 3, the project is

described in a global perspective but also

highlighting relevant system modules; in section 4

the conducted experimental session conditions are

described and in the following section the results are

presented; in the final section, conclusions are drawn

and future work areas revealed.

153

Vinhas V., Silva D., Reis L. and Oliveira E.

REAL-TIME BIOMETRIC EMOTION ASSESSMENT IN AN IMMERSIVE ENVIRONMENT.

DOI: 10.5220/0002174001530158

In Proceedings of the 6th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2009), page

ISBN: 978-989-674-000-9

2 STATE OF THE ART

This section is divided into two subsections: the first

concerning automatic emotion assessment; the

second regarding aeronautical simulation tools.

2.1 Automatic Emotion Assessment

Until a recent past, researchers in the domains

related to emotion assessment had very few solid

ground standards both for specifying the emotional

charge of stimuli and also a reasonable acceptable

emotional state representation model. This issue

constituted a serious obstacle for research

comparison and conclusion validation. The extreme

need of such metrics led to several attempts to

systematize this knowledge domain.

Considering first the definition problem,

Damásio states that an emotional state can be

defined as a collection of responses triggered by

different parts of the body or the brain through both

neural and hormonal networks (Damásio, 1998).

Experiments conducted with patients with brain

lesions in specific areas led to the conclusion that

their social behaviour was highly affective, together

with the emotional responses. It is unequivocal to

state that emotions are essential for humans, as they

play a vital role in their everyday life: in perception,

judgment and action processes (Damásio, 1994).

One of the major models of emotion

representation is the Circumplex Model of Affect

proposed by Russell. This is a spatial model based

on dimensions of affect that are interrelated in a very

methodical fashion (Russell, 1980). Affective

concepts fall in a circle in the following order:

pleasure, excitement, arousal, distress, displeasure,

depression, sleepiness, and relaxation - see Figure 1.

According to this model, there are two components

of affect that exist: the first is pleasure-displeasure,

the horizontal dimension of the model, and the

second is arousal-sleep, the vertical dimension of the

model. Therefore, it seems that any affect stimuli

can be defined in terms of its valence and arousal

components. The remaining variables mentioned

above do not act as dimensions, but rather help to

define the quadrants of the affective space. Although

the existence of criticism concerning the impact

different cultures in emotion expression and

induction, as discussed by Altarriba (Altarriba,

2003), Russell’s model is relatively immune to this

issue if the stimuli are correctly defined in a rather

universal form. Having this in mind, the circumplex

model of affect was the emotion representation

abstraction used in the proposed project.

In order to assess Russell’s model components,

Figure 1: Russell’s Circumplex Model of Affect.

one ought to consider what equipment solutions

were to be selected, considering, simultaneously,

different features such as portability, invasiveness

levels, communication integration and transparency

and direct economical impact.

Emotions assessment requires reliable and

accurate communications with the subject so that the

results are conclusive and the emotions correctly

classified. This communication can occur through

several channels and is supported by specific

equipment. The invasive methods are clearly more

precise, however more dangerous and will not be

considered for this study. Conversely, non invasive

methods such as EEG (Electroencephalography),

GSR (Galvanic Skin Response), oximeter, skin

temperature, ECG (Electrocardiogram), respiration

sensors, amongst others have pointed the way

towards gathering the advantages of low-cost

equipment and non-medical environments with

interesting accuracy levels (Benevoy, 2008).

Some recent studies have successfully used just

EEG information for emotion assessment (Teixeira,

2008). These approaches have the great advantage of

being based on non-invasive solutions, enabling its

usage in general population in a non-medical

environment. Encouraged by these results, the

current research direction seems to be the addition of

other inexpensive, non-invasive hardware to the

equation. Practical examples of this are the

introduction of a full set of non-invasive, low-cost

sensors in several domains by Vinhas (Vinhas,

2008), Kim (Kim, 2008) and Katsis (Katsis, 2008).

The usage of this kind of equipments in such diverse

domains and conditions strongly suggests its high

applicability and progressive migration towards

quotidian handling.

For this study, the Nexus-10 hardware solution

with temperature, GSR and Respiration Rate and

Amplitude sensors shall be used and the data

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

154

communication with the processing unit, fully

described in section 3, shall be based on wireless

Bluetooth technology.

2.2 Aeronautical Simulation Tools

There are two main simulator categories: Game

Engines and Flight Simulators. In game engines, the

most important aspect is an appealing visualization.

Flight Simulators have a different approach – the

main focus is on aerodynamics and flight factors

present in real world, thus trying to achieve as

realistic a flight as possible (Gimenes, 2008). The

academic and business communities have already

begun to use these cost-effective tools, benefitting

from what they have to offer (Lewis, 2002).

The authors, after some consideration and

analysis of available flight simulators, have chosen

to use Microsoft Flight Simulator X (FSX) as the

simulation environment. FSX not only provides a

flexible, well-documented programming interface to

interact with the environment, but also a very

realistic visualization of the simulated world.

Several solutions are offered to treat pterygophobia,

including medication, and some behavior therapies,

together with virtual reality solutions. These are

often used in conjunction with a more conventional

form of therapy (Kazan, 2000), (da Costa, 2008).

Though the authors are cautious regarding any

conclusion about the psychological impact of this

simulation tool, it is believed that this simulation,

together with real-time emotional assessment, may

have a positive impact in treating pterygophobia.

Summarizing, the proposed solution not only

presents an online Russell’s Model emotion

assessment tool, based on minimal-invasive sensors,

but also provides a cost-effective solution for a

virtual reality simulation that can be used for

treating fear of flying.

3 PROJECT DESCRITPION

This section is divided into three subsections,

focusing the global architecture delineation, emotion

assessment module description and aeronautical

simulation component.

3.1 Global Architecture

The system architecture is based on independent and

distributed modules, both in logic and physical

terms. As depicted in Figure 2, and following its

enclosed numeration, biometric data is gathered

Figure 2: System’s Global Architecture.

directly from the subject by Nexus-10 hardware. In

more detail, temperature, GSR and respiration

sensors are used. As to reduce the number of wires

presented to the user, the biometric data is

transmitted by Bluetooth to a computer running the

adequate data driver. The next step is of the

responsibility of BioTrace+ software, and beyond

providing a flexible interface for signal monitoring,

it also records biometric data in a text file.

The BioSignal Collector software was developed

to access the recorded data and make it available for

further processing either by database access or

TCP/IP socket connection. In the last case, lies the

Emotion Classifier, responsible for user’s emotion

state assessment – this process is described below.

The continuously extracted emotional states are

projected into Russell’s model and are filled as input

to the Aeronautical Simulator. The simulation

endpoint has a simple architecture. The main module

communicates with the emotional endpoint,

receiving data from the emotion assessment module,

indicating which of the four quadrants of Russell’s

Model should be active. The module, in turn,

communicates with FSX, changing its internal

variables in order to match the desired quadrant, and

as explained in section 3.3. This module also

produces a permanent log file, with information

collected from the simulator. The simulator interacts

with the user through immersive 3D video hardware,

allowing the user to control simulation visualization.

3.2 Emotion Assessment

The emotion assessment module is based on the

enunciated 4-channel biometric data collected with

Nexus-10 and accessed via text file readings at 10Hz

sample rate – which for the analyzed features is

REAL-TIME BIOMETRIC EMOTION ASSESSMENT IN AN IMMERSIVE ENVIRONMENT

155

perfectly acceptable. At the same rate, emotional

states are assessed and its definition is continuously

uploaded to a database for additional analysis and

third-party tools access. Directly related to the

aeronautical simulation, the GUI also provides an

expedite method to define the session’s emotional

policy – force a specific quadrant, contradict or

maintain the current state or tour the four scenarios.

The remaining of this subsection is divided in three

parts, devoted to emotion model description,

calibration and data fusion, and dynamic scaling.

3.2.1 Base Emotion Model

As previously referred, the adopted emotion model

was Russell’s Circumplex Model of Affect. This

bidimensional approach permits efficient, yet

effective, online emotional assessment with none or

residual historical data as it is based on single

valence and arousal values. The key issue is not the

determination of the subject’s emotional state given

a pair of valence/arousal values, but how to convert

biosignals into valence/arousal pairs.

In order to anticipate the assessment of

emotional data pair values, a normalization process

is conducted, where both valence and arousal values

are fully mapped into the [-1, 1] spectrum. With this

approach, emotional states are believed to be

identified by Cartesian points in a 2D environment.

3.2.2 Calibration & Channel Fusion

Having into consideration the referred normalization

process, one ought to point out the importance of the

calibration process. Although, the 2D point (-¾, ¾)

represents a normalized defined emotional state, it

can be achieved by an infinite conjugation of

biosignals. This reality leads to the necessity of

calibration and biometric channels fusion.

The first procedure consists in, for each subject

and for each session, pinpoint directly in Russell’s

model, what is the predominant emotional state,

through a self-assessment process. By performing

this action, it is possible to define a normalized

emotional baseline point. For each of the four

channels taken into account for emotional state

assessment an initial twenty percent variability is

considered. Whenever overflow is detected, the

dynamic scaling is activated as described below.

The three components were considered to have

similar impact. For the valence values deviation,

only galvanic skin response was considered. For this

computation, the normalized baseline point is

considered as reference. The conjugation of such

weights determines the normalized values of arousal

and valence and hence the current emotional state.

3.2.3 Dynamic Scaling

As a consequence of the emotional classification

process, emerging issue concerns either biosignal

readings’ overflow or underflow, considering user-

defined baseline and initial tolerance allowed.

To overcome this potential limitation, a fully

dynamic scaling approach was considered, that

consists in stretching the biometric signal scale

whenever its readings go beyond the normalized

interval of [-1,1]. This scale update is conducted

independently for each of the analyzed biometric

channels. During this process, a non-linear scale

disruption is created, resulting in greater scale

density towards the limit breach.

In order to better understand this approach, one

shall refer to the set of formulas listed through

Equation 1, depicting an overflow situation.

(a)

[]

),(.

111

IndexcSampleMaxcMaxMathMaxc

(b)

[]

IndexcmplebaseLineSaMaxc

AxisrmbaseLineNo

ScaleUpc

−

(c)

[

][]

IndexcmplebaseLineSaIndexcSamplec

−

(d)

cScaleUpcAxisrmbaseLineNoNormc ×+=

Equation 1: Dynamic Scaling Formulas.

First, c1 (any given biometric channel) maximum

value is determined by comparing current reading

with the stored value – Equation 1(a). If the limit is

broken, the system recalculates the linear scale

factor for values greater than the baseline neutral

value, having as a direct consequence the increasing

of the interval’s density – Equation 1(b). Based on

the new interval definition, subsequent values shall

be normalized accordingly – Equation 1(c) (d). With

this approach, and together with dynamic calibration

and data normalization, it becomes possible for the

system to perform real-time adaptations as a result

of user’s idiosyncrasies and signal deviations, thus

assuring continuous normalized values.

3.3 Aeronautical Simulation

The desired emotional quadrant influences the

simulation in three dimensions: weather, scenery and

maneuvering.

The two quadrants characterized by displeasure

are associated with worse climacteric conditions,

ranging from thunderstorms, for the quadrant with

high arousal levels, to foggy cold fronts, for the one

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

156

with low levels. The two quadrants related to

pleasure are coupled with fair weather, creating a

more stable flight.

The chosen global scenery is an archipelago. For

the two quadrants associated with high arousal

levels, the itinerary takes the plane around an island,

with many closed turns at low altitudes. For the two

quadrants associated with low arousal levels, the

path consists of an oval-shaped route around an

island. The turns in this route have a superior radius

and the altitude variations have smaller amplitude.

As a result, the flight is experienced as a calmer one.

Closely related to the route description is

maneuvering control. For the first route, typical

auto-pilot controls are used, namely speed, heading

and altitude controls. As for the second route, two

additional features are applied – maximum bank and

yaw damper, which limits the maximum roll during

turns, and reduces rolling and yawing oscillations,

making the flight smoother and calmer.

4 EXPERIMENTAL ACTIVITIES

The experiments were conducted using a variety of

equipment, for both the biometrical emotion

assessment module and the simulation module. As

for the first module, sensors for skin temperature,

galvanic skin response and respiration rate and

amplitude were used. In order to present the user

with an immersive experience, 3D video hardware

was used in conjunction with the flight simulator, in

the form of virtual reality video eyewear, which

provides the user with a three degree of freedom

head-tracker, allowing the user to experience the

environment as if he was actually there.

The experiments were conducted among twenty

subjects, 13 males and 7 females, aging between 21

and 56. Four of the subjects stated that they had

some level of fear of flying, while the remaining

declared not to.

After providing background information to

characterize the sample, the subject was connected

to the biometrical equipment, in order to establish an

emotional baseline, as explained in section 3.2.2.

The experiment had three sequential stages. In

the first, the plane takes off from an airport. After

takeoff, a series of closed circuits was performed.

Finally, in the landing phase, the plane lines up with

the selected airport, makes the approach and lands.

After concluding the trial, the subjects described

the experience, and reviewed an animation of the

evolution of both simulation and emotional

assessment, to confirm or refute those assessments.

5 RESULTS

The results are presented and analyzed in two main

groups: emotion assessment and simulation.

In what concerns to emotion assessment, the

validation model was based on user self-assessment,

as previously described. These results were collected

in two forms: concerning single emotions and

specific regions on Russell’s model, and concerning

only the four quadrants. For the first method, a

success rate of 78% was achieved. For the second

one, this number increases to 87%. Table 1 shows

the confusion table with percentages of automatic

assessment versus self-assessment for each quadrant.

Table 1: Emotion Assessment Confusion Table.

Quadrant 2

Quadrant 3

Quadrant 4

Quadrant

30,7 1,8 0,3 1,2

Quadrant

3,1 32,8 10,1

Quadrant

0,2 1,7 10,9 1,2

Quadrant

10,11,612,3

Users

AutomaticAssessment

One additional result to consider is that the

automatic emotion assessment has a lower rate of

failure for opposite quadrants.

Concerning the simulation, users were asked to

describe their experience, and to classify, on a scale

of one to five, the level of immersiveness. The

results show that the majority of the individuals

considered the environment to be highly immersive,

with an average classification of 4,2.

Takeoff and landing are traditionally associated

with higher levels of apprehension and anxiety

among passengers who suffer from pterygophobia, a

fact confirmed by the experimental results. All

subjects that are afraid of flying also stated that

those are in fact the most stressful moments, and the

collected data corroborates this fact. Figure 3 shows

the average arousal levels measured during the

experiments conducted among these individuals. As

can be seen, higher arousal levels were registered

during the initial and final stages of the simulation,

which represent takeoff and landing.

Figure 3: Average Arousal Levels During Simulation.

REAL-TIME BIOMETRIC EMOTION ASSESSMENT IN AN IMMERSIVE ENVIRONMENT

157

6 CONCLUSIONS

From an architectural standpoint, the distributed

architecture with logic and physical module

separation proved to be reliable and efficient. This

approach enabled independence between biometric

data collection, processing and simulation related

computation. It also provided database collection of

both raw biometric channel values and semantic

emotional state information for future analysis and

validation, improving system openness.

At a more significant level, the emotional

assessment layer reached high accuracy levels.

Through the detailed validation process, 78% of the

classified emotional states were considered correct

by the subjects. If simplified to Russell’s four

quadrants, this value reaches 87%, which supports

the conclusion of an effective emotional assessment

process. Still in this category, it is worth to mention

the on-the-fly classification procedure that nearly

suppresses the need to a long baseline data gathering

and user identification as it is performed by the user

at any time. Also, the dynamic scaling was valuable,

as to correctly accommodate outsized signal

deviations without precision loss.

In what regards the aeronautical simulation, all

projected goals where completely fulfilled as users

confirmed their immersion sensation, by both self-

awareness and biological recorded response. It is

believed that the use of 3D glasses as display device

played a particularly important role in creating the

appropriate environment.

Some improvement opportunities have been

identified along the project. It is believed to be

useful, for future system versions, to include

additional biometric channels in the emotional

assessment engine, such as ECG, BVP (Blood

Volume Pulse) and even EEG. This signals

integration would be fairly straightforward as the

current data fusion process and emotional base

model support that kind of enhancement. Still

concerning this module, one shall mention the

possibility to test Russell’s model expansion to 3D

by adding a dominance axis. Regarding the

aeronautical simulator, it would be interesting to

define and test more navigation scenarios. Still in

this point, a more smooth transition between

contexts, especially between quadrants characterized

by high levels of arousal and those with low levels

of arousal would be useful.

As a final project summary, one shall point that

the proposed system has a dual application as a

complete entertainment system with user emotional

awareness that continuously adapts the multimedia

content accordingly, and possibly a more solemn

approach as a phobia treatment auxiliary.

REFERENCES

Altarriba, J., Basnight, D. M., & Canary, T. M., 2003.

Emotion representation and perception across

cultures. In W. J. Lonner, D. L. Dinnel, S. A. Hayes,

& D. N. Sattler (Eds.), Readings in Psychology and

Culture (Unit 4, Chapter 5), Center for Cross-Cultural

Research, Western Washington University,

Bellingham, Washington USA.

Benovoy, M., Cooperstock, J., Deitcher, J., 2008.

Biosignals Analysis and its Application in a

Performance Setting - Towards the Development of an

Emotional-Imaging Generator. In Proceedings of the

First International Conference on Biomedical

Electronics and Devices, pp. 253-258.

da Costa, R., Sardinha, A., Nardi, A., 2008. Virtual Reality

Exposure in the Treatment of Fear of Flying. Aviation,

Space, and Environmental Medicine, 79(9), pp. 899–

903.

Damásio, A. R., 1994. Descartes error: Emotion, reason

and human brain. Europa-América.

Damásio, A. R., 1998. Emotions and the Human Brain.

Iowa, USA: Department of Neurology.

Gimenes, R., Silva, D.C., Reis, L.P., Oliveira, E., 2008.

Flight Simulation Environments Applied to Agent-

Based Autonomous UAVs. In Proceedings of ICEIS

2008, pp. 243–246.

Kahan, M., Tanzer, J., Darvin, D., and Borer, F., 2000.

Virtual Reality-Assisted Cognitive-Behavioral

Treatment for Fear of Flying: Acute Treatment and

Follow-up. CyberPsychology & Behavior, 3(3), pp.

387–392.

Katsis, C., Katertsidis N., Ganiatsas G. and Fotiadis D.,

2008. Towards Emotion Recognition in Car-Racing

Drivers: A Biosignal Processing Approach. IEEE

Tran. on Systems, Man, and Cybernetics – Part A:

Systems and Humans, Vol. 38, No 3, pp. 502-512.

Kim, J., André, E., 2008. Multi-Channel BioSignal

Analysis for Automatic Emotion Recognition. In

Proceedings of the First International Conference on

Biomedical Electronics and Devices.

Lewis, M., Jacobson, J., 2002. Game Engines in Scientific

Research. Communications of the ACM. Vol. 45, Nr.

1, pp. 27-31.

Russell, J. A., 1980. A Circumplex Model of Affect. In

Journal of Personality and Social Psychology, 39, pp.

1161-1178.

Stoller, G., 2006. Fear of Flying can Cripple Workers,

USA Today, March 2006. Available online at

http://www.usatoday.com/educate/college/business/art

icles/20060326.htm. Consulted Dec. 2008.

Teixeira, J., Vinhas, V., Oliveira, E., Reis, LP., 2008.

Emotion Assessment Tool for Human-Machine

Interfaces - Using EEG Data and Multimedia Stimuli

Towards Emotion Classification. In SIGMAP, pp.

185-188.

Vinhas, V., Oliveira, E., Reis, LP., 2008. Realtime

Dynamic Multimedia Storyline Based on Online

Audience Biometric Information. In KES IIMSS, pp.

545-554.

ICINCO 2009 - 6th International Conference on Informatics in Control, Automation and Robotics

158