DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON

REALTIME USER EMOTION ASSESSMENT

Biometric User Data towards Affective Immersive Environments

Vasco Vinhas, Daniel Castro Silva, Eugénio Oliveira and Luís Paulo Reis

FEUP-DEI / LIACC

Rua Dr. Roberto Frias s/n 4200-465 Porto, Portugal

Keywords: Emotion Assessment, Biometric Readings, Immersive Digital Environments, Aeronautical Simulation.

Abstract: Both the academic and industry sectors have increased their attention and investment to the fields of

Affective Computing and immersive digital environments, the latter imposing itself as a reliable domain,

with increasingly cheaper hardware solutions. With all this in mind, the authors envisioned an immersive

dynamic digital environment tied with automatic real-time user emotion assessment through biometric

readings. The environment consisted in an aeronautical simulation, with internal variables such as flight

plan, weather conditions and maneuver smoothness dynamically altered by the assessed emotional state of

the user, based on biometric readings, including galvanic skin response, respiration rate and amplitude and

phalange temperature. The results were consistent with the emotional states reported by the users, with a

success rate of 78%.

1 INTRODUCTION

The presence of dissimulated sensors, actuators and

processing units in unconventional contexts is

becoming consistently inexorable. This fact brings to

both academic and industrial stages the term of

Ubiquitous Computing as a regular one. In a

parallel, yet complementary line, Affective

Computing has recently gained the attention of

researchers and business organizations worldwide.

As a common denominator for these two concepts

resides Emotion Assessment. Although this topic is

no novelty by itself, it has been rediscovered in light

of the mentioned knowledge areas breakthroughs, as

it became theoretically possible to perform real-time

minimal-invasive user emotion assessment based on

live biosignals at economically feasible levels.

Having this in mind, the authors envisioned an

integrated interactive multimedia bidirectional

system where internal parameters would be changed

according to the user’s emotional response. As the

application example in this paper, an immersive

aviation-based environment was considered. The

main reasons behind this decision are related to the

human fascination for everything related to flying.

However, and as with most things, this attraction co-

exists with the fear of flying, usually referred to as

pterygophobia. According to a poll by CNN and

Gallup for the USA Today in March 2006, 27% of

U.S. adults would be at least somewhat fearful of

getting on an airplane (Stoller, 2006). By using the

briefly described multimedia system, the authors

were able to provide distinct practical scenarios to

apply in several situations that range from traditional

entertainment applications to therapeutic phobia

treatment. The conducted experimental protocol was

carried out in a controlled environment where

subjects assumed the pilot’s seat for roughly 25

minutes. Internal variables were unconscientiously

affected by the online assessed user emotions.

The project achieved transversal goals as it was

possible to use it as a fully functional testbed for

online biometric emotion assessment through

galvanic skin response, respiration rate and

amplitude and phalange temperature readings fusion

and its incorporation with Russell’s Circumplex

Model of Affect (Russell, 1980) with success rates

of around 78%. It was found that those without fear

of flying found the experience rather amusing, as

virtual entertainment, while the others considered

the simulation realistic enough to trigger an

emotional response – verified by biometric readings.

The results also suggested a trend to pterygophobia

mitigation.

Vinhas V., Castro Silva D., Oliveira E. and Paulo Reis L. (2009).

DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric User Data towards Affective Immersive

Environments.

In Proceedings of the 11th International Conference on Enterprise Information Systems - Human-Computer Interaction, pages 42-47

DOI: 10.5220/0001984100420047

 SciTePress

2 STATE OF THE ART

This section is divided in three concerning automatic

emotion assessment; aeronautical simulation tools;

and pterygophobia treatments.

2.1 Automatic Emotion Assessment

Until a recent past, researchers in the domains

related to emotion assessment had very few solid

ground standards both for specifying the emotional

charge of stimuli and also a reasonable acceptable

emotional state representation model. This issue

constituted a serious obstacle for research

comparison and conclusion validation. The extreme

need of such metrics led to several attempts to

systematize this knowledge domain.

One of the major models of emotion

representation is the Circumplex Model of Affect

proposed by Russell. This is a spatial model based

on dimensions of affect that are interrelated in a very

methodical fashion (Russell, 1980). Affective

concepts fall in a circle in the following order:

pleasure, excitement, arousal, distress, displeasure,

depression, sleepiness, and relaxation – see Figure 1.

According to this model, there are two components

of affect that exist: the first is pleasure-displeasure,

the horizontal dimension of the model, and the

second is arousal-sleep, the vertical dimension of the

model. Therefore, it seems that any affect stimuli

can be defined in terms of its valence and arousal

components. The remaining variables mentioned

above do not act as dimensions, but rather help to

define the quadrants of the affective space. Although

the existence of criticism concerning the impact

different cultures in emotion expression and

induction, as discussed by Altarriba (Altarriba,

2003), Russell’s model is relatively immune to this

issue if the stimuli are universally defined.

Emotions assessment requires reliable and

accurate communications with the subject so that the

results are conclusive and the emotions correctly

classified. This communication can occur through

several channels and is supported by specific

equipment. The invasive methods are clearly more

precise, however more dangerous and will not be

considered for this study. Conversely, non invasive

methods such as EEG (Electroencephalography),

GSR (Galvanic Skin Response), oximeter, skin

temperature, ECG (Electrocardiogram), respiration

sensors, amongst others have pointed the way

towards gathering the advantages of low-cost

equipment and non-medical environments with

interesting accuracy levels (Benevoy, 2008).

Figure 1: Russell’s Circumplex Model of Affect.

Recent studies have successfully used just EEG

information for emotion assessment (Teixeira,

2008). These approaches have the great advantage of

being based on non-invasive solutions, enabling its

usage in general population in a non-medical

environment. Encouraged by these results, the

current research direction seems to be the addition of

inexpensive, non-invasive hardware to the equation.

Examples of this are the introduction of a full set of

non-invasive, low-cost sensors by Vinhas (Vinhas,

2008), Kim (Kim, 2008) and Katsis (Katsis, 2008).

The usage of such equipments in diverse domains

and conditions suggests its high applicability and

progressive migration towards quotidian handling.

2.2 Aeronautical Simulation Tools

Simulation tools and simulated environments are

used in virtually every field of research, providing

researchers with the necessary means to develop

their work in a time- and cost-effective manner.

In the aviation field, simulation is heavily used,

from multi-million dollar full simulators used to

train professional pilots to freely available flight

simulators used mainly for entertainment purposes

and by aviation enthusiasts. In the past few years,

these low-cost software simulators have achieved a

higher level of realism.

There are two main simulator categories: Game

Engines and Flight Simulators. In game engines, the

most important aspect is an appealing visualization.

Flight Simulators have a different approach – the

main focus is on aerodynamics and flight factors

present in real world, thus trying to achieve as

realistic a flight as possible (Gimenes, 2008). The

academic and business communities have already

begun to use these cost-effective tools, benefitting

from what they have to offer (Lewis, 2002).

DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric

User Data towards Affective Immersive Environments

2.3 Fear of Flight

Several solutions are offered to treat pterygophobia,

including medication, and some behavior therapies,

including virtual reality solutions. These solutions

are often used in conjunction with a more

conventional form of therapy (Kazan, 2000),(da

Costa, 2008). One such example is Virtually Better,

a clinic which offers several solutions based on

virtual reality technology to support therapy in

anxiety disorders (Anderson, 2006),(Rothbaum,

2006). However, and despite having around fifty

clinics worldwide – the majority located within the

United States – it cannot offer its solutions to a very

wide audience at an affordable cost. Some

companies, such as Virtual Aviation, offer an even

more realistic experience, using the same multi-

million dollar simulators used to train professional

pilots (Bird, 2008).

3 PROJECT DESCRITPION

3.1 Global Architecture

The system global architecture is based on

independent and distributed modules, both in logic

and physical terms. As depicted in Figure 2, and

following its enclosed numeration, it is possible to

appreciate that biometric data is gathered directly

from the subject by using Nexus-10 hardware. In

more detail, temperature, GSR and respiration

sensors are used.

The BioSignal Collector software was developed in

order to access the recorded data and make it fully

available for further processing either by database

access or online TCP/IP socket connection. In this

last category, lies the Emotion Classifier, as it is

responsible for user’s emotion state assessment –

how this process is conducted is fully described in

the next subsection. The continuous extracted

emotional states are projected into the Russell’s

model and are filled as inputs for the Aeronautical

Simulator’s Control Software module. This module,

in turn, communicates with FSX, changing its

internal variables in order to match the desired

quadrant, and as explained in more detail in section

3.3. This module also produces a permanent log file,

with information collected from the simulator

regarding location and attitude of the user plane. The

simulator interacts with the user through immersive

3D video hardware, which allows the user to control

the visualization of the simulation.

Figure 2: System’s Global Architecture.

3.2 Emotion Assessment

The emotion assessment module is based on the

enunciated 4-channel biometric data collected with

Nexus-10 and accessed via text file readings at 10Hz

sample rate – which for the analyzed features is

perfectly acceptable. At the same rate, emotional

states are assessed and its definition is continuously

uploaded to a database for additional analysis and

third-party tools access. Directly related to the

aeronautical simulation, the GUI also provides an

expedite method to define the session’s emotional

policy, as it can be defined to force a specific

quadrant, contradict or maintain the current

emotional state or simply tour the four scenarios.

3.2.1 Base Emotion Model

As previously referred, the adopted emotion model

was Russell’s Circumplex Model of Affect. This

bidimensional approach permits efficient, yet

effective, online emotional assessment with none or

residual historical data as it is based on single

valence and arousal values. The key issue is not the

determination of the subject’s emotional state given

a pair of valence/arousal values, but how to convert

biosignals into valence/arousal pairs.

In order to anticipate the assessment of

emotional data pair values, a normalization process

is conducted, where both valence and arousal values

are fully mapped into the [-1,1] spectrum. With this

approach, emotional states are believed to be

identified by Cartesian points in a 2D environment.

ICEIS 2009 - International Conference on Enterprise Information Systems

3.2.2 Calibration & Channel Fusion

Having into consideration the referred normalization

process, one ought to point out the importance of the

calibration process. Although, the 2D point (-¾, ¾)

represents a normalized defined emotional state, it

can be achieved by an infinite conjugation of

biosignals. This reality leads to the necessity of

calibration and biometric channels fusion.

The first procedure consists in for each subject

and for each session, and once the biometric data

stream is enabled, pinpoint directly in Russell’s

model what is his predominant emotional state,

through a self-assessment process. By performing

this action, it is possible to define a normalized

emotional baseline point. For each of the four

channels taken into account for emotional state

assessment an initial twenty percent variability is

considered. Whenever overflow is detected, the

dynamic scaling is activated as described in the

following subsection.

The three components were considered to have

similar impact. For the valence values deviation,

only GSR was considered. For this computation, the

normalized baseline point is considered as reference.

The conjugation of such weights determines the

normalized values of arousal and valence and

therefore the current emotional state.

3.2.3 Dynamic Scaling

As a consequence of the emotional classification

process illustrated in the last two subsections, one

issue that emerges concerns either biosignal

readings’ overflow or underflow considering the

user-defined baseline and initial tolerance allowed.

To overcome this limitation, a fully dynamic

scaling approach was considered, that consists in

stretching the biometric signal scale whenever its

readings go beyond the normalized interval of [-1,1].

This scale update is conducted independently for

each biometric channel. During this process, a non-

linear scale disruption is created, resulting in greater

scale density towards the limit breach.

In order to better understand this approach, one

shall refer to the set of formulas listed through

Equation 1, where four steps are depicted

concerning an overflow situation.

First, c1 – any given particular biometric channel –

maximum value is determined by comparing the

current reading with the stored value – Equation

1(a). If the threshold is broken, the system

recalculates the linear scale factor for values greater

than the baseline neutral value, having as a direct

consequence the increasing of the interval’s density–

Equation 1(b). Based on the new interval definition,

subsequent channel values shall be normalized

accordingly – Equation 1(c) (d). With this approach,

together with dynamic calibration and data

normalization, it becomes possible to perform real-

time adaptations as a result of user peculiarities and

signal deviations, assuring continuous values.

(a)

[]

),(.

111

IndexcSampleMaxcMaxMathMaxc

(b)

[]

IndexcmplebaseLineSaMaxc

AxisrmbaseLineNo

ScaleUpc

−

(c)

[

][]

IndexcmplebaseLineSaIndexcSamplec

−=

(d)

cScaleUpcAxisrmbaseLineNoNormc

Equation 1: Dynamic Scaling Formulas.

3.3 Aeronautical Simulation

The main aeronautical simulation module

communicates with FSX through the SimConnect

API, changing internal variables. The desired

emotional quadrant affects the simulation in three

dimensions: weather, scenery and maneuvering.

The two quadrants characterized by a state of

displeasure are associated with worse climacteric

conditions. The two other quadrants are associated

with fair weather, producing a more stable flight.

The chosen scenery is an archipelago, a set that

can provide both a pleasant flight, with many

sightseeing moments, and an irregular one.

All maneuvers are done via the auto-pilot system

present in the simulated aircraft. Given the desired

waypoint, the heading is calculated, then passed on

to the heading control of the auto-pilot system. For

the first route, typical auto-pilot controls are active,

namely speed, heading and altitude controls. As for

the second route, two additional auto-pilot features

are applied – maximum bank and yaw damper. The

first limits the maximum plane declination during

turns, while the second reduces rolling and yawing

oscillations, making the flight smoother and calmer.

4 EXPERIMENTAL ACTIVITIES

The experiments were conducted using a wide

variety of hardware equipment, for both modules. As

for the first one, sensors for skin temperature, GSR

and respiration rate and amplitude were used. As for

the second module, FSX was used as the simulation

environment, providing access to internal variables

and a realistic visualization. In order to present the

user with an immersive experience, 3D video

hardware was used, in the form of virtual reality

DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric

User Data towards Affective Immersive Environments

video eyewear, which provides the user with a three

DOF head-tracker, allowing the user to experience

the environment as if he was actually there.

The experiment was comprised of three

sequential stages. In the first phase, the plane takes

off from an airport. The choice of the airport to

takeoff from was based on whether the subject

suffered from fear of flying. For individuals

suffering from pterygophobia, the operator forced

either the third of fourth quadrant, providing a calm

takeoff and flight, as not to trigger an anxiety attack.

For the remaining of the individuals, the operator

forced one of the first or second quadrants, trying to

obtain increased amplitude of emotional responses.

After takeoff, a series of closed circuits was

performed. Finally, in the landing phase, the plane

lines up with the selected airport, makes the

approach and lands.

The experiments were conducted among twenty

subjects, 13 male and 7 female, between the ages of

21 and 56. Four of the subjects stated that they had

some level of fear of flying, while the remaining did

not. Of the subjects suffering from pterygophobia,

three of them revealed that they have in fact never

flown, only one actually having suffered from the

symptoms associated with this phobia.

After concluding the trial, the subjects were then

asked to describe the experience, and to review an

animation of the evolution of both the simulation

and the emotional assessment and to confirm or to

refute those assessments. For the case of the four

subjects that stated to suffer from fear of flying, they

were asked to repeat the experiment two more times,

in order to obtain results that could enlighten the

possible use of this tool in treating pterygophobia.

5 RESULTS

In what concerns to emotion assessment, the

validation model was based on user self-assessment,

as previously described. These results were collected

in two forms: the first concerning single emotions

and specific regions on Russell’s model, and the

second concerning only the four quadrants. For the

first method, a success rate of 78% was achieved. If

only the four quadrants are considered, this number

increases to 87%. Table 1 shows the confusion table

with the percentages of automatic assessment versus

user self-assessment for each quadrant.

One additional result is that users tend to locate

their emotional states in the 1

and 2

quadrants.

Another aspect is that the emotion assessment has a

lower failure rate for opposite quadrants.

Table 1: Emotion Assessment Confusion Table.

Quadrant 2

Quadrant 3

Quadrant 4

Quadrant

30,7 1,8 0,3 1,2

Quadrant

3,1 32,8 10,1

Quadrant

0,2 1,7 10,9 1,2

Quadrant

10,11,612,3

Users

AutomaticAssessment

Concerning the simulation, users were asked to

describe their experience, and to classify, in a scale

of 1 to 5, the immersiveness level. The results show

that the majority of the individuals considered the

environment to be highly immersive, with an

average classification of 4,2.

Regarding the subjects with fear of flying, one

outcome that seems to support the fact that this

simulation can be used in pterygophobia treatment is

depicted in Figure 3, which shows the average

emotional response for the three experiments. The

and 3

experiments show an emotional response

that tends to move away from the extreme end of the

quadrant, denoting a reduction in the levels of

fear registered during the latter experiments.

Figure 3: Pterygophobia Subjects’ Emotional Trend.

Takeoff and landing are traditionally associated

with higher levels of apprehension and anxiety

among pterygophobia-suffering passengers, a fact

confirmed by the experimental results. All subjects

afraid of flying stated that those are in fact the most

stressful moments, and the collected data

corroborates this fact. Calculating the average

arousal levels measured during the experiments

conducted among these individuals, higher levels

were registered during the initial and final stages of

the simulation, which represent takeoff and landing.

6 CONCLUSIONS

The distributed architecture proved to be reliable and

efficient, and enabled independence between

biometric data collection and processing, and

simulation computation. It also provided database

collection of both raw biometric values and

emotional state for future analysis and validation.

The emotional assessment layer reached high

levels of accuracy, as the depicted results show.

Through the previously detailed validation process,

ICEIS 2009 - International Conference on Enterprise Information Systems

78% of the emotional states were believed to be

correct by the subjects. If classification is simplified

to quadrant determination, this value reaches 87%,

which supports the conclusion of an effective

emotional assessment process. It is worth to mention

the on-the-fly classification procedure that nearly

suppresses the need to long baseline data gathering

and user identification as this baseline evaluation is

performed by the user at any given time and can be

adjusted. Also, the dynamic scaling was found to be

useful, in order to correctly accommodate outsized

signal deviations without precision loss.

In what regards the aeronautical simulation,

users confirmed their immersion sensation, by both

self-awareness and biological recorded response. It

is believed that the usage of 3D glasses as display

device played a particularly important role in

creating the appropriate environment.

The results seem to suggest that a significant

mitigation of the symptoms of pterygophobia was

achieved among the subjects that referred at least

some level of fear of flying.

In spite of the project’s overall success, several

improvement opportunities have been identified,

such as the inclusion of additional biometric

channels, such as ECG, BVP and EEG. This signals

integration would be fairly transparent as the current

data fusion process and emotional base model

support that kind of enhancement. Regarding the

aeronautical simulator, it would be interesting to

define and test more scenarios with fully automated

and dynamically configured take-off and landing.

As a final project summary, one shall point the

fact that the proposed system has a dual application

as a complete entertainment system with user

emotional awareness that continuously adapt the

multimedia content accordingly to user’s states and a

solemn approach as a phobia treatment auxiliary.

REFERENCES

Altarriba, J., Basnight, D. M., & Canary, T. M., 2003.

Emotion representation and perception across

cultures. In W. J. Lonner, D. L. Dinnel, S. A. Hayes,

& D. N. Sattler (Eds.), Readings in Psychology and

Culture (Unit 4, Chapter 5), Center for Cross-Cultural

Research, WWU, Bellingham, WA, USA.

Anderson, P., Jacobs, C.H., Lindner, G.K., Edwards, S.,

Zimand, E., Hodges, L., Rothbaum, B.O., 2006.

Cognitive behavior therapy for fear of flying:

sustainability of treatment gains after September 11.

Behavior Therapy, Volume 37, Issue 1, pp. 91-97.

Benovoy, M., Cooperstock, J., Deitcher, J., 2008.

Biosignals Analysis and its Application in a

Performance Setting - Towards the Development of an

Emotional-Imaging Generator. In Proceedings of the

First International Conference on Biomedical

Electronics and Devices, pp. 253-258.

Bird, S., 2005. You’re Cleared for Take-Off - How to

Tackle a Fear of Flying on a Virtual Plane. The

Times, March 2005. Available online at

http://www.timesonline.co.uk/tol/life_and_style/health

/features/article549165.ece. Consulted Dec. 2008.

da Costa, R., Sardinha, A., Nardi, A., 2008. Virtual Reality

Exposure in the Treatment of Fear of Flying. Aviation,

Space, and Environmental Medicine, 79(9), pp. 899–

903.

Gimenes, R., Silva D.C.,Reis, LP, Oliveira, E., 2008.

Flight Simulation Environments Applied to Agent-

Based Autonomous UAVs. In Proceedings of ICEIS

2008, pp. 243–246.

Kahan, M., Tanzer, J., Darvin, D., and Borer, F., 2000.

Virtual Reality-Assisted Cognitive-Behavioral

Treatment for Fear of Flying: Acute Treatment and

Follow-up. CyberPsychology & Behavior, 3(3), pp.

387–392.

Katsis, C., Katertsidis N., Ganiatsas G. and Fotiadis D.,

2008. Towards Emotion Recognition in Car-Racing

Drivers: A Biosignal Processing Approach. IEEE

Transactions on Systems, Man, and Cybernetics – Part

A: Systems and Humans, Vol. 38, pp. 502-512.

Kim, J., André, E., 2008. Multi-Channel BioSignal

Analysis for Automatic Emotion Recognition. In

Proceedings of the First International Conference on

Biomedical Electronics and Devices.

Lewis, M., Jacobson, J., 2002. Game Engines in Scientific

Research. Communications of the ACM. Vol. 45, Nr.

1, pp. 27-31.

Rothbaum, B. O., Anderson, P., Zimand, E., Hodges, L.,

Lang, D., & Wilson, J., 2006. Virtual reality exposure

therapy and standard (in vivo) exposure therapy in the

treatment for the fear of flying. Behavior Therapy,

Vol. 37, Issue 1, pp. 80-90.

Russell, J. A., 1980. A Circumplex Model of Affect. In

Journal of Personality and Social Psychology, 39, pp.

1161-1178.

Stoller, G., 2006. Fear of Flying can Cripple Workers,

USA Today, March 2006. Available online at

http://www.usatoday.com/educate/college/business/art

icles/20060326.htm. Consulted Dec. 2008.

Teixeira, J., Vinhas, V., Oliveira, E., Reis, LP., 2008.

Emotion Assessment Tool for Human-Machine

Interfaces - Using EEG Data and Multimedia Stimuli

Towards Emotion Classification. In SIGMAP, pp.

185-188.

Vinhas, V., Oliveira, E., Reis, LP., 2008. Realtime

Dynamic Multimedia Storyline Based on Online

Audience Biometric Information. In KES IIMSS, pp.

545-554.

DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric

User Data towards Affective Immersive Environments