DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON
REALTIME USER EMOTION ASSESSMENT
Biometric User Data towards Affective Immersive Environments
Vasco Vinhas, Daniel Castro Silva, Eugénio Oliveira and Luís Paulo Reis
FEUP-DEI / LIACC
Rua Dr. Roberto Frias s/n 4200-465 Porto, Portugal
Keywords: Emotion Assessment, Biometric Readings, Immersive Digital Environments, Aeronautical Simulation.
Abstract: Both the academic and industry sectors have increased their attention and investment to the fields of
Affective Computing and immersive digital environments, the latter imposing itself as a reliable domain,
with increasingly cheaper hardware solutions. With all this in mind, the authors envisioned an immersive
dynamic digital environment tied with automatic real-time user emotion assessment through biometric
readings. The environment consisted in an aeronautical simulation, with internal variables such as flight
plan, weather conditions and maneuver smoothness dynamically altered by the assessed emotional state of
the user, based on biometric readings, including galvanic skin response, respiration rate and amplitude and
phalange temperature. The results were consistent with the emotional states reported by the users, with a
success rate of 78%.
1 INTRODUCTION
The presence of dissimulated sensors, actuators and
processing units in unconventional contexts is
becoming consistently inexorable. This fact brings to
both academic and industrial stages the term of
Ubiquitous Computing as a regular one. In a
parallel, yet complementary line, Affective
Computing has recently gained the attention of
researchers and business organizations worldwide.
As a common denominator for these two concepts
resides Emotion Assessment. Although this topic is
no novelty by itself, it has been rediscovered in light
of the mentioned knowledge areas breakthroughs, as
it became theoretically possible to perform real-time
minimal-invasive user emotion assessment based on
live biosignals at economically feasible levels.
Having this in mind, the authors envisioned an
integrated interactive multimedia bidirectional
system where internal parameters would be changed
according to the user’s emotional response. As the
application example in this paper, an immersive
aviation-based environment was considered. The
main reasons behind this decision are related to the
human fascination for everything related to flying.
However, and as with most things, this attraction co-
exists with the fear of flying, usually referred to as
pterygophobia. According to a poll by CNN and
Gallup for the USA Today in March 2006, 27% of
U.S. adults would be at least somewhat fearful of
getting on an airplane (Stoller, 2006). By using the
briefly described multimedia system, the authors
were able to provide distinct practical scenarios to
apply in several situations that range from traditional
entertainment applications to therapeutic phobia
treatment. The conducted experimental protocol was
carried out in a controlled environment where
subjects assumed the pilot’s seat for roughly 25
minutes. Internal variables were unconscientiously
affected by the online assessed user emotions.
The project achieved transversal goals as it was
possible to use it as a fully functional testbed for
online biometric emotion assessment through
galvanic skin response, respiration rate and
amplitude and phalange temperature readings fusion
and its incorporation with Russell’s Circumplex
Model of Affect (Russell, 1980) with success rates
of around 78%. It was found that those without fear
of flying found the experience rather amusing, as
virtual entertainment, while the others considered
the simulation realistic enough to trigger an
emotional response – verified by biometric readings.
The results also suggested a trend to pterygophobia
mitigation.
42
Vinhas V., Castro Silva D., Oliveira E. and Paulo Reis L. (2009).
DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric User Data towards Affective Immersive
Environments.
In Proceedings of the 11th International Conference on Enterprise Information Systems - Human-Computer Interaction, pages 42-47
DOI: 10.5220/0001984100420047
Copyright
c
SciTePress
2 STATE OF THE ART
This section is divided in three concerning automatic
emotion assessment; aeronautical simulation tools;
and pterygophobia treatments.
2.1 Automatic Emotion Assessment
Until a recent past, researchers in the domains
related to emotion assessment had very few solid
ground standards both for specifying the emotional
charge of stimuli and also a reasonable acceptable
emotional state representation model. This issue
constituted a serious obstacle for research
comparison and conclusion validation. The extreme
need of such metrics led to several attempts to
systematize this knowledge domain.
One of the major models of emotion
representation is the Circumplex Model of Affect
proposed by Russell. This is a spatial model based
on dimensions of affect that are interrelated in a very
methodical fashion (Russell, 1980). Affective
concepts fall in a circle in the following order:
pleasure, excitement, arousal, distress, displeasure,
depression, sleepiness, and relaxation – see Figure 1.
According to this model, there are two components
of affect that exist: the first is pleasure-displeasure,
the horizontal dimension of the model, and the
second is arousal-sleep, the vertical dimension of the
model. Therefore, it seems that any affect stimuli
can be defined in terms of its valence and arousal
components. The remaining variables mentioned
above do not act as dimensions, but rather help to
define the quadrants of the affective space. Although
the existence of criticism concerning the impact
different cultures in emotion expression and
induction, as discussed by Altarriba (Altarriba,
2003), Russell’s model is relatively immune to this
issue if the stimuli are universally defined.
Emotions assessment requires reliable and
accurate communications with the subject so that the
results are conclusive and the emotions correctly
classified. This communication can occur through
several channels and is supported by specific
equipment. The invasive methods are clearly more
precise, however more dangerous and will not be
considered for this study. Conversely, non invasive
methods such as EEG (Electroencephalography),
GSR (Galvanic Skin Response), oximeter, skin
temperature, ECG (Electrocardiogram), respiration
sensors, amongst others have pointed the way
towards gathering the advantages of low-cost
equipment and non-medical environments with
interesting accuracy levels (Benevoy, 2008).
Figure 1: Russell’s Circumplex Model of Affect.
Recent studies have successfully used just EEG
information for emotion assessment (Teixeira,
2008). These approaches have the great advantage of
being based on non-invasive solutions, enabling its
usage in general population in a non-medical
environment. Encouraged by these results, the
current research direction seems to be the addition of
inexpensive, non-invasive hardware to the equation.
Examples of this are the introduction of a full set of
non-invasive, low-cost sensors by Vinhas (Vinhas,
2008), Kim (Kim, 2008) and Katsis (Katsis, 2008).
The usage of such equipments in diverse domains
and conditions suggests its high applicability and
progressive migration towards quotidian handling.
2.2 Aeronautical Simulation Tools
Simulation tools and simulated environments are
used in virtually every field of research, providing
researchers with the necessary means to develop
their work in a time- and cost-effective manner.
In the aviation field, simulation is heavily used,
from multi-million dollar full simulators used to
train professional pilots to freely available flight
simulators used mainly for entertainment purposes
and by aviation enthusiasts. In the past few years,
these low-cost software simulators have achieved a
higher level of realism.
There are two main simulator categories: Game
Engines and Flight Simulators. In game engines, the
most important aspect is an appealing visualization.
Flight Simulators have a different approach – the
main focus is on aerodynamics and flight factors
present in real world, thus trying to achieve as
realistic a flight as possible (Gimenes, 2008). The
academic and business communities have already
begun to use these cost-effective tools, benefitting
from what they have to offer (Lewis, 2002).
DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric
User Data towards Affective Immersive Environments
43
2.3 Fear of Flight
Several solutions are offered to treat pterygophobia,
including medication, and some behavior therapies,
including virtual reality solutions. These solutions
are often used in conjunction with a more
conventional form of therapy (Kazan, 2000),(da
Costa, 2008). One such example is Virtually Better,
a clinic which offers several solutions based on
virtual reality technology to support therapy in
anxiety disorders (Anderson, 2006),(Rothbaum,
2006). However, and despite having around fifty
clinics worldwide – the majority located within the
United States – it cannot offer its solutions to a very
wide audience at an affordable cost. Some
companies, such as Virtual Aviation, offer an even
more realistic experience, using the same multi-
million dollar simulators used to train professional
pilots (Bird, 2008).
3 PROJECT DESCRITPION
3.1 Global Architecture
The system global architecture is based on
independent and distributed modules, both in logic
and physical terms. As depicted in Figure 2, and
following its enclosed numeration, it is possible to
appreciate that biometric data is gathered directly
from the subject by using Nexus-10 hardware. In
more detail, temperature, GSR and respiration
sensors are used.
The BioSignal Collector software was developed in
order to access the recorded data and make it fully
available for further processing either by database
access or online TCP/IP socket connection. In this
last category, lies the Emotion Classifier, as it is
responsible for user’s emotion state assessment –
how this process is conducted is fully described in
the next subsection. The continuous extracted
emotional states are projected into the Russell’s
model and are filled as inputs for the Aeronautical
Simulator’s Control Software module. This module,
in turn, communicates with FSX, changing its
internal variables in order to match the desired
quadrant, and as explained in more detail in section
3.3. This module also produces a permanent log file,
with information collected from the simulator
regarding location and attitude of the user plane. The
simulator interacts with the user through immersive
3D video hardware, which allows the user to control
the visualization of the simulation.
Figure 2: System’s Global Architecture.
3.2 Emotion Assessment
The emotion assessment module is based on the
enunciated 4-channel biometric data collected with
Nexus-10 and accessed via text file readings at 10Hz
sample rate – which for the analyzed features is
perfectly acceptable. At the same rate, emotional
states are assessed and its definition is continuously
uploaded to a database for additional analysis and
third-party tools access. Directly related to the
aeronautical simulation, the GUI also provides an
expedite method to define the session’s emotional
policy, as it can be defined to force a specific
quadrant, contradict or maintain the current
emotional state or simply tour the four scenarios.
3.2.1 Base Emotion Model
As previously referred, the adopted emotion model
was Russell’s Circumplex Model of Affect. This
bidimensional approach permits efficient, yet
effective, online emotional assessment with none or
residual historical data as it is based on single
valence and arousal values. The key issue is not the
determination of the subject’s emotional state given
a pair of valence/arousal values, but how to convert
biosignals into valence/arousal pairs.
In order to anticipate the assessment of
emotional data pair values, a normalization process
is conducted, where both valence and arousal values
are fully mapped into the [-1,1] spectrum. With this
approach, emotional states are believed to be
identified by Cartesian points in a 2D environment.
ICEIS 2009 - International Conference on Enterprise Information Systems
44
3.2.2 Calibration & Channel Fusion
Having into consideration the referred normalization
process, one ought to point out the importance of the
calibration process. Although, the 2D point (-¾, ¾)
represents a normalized defined emotional state, it
can be achieved by an infinite conjugation of
biosignals. This reality leads to the necessity of
calibration and biometric channels fusion.
The first procedure consists in for each subject
and for each session, and once the biometric data
stream is enabled, pinpoint directly in Russell’s
model what is his predominant emotional state,
through a self-assessment process. By performing
this action, it is possible to define a normalized
emotional baseline point. For each of the four
channels taken into account for emotional state
assessment an initial twenty percent variability is
considered. Whenever overflow is detected, the
dynamic scaling is activated as described in the
following subsection.
The three components were considered to have
similar impact. For the valence values deviation,
only GSR was considered. For this computation, the
normalized baseline point is considered as reference.
The conjugation of such weights determines the
normalized values of arousal and valence and
therefore the current emotional state.
3.2.3 Dynamic Scaling
As a consequence of the emotional classification
process illustrated in the last two subsections, one
issue that emerges concerns either biosignal
readings’ overflow or underflow considering the
user-defined baseline and initial tolerance allowed.
To overcome this limitation, a fully dynamic
scaling approach was considered, that consists in
stretching the biometric signal scale whenever its
readings go beyond the normalized interval of [-1,1].
This scale update is conducted independently for
each biometric channel. During this process, a non-
linear scale disruption is created, resulting in greater
scale density towards the limit breach.
In order to better understand this approach, one
shall refer to the set of formulas listed through
Equation 1, where four steps are depicted
concerning an overflow situation.
First, c1 – any given particular biometric channel
maximum value is determined by comparing the
current reading with the stored value – Equation
1(a). If the threshold is broken, the system
recalculates the linear scale factor for values greater
than the baseline neutral value, having as a direct
consequence the increasing of the interval’s density–
Equation 1(b). Based on the new interval definition,
subsequent channel values shall be normalized
accordingly – Equation 1(c) (d). With this approach,
together with dynamic calibration and data
normalization, it becomes possible to perform real-
time adaptations as a result of user peculiarities and
signal deviations, assuring continuous values.
(a)
[]
),(.
111
IndexcSampleMaxcMaxMathMaxc
=
(b)
[]
IndexcmplebaseLineSaMaxc
AxisrmbaseLineNo
ScaleUpc
11
1
.1
=
(c)
[
][]
IndexcmplebaseLineSaIndexcSamplec
11
=
(d)
cScaleUpcAxisrmbaseLineNoNormc
×
+
=
11
.
Equation 1: Dynamic Scaling Formulas.
3.3 Aeronautical Simulation
The main aeronautical simulation module
communicates with FSX through the SimConnect
API, changing internal variables. The desired
emotional quadrant affects the simulation in three
dimensions: weather, scenery and maneuvering.
The two quadrants characterized by a state of
displeasure are associated with worse climacteric
conditions. The two other quadrants are associated
with fair weather, producing a more stable flight.
The chosen scenery is an archipelago, a set that
can provide both a pleasant flight, with many
sightseeing moments, and an irregular one.
All maneuvers are done via the auto-pilot system
present in the simulated aircraft. Given the desired
waypoint, the heading is calculated, then passed on
to the heading control of the auto-pilot system. For
the first route, typical auto-pilot controls are active,
namely speed, heading and altitude controls. As for
the second route, two additional auto-pilot features
are applied – maximum bank and yaw damper. The
first limits the maximum plane declination during
turns, while the second reduces rolling and yawing
oscillations, making the flight smoother and calmer.
4 EXPERIMENTAL ACTIVITIES
The experiments were conducted using a wide
variety of hardware equipment, for both modules. As
for the first one, sensors for skin temperature, GSR
and respiration rate and amplitude were used. As for
the second module, FSX was used as the simulation
environment, providing access to internal variables
and a realistic visualization. In order to present the
user with an immersive experience, 3D video
hardware was used, in the form of virtual reality
DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric
User Data towards Affective Immersive Environments
45
video eyewear, which provides the user with a three
DOF head-tracker, allowing the user to experience
the environment as if he was actually there.
The experiment was comprised of three
sequential stages. In the first phase, the plane takes
off from an airport. The choice of the airport to
takeoff from was based on whether the subject
suffered from fear of flying. For individuals
suffering from pterygophobia, the operator forced
either the third of fourth quadrant, providing a calm
takeoff and flight, as not to trigger an anxiety attack.
For the remaining of the individuals, the operator
forced one of the first or second quadrants, trying to
obtain increased amplitude of emotional responses.
After takeoff, a series of closed circuits was
performed. Finally, in the landing phase, the plane
lines up with the selected airport, makes the
approach and lands.
The experiments were conducted among twenty
subjects, 13 male and 7 female, between the ages of
21 and 56. Four of the subjects stated that they had
some level of fear of flying, while the remaining did
not. Of the subjects suffering from pterygophobia,
three of them revealed that they have in fact never
flown, only one actually having suffered from the
symptoms associated with this phobia.
After concluding the trial, the subjects were then
asked to describe the experience, and to review an
animation of the evolution of both the simulation
and the emotional assessment and to confirm or to
refute those assessments. For the case of the four
subjects that stated to suffer from fear of flying, they
were asked to repeat the experiment two more times,
in order to obtain results that could enlighten the
possible use of this tool in treating pterygophobia.
5 RESULTS
In what concerns to emotion assessment, the
validation model was based on user self-assessment,
as previously described. These results were collected
in two forms: the first concerning single emotions
and specific regions on Russell’s model, and the
second concerning only the four quadrants. For the
first method, a success rate of 78% was achieved. If
only the four quadrants are considered, this number
increases to 87%. Table 1 shows the confusion table
with the percentages of automatic assessment versus
user self-assessment for each quadrant.
One additional result is that users tend to locate
their emotional states in the 1
st
and 2
nd
quadrants.
Another aspect is that the emotion assessment has a
lower failure rate for opposite quadrants.
Table 1: Emotion Assessment Confusion Table.
1
st
Quadrant 2
nd
Quadrant 3
rd
Quadrant 4
th
Quadrant
1
st
Quadrant
30,7 1,8 0,3 1,2
2
nd
Quadrant
3,1 32,8 10,1
3
rd
Quadrant
0,2 1,7 10,9 1,2
4
th
Quadrant
10,11,612,3
Users
AutomaticAssessment
Concerning the simulation, users were asked to
describe their experience, and to classify, in a scale
of 1 to 5, the immersiveness level. The results show
that the majority of the individuals considered the
environment to be highly immersive, with an
average classification of 4,2.
Regarding the subjects with fear of flying, one
outcome that seems to support the fact that this
simulation can be used in pterygophobia treatment is
depicted in Figure 3, which shows the average
emotional response for the three experiments. The
2
nd
and 3
rd
experiments show an emotional response
that tends to move away from the extreme end of the
2
nd
quadrant, denoting a reduction in the levels of
fear registered during the latter experiments.
Figure 3: Pterygophobia Subjects’ Emotional Trend.
Takeoff and landing are traditionally associated
with higher levels of apprehension and anxiety
among pterygophobia-suffering passengers, a fact
confirmed by the experimental results. All subjects
afraid of flying stated that those are in fact the most
stressful moments, and the collected data
corroborates this fact. Calculating the average
arousal levels measured during the experiments
conducted among these individuals, higher levels
were registered during the initial and final stages of
the simulation, which represent takeoff and landing.
6 CONCLUSIONS
The distributed architecture proved to be reliable and
efficient, and enabled independence between
biometric data collection and processing, and
simulation computation. It also provided database
collection of both raw biometric values and
emotional state for future analysis and validation.
The emotional assessment layer reached high
levels of accuracy, as the depicted results show.
Through the previously detailed validation process,
ICEIS 2009 - International Conference on Enterprise Information Systems
46
78% of the emotional states were believed to be
correct by the subjects. If classification is simplified
to quadrant determination, this value reaches 87%,
which supports the conclusion of an effective
emotional assessment process. It is worth to mention
the on-the-fly classification procedure that nearly
suppresses the need to long baseline data gathering
and user identification as this baseline evaluation is
performed by the user at any given time and can be
adjusted. Also, the dynamic scaling was found to be
useful, in order to correctly accommodate outsized
signal deviations without precision loss.
In what regards the aeronautical simulation,
users confirmed their immersion sensation, by both
self-awareness and biological recorded response. It
is believed that the usage of 3D glasses as display
device played a particularly important role in
creating the appropriate environment.
The results seem to suggest that a significant
mitigation of the symptoms of pterygophobia was
achieved among the subjects that referred at least
some level of fear of flying.
In spite of the project’s overall success, several
improvement opportunities have been identified,
such as the inclusion of additional biometric
channels, such as ECG, BVP and EEG. This signals
integration would be fairly transparent as the current
data fusion process and emotional base model
support that kind of enhancement. Regarding the
aeronautical simulator, it would be interesting to
define and test more scenarios with fully automated
and dynamically configured take-off and landing.
As a final project summary, one shall point the
fact that the proposed system has a dual application
as a complete entertainment system with user
emotional awareness that continuously adapt the
multimedia content accordingly to user’s states and a
solemn approach as a phobia treatment auxiliary.
REFERENCES
Altarriba, J., Basnight, D. M., & Canary, T. M., 2003.
Emotion representation and perception across
cultures. In W. J. Lonner, D. L. Dinnel, S. A. Hayes,
& D. N. Sattler (Eds.), Readings in Psychology and
Culture (Unit 4, Chapter 5), Center for Cross-Cultural
Research, WWU, Bellingham, WA, USA.
Anderson, P., Jacobs, C.H., Lindner, G.K., Edwards, S.,
Zimand, E., Hodges, L., Rothbaum, B.O., 2006.
Cognitive behavior therapy for fear of flying:
sustainability of treatment gains after September 11.
Behavior Therapy, Volume 37, Issue 1, pp. 91-97.
Benovoy, M., Cooperstock, J., Deitcher, J., 2008.
Biosignals Analysis and its Application in a
Performance Setting - Towards the Development of an
Emotional-Imaging Generator. In Proceedings of the
First International Conference on Biomedical
Electronics and Devices, pp. 253-258.
Bird, S., 2005. You’re Cleared for Take-Off - How to
Tackle a Fear of Flying on a Virtual Plane. The
Times, March 2005. Available online at
http://www.timesonline.co.uk/tol/life_and_style/health
/features/article549165.ece. Consulted Dec. 2008.
da Costa, R., Sardinha, A., Nardi, A., 2008. Virtual Reality
Exposure in the Treatment of Fear of Flying. Aviation,
Space, and Environmental Medicine, 79(9), pp. 899–
903.
Gimenes, R., Silva D.C.,Reis, LP, Oliveira, E., 2008.
Flight Simulation Environments Applied to Agent-
Based Autonomous UAVs. In Proceedings of ICEIS
2008, pp. 243–246.
Kahan, M., Tanzer, J., Darvin, D., and Borer, F., 2000.
Virtual Reality-Assisted Cognitive-Behavioral
Treatment for Fear of Flying: Acute Treatment and
Follow-up. CyberPsychology & Behavior, 3(3), pp.
387–392.
Katsis, C., Katertsidis N., Ganiatsas G. and Fotiadis D.,
2008. Towards Emotion Recognition in Car-Racing
Drivers: A Biosignal Processing Approach. IEEE
Transactions on Systems, Man, and Cybernetics – Part
A: Systems and Humans, Vol. 38, pp. 502-512.
Kim, J., André, E., 2008. Multi-Channel BioSignal
Analysis for Automatic Emotion Recognition. In
Proceedings of the First International Conference on
Biomedical Electronics and Devices.
Lewis, M., Jacobson, J., 2002. Game Engines in Scientific
Research. Communications of the ACM. Vol. 45, Nr.
1, pp. 27-31.
Rothbaum, B. O., Anderson, P., Zimand, E., Hodges, L.,
Lang, D., & Wilson, J., 2006. Virtual reality exposure
therapy and standard (in vivo) exposure therapy in the
treatment for the fear of flying. Behavior Therapy,
Vol. 37, Issue 1, pp. 80-90.
Russell, J. A., 1980. A Circumplex Model of Affect. In
Journal of Personality and Social Psychology, 39, pp.
1161-1178.
Stoller, G., 2006. Fear of Flying can Cripple Workers,
USA Today, March 2006. Available online at
http://www.usatoday.com/educate/college/business/art
icles/20060326.htm. Consulted Dec. 2008.
Teixeira, J., Vinhas, V., Oliveira, E., Reis, LP., 2008.
Emotion Assessment Tool for Human-Machine
Interfaces - Using EEG Data and Multimedia Stimuli
Towards Emotion Classification. In SIGMAP, pp.
185-188.
Vinhas, V., Oliveira, E., Reis, LP., 2008. Realtime
Dynamic Multimedia Storyline Based on Online
Audience Biometric Information. In KES IIMSS, pp.
545-554.
DYNAMIC MULTIMEDIA ENVIRONMENT BASED ON REALTIME USER EMOTION ASSESSMENT - Biometric
User Data towards Affective Immersive Environments
47