AUTOMATIC EMOTION INDUCTION AND
ASSESSMENT FRAMEWORK
Enhancing User Interfaces by Interperting Users Multimodal Biosignals
Jorge Teixeira, Vasco Vinhas, Luís Paulo Reis and Eugénio Oliveira
FEUP - Faculdade de Engenharia da Universidade do Porto, Rua Dr. Roberto Frias s/n, Porto, Portugal
DEI - Departamento de Engenharia Informática, Rua Dr. Roberto Frias s/n, Porto, Portugal
LIACC - Laboratório de Inteligência Artificial e Ciência de Computadores, Rua do Campo Alegre 823, Porto, Portugal
Keywords: Biosignals, Emotions, Classification, Multimedia, Clustering.
Abstract: Emotion’s definition, identification, systematic induction and efficient and reliable classification have been
themes to which several complementary knowledge areas such as psychology, medicine and computer
science have been dedicating serious investments. This project consists in developing an automatic tool for
emotion assessment based on a dynamic biometric data acquisition set as galvanic skin response and
electroencephalography are practical examples. The output of standard emotional induction methods is the
support for classification based on data analysis and processing. The conducted experimental sessions,
alongside with the developed support tools, allowed the extraction on conclusions such as the capability of
effectively performing automatic classification of the subject’s predominant emotional state. Self
assessment interviews validated the developed tool's success rate of approximately 75%. It was also
experimentally strongly suggested that female subjects are emotionally more active and easily induced than
males.
1 INTRODUCTION
Emotions play an important role in all human
activities, from the trivial to the most complex ones.
This significance is translated both in terms of
reality perception and even in the cognitive decision
process. Meanwhile, computers have gained such a
relevant presence in the modern society that they
have been introduced in almost every aspect of it,
enhancing the magnitude of ubiquitous computing.
Having these two realities in mind – the
importance of emotional states and the necessity of
daily interaction with multiple devices – merging
them would be a great improvement. By providing
the distributed computer systems with the perception
of their users’ emotions, the applications would be
able to adjust their interface, promote and suggest
functionalities accordingly. It is believed that this
approach would increase the global system’s
transparency and efficiency as its dynamism would
follow in the encounter to user’s intentions and
temper.
Alongside the ubiquitous computing, multimedia
contents are becoming constantly more complex and
seemlier to reality, enabling a greater action
immersion sensation, the primitive absolute need of
achieving a perfect match between audiovisual
contents and the audience desires is still present and
constitutes the main key to the industry success.
The alliance between the multimedia contents
choice possibility that enables the audience to
individually presence what desires and accurate
emotional states detection systems leads to
subconscious individual interaction between the
audience and the multimedia control system,
potentiating the perfect match between content and
individual audience desires.
This study illustrates a proposal for an
application that enables automatic emotional state
assessment using minimal invasive solutions.
The rest of the paper is organized as follows: in
the next section the current state of the art is
presented; in section 3 a project description is given;
in section 4 the study’s results are depicted and,
consequently, the project’s conclusions are listed
and future work areas are identified in the final
section.
487
Teixeira J., Vinhas V., Reis L. and Oliveira E. (2009).
AUTOMATIC EMOTION INDUCTION AND ASSESSMENT FRAMEWORK - Enhancing User Interfaces by Interperting Users Multimodal Biosignals.
In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing, pages 487-490
DOI: 10.5220/0001542104870490
Copyright
c
SciTePress
2 STATE OF THE ART
An emotional state can be defined as a collection of
responses triggered by different parts of the body or
the brain through both neural and hormonal
networks (Damásio, 1998). Experiments conducted
with patients with brain lesions in specific areas led
to the conclusion that their social behaviour was
highly affective, together with the emotional
responses. It is unequivocal to state that emotions
are essential for humans, as they play a vital role in
their everyday life: in perception, judgment and
action processes (Damásio, 1994).
Due to the complexity of human emotions, their
analysis is often based on the identification of
distinct basic emotional states so that the analysis
and processing is simplified. For this project, three
emotional states will be studied: joy, sadness and a
neutral emotional state.
In order to analyse biometric data that contains
mainly positive and negative emotional states, it is
essential to create and define an experimental
environment that is able to induce a subject in a
specific and controlled emotional state. Nowadays, it
is common to use an actor as one possible approach
to human beings emotions’ simulation (Chanel et al.,
2005). As the actor predicts specific emotions,
outside aspects as facial expression or voice change
accordingly. However, the physiological responses
will not suffer any variations, which lead to one of
the biggest disadvantages of this approach, as the
gathered biometric information does not represent
the real emotional state of the actor.
An alternative method, adopted in this study, is
the use of multimedia stimuli (Chanel et al., 2005).
These stimuli contain a variety of contents such as
music, videos, text and images. The main advantage
of this method resides in the strong correlation
between the induced emotional states and the
physiological responses, as the emotions are no
longer simulated.
The electroencephalograph used on this project
was the NeurobitLite and is composed by 3
electrodes, one functioning as and active one and the
other two as references, using a monopolar method
strategy.
3 PROJECT DESCRIPTION
The development of an automatic tool that able to
determine the emotional state of a subject through
EEG biometric information was based on data
analysis’ methods. These methods included the
decimation; the weighted average; spikes’ removal;
and clusters for the final emotion’s assessment.
The implementation of the weighted average
technique culminated with the enunciation of a
hypothesis concerning the behaviour of the electrical
brain waves when subjects are emotionally induced.
The hypothesis follows a specific temporal
distribution and a pattern that was observed in the
majority of the experimental sessions. Figure 3
represents the evolution of Beta and Gamma
amplitudes’ over the entire experimental session.
This behaviour is considered as the pattern
behaviour for high frequency brain waves (Teixeira,
Vinhas, 2008).
The enunciated expected behaviour represents
a three step chart with each step having the duration
of two minutes. This data treatment had in mind the
three steps took into account to IAPS session
management – three sets of twenty pictures lasting
for two minutes with a grading emotional effect
from joy to sadness passing through an intermediate
neutral state.
Figure 1: Expected behaviour for high frequency waves.
In what concerns to cluster analysis, the
emotional induction results in an amplitude’s
variation according to a specific emotional state.
Having these concepts in mind, and based on the
pattern-behaviour for the EEG data previously
described, three distinct groups of data were created
based on the brain waves’ mean amplitude. Each of
these three groups have one specific centroide, a
point that is used as a reference for the neighbours of
the same cluster.
The emotional state classification, performed by the
EAT, is based on the predominant emotional state of
the subject, as previously described. In order to
evaluate the success rate of the EAT classification,
at the end of each experimental session, self-
assessment interviews were performed to subjects.
The main objective of these interviews was to attain
information concerning the predominant emotional
state, so that it could be later compared with the
results obtained from the EAT analysis. Besides this
fact, other important aspects like the apreciation of
the presented visual stimuli sequence, the
environment conditions and any disturbs during the
experimental session have been collected.
BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing
488
These interviews constituted an essential
database for project results’ validation purposes.
The emotional induction is divided in three
distinct and perfectly defined stages: joy, sadness
and a neutral state. This allows determining which of
the stages is more efficient during the whole
experimental session or has stronger and more
coherent effect on the subject. This hypothesis is the
base of the concept for the EAT, so that it is able to
determine, based on the gathered EEG biometric
data, the predominant emotional state of the subject.
Based on the statistical analysis, each of the steps
is directly associated with one of the clusters, so that
there are three distinct clusters per experimental
session. The clusters’ analysis is based on the
centroides’ values and the number of samples, so
that the global organization of the samples is
different for each emotional state.
Aside with these characteristics, the emotions’
assessment was performed for both Beta and
Gamma brain waves, since high frequency brain
waves are believed to have the most noteworthy
changes due to emotional states transition (Teixeira,
Vinhas et al, 2008).
4 RESULTS
From the experimental sessions conducted two
different kind of results were achieved: the first
belongs to the visual analysis performed and is
based on the pattern-behaviour defined for the high
frequency brain waves; the other concerns to the
results obtained from the application of the EAT to
the biomatric data captured during the experimental
sessions.
The enunciated hypothesis was proved and
validated by the application of the data analysis’ of
the experimental sessions. Gathered biometric data
showed a high degree of similarity between the
behaviour of the high frequency brain waves and the
hypothesis previously stated when the predominant
emotional state of the subject is coincident.
Figure 2: Beta wave (a) and Gamma wave (b) comparison
for men and women.
Apart from these results, the comparasion of men
and women brain wave behaviour, presented in
Figure 4, indicates differences in amplitude directly
related with the sex of the subject. The presented
charts were based on the average amplitude of all
subjects, male and female seperatly, for the three
distinct session stages.
Through this ilustration, it is shown a slighty
decreasing variation of the average amplitude along
the entire experimental session, which proves the
pattern-behaviour previously described. Together
with these results, also an amplitude diference
between male and female waves is presented during
the session, with a higher amplitude for the female
behaviour.
Based on the previously described results, as
well as the statistical analysis, the EAT was able to
follow the initial hypothesis. The emotional state
decision taken by the aplicaiton is based on the
clusters’ length, centroides’ value and the respective
high frequency brain wave.
Figure 3: Cluster analysis for the Gamma wave.
Figure 5 represents the cluster analysis of one
experimental session for the Gamma wave. These
results indicate a clear majority of data associated
with the lowest centroide value, and a small density
of data near the high value centroide. Based on this
approach, the emotional state classification is based
on the clusters’ length and its degree of correlation
between each others.
For this specific project, there were studied two
different emotional states plus the neutral one, so
that three clusters were adopted for the statistical
analysis. The integration of this tool in a project with
a bigger scope is suitable and advantageous, since
the number of clusters is dependent on the number
of emotional states to analyse and the specific
multimedia content used for the emotional induction.
Besides the emotion assessment functionality,
this tool also integrates some important features for
data ane quem alysis as: plotting the original
biometric data gathered from the EEG; calculate the
weighted means directly from the original signal in
intervals of 5, 10 and 20 seconds, defined by the
user; activate or deactivate the spikes removal
technique, affecting and improving the assessment
results for more unstable experimental sessions.
AUTOMATIC EMOTION INDUCTION AND ASSESSMENT FRAMEWORK - Enhancing User Interfaces by
Interperting Users Multimodal Biosignals
489
Figure 4: EAT running screenshot.
Figure 6 represents an EAT running screenshot,
where the final conclusion led to a predominant
emotional state of sadness after a decimation of 20
seconds of the load of raw session data file. The
performance attained through the application of the
EAT is directly related with the success rate of the
emotional assessment and is a determinant factor for
the verification and validation of this tool for future
work. Accordingly to Table 1, where the confusion
table is presented, the final rate of success is 74%
and all the failures of the EAT are related with the
sadness emotional state, with low values for the
brain waves amplitude.
Table 1: Confusion Table.
EAT
Joy Neutral Sadness
Subject
Joy
11%
0% 11%
Neutral 0%
0%
0%
Sadness 0% 16%
63%
The application of the EAT for the automatic
assessment was, for one of the experimental
sessions, able to determine the correct predominant
emotional state, which wasn’t possible through the
empirical visual inspection analysis of the biometric
data after processing it. This indicates that the 16%
of failure of the EAT are related to a discrepancy
between the analysis of the Beta and Gamma brain
waves’ behaviour.
5 CONCLUSIONS
The execution of twenty eight experimental sessions
based on a predefined induction method resulted on
a vast collection of biometric data. The initial
enunciated hypothesis was validated and the
majority of the subjects included in this project have
reacted in a similar way to the multimedia content
presented. Starting with light, enjoyable contents
and finishing with sad ones, it was able to conclude
that the average amplitude of the high frequency
brain waves decreased along the entire session based
on the emotional state induced on the subject. Two
different emotional states – joy and sadness – plus a
neutral one led to the use of three clusters,
characterized by its centroides’ value, the number of
samples included and the degree of correlation
between them. EAT was able to classify the
predominant emotional state, out of three, with an
accuracy of almost 75%.
One shall consider system adaptations in order to
accommodate psychiatric diagnosis and treatment
procedures, either by simple emotional state
assessment or by complementing this feature with
audiovisual adequate contents. Videogame related
entertainment industry is also a potential target with
the introduction of emotional state information as an
extra variable for game play enhancement. There
have been identified some future work such as the
necessity of performing the referred assessment in
real-time, following a sliding window approach for
containing some historical and contextual
information. Once this enhancement becomes real, it
would be interesting to expand the number of
emotional states detectable by the application, such
as anger and excitement.
As a final remark, one shall state that the
presented study achieved to develop an automatic
tool for basic emotional states detection with high
rates of success based on pre-defined stable
experimental methodologies of emotion induction
and data processing and validation. The used
hardware solutions are believed to be minimal
invasive and are not costly which enables its vast
application at a larger scale.
REFERENCES
Chanel, G., et al. 2005. Emotion Assessment: Arousal
Evaluation using EEG's and Peripheral Physiological
Signals. University of Geneva, Switzerland: Computer
Science Department, pp 530-537, vol 4105/2006.
Damásio, A. R. 1994. Descartes error: Emotion, reason
and human brain. Europa-América.
Damásio, A. R. 1998. Emotions and the Human Brain.
Iowa, USA: Department of Neurology.
Teixeira, J., Vinhas, V. et al. 2008. Multichannel Emotion
Assessment Framework: Gender and High-Frequency
Electroencephalography as Key-Factors. ICINCO
2008, 5
th
International Conference on Informatics in
Control, Automation and Robotics.
BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing
490