AUTOMATIC EMOTION INDUCTION AND

ASSESSMENT FRAMEWORK

Enhancing User Interfaces by Interperting Users Multimodal Biosignals

Jorge Teixeira, Vasco Vinhas, Luís Paulo Reis and Eugénio Oliveira

FEUP - Faculdade de Engenharia da Universidade do Porto, Rua Dr. Roberto Frias s/n, Porto, Portugal

DEI - Departamento de Engenharia Informática, Rua Dr. Roberto Frias s/n, Porto, Portugal

LIACC - Laboratório de Inteligência Artificial e Ciência de Computadores, Rua do Campo Alegre 823, Porto, Portugal

Keywords: Biosignals, Emotions, Classification, Multimedia, Clustering.

Abstract: Emotion’s definition, identification, systematic induction and efficient and reliable classification have been

themes to which several complementary knowledge areas such as psychology, medicine and computer

science have been dedicating serious investments. This project consists in developing an automatic tool for

emotion assessment based on a dynamic biometric data acquisition set as galvanic skin response and

electroencephalography are practical examples. The output of standard emotional induction methods is the

support for classification based on data analysis and processing. The conducted experimental sessions,

alongside with the developed support tools, allowed the extraction on conclusions such as the capability of

effectively performing automatic classification of the subject’s predominant emotional state. Self

assessment interviews validated the developed tool's success rate of approximately 75%. It was also

experimentally strongly suggested that female subjects are emotionally more active and easily induced than

males.

1 INTRODUCTION

Emotions play an important role in all human

activities, from the trivial to the most complex ones.

This significance is translated both in terms of

reality perception and even in the cognitive decision

process. Meanwhile, computers have gained such a

relevant presence in the modern society that they

have been introduced in almost every aspect of it,

enhancing the magnitude of ubiquitous computing.

Having these two realities in mind – the

importance of emotional states and the necessity of

daily interaction with multiple devices – merging

them would be a great improvement. By providing

the distributed computer systems with the perception

of their users’ emotions, the applications would be

able to adjust their interface, promote and suggest

functionalities accordingly. It is believed that this

approach would increase the global system’s

transparency and efficiency as its dynamism would

follow in the encounter to user’s intentions and

temper.

Alongside the ubiquitous computing, multimedia

contents are becoming constantly more complex and

seemlier to reality, enabling a greater action

immersion sensation, the primitive absolute need of

achieving a perfect match between audiovisual

contents and the audience desires is still present and

constitutes the main key to the industry success.

The alliance between the multimedia contents

choice possibility that enables the audience to

individually presence what desires and accurate

emotional states detection systems leads to

subconscious individual interaction between the

audience and the multimedia control system,

potentiating the perfect match between content and

individual audience desires.

This study illustrates a proposal for an

application that enables automatic emotional state

assessment using minimal invasive solutions.

The rest of the paper is organized as follows: in

the next section the current state of the art is

presented; in section 3 a project description is given;

in section 4 the study’s results are depicted and,

consequently, the project’s conclusions are listed

and future work areas are identified in the final

section.

487

Teixeira J., Vinhas V., Reis L. and Oliveira E. (2009).

AUTOMATIC EMOTION INDUCTION AND ASSESSMENT FRAMEWORK - Enhancing User Interfaces by Interperting Users Multimodal Biosignals.

In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing, pages 487-490

DOI: 10.5220/0001542104870490

 SciTePress

2 STATE OF THE ART

An emotional state can be defined as a collection of

responses triggered by different parts of the body or

the brain through both neural and hormonal

networks (Damásio, 1998). Experiments conducted

with patients with brain lesions in specific areas led

to the conclusion that their social behaviour was

highly affective, together with the emotional

responses. It is unequivocal to state that emotions

are essential for humans, as they play a vital role in

their everyday life: in perception, judgment and

action processes (Damásio, 1994).

Due to the complexity of human emotions, their

analysis is often based on the identification of

distinct basic emotional states so that the analysis

and processing is simplified. For this project, three

emotional states will be studied: joy, sadness and a

neutral emotional state.

In order to analyse biometric data that contains

mainly positive and negative emotional states, it is

essential to create and define an experimental

environment that is able to induce a subject in a

specific and controlled emotional state. Nowadays, it

is common to use an actor as one possible approach

to human beings emotions’ simulation (Chanel et al.,

2005). As the actor predicts specific emotions,

outside aspects as facial expression or voice change

accordingly. However, the physiological responses

will not suffer any variations, which lead to one of

the biggest disadvantages of this approach, as the

gathered biometric information does not represent

the real emotional state of the actor.

An alternative method, adopted in this study, is

the use of multimedia stimuli (Chanel et al., 2005).

These stimuli contain a variety of contents such as

music, videos, text and images. The main advantage

of this method resides in the strong correlation

between the induced emotional states and the

physiological responses, as the emotions are no

longer simulated.

The electroencephalograph used on this project

was the NeurobitLite and is composed by 3

electrodes, one functioning as and active one and the

other two as references, using a monopolar method

strategy.

3 PROJECT DESCRIPTION

The development of an automatic tool that able to

determine the emotional state of a subject through

EEG biometric information was based on data

analysis’ methods. These methods included the

decimation; the weighted average; spikes’ removal;

and clusters for the final emotion’s assessment.

The implementation of the weighted average

technique culminated with the enunciation of a

hypothesis concerning the behaviour of the electrical

brain waves when subjects are emotionally induced.

The hypothesis follows a specific temporal

distribution and a pattern that was observed in the

majority of the experimental sessions. Figure 3

represents the evolution of Beta and Gamma

amplitudes’ over the entire experimental session.

This behaviour is considered as the pattern

behaviour for high frequency brain waves (Teixeira,

Vinhas, 2008).

The enunciated expected behaviour represents

a three step chart with each step having the duration

of two minutes. This data treatment had in mind the

three steps took into account to IAPS session

management – three sets of twenty pictures lasting

for two minutes with a grading emotional effect

from joy to sadness passing through an intermediate

neutral state.

Figure 1: Expected behaviour for high frequency waves.

In what concerns to cluster analysis, the

emotional induction results in an amplitude’s

variation according to a specific emotional state.

Having these concepts in mind, and based on the

pattern-behaviour for the EEG data previously

described, three distinct groups of data were created

based on the brain waves’ mean amplitude. Each of

these three groups have one specific centroide, a

point that is used as a reference for the neighbours of

the same cluster.

The emotional state classification, performed by the

EAT, is based on the predominant emotional state of

the subject, as previously described. In order to

evaluate the success rate of the EAT classification,

at the end of each experimental session, self-

assessment interviews were performed to subjects.

The main objective of these interviews was to attain

information concerning the predominant emotional

state, so that it could be later compared with the

results obtained from the EAT analysis. Besides this

fact, other important aspects like the apreciation of

the presented visual stimuli sequence, the

environment conditions and any disturbs during the

experimental session have been collected.

BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing

488

These interviews constituted an essential

database for project results’ validation purposes.

The emotional induction is divided in three

distinct and perfectly defined stages: joy, sadness

and a neutral state. This allows determining which of

the stages is more efficient during the whole

experimental session or has stronger and more

coherent effect on the subject. This hypothesis is the

base of the concept for the EAT, so that it is able to

determine, based on the gathered EEG biometric

data, the predominant emotional state of the subject.

Based on the statistical analysis, each of the steps

is directly associated with one of the clusters, so that

there are three distinct clusters per experimental

session. The clusters’ analysis is based on the

centroides’ values and the number of samples, so

that the global organization of the samples is

different for each emotional state.

Aside with these characteristics, the emotions’

assessment was performed for both Beta and

Gamma brain waves, since high frequency brain

waves are believed to have the most noteworthy

changes due to emotional states transition (Teixeira,

Vinhas et al, 2008).

4 RESULTS

From the experimental sessions conducted two

different kind of results were achieved: the first

belongs to the visual analysis performed and is

based on the pattern-behaviour defined for the high

frequency brain waves; the other concerns to the

results obtained from the application of the EAT to

the biomatric data captured during the experimental

sessions.

The enunciated hypothesis was proved and

validated by the application of the data analysis’ of

the experimental sessions. Gathered biometric data

showed a high degree of similarity between the

behaviour of the high frequency brain waves and the

hypothesis previously stated when the predominant

emotional state of the subject is coincident.

Figure 2: Beta wave (a) and Gamma wave (b) comparison

for men and women.

Apart from these results, the comparasion of men

and women brain wave behaviour, presented in

Figure 4, indicates differences in amplitude directly

related with the sex of the subject. The presented

charts were based on the average amplitude of all

subjects, male and female seperatly, for the three

distinct session stages.

Through this ilustration, it is shown a slighty

decreasing variation of the average amplitude along

the entire experimental session, which proves the

pattern-behaviour previously described. Together

with these results, also an amplitude diference

between male and female waves is presented during

the session, with a higher amplitude for the female

behaviour.

Based on the previously described results, as

well as the statistical analysis, the EAT was able to

follow the initial hypothesis. The emotional state

decision taken by the aplicaiton is based on the

clusters’ length, centroides’ value and the respective

high frequency brain wave.

Figure 3: Cluster analysis for the Gamma wave.

Figure 5 represents the cluster analysis of one

experimental session for the Gamma wave. These

results indicate a clear majority of data associated

with the lowest centroide value, and a small density

of data near the high value centroide. Based on this

approach, the emotional state classification is based

on the clusters’ length and its degree of correlation

between each others.

For this specific project, there were studied two

different emotional states plus the neutral one, so

that three clusters were adopted for the statistical

analysis. The integration of this tool in a project with

a bigger scope is suitable and advantageous, since

the number of clusters is dependent on the number

of emotional states to analyse and the specific

multimedia content used for the emotional induction.

Besides the emotion assessment functionality,

this tool also integrates some important features for

data ane quem alysis as: plotting the original

biometric data gathered from the EEG; calculate the

weighted means directly from the original signal in

intervals of 5, 10 and 20 seconds, defined by the

user; activate or deactivate the spikes removal

technique, affecting and improving the assessment

results for more unstable experimental sessions.

AUTOMATIC EMOTION INDUCTION AND ASSESSMENT FRAMEWORK - Enhancing User Interfaces by

Interperting Users Multimodal Biosignals

489

Figure 4: EAT running screenshot.

Figure 6 represents an EAT running screenshot,

where the final conclusion led to a predominant

emotional state of sadness after a decimation of 20

seconds of the load of raw session data file. The

performance attained through the application of the

EAT is directly related with the success rate of the

emotional assessment and is a determinant factor for

the verification and validation of this tool for future

work. Accordingly to Table 1, where the confusion

table is presented, the final rate of success is 74%

and all the failures of the EAT are related with the

sadness emotional state, with low values for the

brain waves amplitude.

Table 1: Confusion Table.

EAT

Joy Neutral Sadness

Subject

Joy

11%

0% 11%

Neutral 0%

Sadness 0% 16%

63%

The application of the EAT for the automatic

assessment was, for one of the experimental

sessions, able to determine the correct predominant

emotional state, which wasn’t possible through the

empirical visual inspection analysis of the biometric

data after processing it. This indicates that the 16%

of failure of the EAT are related to a discrepancy

between the analysis of the Beta and Gamma brain

waves’ behaviour.

5 CONCLUSIONS

The execution of twenty eight experimental sessions

based on a predefined induction method resulted on

a vast collection of biometric data. The initial

enunciated hypothesis was validated and the

majority of the subjects included in this project have

reacted in a similar way to the multimedia content

presented. Starting with light, enjoyable contents

and finishing with sad ones, it was able to conclude

that the average amplitude of the high frequency

brain waves decreased along the entire session based

on the emotional state induced on the subject. Two

different emotional states – joy and sadness – plus a

neutral one led to the use of three clusters,

characterized by its centroides’ value, the number of

samples included and the degree of correlation

between them. EAT was able to classify the

predominant emotional state, out of three, with an

accuracy of almost 75%.

One shall consider system adaptations in order to

accommodate psychiatric diagnosis and treatment

procedures, either by simple emotional state

assessment or by complementing this feature with

audiovisual adequate contents. Videogame related

entertainment industry is also a potential target with

the introduction of emotional state information as an

extra variable for game play enhancement. There

have been identified some future work such as the

necessity of performing the referred assessment in

real-time, following a sliding window approach for

containing some historical and contextual

information. Once this enhancement becomes real, it

would be interesting to expand the number of

emotional states detectable by the application, such

as anger and excitement.

As a final remark, one shall state that the

presented study achieved to develop an automatic

tool for basic emotional states detection with high

rates of success based on pre-defined stable

experimental methodologies of emotion induction

and data processing and validation. The used

hardware solutions are believed to be minimal

invasive and are not costly which enables its vast

application at a larger scale.

REFERENCES

Chanel, G., et al. 2005. Emotion Assessment: Arousal

Evaluation using EEG's and Peripheral Physiological

Signals. University of Geneva, Switzerland: Computer

Science Department, pp 530-537, vol 4105/2006.

Damásio, A. R. 1994. Descartes error: Emotion, reason

and human brain. Europa-América.

Damásio, A. R. 1998. Emotions and the Human Brain.

Iowa, USA: Department of Neurology.

Teixeira, J., Vinhas, V. et al. 2008. Multichannel Emotion

Assessment Framework: Gender and High-Frequency

Electroencephalography as Key-Factors. ICINCO

2008, 5

International Conference on Informatics in

Control, Automation and Robotics.

BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing

490