MULTICHANNEL EMOTION ASSESSMENT FRAMEWORK

Positive and Negative Emotional Dichotomy

Jorge Teixeira

Faculdade de Engenharia da Universidade do Porto, Rua Dr. Roberto Frias s/n, Porto, Portugal

Vasco Vinhas, Eugenio Oliveira, Luis Paulo Reis

Faculdade de Engenharia da Universidade do Porto, Rua Dr. Roberto Frias s/n, Porto, Portugal

LIACC - Artiﬁcial Intelligence and Computer Science Laboratory, Rua Campo Alegre 823, Porto, Portugal

Keywords:

Medical Signal Acquisition, Data Analysis and Processing, Emotion Assessment, Electroencephalography,

Galvanic Skin Response.

Abstract:

While affective computing and the entertainment industry still maintain a substantial gap between themselves,

biosignals are subject of digital acquisition through low budget technologic solutions at neglectable invasive

levels preventing users from focusing their awareness in the equipment. The integration of electroencephalog-

raphy, galvanic skin response and oximeter in a multichannel framework constitutes an effort in the path to

identify emotional states via biosignals expression. In order to induce and detect speciﬁc emotions, gender

speciﬁc sessions were deﬁned based on the International Affective Picture System and performed in a con-

trolled environment. Results granted by distinct analysis techniques showed that high frequency EEG waves

are strongly related to emotions and are a solid ground to perform accurate emotion classiﬁcation. They have

also given strong indications that females are more sensitive to emotion induction. On the other hand, one

might conclude that the attained success levels concerning relating emotions to biosignals are extremely en-

couraging not only to the continuation of this research topic but also to the application of these results in

domains such as multimedia entertainment, advertising and medical treatments.

1 INTRODUCTION

Affective computing is consistently becoming a con-

ﬁrmed scientiﬁc domain with practical applications

while the entertainment industry as a whole, and spe-

cially the cinematographic and videogame branches,

which have been closing the semantic gap between

them, constitute an economic giant. Having this

macrocontextualization in mind, the authors have al-

ready engaged a research project with the main inten-

tion of using emotion assessment through biosignals

to promote both subconscious interaction and individ-

ual speciﬁc appropriated content delivery.

The presented study ﬁnds itself integrated in this

scope, as it perfectly falls in the emotion assessment

research module. The proposed system constitutes a

solid technologic framework that intends to enable bi-

ological information acquisition in a controlled en-

vironment having as initial hypothesis the existence

of human physical expression of emotional states that

can be objectively measured by relatively inexpensive

equipment. The multichannel structure was deﬁned

by exploiting known techniques, namely electroen-

cephalography, galvanic skin response and heart rate

monitoring, and emotional states were induced using

third-party catalogued pictures.

The ﬁrst goal was to effectively deﬁne, build and

test an experimental session framework where sub-

jects followed a given strict protocol in order to vi-

sualize and/or interact with multimedia content. This

framework was designed not only to collect data but

also to constitute a validation environment. The sec-

ond objective was to, using the given platform, iden-

tify speciﬁc, controlled and extractable biological sig-

nals that could be used as emotion index factors ap-

plied to all subjects or to a characteristic group of

equals.

The conﬁrmation of the initial hypothesis and

project goals would enable its immediate application

in developing a full or semi-automatic emotion clas-

siﬁcation engine that would be able to apply and use

the identiﬁed signal patterns to, in real-time, identify

249

Teixeira J., Vinhas V., Oliveira E. and Paulo Reis L. (2008).

MULTICHANNEL EMOTION ASSESSMENT FRAMEWORK - Positive and Negative Emotional Dichotomy.

In Proceedings of the Fifth International Conference on Informatics in Control, Automation and Robotics - SPSMC, pages 249-252

DOI: 10.5220/0001486902490252

 SciTePress

the subject’s emotional states with high accuracy.

In order to better detail the presented study, this

document is structured as follows: the domain state of

the art is described in the next section, in section 3 the

multichannel emotion assessment framework is pre-

sented with special emphasis in the most signiﬁcant

decisions. Still in that section, results are presented

and related conclusions are extracted in section 4 as

well as are identiﬁed future work areas and practical

domains of application are suggested.

2 STATE OF THE ART

The emotional state of human beings belongs to a

complex theme since its deﬁnition is not unique and

its essence not consensual. An overview of the emo-

tion assessment is presented in the next subsection, as

well as a brief description of the most common ap-

proaches to emotional induction, and ﬁnally a refer-

ence to equipment solutions.

2.1 Emotion Assessment

The emotion itself can be seen as a consequence of

an action or an environment cause so that the induc-

tion of a speciﬁc emotional state is tightly connected

with an arousal procedure. In order to identify and

assess an emotion, patterns are used and they con-

stitute different approaches to the emotional induc-

tion, which will be discussed in the next subsection.

Apart from the induction, the classiﬁcation is essen-

tial, and can be accomplished based on a coincidence

of values on a strategic number of dimensions (Lo-

gothetis, 1957). Based on this study, the emotion as-

sessment can be analyzed through three distinct di-

mensions. The two primary levels are the valence

and the arousal, and the secondary one is the domi-

nance, which has a weaker relationship with the oth-

ers (P.J. Lang, 2005)(A. Mehrabian, 1974).

In order to best analyze the assessment of the pic-

tures, it is generally used an affective space. This is a

standardized method to graphically display the emo-

tional assessment results of the pictures. According

to the valence and arousal mean values, it is plotted

a bidimensional graph where the horizontal-axis rep-

resents the arousal and the vertical-axis the valence,

both scaled from 1 to 9.

2.2 Generic Approach to Emotion

Induction

There is not a process for emotional induction that is

perfectly suitable for all cases, but a group of different

approaches to achieve the same objective. A prevalent

method to induce emotional processes consists of ask-

ing an actor to feel or express a particular mood. This

strategy has been widely used for emotion assessment

from facial expressions and to some extent from phys-

iological signals (G. Chanel, 2005). However, even

for expert actors for whom the capacity to achieve a

speciﬁc emotional state is obvious, it is hard to guar-

antee that the physiological responses are consistent

and reproducible by other non-actor people.

An alternative approach to the emotional induc-

tion is composed by multimedia stimuli. Music, im-

ages, videos and video-gamesbelongsto a category of

stimuli that has signiﬁcant advantages compared with

the induction through actors, since there is no need

of actors and the quality of the induced emotions is

greater as they are more realistic.

2.3 Equipment Solutions

Emotions’ assessment needs reliable and accurate

communications with the subject so that the results

are conclusive and the emotions correctly classiﬁed.

This communication can occur through several chan-

nels and is supported by speciﬁc equipment. The BCI

- Brain Computer Interface - is directly connected

to this area and uses two different approaches, inva-

sive and non-invasive methods. The invasive meth-

ods are clearly more precise, however more danger-

ous and will not be considered for this study. On the

other hand, non invasive methods such as EEG, fMRI,

GSR, oximeter and others have shorten the distance

between the utopia and the truth of understanding the

human brain behaviour, gathering together the advan-

tages of inexpensive equipment and non-medical en-

vironments.

Due to the medical community skepticism, EEG,

in clinical use, it is considered a gross correlate of

brain activity (Ebersole, 2002). In spite of this reality,

recent medical research studies (Pascalis, 1998)(Af-

tanas, 1997) have been trying to revert this scenario

by suggesting that increased cortical dynamics, up to

a certain level, are probably necessary for emotion

functioning and by relating EEG activity and heart

rate during recall of emotional events. Similar efforts,

but using invasive technology like Electrocorticogra-

phy (ECoG), have enable complex BCI like playing a

videogame or operating a robot (Leuthardt, 2004).

Some more recent studies have successfully

used just EEG information for emotion assessment

(K. Ishino, 2003). These approaches have the great

advantage of being based on non-invasive solutions,

enabling its usage in general population in a non-

medical environment. Encouraged by these results,

ICINCO 2008 - International Conference on Informatics in Control, Automation and Robotics

250

the current research direction seems to be the addi-

tion of other inexpensive, non-invasive hardware to

the equation. Practical examples of this are the intro-

duction of GSR and oximeters by Takahashi (Taka-

hashi, 2004) and Chanel et al(G. Chanel, 2005).

On this study three non-invasive equipments will

be used in parallel so that the reliability of all the

procedures is guaranteed. A Neurobit Lite EEG de-

vice with one active electrode and two references, a

Thoughtstream biofeedback system Galvanic Skin

Response with two dry electrodes and an oximeter

with a ﬁnger sensor.

3 PROJECT DESCRIPTION AND

RESULTS

In this section it will be given a brief overview of

the whole project development.The procedures and

methods involved on this study were grouped into two

parts. The ﬁrst one deals with the emotional induc-

tion approach used on this study and the last one re-

veals the experimental conditions used along the ex-

perimental sessions.

A good experimental control and an easy, yet ef-

ﬁcient, method for results comparison are key factors

that demand an effective set of visual stimuli. The

IAPS library is so the most indicated emotional in-

duction method, as it has been widely used through

the research community and all the pictures classiﬁed

according to valence, arousal and dominance (L. Af-

tanas, 2001)(G. Chanel, 2005)(M. Muller, 1999).

The experimental conditions are an essential is-

sue to the validation and acceptance of the results ob-

tained.

A total of twenty eight subjects, seventeen males

and eleven females, all right-handed aged eighteen-

thirty years old took part in this study. All subjects

had access to an introductory text for the experimen-

tal session in order to access the essential information

about the main procedures involved and a question-

naire ﬁlled before each session to avoid possible bar-

riers as mental diseases.

The experimental results of this study are based

on a statistical analysis.

This analysis indicates that mens behaviour is dif-

ferent from the womens one, since the mean ampli-

tude of the high frequency brain waves is higher along

the entire experimental session.

Considering the GSR data, it is presented the slope

variation between the three stages - happiness, neutral

and sadness - and analyzed its behaviour along the

complete session.

The heart rate analysis indicates an unexpected

and almost undetectable variation of its value.

4 CONCLUSIONS

In what regards to the ﬁrst topic, one ought to assert

that the initial study’s main goals were completed un-

dertaken. The described experiments achieved to de-

velop and test a solid framework to conduct controlled

emotionally-evocative experiments enabling ﬂexible

capability of recording and monitoring biosignals in

real-time.

At a more detailed level, it was possible to deﬁne

dynamic gender-designed emotional sessions, with

ﬁne tuning capabilities, in order to trigger speciﬁc

emotional states. The data provided from the con-

ducted experiments was input to the detailed sys-

tem’s architecture that provedto be able to supply ses-

sions with biosignals real-time monitoring and stor-

age for physical and temporal independent process-

ing. The achieved results also demonstrated that

the postsession data processing techniques were ef-

ﬁcient and effective in what concerns to emotional

states/biosignals correlation identiﬁcation.

Having the study’s results, presented in the pre-

vious section, as solid ground, it is possible to con-

ﬁrm that basic emotional states have biological man-

ifestations capable of being captured and recorded by

the selected equipments, specially with EEG and GSR

techniques.

Concerning subject variables, it is plausible to

state that females react more aggressively to the pre-

sented pictures, triggering with more expression and

effectiveness the desired emotional states. These ob-

jective data was also corroborated by the interviews

conducted at the end of each sessions, where female

subjects consistently stated that they felt happy and in

a good mood in the ﬁrst stage of the session and sad

at the end. In the same interviews, a high percent-

age of male subjects afﬁrmed that they did not felt a

deep emotional commitment along the picture presen-

tation.

The fact that gender is a key factor in what con-

cerns emotional state triggering through multimedia

content constitutes the ﬁrst major contribution of the

present study.

Exploiting the evidence that EEG signals were

strongly inﬂuenced by the subject’s emotional states,

a more detailed data analysis was performed, as il-

lustrated in the previous section, with special focus

to high frequency signals, namely beta and gamma

waves. The data provided strongly suggests that,

specially in female subjects, high frequency relative

MULTICHANNEL EMOTION ASSESSMENT FRAMEWORK - Positive and Negative Emotional Dichotomy

251

EEG values are directly correlated to valence, inde-

pendently of their initial standard level. The data pro-

vided suggests that beta and gamma waves strongly

seam to vary directly with valence, enabling, indi-

rectly and with conjugation with other inputs, emo-

tional state detection.

Despite the described positive outcome, there

were identiﬁed some features that, although did not

match the initial assumptions, have already been sub-

ject of turnaround strategy deﬁnition. The ﬁrst one

is related to the inexistence of generic signiﬁcative

changes in heart rate values along experimental ses-

sions and the second resides in the fact that the ex-

pected GSR readings curve – high conductivity with

high arousal situations – was not recorded as often

as predicted. The authors believe that the existence

of these issues is grounded on the fact that the de-

signed emotionally-evocative sessions based only on

pictures do not trigger such strong emotions capable

of signiﬁcantly inﬂuence biosignals such as heart rate.

It is believed that it would be necessary much strong

multimedia content provided in a more immersive en-

vironment so that subjects could be more deeply in-

volved.

4.1 Future Work

The main future work topics are not only related to

this particular study, once it is a spin off/module of a

major one, but also with the main global project. With

this in mind, there were identiﬁed the following areas:

• More Sophisticated Equipment Reinforcement: It

is intended to acquire more sophisticated equip-

ment, specially and speciﬁcally in what concerns

to a multi-channel EEG and a more sensible and

reliable GSR;

• Equipment Diversity: It would be useful to inte-

grate in the developed framework new equipments

capable of reading and extract more biosignals,

namely pupil dilatation, voice analysis and facial

expression recognition;

• More Detailed Emotion Classiﬁcation: using the

depicted key factors with conjugation with others

provided by the study continuation and data vol-

ume and diversity enhancement brought, by new

equipment acquisition, it would be plausible to

perform automatic subject emotion classiﬁcation

with deeper detail levels.

• Software Control: The accomplishment of the

previous items would enable both conscious and

subconscious control of several tools and/or mul-

timedia contents;

Considering the studied problem as a whole, specif-

ically the emotion classiﬁcation topic, several practi-

cal domain applications are not only feasible but also

attractive. Most of the immediate technology adapta-

tions shall reside in the entertainment industry, both in

audiovisual and videogame branches through multi-

media content adaptability to user’s emotional states.

Other possible application areas are user interface en-

hancement, direct advertising and medical applica-

tions, namely in phobia treatments and psychological

evaluations.

REFERENCES

A. Mehrabian, J. R. (1974). An approach to environmental

psychology. In The MIT Press.

Aftanas, L. (1997). Nonlinear forecasting measurements

of the human eeg during evoked emotions. In Brain

Topography, volume 10, pages 155–162.

Ebersole, J. (2002). Current Practice of Clinical Electroen-

cephalography. Lippincott Williams & Wilkins.

G. Chanel, J. Kronegg, D. G. (2005). Emotion assessment:

Arousal evalutation using eeg’s and peripheral physi-

ological signals. In Technical Report.

J. Allen, J. K. (2006). Frontal eeg asymmetry, emotion, and

phychopathology: the ﬁrst, and the next 25 years. In

Biological Psychology, volume 67, pages 1–5.

K. Ishino, M. H. (2003). A feeling estimation system us-

ing a simple electroencephalograph. In Proceedings

of 2003 IEEE Int. Conference on Systems, Man, and

Cybernetics, pages 4204–4209.

L. Aftanas, A. A. (2001). Time-dependent cortical asym-

metries induced by emotional arousal: Eeg analysis of

event-related synchronization and desynchronization

in individually deﬁned frequency bands. In Int. Jour-

nal of Psychophysiology, volume 44, pages 67–82.

Leuthardt, E. (2004). A braincomputer interface using elec-

trocorticographic signals in humans. In Journal of

Neural Engineering, pages 63–71.

Logothetis, N. (1957). The measurement of meaning. In

University of Illinois Press.

M. Muller, A. K. (1999). Processing of affective pictures

modulates right-hemispheric gamma band eeg activ-

ity. In Clinical Neurophysiology, volume 110, pages

1913–1920.

Pascalis, V. D. (1998). Eeg activity and heart rate during

recall of emotional events in hypnosis: relationships

with hypnotizability and suggestibility. In Int. Journal

of Psychophysiology, volume 29, pages 255–275.

P.J. Lang, M. Bradley, B. C. (2005). International affective

picture system (iaps): Affective ratings of pictures and

instruction manual. In Technical Report.

Takahashi, K. (2004). Remarks on emotion recognition

from bio-potencial signals. In The second Int. Con-

ference on Autonomous Robots and Agents.

ICINCO 2008 - International Conference on Informatics in Control, Automation and Robotics

252