Towards Emotion Related Feature Extraction based on

Generalized Source-Independent Event Detection

Rui Santos

, Joana Sousa

, Carlos J. Marques

3,4

, Hugo Gamboa

1,2

and Hugo Silva

2,5

Physics Department, FCT-UNL, Lisbon, Portugal

PLUX - Wireless Biosignals, S.A., Lisbon, Portugal

Faculty of Human Kinetics at the Technical University of Lisbon, Lisbon, Portugal

Physical Therapy and Rehabilitation Department at the Sch

on Klinik, Hamburg, Germany

Instituto de Telecomunicac¸

oes, IST-UTL, Lisbon, Portugal

Abstract. Emotion recognition is of major importance towards the acceptabil-

ity of Human-Computer Interaction systems, and several approaches to emo-

tion classiﬁcation using features extracted from biosignals have already been de-

veloped. This analysis is, in general, performed on a signal-speciﬁc basis, and

can bring a signiﬁcant complexity to those systems. In this paper we propose

a signal-independent approach on marking speciﬁc signal events. In this prelimi-

nary study, the developed algorithm was applied on ECG and EMG signals. Based

on a morphological analysis of the signal, the algorithm allows the detection of

signiﬁcant events within those signals. The performance of our algorithm proved

to be comparable with that achieved by signal-speciﬁc processing techniques on

events detection. Since no previous knowledge or signal-speciﬁc pre-processing

steps are required, the presented approach is particularly interesting for automatic

feature extraction in the context of emotion recognition systems.

1 Introduction

The ability to recognize emotion is of upmost importance with the increasing develop-

ment of intelligent and adaptive computer systems, allowing them to sense and respond

appropriately to user’s affective feedback [1]. Emotions are one of the least explored

frontiers of intuitive Human-Computer Interaction (HCI) [2], and its understanding is

expected to improve the acceptability of those systems. This requires, however, robust

emotion recognition systems, capable of guaranteeing acceptable recognition accuracy,

and adaptable to practical applications [3].

Emotion recognition systems still present relevant challenges, as it is very hard

to uniquely correlate emotion-relevant signal patterns with a certain emotional state.

Furthermore, these patterns may widely differ from person to person, and between

different situations [3]; there is a lack of ground-truth datasets in order to develop

user-independent systems, which would be essential for practical applications [4, 5].

There have been many attempts to built automatic emotion recognition systems [3, 6–

9], which mostly rely on supervised pattern classiﬁcation approaches. However, some of

the major problems still veriﬁed are the low recognition rates when considering subject-

independent classiﬁcation and generalization to more than one task [10]. A generally

Santos R., Sousa J., J. Marques C., Gamboa H. and Silva H..

Towards Emotion Related Feature Extraction based on Generalized Source-Independent Event Detection.

DOI: 10.5220/0003891900710078

In Proceedings of the 2nd International Workshop on Computing Paradigms for Mental Health (MindCare-2012), pages 71-78

ISBN: 978-989-8425-92-8

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

applicable recognition system, for realistic online applications, would have to automati-

cally select the most signiﬁcant features and speciﬁc classiﬁers to several data sets ob-

tained from different natural contexts [8].

Emotions are inherently multi-modal, and several studies on their recognition work

by fusing features extracted from multiple modalities (facial expression [11], voice [12]

or gesture [13] and biosignals). The work around biosignals include the heart rate, skin

temperature, electrodermal and electromyographic activity or respiration rate [1, 14].

The areas of speech and face recognition are far more explored, mostly because it is a

very hard task to uniquely map physiological patterns onto speciﬁc emotional states and

some of the sensors used to acquire those biosignals may be sensitive to motion artifacts

[6]. However, the size of such sensors is decreasing and this method is considered to

be less disturbing than the standard audiovisual techniques [2]. Moreover, biosignals

allow us to continuously gather information on the user’s affective state, and should be

more robust against possible artifacts of human social masking, since they are directly

controlled by the human autonomous nervous system [3, 8].

Feature extraction from biosignals is a complex multivariate task, requiring a broad

insight into biological processes related with neuropsychological functions [6]. Stan-

dard speciﬁc processing techniques in which the results are highly dependent on the

input parameters are usually applied, bringing an added complexity to these feature

extraction systems. In this paper we propose a signal-independent approach to the de-

tection of speciﬁc signal events, in order to accomplish an accurate feature extraction

based on the results of a generic algorithm. The performance of the generic events de-

tection approach is also compared with that of signal-speciﬁc standard algorithms.

The following section presents a brief description of the biosignals considered in

this preliminary approach, as well as the applied signal processing methodologies. The

obtained results are presented and discussed in section 3. In the last section some ﬁnal

remarks and future work steps are referred.

2 Materials and Methods

2.1 Biosignals

This subsection introduces the biosignals that were considered in this preliminary study,

including the main features usually extracted in the context of emotion classiﬁcation

systems. Those types of signals were selected based on their characteristic waveshapes,

in which the events are clearly distinguishable. As such, an intuitive evaluation of the

algorithm performance on that signals is accomplished. Furthermore, these signals are

strong related to mental diseases and they are very helpful into infers about the mental

health condition of the patients.

With these signals we only intend to exemplify the application of the developed

algorithm, since their origin, as respectively described, is not related to the testing pro-

tocols usually followed in emotion classiﬁcation studies [1, 3].

Electrocardiography. The electrocardiogram (ECG) is the recording, on the body sur-

face, of the electrical activity generated by the heart. From the ECG processing one can

extract features as the Heart Rate (HR) and the Heart Rate Variability (HRV). HRV has

become the conventionally accepted term to describe variations of both instantaneous

heart rate and RR interval. In a continuous ECG record, each QRS complex is detected,

and the so-called normal-to-normal (NN) intervals or the instantaneous heart rate is de-

termined. Simple time domain variables that can be calculated include the mean NN

interval, the standard deviation of the NN intervals (SDNN) or the mean heart rate.

Spectral analysis is also usually performed, since there is a correlation between the

relative power of the low frequencies (LF) and high frequencies (HF) ranges and the

sympathetic and parasympathetic nervous systems activity. Non-linear methods based

on chaos theory and fractal analysis are also used on HRV analysis to better understand

the HR ﬂuctuations. The common example of non-linear methods is the Poincar

e plot,

which reﬂects the graphical correlation between consecutive RR intervals [15].

Since HRV and the automatic heart modulation are correlated, HRV analysis is a

powerful tool for clinical use. The HRV analysis allows a better understanding of the

SA node automatic modulation. For example, the spectral analysis reveals that the vagal

activity is the major contribution to the high frequencies component. Furthermore, it is

also argued that the LF and HF ratio reﬂects the sympathovagal balance or the sym-

pathetic modulation. These parameters are very important since some authors defend

that there is a signiﬁcant increase of LF band and signiﬁcant increment of the HR for

panic disorders patients and signiﬁcant lower values of R-R intervals and HF peak of

spectral analysis in depressive patients than in the health people. In general, both the

HR and the HRV are dependent on the activity level of the autonomic nervous system,

which in turn is dependent on emotional stimuli [4]. A low HRV can indicate a state of

relaxation, while an increased HRV can be caused by mental stress or frustration [2].

The ECG signals here considered were obtained from a public database (PhysioBank)

at PhysioNet library [16]. Those were acquired from healthy people, with normal sinus

rhythm, during 4 seconds and at a sampling frequency of 125 Hz. A total of 26 ECG

signals was considered.

Electromyography. Electromyography (EMG) signal arises from the ﬂow of charged

particles across the muscle membrane when its cells are electrically activated. This

biosignals can record both voluntary and involuntary muscle activation, in addition to

the action potentials produced by external stimulation, such as motor evoked potentials

after central or peripheral nerve stimulation [17]. EMG signals have been shown to

correlate with negatively valenced emotions [18]. The signals here considered were

acquired in the context of a study aiming at accessing the performance of the right leg

during the execution of an emergency brake in a car simulator [19]. EMG signals of the

Rectus Femoris, Vastus Medialis, Tibialis Anterior and Gastrocnemius muscles were

acquired while 3 subjects performed the emergency brake test. From that data set, a

total of 40 EMG signals was considered in this study, comprising 10 signals of each

considered muscle. Bipolar EMG sensors (emgPLUX) were used to access the muscle

activation. The sensors were connected to a wireless acquisition unit, bioPLUX research

[20], which performed the acquisition at a sampling frequency of 1000 Hz.

2.2 Generalized Signal Processing Approach

The main features that guided the development of the introduced signal processing

methodology have been the signal-independence, implying no speciﬁc pre-processing

steps. The immunity to noise and artifacts and the simplicity, in order to allow a real-

time implementation, were also considered.

The basis of the developed algorithm is the identiﬁcation of time-domain speciﬁc

morphological parameters that can clearly distinguish events, such as onset and offset

instants and transient waveshapes, within a signal. In practise, after a signal segmenta-

tion, those events are computed as the split points for which the absolute values of the

difference between the standard deviation of the successive segments are maximized.

A further optimization step is then applied through an iterative change of the input pa-

rameters of the previously described processing steps. The optimal solution is selected

as the one which assures a better ﬁtting of all the signal segments, between the detected

events, to linear regressions models.

A detailed description and performance evaluation of this signal processing ap-

proach can be found in [21]. This paper extends that events detection approach to

application in different biosignals and evaluates its performance by comparison with

standard signal-speciﬁc algorithms.

2.3 Standard Signal-speciﬁc Algorithms

In order to evaluate the performance of the proposed signal processing approach, its

results were compared with those from signal-speciﬁc standard techniques for EMG

onset and ECG waveshape detection.

An adaptation of the QRS complex detector algorithm proposed by Pan and Tomp-

kins [22] was implemented as a standard method for performance evaluation on de-

tecting heartbeats within an ECG signal. ECG signal pre-processing steps included

the application of a 2

order Butterworth lowpass ﬁlter with a 30 Hz cutoff fre-

quency, to remove some possible high frequency noise due to electronic devices inter-

ference. A 2

order Butterworth high pass ﬁlter with 2 Hz cutoff frequency was also

applied to remove some possible baseline ﬂuctuations and other respiration artifacts.

Considering EMG signals, a standard onset detector proposed by Hodges et al.[23]

was implemented. This approach implies the EMG signal to be pre-processed in order

to reduce the high frequency noise and obtain the signal’s envelope. After subtracting

the mean value from the EMG signal and obtaining the rectiﬁed wave, a sixth-order

digital butterworth ﬁlter with 50 Hz cutoff frequency was applied, following the imple-

mentation procedure described in [24], completing the envelope detection.

The muscle activity onset was identiﬁed as the instant from which the mean of the

following N samples from the EMG envelope exceeds the baseline activity level by

a speciﬁed multiple h of standard deviations. The baseline activity was adaptively de-

termined, at each instant, by averaging M previous signal samples [24]. With a cutoff

frequency of 50Hz, the set of criteria {N=25, h=3} and {N=50, h=1} are those reported

as able to identify the EMG onset more accurately [23]. Hodges detector was imple-

mented with both set of parameters, considering the baseline activity level computed

from M=100 and M=200, respectively [24].

3 Results and Discussion

The signal processing methodology described in sub-section 2.2 was applied to the

selected raw biosignals. No signal-speciﬁc pre-processing steps were applied. Graphic

representations exemplifying the obtained results are presented in Figure 1.

Fig. 1. Graphic example of: a) detected ECG waveshapes and b) EMG onset and offset points

applying the developed events detection algorithm. The events are marked by vertical red marks.

As exempliﬁed in Figure 1a), the developed algorithm allows the ECG waveshape

discrimination, from the electrical baseline, detecting two events around it. Choosing

one of those instants, parameters such as the HR and the HRV can be easily accessed.

Furthermore, this events marking could be used for a further and more complete fea-

tures extraction based on the waveshape morphology, such as the position of each of the

P, Q, R, S and T waves and the respective amplitude. After applying Pan and Tompkins

adapted algorithm to ECG signals, the resulting graphics were then visually examined

and, for both algorithms, the percentage of detected ECG waveshapes and the num-

ber of extra detected events were registered. Table 1 exposes the mean values of those

parameters.

Table 1. Results obtained from the proposed events detection algorithm and from Pan and Tomp-

kins adapted algorithm while performing the ECG signals QRS complex detection. The results

follow the format mean(±standard deviation).

Events detection Pan and Tompkins

algorithm algorithm

Percentage of detected 93.84% 92.63%

ECG waveshapes (± 15.64 %) (± 21.90%)

Number of 1.35 0.38

extra detections (± 1.52) (± 1.06)

These results show that the mean percentage of detected events within a given sig-

nal is signiﬁcant for both algorithm, being slightly bigger for the events detection algo-

rithm proposed in this paper. Considering the number of extra detections, however, this

is more signiﬁcant when considering our events detection algorithm than for Pan and

Tompkins adapted algorithm. This extra events detection is mostly due to the detection

of some pronounced T waves in some signals.

Figure 1b) exempliﬁes our tool’s ability to accurately mark both onset and offset

instants in EMG signals. This is extremely important towards an accurate posterior fea-

tures extraction, such as the signal amplitude and the duration of each activation. For

performance evaluation on onset detection, the results of our approach and those of

Hodges detector were compared with those obtained previously by visual inspection. In

order to minimize the error from intra-rater variability, results from 3 examiners were

considered and averaged, in each signal, to deﬁne the ”true” onset value. For each signal

the difference between the ”true” onset and those detected for each one of the imple-

mented computational methods was computed. In case that a computational method

detected no onset time, that was also registered. Table 2 exposes the mean error (con-

sidering only the signals for which the onset detection was achieved) and the percentage

of missing detections veriﬁed for each of compared methods.

Table 2. Results obtained from the proposed events detection algorithm and Hodges detector

while performing the EMG signals onset detection. The ”true” onset values were determined by

visual inspection. The results follow the format mean(±standard deviation).

Events detection Hodges detector Hodges detector

algorithm {N=25, h=3} {N=50,h=1}

Mean error (samples)

-22.94

—

12.65

(± 115.09) (± 81.85)

Percentage of

0% 100% 5%

missing detections

These results show that the proposed events detection algorithm achieved a superior,

though not signiﬁcant, absolute mean error than that veriﬁed for Hodges detector im-

plemented with the parameter set {N=50, h=1}. An high standard deviation value was

veriﬁed for mean error values of both approaches. Therefore, there is no clear tendency

of either the algorithms to perform an early or late onset detection.

Considering the events detection algorithm results, no missing onset detections were

veriﬁed. Hodges detector implemented with the parameters set {N=50, h=1} missed 2

out of 40 onset detections. However, when applied with the criteria {N=25, h=3}, no

onset was detected applying Hodges algorithm. This clearly exempliﬁes the sensibility

of this single-threshold based algorithm to its input parameters, in agreement with that

previously reported in other studies [23, 24].

In general, the algorithm here proposed showed to have a comparable efﬁciency

and higher reliability than that of standard signal events detectors, without the previous

knowledge or speciﬁc pre-processing steps required in those approaches.

4 Conclusions and Future Work

In this paper we have introduced a generalized approach on biosignals analysis, aim-

ing at an accurate events detection for posterior features extraction. The performance

of our algorithm proved to be comparable with that achieved by ECG and EMG sig-

nals speciﬁc processing techniques on events detection. Since no previous knowledge

or signal-speciﬁc processing steps are required, our approach is particularly suited to

application in the context of automatic emotion recognition, bringing simplicity and

scalability to those systems.

In future work it is our intention to perform its application on a wider range of

biosignals used for emotion recognition, such as the electrodermal response and the

respiration signals. The integration of the developed algorithm with features classiﬁ-

cation tools will also provide means to evaluate its performance towards an automatic

emotion recognition system, when compared with standard signal-speciﬁc features ex-

traction techniques.

Acknowledgements

This work was partially supported by National Strategic Reference Framework (NSRF-

QREN) under projects ”LUL” and ”Affective Mouse”, by Seventh Framework Pro-

gramme (FP7) program under project ICT4Depression and under the grant SFRH/BD/

65248/2009 from Fundac¸

ao para a Ci

encia e Tecnologia (FCT), whose support the au-

thors gratefully acknowledge.

References

1. Picard, R.W. and Vyzas, E. and Healey, J.: Toward machine emotional intelligence: Anal-

ysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine

Intelligence, Vol. 23. Published by the IEEE Computer Society (2001) 1175–1191

2. Haag, A. and Goronzy, S. and Schaich, P. and Williams, J.: Emotion recognition using

bio-sensors: First steps towards an automatic system. Affective Dialogue SystemsBME-32.

Springer (2004) 36–48

3. Kim, J.: Bimodal emotion recognition using speech and physiological changes. Robust

Speech Recognition and Understanding. Citeseer (2007) 265–280

4. Kim, KH and Bang, SW and Kim, SR: Emotion recognition system using short-term moni-

toring of physiological signals. Medical and biological engineering and computing, Vol. 42.

Springer (2004) 419–427

5. Van Den Broek, E. L. and Lis

y, V. and Westerink, J. H. D. M. and Schut, M. H. and Tuinen-

breijer, K.: Biosignals as an Advanced Man-Machine Interface. BIOSTEC International Joint

Conference on Biomedical Engineering Systems and Technologies. Citeseer (2009) 15–24

6. Kim, J. and Andr

e, E.: Emotion recognition based on physiological changes in music listen-

ing. IEEE Transactions on Pattern Analysis and Machine Intelligence , Vol. 30. Published by

the IEEE Computer Society (2008) 2067–2083

7. Heo, H. and Lee, E. C. and Park, K. R. and Kim, C. J. and Whang, M.: A realistic game sys-

tem using multi-modal user interfaces Consumer Electronics, IEEE Transactions on, Vol. 56.

IEEE (2010) 1364–1372

8. Wagner, J. and Andr

e, E.: From physiological signals to emotions: Implementing and com-

paring selected methods for feature extraction and classiﬁcation. 2005 IEEE International

Conference on Multimedia and Expo. IEEE (2005) 940–943

9. Rigas, G. and Katsis, C. and Ganiatsas, G. and Fotiadis, D.: A user independent, biosignal

based, emotion recognition method. User Modeling 2007, Vol. 4511. Springer (2007) 314–

318

10. Hristova, E. and Grinberg, M. and Lalev, E.: Biosignal Based Emotion Analysis of Human-

Agent Interactions. Cross-Modal Analysis of Speech, Gestures, Gaze and Facial Expres-

sions, Vol. 5641. Springer(2009) 63–75

11. Matsumoto, D. and Keltner, D. and Shiota, M. N. and O’Sullivan, M. and Frank, M.: Facial

expressions of emotion. Handbook of emotions. The Guilford Press (2008) 211–234

12. Vogt, T. and Andr

e, E. and Wagner, J.: Automatic recognition of emotions from speech: a

review of the literature and recommendations for practical realization. Affect and Emotion

in Human-Computer Interaction, Vol. 5078. Springer (2008) 75–91

13. Kipp, M. and Martin, J. C.: Gesture and Emotion: Can basic gestural form features dis-

criminate emotions? ACII 2009. 3rd International Conference on Affective Computing and

Intelligent Interaction and Workshops, 2009. IEEE (2009) 1–8

14. Canento, F. and Fred, A. and Silva, H. and Gamboa, H. and Lourenc¸o, A.: Multimodal

Biosignal Sensor Data Handling for Emotion Recognition IEEE Sensors 2011. IEEE In-

ternational Sensors Conference, 2011.

15. Camm, A. J. and Malik, M. and Bigger, JT and Breithardt, G. and Cerutti, S. and Cohen, RJ

and Coumel, P. and Fallen, EL and Kennedy, HL and Kleiger, RE and others: Heart rate vari-

ability: standards of measurement, physiological interpretation, and clinical use. Circulation,

Vol. 93. (1996)1043–1065

16. PhysioBank - physiologic signal archives for biomedical research (2009) [online] Available

at: http://physionet.ph.biu.ac.il/physiobank/ [Accessed 8 November 2011]

17. Hadjileontiadis, L.J.: Biosignals and compression standards. M-Health Springer(2006) 277–

292

18. Nakasone, A. and Prendinger, H. and Ishizuka, M.: Emotion recognition from electromyo-

graphy and skin conductance. The Fifth International Workshop on Biosignal Interpretation

(BSI-05), Tokyo, Japan Citeseer (2005) 219–222

19. Marques, C. J. and Gamboa, H. and Lampe, F. and Barreiros, J. and Cabri, J.: Muscle activa-

tion thresholds before and after total knee arthoplasty - Protocol of a Randomized Compar-

ison of Minimally Invasive vs. Standard Approach Proceedings of the International Confer-

ence on Bio-inspired Systems and Signal Processing (BIOSTEC 2011) 544-547

20. PLUX - Wireless Biosignals, S.A. (2007) [online] Available at: http://plux.info/ [Accessed 8

November 2011]

21. Santos, R. and Sousa, J. and Sa

nudo, B. and Marques, C. J. and Gamboa, H.: Biosignals

events detection - a morphological signal-independent approach. Proceedings of the Interna-

tional Conference on Bio-inspired Systems and Signal Processing (BIOSTEC 2012)

22. Pan, J. and Tompkins, W. J.: A real-time QRS detection algorithm. IEEE Transactions on

Biomedical Engineering, Vol. BME-32. IEEE (1985) 230 - 236

23. Hodges, P. W. and Bui, B. H.: A comparison of computer-based methods for the determi-

nation of onset of muscle contraction using electromyography. Electroencephalography and

Clinical Neurophysiology/Electromyography and Motor Control, Vol. 101. Elsevier (1996)

511–519

24. Staude, G. and Flachenecker, C. and Daumer, M. and Wolf, W.: Onset detection in surface

electromyographic signals: a systematic comparison of methods. EURASIP Journal on Ap-

plied Signal Processing, Vol. 2001. Hindawi Publishing Corp. (2001) 67–81