Towards Emotion Related Feature Extraction based on
Generalized Source-Independent Event Detection
Rui Santos
1
, Joana Sousa
2
, Carlos J. Marques
3,4
, Hugo Gamboa
1,2
and Hugo Silva
2,5
1
Physics Department, FCT-UNL, Lisbon, Portugal
2
PLUX - Wireless Biosignals, S.A., Lisbon, Portugal
3
Faculty of Human Kinetics at the Technical University of Lisbon, Lisbon, Portugal
4
Physical Therapy and Rehabilitation Department at the Sch
¨
on Klinik, Hamburg, Germany
5
Instituto de Telecomunicac¸
˜
oes, IST-UTL, Lisbon, Portugal
Abstract. Emotion recognition is of major importance towards the acceptabil-
ity of Human-Computer Interaction systems, and several approaches to emo-
tion classification using features extracted from biosignals have already been de-
veloped. This analysis is, in general, performed on a signal-specific basis, and
can bring a significant complexity to those systems. In this paper we propose
a signal-independent approach on marking specific signal events. In this prelimi-
nary study, the developed algorithm was applied on ECG and EMG signals. Based
on a morphological analysis of the signal, the algorithm allows the detection of
significant events within those signals. The performance of our algorithm proved
to be comparable with that achieved by signal-specific processing techniques on
events detection. Since no previous knowledge or signal-specific pre-processing
steps are required, the presented approach is particularly interesting for automatic
feature extraction in the context of emotion recognition systems.
1 Introduction
The ability to recognize emotion is of upmost importance with the increasing develop-
ment of intelligent and adaptive computer systems, allowing them to sense and respond
appropriately to user’s affective feedback [1]. Emotions are one of the least explored
frontiers of intuitive Human-Computer Interaction (HCI) [2], and its understanding is
expected to improve the acceptability of those systems. This requires, however, robust
emotion recognition systems, capable of guaranteeing acceptable recognition accuracy,
and adaptable to practical applications [3].
Emotion recognition systems still present relevant challenges, as it is very hard
to uniquely correlate emotion-relevant signal patterns with a certain emotional state.
Furthermore, these patterns may widely differ from person to person, and between
different situations [3]; there is a lack of ground-truth datasets in order to develop
user-independent systems, which would be essential for practical applications [4, 5].
There have been many attempts to built automatic emotion recognition systems [3, 6–
9], which mostly rely on supervised pattern classification approaches. However, some of
the major problems still verified are the low recognition rates when considering subject-
independent classification and generalization to more than one task [10]. A generally
Santos R., Sousa J., J. Marques C., Gamboa H. and Silva H..
Towards Emotion Related Feature Extraction based on Generalized Source-Independent Event Detection.
DOI: 10.5220/0003891900710078
In Proceedings of the 2nd International Workshop on Computing Paradigms for Mental Health (MindCare-2012), pages 71-78
ISBN: 978-989-8425-92-8
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
applicable recognition system, for realistic online applications, would have to automati-
cally select the most significant features and specific classifiers to several data sets ob-
tained from different natural contexts [8].
Emotions are inherently multi-modal, and several studies on their recognition work
by fusing features extracted from multiple modalities (facial expression [11], voice [12]
or gesture [13] and biosignals). The work around biosignals include the heart rate, skin
temperature, electrodermal and electromyographic activity or respiration rate [1, 14].
The areas of speech and face recognition are far more explored, mostly because it is a
very hard task to uniquely map physiological patterns onto specific emotional states and
some of the sensors used to acquire those biosignals may be sensitive to motion artifacts
[6]. However, the size of such sensors is decreasing and this method is considered to
be less disturbing than the standard audiovisual techniques [2]. Moreover, biosignals
allow us to continuously gather information on the user’s affective state, and should be
more robust against possible artifacts of human social masking, since they are directly
controlled by the human autonomous nervous system [3, 8].
Feature extraction from biosignals is a complex multivariate task, requiring a broad
insight into biological processes related with neuropsychological functions [6]. Stan-
dard specific processing techniques in which the results are highly dependent on the
input parameters are usually applied, bringing an added complexity to these feature
extraction systems. In this paper we propose a signal-independent approach to the de-
tection of specific signal events, in order to accomplish an accurate feature extraction
based on the results of a generic algorithm. The performance of the generic events de-
tection approach is also compared with that of signal-specific standard algorithms.
The following section presents a brief description of the biosignals considered in
this preliminary approach, as well as the applied signal processing methodologies. The
obtained results are presented and discussed in section 3. In the last section some final
remarks and future work steps are referred.
2 Materials and Methods
2.1 Biosignals
This subsection introduces the biosignals that were considered in this preliminary study,
including the main features usually extracted in the context of emotion classification
systems. Those types of signals were selected based on their characteristic waveshapes,
in which the events are clearly distinguishable. As such, an intuitive evaluation of the
algorithm performance on that signals is accomplished. Furthermore, these signals are
strong related to mental diseases and they are very helpful into infers about the mental
health condition of the patients.
With these signals we only intend to exemplify the application of the developed
algorithm, since their origin, as respectively described, is not related to the testing pro-
tocols usually followed in emotion classification studies [1, 3].
Electrocardiography. The electrocardiogram (ECG) is the recording, on the body sur-
face, of the electrical activity generated by the heart. From the ECG processing one can
72
extract features as the Heart Rate (HR) and the Heart Rate Variability (HRV). HRV has
become the conventionally accepted term to describe variations of both instantaneous
heart rate and RR interval. In a continuous ECG record, each QRS complex is detected,
and the so-called normal-to-normal (NN) intervals or the instantaneous heart rate is de-
termined. Simple time domain variables that can be calculated include the mean NN
interval, the standard deviation of the NN intervals (SDNN) or the mean heart rate.
Spectral analysis is also usually performed, since there is a correlation between the
relative power of the low frequencies (LF) and high frequencies (HF) ranges and the
sympathetic and parasympathetic nervous systems activity. Non-linear methods based
on chaos theory and fractal analysis are also used on HRV analysis to better understand
the HR fluctuations. The common example of non-linear methods is the Poincar
´
e plot,
which reflects the graphical correlation between consecutive RR intervals [15].
Since HRV and the automatic heart modulation are correlated, HRV analysis is a
powerful tool for clinical use. The HRV analysis allows a better understanding of the
SA node automatic modulation. For example, the spectral analysis reveals that the vagal
activity is the major contribution to the high frequencies component. Furthermore, it is
also argued that the LF and HF ratio reflects the sympathovagal balance or the sym-
pathetic modulation. These parameters are very important since some authors defend
that there is a significant increase of LF band and significant increment of the HR for
panic disorders patients and significant lower values of R-R intervals and HF peak of
spectral analysis in depressive patients than in the health people. In general, both the
HR and the HRV are dependent on the activity level of the autonomic nervous system,
which in turn is dependent on emotional stimuli [4]. A low HRV can indicate a state of
relaxation, while an increased HRV can be caused by mental stress or frustration [2].
The ECG signals here considered were obtained from a public database (PhysioBank)
at PhysioNet library [16]. Those were acquired from healthy people, with normal sinus
rhythm, during 4 seconds and at a sampling frequency of 125 Hz. A total of 26 ECG
signals was considered.
Electromyography. Electromyography (EMG) signal arises from the flow of charged
particles across the muscle membrane when its cells are electrically activated. This
biosignals can record both voluntary and involuntary muscle activation, in addition to
the action potentials produced by external stimulation, such as motor evoked potentials
after central or peripheral nerve stimulation [17]. EMG signals have been shown to
correlate with negatively valenced emotions [18]. The signals here considered were
acquired in the context of a study aiming at accessing the performance of the right leg
during the execution of an emergency brake in a car simulator [19]. EMG signals of the
Rectus Femoris, Vastus Medialis, Tibialis Anterior and Gastrocnemius muscles were
acquired while 3 subjects performed the emergency brake test. From that data set, a
total of 40 EMG signals was considered in this study, comprising 10 signals of each
considered muscle. Bipolar EMG sensors (emgPLUX) were used to access the muscle
activation. The sensors were connected to a wireless acquisition unit, bioPLUX research
[20], which performed the acquisition at a sampling frequency of 1000 Hz.
73
2.2 Generalized Signal Processing Approach
The main features that guided the development of the introduced signal processing
methodology have been the signal-independence, implying no specific pre-processing
steps. The immunity to noise and artifacts and the simplicity, in order to allow a real-
time implementation, were also considered.
The basis of the developed algorithm is the identification of time-domain specific
morphological parameters that can clearly distinguish events, such as onset and offset
instants and transient waveshapes, within a signal. In practise, after a signal segmenta-
tion, those events are computed as the split points for which the absolute values of the
difference between the standard deviation of the successive segments are maximized.
A further optimization step is then applied through an iterative change of the input pa-
rameters of the previously described processing steps. The optimal solution is selected
as the one which assures a better fitting of all the signal segments, between the detected
events, to linear regressions models.
A detailed description and performance evaluation of this signal processing ap-
proach can be found in [21]. This paper extends that events detection approach to
application in different biosignals and evaluates its performance by comparison with
standard signal-specific algorithms.
2.3 Standard Signal-specific Algorithms
In order to evaluate the performance of the proposed signal processing approach, its
results were compared with those from signal-specific standard techniques for EMG
onset and ECG waveshape detection.
An adaptation of the QRS complex detector algorithm proposed by Pan and Tomp-
kins [22] was implemented as a standard method for performance evaluation on de-
tecting heartbeats within an ECG signal. ECG signal pre-processing steps included
the application of a 2
nd
order Butterworth lowpass filter with a 30 Hz cutoff fre-
quency, to remove some possible high frequency noise due to electronic devices inter-
ference. A 2
nd
order Butterworth high pass filter with 2 Hz cutoff frequency was also
applied to remove some possible baseline fluctuations and other respiration artifacts.
Considering EMG signals, a standard onset detector proposed by Hodges et al.[23]
was implemented. This approach implies the EMG signal to be pre-processed in order
to reduce the high frequency noise and obtain the signal’s envelope. After subtracting
the mean value from the EMG signal and obtaining the rectified wave, a sixth-order
digital butterworth filter with 50 Hz cutoff frequency was applied, following the imple-
mentation procedure described in [24], completing the envelope detection.
The muscle activity onset was identified as the instant from which the mean of the
following N samples from the EMG envelope exceeds the baseline activity level by
a specified multiple h of standard deviations. The baseline activity was adaptively de-
termined, at each instant, by averaging M previous signal samples [24]. With a cutoff
frequency of 50Hz, the set of criteria {N=25, h=3} and {N=50, h=1} are those reported
as able to identify the EMG onset more accurately [23]. Hodges detector was imple-
mented with both set of parameters, considering the baseline activity level computed
from M=100 and M=200, respectively [24].
74
3 Results and Discussion
The signal processing methodology described in sub-section 2.2 was applied to the
selected raw biosignals. No signal-specific pre-processing steps were applied. Graphic
representations exemplifying the obtained results are presented in Figure 1.
Fig. 1. Graphic example of: a) detected ECG waveshapes and b) EMG onset and offset points
applying the developed events detection algorithm. The events are marked by vertical red marks.
As exemplified in Figure 1a), the developed algorithm allows the ECG waveshape
discrimination, from the electrical baseline, detecting two events around it. Choosing
one of those instants, parameters such as the HR and the HRV can be easily accessed.
Furthermore, this events marking could be used for a further and more complete fea-
tures extraction based on the waveshape morphology, such as the position of each of the
P, Q, R, S and T waves and the respective amplitude. After applying Pan and Tompkins
adapted algorithm to ECG signals, the resulting graphics were then visually examined
and, for both algorithms, the percentage of detected ECG waveshapes and the num-
ber of extra detected events were registered. Table 1 exposes the mean values of those
parameters.
Table 1. Results obtained from the proposed events detection algorithm and from Pan and Tomp-
kins adapted algorithm while performing the ECG signals QRS complex detection. The results
follow the format mean(±standard deviation).
Events detection Pan and Tompkins
algorithm algorithm
Percentage of detected 93.84% 92.63%
ECG waveshapes (± 15.64 %) (± 21.90%)
Number of 1.35 0.38
extra detections (± 1.52) (± 1.06)
These results show that the mean percentage of detected events within a given sig-
nal is significant for both algorithm, being slightly bigger for the events detection algo-
rithm proposed in this paper. Considering the number of extra detections, however, this
is more significant when considering our events detection algorithm than for Pan and
Tompkins adapted algorithm. This extra events detection is mostly due to the detection
75
of some pronounced T waves in some signals.
Figure 1b) exemplifies our tool’s ability to accurately mark both onset and offset
instants in EMG signals. This is extremely important towards an accurate posterior fea-
tures extraction, such as the signal amplitude and the duration of each activation. For
performance evaluation on onset detection, the results of our approach and those of
Hodges detector were compared with those obtained previously by visual inspection. In
order to minimize the error from intra-rater variability, results from 3 examiners were
considered and averaged, in each signal, to define the ”true” onset value. For each signal
the difference between the ”true” onset and those detected for each one of the imple-
mented computational methods was computed. In case that a computational method
detected no onset time, that was also registered. Table 2 exposes the mean error (con-
sidering only the signals for which the onset detection was achieved) and the percentage
of missing detections verified for each of compared methods.
Table 2. Results obtained from the proposed events detection algorithm and Hodges detector
while performing the EMG signals onset detection. The ”true” onset values were determined by
visual inspection. The results follow the format mean(±standard deviation).
Events detection Hodges detector Hodges detector
algorithm {N=25, h=3} {N=50,h=1}
Mean error (samples)
-22.94
12.65
(± 115.09) (± 81.85)
Percentage of
0% 100% 5%
missing detections
These results show that the proposed events detection algorithm achieved a superior,
though not significant, absolute mean error than that verified for Hodges detector im-
plemented with the parameter set {N=50, h=1}. An high standard deviation value was
verified for mean error values of both approaches. Therefore, there is no clear tendency
of either the algorithms to perform an early or late onset detection.
Considering the events detection algorithm results, no missing onset detections were
verified. Hodges detector implemented with the parameters set {N=50, h=1} missed 2
out of 40 onset detections. However, when applied with the criteria {N=25, h=3}, no
onset was detected applying Hodges algorithm. This clearly exemplifies the sensibility
of this single-threshold based algorithm to its input parameters, in agreement with that
previously reported in other studies [23, 24].
In general, the algorithm here proposed showed to have a comparable efficiency
and higher reliability than that of standard signal events detectors, without the previous
knowledge or specific pre-processing steps required in those approaches.
4 Conclusions and Future Work
In this paper we have introduced a generalized approach on biosignals analysis, aim-
ing at an accurate events detection for posterior features extraction. The performance
of our algorithm proved to be comparable with that achieved by ECG and EMG sig-
nals specific processing techniques on events detection. Since no previous knowledge
76
or signal-specific processing steps are required, our approach is particularly suited to
application in the context of automatic emotion recognition, bringing simplicity and
scalability to those systems.
In future work it is our intention to perform its application on a wider range of
biosignals used for emotion recognition, such as the electrodermal response and the
respiration signals. The integration of the developed algorithm with features classifi-
cation tools will also provide means to evaluate its performance towards an automatic
emotion recognition system, when compared with standard signal-specific features ex-
traction techniques.
Acknowledgements
This work was partially supported by National Strategic Reference Framework (NSRF-
QREN) under projects ”LUL” and ”Affective Mouse”, by Seventh Framework Pro-
gramme (FP7) program under project ICT4Depression and under the grant SFRH/BD/
65248/2009 from Fundac¸
˜
ao para a Ci
ˆ
encia e Tecnologia (FCT), whose support the au-
thors gratefully acknowledge.
References
1. Picard, R.W. and Vyzas, E. and Healey, J.: Toward machine emotional intelligence: Anal-
ysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine
Intelligence, Vol. 23. Published by the IEEE Computer Society (2001) 1175–1191
2. Haag, A. and Goronzy, S. and Schaich, P. and Williams, J.: Emotion recognition using
bio-sensors: First steps towards an automatic system. Affective Dialogue SystemsBME-32.
Springer (2004) 36–48
3. Kim, J.: Bimodal emotion recognition using speech and physiological changes. Robust
Speech Recognition and Understanding. Citeseer (2007) 265–280
4. Kim, KH and Bang, SW and Kim, SR: Emotion recognition system using short-term moni-
toring of physiological signals. Medical and biological engineering and computing, Vol. 42.
Springer (2004) 419–427
5. Van Den Broek, E. L. and Lis
`
y, V. and Westerink, J. H. D. M. and Schut, M. H. and Tuinen-
breijer, K.: Biosignals as an Advanced Man-Machine Interface. BIOSTEC International Joint
Conference on Biomedical Engineering Systems and Technologies. Citeseer (2009) 15–24
6. Kim, J. and Andr
´
e, E.: Emotion recognition based on physiological changes in music listen-
ing. IEEE Transactions on Pattern Analysis and Machine Intelligence , Vol. 30. Published by
the IEEE Computer Society (2008) 2067–2083
7. Heo, H. and Lee, E. C. and Park, K. R. and Kim, C. J. and Whang, M.: A realistic game sys-
tem using multi-modal user interfaces Consumer Electronics, IEEE Transactions on, Vol. 56.
IEEE (2010) 1364–1372
8. Wagner, J. and Andr
´
e, E.: From physiological signals to emotions: Implementing and com-
paring selected methods for feature extraction and classification. 2005 IEEE International
Conference on Multimedia and Expo. IEEE (2005) 940–943
9. Rigas, G. and Katsis, C. and Ganiatsas, G. and Fotiadis, D.: A user independent, biosignal
based, emotion recognition method. User Modeling 2007, Vol. 4511. Springer (2007) 314–
318
77
10. Hristova, E. and Grinberg, M. and Lalev, E.: Biosignal Based Emotion Analysis of Human-
Agent Interactions. Cross-Modal Analysis of Speech, Gestures, Gaze and Facial Expres-
sions, Vol. 5641. Springer(2009) 63–75
11. Matsumoto, D. and Keltner, D. and Shiota, M. N. and O’Sullivan, M. and Frank, M.: Facial
expressions of emotion. Handbook of emotions. The Guilford Press (2008) 211–234
12. Vogt, T. and Andr
´
e, E. and Wagner, J.: Automatic recognition of emotions from speech: a
review of the literature and recommendations for practical realization. Affect and Emotion
in Human-Computer Interaction, Vol. 5078. Springer (2008) 75–91
13. Kipp, M. and Martin, J. C.: Gesture and Emotion: Can basic gestural form features dis-
criminate emotions? ACII 2009. 3rd International Conference on Affective Computing and
Intelligent Interaction and Workshops, 2009. IEEE (2009) 1–8
14. Canento, F. and Fred, A. and Silva, H. and Gamboa, H. and Lourenc¸o, A.: Multimodal
Biosignal Sensor Data Handling for Emotion Recognition IEEE Sensors 2011. IEEE In-
ternational Sensors Conference, 2011.
15. Camm, A. J. and Malik, M. and Bigger, JT and Breithardt, G. and Cerutti, S. and Cohen, RJ
and Coumel, P. and Fallen, EL and Kennedy, HL and Kleiger, RE and others: Heart rate vari-
ability: standards of measurement, physiological interpretation, and clinical use. Circulation,
Vol. 93. (1996)1043–1065
16. PhysioBank - physiologic signal archives for biomedical research (2009) [online] Available
at: http://physionet.ph.biu.ac.il/physiobank/ [Accessed 8 November 2011]
17. Hadjileontiadis, L.J.: Biosignals and compression standards. M-Health Springer(2006) 277–
292
18. Nakasone, A. and Prendinger, H. and Ishizuka, M.: Emotion recognition from electromyo-
graphy and skin conductance. The Fifth International Workshop on Biosignal Interpretation
(BSI-05), Tokyo, Japan Citeseer (2005) 219–222
19. Marques, C. J. and Gamboa, H. and Lampe, F. and Barreiros, J. and Cabri, J.: Muscle activa-
tion thresholds before and after total knee arthoplasty - Protocol of a Randomized Compar-
ison of Minimally Invasive vs. Standard Approach Proceedings of the International Confer-
ence on Bio-inspired Systems and Signal Processing (BIOSTEC 2011) 544-547
20. PLUX - Wireless Biosignals, S.A. (2007) [online] Available at: http://plux.info/ [Accessed 8
November 2011]
21. Santos, R. and Sousa, J. and Sa
˜
nudo, B. and Marques, C. J. and Gamboa, H.: Biosignals
events detection - a morphological signal-independent approach. Proceedings of the Interna-
tional Conference on Bio-inspired Systems and Signal Processing (BIOSTEC 2012)
22. Pan, J. and Tompkins, W. J.: A real-time QRS detection algorithm. IEEE Transactions on
Biomedical Engineering, Vol. BME-32. IEEE (1985) 230 - 236
23. Hodges, P. W. and Bui, B. H.: A comparison of computer-based methods for the determi-
nation of onset of muscle contraction using electromyography. Electroencephalography and
Clinical Neurophysiology/Electromyography and Motor Control, Vol. 101. Elsevier (1996)
511–519
24. Staude, G. and Flachenecker, C. and Daumer, M. and Wolf, W.: Onset detection in surface
electromyographic signals: a systematic comparison of methods. EURASIP Journal on Ap-
plied Signal Processing, Vol. 2001. Hindawi Publishing Corp. (2001) 67–81
78