PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP)

Egon L. van den Broek

∗

Center for Telematics and Information Technology (CTIT), University of Twente

P.O. Box 217, 7500 AE Enschede, The Netherlands

Joris H. Janssen

∗

, Joyce H. D. M. Westerink

User Experience Group, Philips Research Europe, High Tech Campus 34, 5656 AE Eindhoven, The Netherlands

Jennifer A. Healey

Intel Corporation, Corporate Technology Group, 3600 Juliette Lane SC12-319 Santa Clara CA 95054, U.S.A.

Keywords:

Affective signal processing (ASP), Emotion, Validation, Physiology-driven, Triangulation.

Abstract:

Although emotions are embraced by science, their recognition has not reached a satisfying level. Through a

concise overview of affect, its signals, features, and classiﬁcation methods, we provide understanding for the

problems encountered. Next, we identify the prerequisites for successful Affective Signal Processing: valida-

tion (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions

of the signal processing community. Using these directives, a critical analysis of a real-world case is provided.

This illustrates that the prerequisites can become a valuable guide for Affective Signal Processing.

When dealing with people, let us remember that we

are not dealing with creatures of logic; we are dealing

with creatures of emotion ... (Dale Carnegie, 1936)

Dale Carnegie (1888-1955)

1 INTRODUCING EMOTIONS

While a century ago emotions were considered as too

spiritual and human’s health was solely explained in

physical (e.g., injuries) and physiological terms (e.g.,

bacteria, viruses), it is now generally acknowledged

that emotions have their impact on health and illness.

It has been shown that emotions inﬂuence our cog-

nitive and social functioning as well as our cardio-

vascular system (Schuler and O’Brien, 1997) and, as

a consequence, can even either shorten or prolong

∗

Joris H. Janssen and Egon L. van den Broek did equally

contribute to this article; hence, they are shared ﬁrst authors.

For the interested reader, we refer to “Historical foun-

dations of social effectiveness? Dale Carnegie’s princi-

ples” (Duke and Novicevic, 2008), which illustrates the

timeless signiﬁcance of Carnegie’s work.

life (Frederickson et al., 2000).

Medicine’s interest in emotions was followed by

that of Artiﬁcial Intelligence (AI), which envisioned

that emotions both lead the path to true AI and en-

hance the communication between man and machine

(or even environment) (Picard, 1997; Minsky, 2006).

This expresses the intrinsic need for automated sens-

ing of emotions. Often this is done through speech

or face analysis; see Cowie et al. (2001) and Zeng

et al. (in press) for reviews. Alternatively, physio-

logical signals are used to identify emotions; see also

Table 1 and 2, Box 1, and (Westerink et al., 2008a).

This paper discusses the this last approach, which we

will denote as Affective Signal Processing (ASP).

Physiological signals have the advantage that they

are free from social masking and have the potential of

being measured by non-invasive sensors; e.g., (Sam-

boa et al., 2008; Westerink et al., 2008a), making

them suited for a wide range of applications. In

contrast, recognizing facial expressions is notoriously

problematic and speech is often either absent or suf-

fers from severe distortions in many in real-world ap-

plications (Cowie et al., 2001; Zeng et al., in press).

In the next section, we provide both an overview

426

L. van den Broek E., H. Janssen J., H. D. M. Westerink J. and A. Healey J. (2009).

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP).

In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing , pages 426-433

DOI: 10.5220/0001780504260433

 SciTePress

and a review of ASP. After that, we introduce four

prerequisites for successful ASP. Moreover, a real

world case will be analyzed, using these prerequisites

(Box 1). We ﬁnish the paper with a brief conclusion

in which we denote some applications of ASP.

2 AFFECTIVE SIGNAL

PROCESSING (ASP)

A broad range of affective signals are used in affec-

tive sciences. When processing such signals some

general issues have to be taken in consideration: 1)

Affective signals are typically derived through non-

invasive methods to determine changes in physiology

and, as such, are indirect measures. Hence, a delay

between the actual change in emotional state and the

recorded change in signal has to be taken into ac-

count. 2) Physiological sensors are unreliable: they

contain movement artifacts and are sensitive to dif-

ferences in bodily position. 3) Most sensors are ob-

trusive, preventing their integration in real world ap-

plications. 4) Affective signals are inﬂuenced by (the

interaction among) a variety of factors (Cacioppo and

Tassinary, 1990). Some of these sources are located

internally (e.g., a thought) and some are among the

broad range of possible external factors (e.g., a sig-

nal outside). This makes affective signals inherently

noisy, which is most prominent in real world research.

5) Physiological changes can evolve in a matter of

milliseconds, seconds, minutes or even longer. Some

changes hold for only a brief moment, while others

can even be permanent. Although seldom reported,

the expected time windows of change are of interest.

In particular since changes can add to each other, even

when having a different origin. 6) Humans are no lin-

ear time (translationor shift) invariant systems (Bouc-

sein, 1992), they habituate. This increases the com-

plexity of ASP substantially, since most signal pro-

cessing techniques rely on this assumption. 7) Af-

fective signals have large individual differences. This

calls for methods and models tailored to the indi-

vidual. It has been shown that personal approaches

increase the performance of ASP (Bailenson et al.,

2008).

2.1 Classiﬁcation of Affective Signals

To enable processing of the signals, in most cases

comprehensive sets of features have to be identiﬁed

for each affective signal. To extract these features, the

affective signals are processed in the time (e.g., statis-

tical moments (Westerink et al., 2008b)), frequency

(e.g., Fourier), time-frequency (e.g., Wavelets), or

power domain (e.g., Periodogram and Autoregres-

sion). In Table 1, we provide a brief overview of the

signals most often applied, including their best known

features, with reference to their physiological source.

The features obtained from the affective signals

(see Table 1) are fed to pattern recognition methods,

which can be classiﬁed as: template matching, syn-

tactic or structural matching, and statistical classiﬁca-

tion; e.g., artiﬁcial neural networks (ANN). The for-

mer two are not or seldom used in affective signal pro-

cessing; most ASP schemes use the latter. Statistical

pattern recognition distinguishes supervised and un-

supervised (e.g., clustering) pattern recognition; i.e.,

respectively, with or without a set of (labeled) training

data. With unsupervised pattern recognition, the dis-

tance/similarity measure used and the algorithm ap-

plied to generate the clusters are key elements. Super-

vised pattern recognition relies on learning from a set

of examples (training set). Statistical pattern recog-

nition uses input features, a discriminant function (or

network function for ANN) to recognize the features,

and an error criterion in its classiﬁcation process.

In the ﬁeld of ASP, several studies have been con-

ducted, using a broad range of signals, features, and

classiﬁers; see Table 2 for an overview. Nonetheless,

both the recognition performance and the number of

emotions that the classiﬁers were able to discriminate

are disappointing. Moreover, comparing the different

studies is problematic because of the different settings

the research was applied in, ranging from controlled

lab studies to real world testing, the type of emotion

triggers used, the number of target states to be dis-

criminated, and the signals and features employed.

This illustrates the need for a set of prerequisites for

ASP.

3 PREREQUISITES

3.1 Validity

In the pursuit to trigger emotions in a more or less

controlled manner, a range of methods have been

applied: actors, images (IAPS), sounds (e.g., mu-

sic), (fragments of) movies (Westerink et al., 2008b),

speech (Van den Broek, 2004), commercials (Ha-

zlett and Hazlett, 1999; Poels and Dewitte, 2006),

games, agents / serious gaming / virtual reality (Slater

et al., 2006; Westerink et al., 2008a), reliving of emo-

tions (Van den Broek, 2004), and real world expe-

riences (Healey and Picard, 2005); see also Box 1.

However, how to know which of these methods ac-

tually triggered participants’ true emotions? This is

a typical concern of validity, which is a crucial issue

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP)

427

Table 1: An overview of common physiological signals and features used in ASP.

Physiology Features Unit Remark

Cardiovascular activity Heart rate (HR) beats / min

through ECG or BVP SD IBIs s HRV index

(Berntson et al., 1997) RMSSD IBIs s HRV index

LF power (0.05Hz - 0.15Hz) ms

Sympathetic activity

HF power (0.15HZ - 0.40Hz) ms

Parasympathetic activity

VLF power ( < 0.05Hz) ms

LF/HF

Pulse Transit Time (PTT) ms

Electrodermal Activity (EDA) Mean, SD SCL µS Tonic Sympathetic Activity

(Boucsein, 1992) Nr of SCRs Rate Phasic Activity

SCR amplitude µS Phasic Activity

SCR 1/2 recovery time s

SCR rise time s

Skin temperature (ST) Mean, SD temp

Respiration Respiration rate

(Grossman and Taylor, 2007) Amplitude Resps

Muscle activity Mean, SD corrugator supercilii µV Frowning

through EMG Mean, SD zygomaticus major µV Smiling

(Reaz et al., 2006) Mean, SD upper trapezius µV

(Westerink et al., 2008b) Mean, SD inter-blink interval ms

Notes. SD: Standard deviation; RMSSD: Root Mean Sum of Square Differences; IBI: Inter-beat interval; LF: Low frequency;

HF: High frequency; VLF: Very Low Frequency; SCL: Skin Conductance Level; SCR: Skin Conductance Response; ECG:

Electrocardiogram; EMG: Electromyogram; BVP: Blood volume pulse.

for ASP. Validity can be best obtained through four

approaches: content, criteria-related, construct, and

ecological validation, which we will discuss in rela-

tion to ASP; in addition, see Box 1.

Content validity refers to a) The agreement of ex-

perts on the domain of interest; e.g., limited to a spe-

ciﬁc application or group of patients; b) The degree

to which a feature (or its parameters) of a given signal

represents a construct; and c) The degree to which a

set of features (or their parameters) of a given set of

signals adequately represents all facets of the domain.

For instance, employing only skin conductance level

(SCL) for ASP will lead to a weak content validity

when trying to measure emotion, as SCL is known to

relate to the arousal component of an emotion, but not

to the valence component. However, when trying to

measure only emotional arousal, measuring only SCL

may form strong content validity; see also Box 1.

Criteria-related validity handles the quality of the

translation from the preferred measurement to an al-

ternative, rather than to what extend the measure-

ment represents a construct. Emotions are prefer-

ably measured at the moment they occur; however,

measurements before (predictive) or after (postdic-

tive) the particular event are sometimes more feasible;

e.g., through subjectivequestionnaires. The quality of

these translations are referred to as predictive or post-

dictive validity. A third form of criteria-related valid-

ity is concurrent validity: a metric for the reliability

of measurements applied in relation to the preferred

standard. For instance, the more affective states are

discriminated the higher the concurrent validity.

A construct validation process aims to develop a

nomological network (i.e., a ground truth), or possi-

bly an ontology or semantic network, build around

the construct of interest. Such a network requires

theoretically grounded, observable, operational def-

initions of all constructs and the relations between

them. Such a network aims to provide a veriﬁable

theoretical framework. The lack of such a network

is one of the most pregnant problems ASP is coping

with; e.g., see Box 1. A frequently occurring mis-

take is that emotions are denoted, where moods (i.e.,

longer object-unrelated affective states with very dif-

ferent physiology) are meant. This is very relevant

for ASP, as it is known that moods are accompanied

by very different physiological patterns than emotions

are (Gendolla and Brinkman, 2005).

Ecological validity refers to the inﬂuence of the

context on measurements. We identify two issues:

1) Natural affective events are sparse, which makes it

hard to let participants cycle through a range of affec-

BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing

428

Table 2: A summary of 18 studies that have tried to infer affect from physiological signals.

Information source Signals Part Fea Sel / Red Classiﬁers Target Result

(Sinha and Parsons, 1996) M 27 18 LDA 2 emotions 86%

(Picard et al., 2001) C ,E ,R ,M 1 40 SFS,

Fisher

LDA 8 emotions 81%

(Scheirer et al., 2002) C ,E 24 5 Viterbi HMM 2 frustrations 64%

(Nasoz et al., 2003) C ,E ,S 31 3 kNN, LDA 6 emotions 69%

(Takahashi, 2003) C ,E ,B 12 18 SVM 6 emotions 42%

(Haag et al., 2004) C ,E ,S ,M ,R 1 13 MLP val / aro 90%

97%

(Kim et al., 2004) C ,E ,S 175 SVM 3 emotions 78%

(Lisetti and Nasoz, 2004) C ,E ,S 29 kNN, LDA,

MLP

6 emotions 84%

(Wagner et al., 2005) C ,E ,R ,M 1 32 SFS,

Fisher

kNN, LDA,

MLP

4 emotions 92%

(Yoo et al., 2005) C ,E 6 5 MLP 4 emotions 80%

(Choi and Woo, 2005) G 3 PCA MLP 4 emotions 90%

(Healey and Picard, 2005) C ,G ,R ,M 9 22 Fisher LDA 3 stress levels 97%

(Rani et al., 2006) C ,G ,S ,M ,P 15 46 kNN, SVM,

RT, BN

3 emotions 85%

(Zhai and Barreto, 2006) C ,G ,S ,P 32 11 SVM 2 stress levels 90%

(Leon et al., 2007) C ,E 8 5 DBI AANN 3 emotions 71%

(Liu et al., 2008) C ,E ,S ,M 6 35 SVM 3 affect states 83%

(Katsis et al., 2008) C ,E ,M ,R 10 15 SVM, ANFIS 4 affect states 79%

(Yannakakis and Hallam,

2008)

C ,E 72 20 ANOVA SVM, MLP 2 fun levels 70%

(Kim and Andr´e, in press) C ,E ,M ,R 3 110 SBS LDA 2 fun levels 70%

Notes. Part: the number of participants; Fea: the number of features; Sel / Red: Algorithms used for selection or reduction of

features; C : Cardiovascular activity; E : Electrodermal activity; R : Respiration; M : Electromyogram; B : Electroencephalo-

gram; S : Skin temperature; P : Pupil Diameter; MLP: MultiLayer Perceptron; HMM: Hidden Markov Model; RT: Regression

Tree; BN: Bayesian Network; AANN: Auto-Associative Neural Network; SVM: Support Vector Machine; LDA: Linear

Discriminant Analysis; kNN: k Nearest Neighbors; ANFIS: Adaptive Neuro-Fuzzy Inference System; DBI: Davies-Bouldin

Index; PCA: Principal Component Analysis; SFS: Sequential Forward Selection; SBS: Sequential Backward Selection.

tive states in a limited time frame and 2) The affective

signals that occur are easily contaminated by contex-

tual factors; so, using a similar context as the intended

ASP application for initial learning is of vital impor-

tance; see also Box 1. Although understandable from

a measurement-feasibility perspective, emotion mea-

surements are often done in controlled laboratory set-

tings. This makes results poorly generalizable to real-

world applications. Hence, the need for unobtrusive

sensors and methods to make longitudinal real-world

studies possible is pressing.

3.2 Triangulation

We propose to adopt the principle of Triangulation

on ASP, as applied in social sciences and human-

computer interaction. Heath (2001) deﬁnes triangula-

tion as “the strategy of using multiple operationaliza-

tions of constructs to help separate the construct un-

der consideration from other irrelevancies in the op-

erationalization”. Using this strategy provides several

advantages: 1) Distinct signals can be used to vali-

date each other; 2) Extrapolations can be made based

on multiple data sets, providing more certainty. In

turn, corrections can be made to errors in a result set

that clearly defy from other results; and 3) More solid

ground, or even a ground truth (see also Section 3.1),

is obtained for the interpretation of signals, as multi-

ple perspectives are used.

Triangulation was, for example, successfully em-

ployed by Bailenson et al. (2008) and Healey and

Picard (2005) . As (one of) the ﬁrst, Bailenson

et al. (2008) has shown that using both physio-

logical signals and facial expressions leads to bet-

ter ASP than using one of them; see also Section 1.

Hence, we advise to record three affective signals,

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP)

429

or have at least three features derived from them,

for each construct under investigation, in well con-

trolled research. Moreover, qualitative and subjective

measures should accompany the signals (e.g., ques-

tionnaires, video recordings, interviews, and Likert

scales); e.g., see (Hazlett and Hazlett, 1999; Healey

and Picard, 2005; Slater et al., 2006; Van den Broek,

2004; Westerink et al., 2008a). Please consult also

Section 3.1 on this topic.

3.3 A Physiology-driven Approach

A ﬁnal prerequisite stems from the idea that ASP can

never be entirely based on psychological changes. As

discussed in Section 2, there are many factors out-

side one’s affective state that contaminate affective

signals. Beside validation and triangulation, another

way to deal with this is to employ a more physiology-

driven perspective (Tractinsky, 2004). Instead of ex-

pressing the goals of ASP directly in terms of affec-

tive states, they can often be stated in terms of the af-

fective signals themselves (Slater et al., 2006). For in-

stance, instead of inferring an air-trafﬁccontroller’s or

driver’s stress level, thresholding SCL might be sufﬁ-

cient; see also Box 1.

Note that there always remains an interpretation

in affective states. Then, the use of syntactic or struc-

tural pattern recognition for ASP should be (further)

explored. Its hierarchical approach to simplifying

complex patterns in affective signals is expected to be

valuable for ASP.

3.4 Contributions of Signal Processing

The majority of research on ASP is conducted by psy-

chology, physiology, medicine, human-computer in-

teraction, or artiﬁcial intelligence. Hence, true sig-

nal processing expertise is often missing; see also

Box 1. In particular, expertise from biomedical sig-

nal processing could signiﬁcantly contribute to ASP’s

progress. We will now address a triplet of issues:

The development of ﬁlters tailored to the speci-

ﬁcations of ASP sensors and to ASP’s applications

could signiﬁcantly boost the performance of ASP.

Affective signals are mostly sampled in a discrete

fashion. The required AD conversion, however, can

distort the signal; i.e., aliasing. For all possible sig-

nals, with all possible ampliﬁers and sensors, it is rec-

ommended to determine the relation between sample

frequency and signal loss / distortion. So far, this has

not been done and guidelines are provided founded

on weak assumptions; see also Box 1. Then, for all

signals, also the Nyquist frequency could be deﬁned.

A benchmark should be founded with veriﬁed af-

fective signals. This would enable objective perfor-

mance measurements of signal processing and pattern

recognition techniques. The principle of triangulation

(see Section 3.2) could be applied using it and the

generic applicability of techniques could be tested.

Moreover, it could be used for concurrent validation

(see Section 3.1); e.g., through comparing different

signals or apparatus that can substitute each other.

4 CONCLUSIONS

This paper provided both an overview and a review

of ASP and explained the lack of success of ASP;

see Section 2 and Tables 1 and 2. Next, in Sec-

tion 3, four prerequisites are introduced from which

ASP is expected to beneﬁt signiﬁcantly: validation,

the principle of triangulation, and a physiological-

driven approach, and contributions of the signal pro-

cessing community. In addition, a real world case is

discussed in Box 1, which illustrated the use of the

proposed prerequisites.

With the guidelines provided and the future’s

progress ahead, we envision embedding of ASP in

various professional and consumer settings, as a key

factor of our every day life. A broad range of probes

have been developed over the years (Westerink et al.,

2008a), which illustrate the feasibility of embedding

ASP in various settings. Let us brieﬂy denote three

of them: 1) For more than a decade, ASP is already

applied to determine the impact of advertisements on

people (Hazlett and Hazlett, 1999); for a review on

emotion measurement in advertising, see (Poels and

Dewitte, 2006). 2) Almost half a century ago, with

the development of Eliza (Weizenbaum, 1966) both

the possible implications of AI (for medicine) and the

limitations of (classic) AI became apparent. Among

many others, Liu et al. (2008), Slater et al. (2006),

and Van den Broek (2004) denote how ASP can help

AI to mature and to be of real value in this ﬁeld. 3)

ASP has often been applied with pilots and in auto-

motive industry (Healey and Picard, 2005; Westerink

et al., 2008a), as is also denoted in Box 1. Healey and

Picard (2005) also provide a brief overview of litera-

ture on this topic.

With Ambient Intelligence evolving, (wireless)

sensor networks becoming more mature, and with

prerequisites that enable the exploitation of ASP’s

full potential, empathic machines should come within

reach. And would it not be an appealing idea to live

in an empathic surrounding that adapts to your mood

and emotions, which can calm you or help you to con-

centrate when required?

BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing

430

Box 1: Application of the prerequisites in practice

One of the ﬁrst large-scale real-world cases in which ASP is applied is the driving application of Healey and Picard (2005). They

applied ASP on ECG, EMG, EDA, and respiration to determine the stress of 24 participants, during at least 50 minutes real world

driving, completed by questionnaires and video recordings. Healey and Picard (2005) were able to distinguish between three stress

levels, using ﬁve minute time windows. In addition, through ASP, they developed a continuous stress metric. In this box, we apply

our ASP prerequisites for a critical analysis of this case; each prerequisite is denoted separately in relation to the case.

Validation. Section 3.1 denotes content, criteria-related, construct, and ecological validation. The continuous stress metric developed

by Healey and Picard predicts three situations: sitting still, driving on the highway, and driving in town. These situations are likely to

be confounded by movement. Movement, however, is not necessarily correlated with stress level. And even when movement reﬂects

a more excited state, it does not always come with a negative bias. As affective signals are also inﬂuenced by movement, this makes it

uncertain whether the actually measured construct is movement, stress, or a combination of both. This, together with a weak deﬁnition,

leads to a limited content validity.

For the continuous stress metric employed, Healey and Picard show a very high inter-observer reliability. Moreover, this stress

metric has a very high temporal resolution. Nonetheless, as the authors elaborate, this was in some cases not enough to deal with the

short latencies of some physiological signals. In addition, subjective stress reports could not be done during driving, so they were

conducted after the drive. As the drives were quite long, this might have lead to a decrease in postdictive validity. However, they were

congruent with averages of the continuous stress metric. Taken together, the criteria-related validity of the measurements is high.

Construct validation refers to a nomological framework built around the construct of interest. They nicely describe their stress

hypothesis in terms of the activity of the sympathetic and parasympathetic nervous systems. However, a description of the relation

between most of the physiological signals and these dimensions is only elaborated to alimited extent. Moreover, the HRVmeasurement

is questionable as it was not corrected for respiration (Grossman and Taylor, 2007). All in all, the construct validity is poor.

The ecological validity of Healey and Picard (2005) is excellent: the research results can be generalized to driving in general, most

likely to all driving situations. One could argue that the results are hard to generalize to other domains of application. This is indeed a

valid observation; however, the authors do (correctly) not claim to want so.

Triangulation. Healey and Picard (2005) use multiple signals for the construct under investigation, complemented by subjective

self-reports and observer scores. Healey and Picard correlated different physiological signals to see to what extent they describe the

same construct and used self-reports and observer scores to validate the physiological changes and, hence, created a higher validity;

see also Section 3.2. Hence, they successfully adopted the principle of triangulation.

Physiology-driven approach. Instead of continuously inferring the stress level of the driver, it might be sufﬁcient to express a

high stress level in terms of one or a few physiological signals. For instance, (Healey and Picard, 2005) show that EDA is strongly

correlated with their continuous stress metric. So, using EDA as a stress scale might be sufﬁcient for actual applications. By setting

some thresholds in the EDA, it can also be used for music selection or distraction management (e.g., cell phones), as the authors

propose. This reduces the amount of sensors and computing power needed, making it more feasible for practical application; see also

Section 3.3.

Signal processing contributions. Healey and Picard (2005) state that “Each signal was sampled at a rate appropriate for capturing

the information contained in the signal ...”. Such an ill speciﬁed statement is in line with other research where similar statements are

made or the subject is ignored completely. However, Healey and Picard (2005) continue with “ . . . constrained by the sampling rates

available ...”, which explains at least partly the reported sample frequencies; i.e., ECG: 496 Hz; EDA and respiration: 31 Hz; and

EMG: 15.5 Hz. These are not reported previously as standard frequencies, if reported at all.

Tailored ﬁlters as proposed in the current paper are absent. The only ﬁlter deﬁned is the 0.5 seconds averaging ﬁlter. Why this is

applied is not reported. Possibly, more ﬁlters were applied; however, their speciﬁcations are omitted. On the one hand, this makes it

hard to reproduce the research; on the other hand, it illustrates the lack of attention for ﬁltering.

On several occasions throughout the paper, comparable work on aircraft pilots was mentioned. It would be of interest to compare

the rich set of data Healey and Picard (2005) gathered with other data sets. Then, the robustness of the variety of signals, their features

and parameters could be fully accessed. To extend from a few single initiatives to a more general practice, a benchmark should be set

up; see also Section 3.4.

Conclusion. The work of Healey and Picard (2005) is already a showcase for triangulation. Their results illustrate that their case is

per excellence also suitable for a physiologically driven approach. An additional advantage of this direction is that it can be further

explored even without the preferred thorough theoretical framework. This would ease the way of their work to possible application

areas. Moreover, the applicability of the processing schemes proposed by Healey and Picard (2005) could be veriﬁed with future

benchmarks and the robustness of their approach could be strengthened through the usage of tailored ﬁlters and ASP techniques.

Hence, using the framework introduced in this paper, the case discussed could be substantially improved, even post-hoc, although its

foundation is already good and unique in its kind.

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP)

431

ACKNOWLEDGEMENTS

The authors would like to thank Marjolein van der

Zwaag and Stijn de Waele for their comments on an

earlier version of this paper. Furthermore, we would

like to thank the anonymous reviewers, who provided

us the opportunity to improve this paper.

REFERENCES

Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J.,

Jabon, M. E., Hutcherson, C. A., Nass, C., and John,

O. (2008). Real-time classiﬁcation of evoked emo-

tions using facial feature tracking and physiological

responses. International Journal of Human-Computer

Studies, 66(5):303–317.

Berntson, G. G., Bigger, J. T., Eckberg, D. L., Grossman,

P., Kaufmann, P. G., Malik, M., Nagaraja, H. N.,

Porges, S. W., Saul, J. P., Stone, P. H., and van der

Molen, M. W. (1997). Heart rate variability: Origins,

methods, and interpretive caveats. Psychophysiology,

34(6):623–648.

Boucsein, W. (1992). Electrodermal activity. New York,

NY, USA: Plenum Press.

Cacioppo, J. and Tassinary, L. (1990). Inferring psychologi-

cal signiﬁcance from physiological signals. American

Psychologist, 45(1):16–28.

Choi, A. and Woo, W. (2005). Physiological sensing and

feature extraction for emotion recognition by exploit-

ing acupuncture spots. Lecture Notes in Computer

Science (Affective Computing and Intelligent Interac-

tion), 3784:590–597.

Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis,

G., Kollias, S., Fellenz, W., and Taylor, J. G. (2001).

Emotion recognition in human–computer interaction.

IEEE Signal Processing Magazine, 18(1):32–80.

Duke, A. and Novicevic, M. M. (2008). Historical founda-

tions of social effectiveness? Dale Carnegie’s princi-

ples. Social Inﬂuences, 3(2):132–142.

Frederickson, B. L., Manusco, R. A., Branigan, C., and Tu-

gade, M. M. (2000). The undoing effect of positive

emotions. Motivation and Emotion, 24(4):237–257.

Gendolla, G. H. E. and Brinkman, K. (2005). The role

of mood states in self-regulation: Effects on action

preferences and resource mobilization. European Psy-

chologist, 10(3):187–198.

Grossman, P. and Taylor, E. W. (2007). Toward understand-

ing respiratory sinus arrhythmia: Relations to cardiac

vagal tone, evolution and biobehavioral functions. Bi-

ological Psychology, 74(2):263–285.

Haag, A., Goronzy, S., Schaich, P., and Williams, J. (2004).

Emotion recognition using bio-sensors: First steps to-

wards an automatic system. Lecture notes in computer

science (Affective Dialogue Systems), 3068:36–48.

Hazlett, R. L. and Hazlett, S. Y. (1999). Emotional response

to television commercials: Facial EMG vs. self-report.

Journal of Advertising Research, 39(2):7–23.

Healey, J. A. and Picard, R. W. (2005). Detecting stress dur-

ing real-world driving tasks using physiological sen-

sors. IEEE Transactions on Intelligent Transportation

Systems, 6(2):156–166.

Heath, L. (2001). Triangulation: Methodology, pages

15901–15906. Elsevier Science Ltd.: Oxford, UK,

1 edition. ISBN: 978-0-08-043076-8.

Katsis, C. D., Katertsidis, N., Ganiatsas, G., and Fotiadis,

D. I. (2008). Toward emotion recognition in car-racing

drivers: A biosignal processing approach. IEEE

Transactions on Systems, Man, and Cybernetics–Part

A: Systems and Humans, 38(3):502–512.

Kim, J. and Andr´e, E. (in press). Emotion recognition

based on physiological changes in music listening.

IEEE Transactions on Pattern Analysis Machine In-

telligence.

Kim, K. H., Bang, S. W., and Kim, S. R. (2004). Emo-

tion recognition system using short-term monitoring

of physiological signals. Medical & Biological Engi-

neering & Computing, 42(3):419–427.

Leon, E., Clarke, G., Callaghan, V., and Sepulveda, F.

(2007). A user-independent real-time emotion recog-

nition system for software agents in domestic environ-

ments. Engineering Applications of Artiﬁcial Intelli-

gence, 20(3):337–345.

Lisetti, C. L. and Nasoz, F. (2004). Using noninvasive wear-

able computers to recognize human emotions from

physiological signals. EURASIP Journal on Applied

Signal Processing, 2004(11):1672–1687.

Liu, C., Conn, K., Sarkar, N., and Stone, W. (2008).

Physiology-based affect recognition for computer-

assisted intervention of children with Autism Spec-

trum Disorder. International Journal of Human-

Computer Studies, 66(9):662–677.

Minsky, M. (2006). The Emotion Machine: Commonsense

Thinking, Artiﬁcial Intelligence, and the Future of the

Human Mind. New York, NY, USA: Simon & Schus-

ter.

Nasoz, F., Alvarez, K., Lisetti, C. L., and Finkelstein, N.

(2003). Emotion recognition from physiological sig-

nals for presence technologies. International Journal

of Cognition, Technology and Work, 6:4–14.

Picard, R. W. (1997). Affective Computing. Boston MA,

USA: MIT Press.

Picard, R. W., Vyzas, E., and Healey, J. (2001). Toward

machine emotional intelligence: Analysis of affec-

tive physiological state. IEEE Transactions on Pat-

tern Analysis and Machine Intelligence, 23(10):1175–

1191.

Poels, K. and Dewitte, S. (2006). How to capture the heart?

reviewing 20 years of emotion measurement in adver-

tising. Journal of Advertising Research, 46(1):18–37.

Rani, P., Liu, C., Sarkar, N., and Vanman, E. (2006). An em-

pirical study of machine learning techniques for affect

recognition in human-robot interaction. Pattern Anal-

ysis & Applications, 9(1):58–69.

BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing

432

Reaz, M. B. I., Hussain, M. S., and Mohd-Yasin, F. (2006).

Techniques of EMG signal analysis: detection, pro-

cessing, classiﬁcation and applications. Biological

Procedures Online, 8(1):11–35.

Samboa, H., Silva, F., Silva, H., and ao, R. F. (2008). PLUX

– Biosignals Aquisition and Processing. URL:

http:

//www.plux.info

[Last accessed on November 03,

2008].

Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W.

(2002). Frustrating the user on purpose: a step to-

ward building an affective computer. Interacting with

Computers, 14(2):93–118.

Schuler, J. L. H. and O’Brien, W. H. (1997). Cardiovascu-

lar recovery from stress and hypertension factors: A

meta-analytic view. Psychophysiology, 34:649–659.

Sinha, R. and Parsons, O. A. (1996). Multivariate response

patterning of fear. Cognition and Emotion, 10(2):173–

198.

Slater, M., Guger, C., Edlinger, G., Leeb, R., Pfurtscheller,

G., Antley, A., Garau, M., Brogni, A., and Friedman,

D. (2006). Analysis of physiological responses to a

social situation in an immersive virtual environment.

Presence: Teleoperators and Virtual Environments,

15(5):553–569.

Takahashi, K. (2003). Remarks on emotion recognition

from bio-potential signals. In Proceedings of the IEEE

International Conference on Systems, Man and Cy-

bernetics, volume 2, pages 1655–1659, Palmerston

North, New Zealand.

Tractinsky, N. (2004). Tools over solutions? comments on

interacting with computers special issue on affective

computing. Interacting with Computers, 16(4):751–

757.

Van den Broek, E. L. (2004). Emotional Prosody Measure-

ment (EPM): A voice-based evaluation method for

psychological therapy effectiveness. Studies in Health

Technology and Informatics (Medical and Care Com-

punetics), 103:118–125.

Wagner, J., Kim, J., and Andr´e, E. (2005). From physiologi-

cal signals to emotions: Implementing and comparing

selected methods for feature extraction and classiﬁca-

tion. In Proceedings of the IEEE International Con-

ference on Multimedia and Expo (ICME).

Weizenbaum, J. (1966). ELIZA – a computer program for

the study of natural language communication between

man and machine. Communications of the ACM,

9(1):36–45.

Westerink, J. H. D. M., Ouwerkerk, M., overbeek, T.,

Pasveer, W. F., and de Ruyter, B. (2008a). Probing

Experiences: From Academic Research to Commer-

cial Propositions, volume 8 of Philips Research Book

Series. Springer: Dordrecht, The Netherlands.

Westerink, J. H. D. M., van den Broek, E. L., Schut,

M. H., van Herk, J., and Tuinenbreijer, K. (2008b).

Computing emotion awareness through galvanic skin

response and facial electromyography, volume 8 of

Philips Research Book Series, chapter 14 (Part II:

Probing in order to Feed Back), pages 137–150.

Springer: Dordrecht, The Netherlands.

Yannakakis, G. N. and Hallam, J. (2008). Entertain-

ment modeling through physiology in physical play.

International Journal of Human-Computer Studies,

66(10):741–755.

Yoo, S. K., Lee, C. K., Park, J. Y., Kim, N. H., Lee, B. C.,

and Jeong, K. S. (2005). Neural network based emo-

tion estimation using heart rate variability and skin

resistance. Lecture Notes in Computer Science (Ad-

vances in Natural Computation), 3610:818–824.

Zeng, Z., Pantic, M., Roisman, G. I., and Huang, T. S.

(in press). A survey of affect recognition meth-

ods: Audio, visual, and spontaneous expressions.

IEEE Transactions on Pattern Analysis Machine In-

telligence.

Zhai, J. and Barreto, A. (2006). Stress detection in com-

puter users through noninvasive monitoring of physi-

ological signals. Biomedical Science Instrumentation,

42:495–500.

PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP)

433