Revealing Psychophysiology and Emotions through Thermal
Infrared Imaging
Arcangelo Merla
Institute of Infrared Imaging Lab., ITAB – Insitute for Advanced Biomedical Technologies,
Department of Neuroscience and Imaging, University of Chieti-Pescara, Via dei Vestini 33, Chieti, Italy
Keywords: Autonomic Nervous System, Computational Physiology, Emotions, Human-Machine Interaction,
Psychophysiology, Thermal Infrared Imaging.
Abstract: Thermal infrared imaging has been proposed as a tool for the non-invasive and contact-less evaluation of
vital signs, psychophysiological responses and states. Several applications have been so far developed in
many diversified fields, like social and developmental psychology, psychometrics, human-computer
interaction, continuous monitoring of vital signs, stress and, even, deception detection. Thermal infrared
imaging has been poorly exploited in the field of human-robot interaction. Therefore, the state of the art of
thermal infrared imaging in computational physiology and psychophysiology is discussed in order to
provide insights about its potentialities and limits for human-robot interaction and applications with
affective robots.
1 INTRODUCTION
Understanding the psychophysiological state of
other individuals plays an essential role for planning
or adopting congruent strategies for social
interaction. To endow artificial agents with the
capability of reading and interpreting human
psychophysiological and emotional states represents
a major issue in the field of human-machine
interaction. In addition, in order to favor the
ecological dimension of such interaction, it is
desirable to non-invasively assess human
psychophysiological and emotional states.
Monitoring psychophysiological and emotional
states is usually performed through the
measurements of several autonomic nervous system
(ANS) parameters, like skin conductance response,
hand palm temperature, heart beat and/or breath rate
modulations, peripheral vascular tone, facial
expression and electromyography activity. Classical
technology for monitoring ANS activity usually
requires contact sensors or devices, thus resulting
somehow invasive and potentially biasing the
estimation of the state, as the compliant participation
of the individual is required. Thermal infrared (IR)
imaging has been proposed as a potential solution
for recording thermal signatures of ANS activity
non-invasively (Merla, 2004). Thermal IR imaging,
in fact, allows the contact-less and non-invasive
recording of the cutaneous temperature through the
measurement of the spontaneous body thermal
irradiation; it has been proposed for monitoring
cutaneous thermal effects associated with emotional
response and neurovegetative activity thanks to the
integrated use of advanced thermal imaging
technology, bioheat transfer modeling and
computational physiology (Buddharaju, 2005;
Garbey, 2007; Merla, 2004, 2007a, 2007b; Murthy,
2006; Pavlidis, 2007; Shastri, 2009).
As the face is usually exposed to social
communication and interaction, thermal imaging for
psychophysiology is performed on the subject’s
face. Provided the proper choice of infrared imaging
systems, optics, and solutions for tracking the
regions of interest, it is possible to avoid any motor
or behavioral restriction on the subject (Dowdall,
2006; Zhou, 2009).
Automatic recording and processing of thermal
IR imaging data for psychophysiology is possible.
Therefore, it seems that this technology, in
combination or in addition with other existing
technologies, could potentially contribute to endow
artificial agents with the capability of getting
insights into the psychophysiological state of the
human interlocutor. To this goal, a description of the
368
Merla A..
Revealing Psychophysiology and Emotions through Thermal Infrared Imaging.
DOI: 10.5220/0004900803680377
In Proceedings of the International Conference on Physiological Computing Systems (OASIS-2014), pages 368-377
ISBN: 978-989-758-006-2
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
state of the art of thermal imaging in computational
physiology and psychophysiology is presented.
2 THERMAL INFRARED
IMAGING DATA AND
COMPUTATIONAL
PHYSIOLOGY
The Autonomic Nervous System has been the object
of intense study in psychophysiology. The
sympathetic division readies the body for a crisis
that may require sudden, intense physical activity
and provides a primal survival mechanism. The
parasympathetic prompts the body for social
relationships. When autonomic activation occurs, an
individual experiences changes of the cardiovascular
and respiratory activity, with variations in blood
pressure, heart rate, breathing rate, and depth of
respiration. Thermal signatures of a variety of
psychophysiological signals have been identified. In
particular, it has been demonstrated and validated
that through thermal IR imaging it is possible to
compute at a distance the cardiac pulse, the
breathing rate, the cutaneous blood perfusion rate,
and the electro-dermal response (Garbey, 2007;
Merla, 2007a, 2007b; Murthy, 2006; Pavlidis 2007;
Shastri 2009). This section summarizes methods and
results in the field of computational physiology
based on thermal IR imaging.
2.1 Breathing Rate
Breathing consists of inspiration and expiration
cycles. During the inspiration, environmental air
flows via the nostrils to the lungs. Conversely, in the
expiration, air that was heated through its contact
with the lungs flows via the nostrils to the
environment. This creates a periodic or quasi-
periodic thermal signal in the proximity of the
nostrils that oscillates between high (expiration) and
low (inspiration) values. In conventional respiratory
studies, a thermistor is attached near the nostrils to
capture this phenomenon and produce a
representative breath signal.
Thermal imaging can act as a virtual thermistor,
since it captures the same phenomenon, but at a
distance (Murthy, 2006). As a periodic signal, the
breath signal can be analysed through Fourier
transformation on sliding segments (windows) of the
normalized breath thermal signal.
The estimation of breathing rate through thermal
imaging is very accurate as proved by comparison
with respiratory signals taken from respiratory belt
at the thorax (Murthy, 2009) (Figure 1), up to
achieve correlation values between thermally and
mechanically (LifeShirt technology, see Lewis,
2011) recorded breath rate signals as high as 1 over
a sample of 25 subjects, in both shallow, normal,
and forced ventilation (Lewis, 2011).
2.2 Cardiac Pulse
Thermal IR imaging allows the computation of the
cardiac pulse through the spectral analysis of the
thermal signature of the superficial vessels’ blood
flow pulsation (Garbey, 2007). The method is based
on the hypothesis that the temperature modulation
due to pulsating blood flow produces the strongest
variation on a superficial vessel’s temperature
signal. Garbey and colleagues (2004) proposed a
model to simulate the heat diffusion process on the
skin initiated by the core tissue and a major
superficial blood vessel. They took into account
noise effects due to the environment and instability
in the blood flow. Their simulation demonstrated
that the skin temperature waveform is directly
analogous to the pulse waveform, but its exact shape
is smoothed, shifted, and noisy with respect to the
originating pulse waveform due to the diffusion
process. This indicates that the pulse can be
recovered from the skin temperature modulation
recorded with a highly sensitive thermal camera and
processed through an appropriate signal analysis
method, as the overall thermal signal that is sensed
by the infrared camera is a composite signal, with
the pulse being one of its components.
In subsequent works, Sun (2006) and Garbey
(2007) proposed a method that, based on the
outcome of repeated Fourier analysis and proper
filtering of the raw signal, computes the cardiac
pulse through an estimation function. In real
environment settings, the performance of the
proposed method, with respect to standard fingertip
laser transducer, ranged from 88.52% to 90.33%,
depending on the clarity of the vessel’s thermal
imprint, over a sample of 34 subjects. Vessels from
jugular, wrist and fronto-temporal regions were used
for pulse assessment through thermal infrared
imaging (Figure 2).
RevealingPsychophysiologyandEmotionsthroughThermalInfraredImaging
369
Figure 1: Thermal imaging data. Left: Thermal image showing the thermal track of the airflow. Right: Raw temperature vs.
time profile for a region of interest close to the nose tip (upper panel); Signal from thorax respiratory belt (bottom panel).
Figure 2: Pulse computation from thermal imaging data. Upper panel: Collection point on the carotid artero-venous
complex, the fronto-temporal region and the wrist of the subject. Bottom panel: Temperature profile after removing
frequency signals lower than 0.67 Hz (40 bmp) and higher than1.67 Hz (100 bmp). (Adapted from Garbey, 2007).
2.3 Cutaneous Blood Perfusion Rate
Bio-heat transfer models permit the calculation of
the cutaneous perfusion from high-resolution IR
image series (Pavlidis, 2002; Merla, 2008) (Figure
3). Cutaneous perfusion is a strong indicator of
psychophysiological states, being it related to
cutaneous vasoconstriction and vasodilation.
Two major advantages for computing cutaneous
perfusion from thermal imagery are the achievable
frame rate and spatial resolution (up to 100 complete
524x524 pixel images per second using the most
advanced commercially available thermal cameras),
thus overcoming two of the main limitations of the
laser Doppler technique, that is the classical
technology for assessing cutaneous perfusion. The
models adopted derive from previous works by
Fujimasa (1995) and provide a proper estimation for
cutaneous perfusion rate in healthy individuals
(Merla, 2008). Pavlidis (2002) even suggested to use
cutaneous perfusion rate changes in the periorbital
region as a performing channel for a new generation
of deception detection systems, based on the flight-
fight response of the inquired subject to sensitive
questions (see section 3.4).
2.4 Electro-dermal Activity and
Sudomotor Response
Determination of sympathetic activation through
vital sign monitoring is not always straightforward.
As an alternative, sympathetic manifestations
through cholinergic postganglionic fibres could be
recorded. These fibres innervate sweat glands of the
skin and the blood vessels to skeletal muscles and
the brain and provide a pathway to selectively
enhancing blood flow to muscles and stimulating
sweat gland secretion.
In this context, Electro-Dermal Activity (EDA)
has been the gold standard for peripheral monitoring
of sympathetic responses. EDA is measured through
the Galvanic Skin Response (GSR) or the Skin
PhyCS2014-InternationalConferenceonPhysiologicalComputingSystems
370
Figure 3: Cutaneous Perfusion Rate computed from thermal IR data. On the left: average rate during the vision of neutral-
content movie; on the right: average rate while watching erotic clip (see section 3.3; adapted from Merla, 2007b).
Figure 4: Emotional sweating and sudomotor response. The delivery of emotional pressure (see section 3.2) or stress
stimulation (on the right) changes the rest (on the left) temperature distribution. The spotted dark signature is associated
with the activity of the sweating glands. (Adapted from Merla, 2007a).
Conductance Response (SCR), which is method for
quantifying sweat gland activation in the palm
through measurement of the change of the cutaneous
electrical conductivity.
Recent researches have demonstrated that facial
perspiration activity (i.e., sudomotor response)
associated to EDA can be appreciated, recorded and
quantified by means of thermal IR imaging (Merla,
2004, 2007a, 2007b; Shastri, 2009). Concomitantly
to the palm area, strong sweat gland activation is
manifested in the maxillary, perioral, and nose tip
regions (Figure 4).
The temperature changes reveal tonic (baseline
and /or general) and phasic (event-related)
components strongly correlated with GSR
sympathetic constituents (Merla, 2007b; Shastri,
2009).
3 THERMAL INFRARED
IMAGING IN
PSYCHOPHYSIOLOGY
The possibility of recording and monitoring
psychophysiological signals in non-invasive and
touch-less manner opens the way to the application
of thermal IR imaging in psychophysiology.
Together with the characterization of the thermal
signal in facial regions of autonomic valence (nose
or nose tip, perioral or maxillary areas, periorbital
and supraorbital areas associated with the activity of
the periocular and corrugator muscle, and forehead),
to monitor the modulation of the autonomic activity,
thermal IR imaging has been indicated as a potential
tool to build up, given the use of proper
RevealingPsychophysiologyandEmotionsthroughThermalInfraredImaging
371
classification algorithms, an atlas of the thermal
expression of emotional states (Nhan, 2010).
In this section, an overview of the applications of
thermal IR imaging in psychophysiology is
proposed.
3.1 Startle
Startle is an uncontrolled reflex that occurs when
individuals are engaged with a cognitive task and an
unexpected external stimulus or event requires
immediate shift of attention, generally followed by
autonomic and behavior responses such as increased
heart beat rate and sudomotor activity. Startle is part
of the flight or fight response and can be easily
evoked by using loud and unexpected sounds.
Pavlidis (2001) reported that, during startles,
sudomotor response occurs as perspiration pores on
the perioral, maxillary and nose area became active
decreasing the cutaneous temperature. Temperature
increases were observed on the periorbital and neck
areas (over the carotid) in contrast to cooling of the
cheeks. Researchers explained their observations on
the basis of the activation of the adrenergic system,
further suggesting the redirection of blood from the
cheeks to the periorbital region. Gane (2011)
reported a similar temperature drop for the maxillary
region while no temperature changes in the
periorbital regions could be appreciated.
Shastri (2009) induced startle response by using
natural sounds (i.e., glass breaking and phone rings)
on subjects engaged in a counting task. The results
confirmed the onset of sudomotor response on the
maxillary area (Figure 4). In addition, the detection
power of the sudomotor response by thermal
imaging was found to be similar to that of standard
GSR recording.
Coli (2007), within a classic repeated arousal
experiment, proved that the thermal signal from the
maxillary region and the GSR measurements reveal
a high level of affinity in terms of both tonic and
phasic components.
3.2 Distress and Fear
Thermal IR imaging has been proposed as a non-
intrusive method for assessing distress and mental
workload. In a study by Puri (2005) and in a
following one by Zhu (2008) signs of distress and
frustration in the human-computer interaction were
assessed during a stroop task. Based on the frontal
forehead temperature, the authors reported an
increased blood volume to supraorbital vessels with
respect to the rest condition.
Mental workload has been assessed in
professional drivers. Participants were exposed to
simulator driving tasks while cognitively challenged
with a mental loading task. Compared to baseline,
significant differences in nose tip temperature were
observed on the nose temperature along the
simulation procedure in agreement with the required
mental load (Calvin, 2007). As for the occupational
distress, in a seminal study, levels of stress in expert
and novice surgeons were measured during training
on three different drilling tasks designed for
laparoscopic surgery. The authors, by monitoring the
perioral and nose regions of the participants,
observed higher levels of distress in novice
compared to expert surgeons. Distress signs were
assessed by lower temperatures on the peri-nasal
region along with the activation of perspiration pores
(Pavlidis, 2012).
Thermal IR Imaging has also been used to assess
training times by studying learning proficiency
patterns on an alphabet-arithmetic task. During the
first trials nose temperatures were lower with respect
to the baseline. With repeated experience and
training, the nose temperatures rose as individuals
became more accurate and quicker in their responses
(Kang, 2006).
Early evidence of peripheral thermal patterns
associated with fear date back to 1998. Kistler and
colleagues induced fear in participants by showing
to them scenes from thriller movies. They found
dramatic decreases of fingertips temperature during
the most scaring scenes of the movies.
Merla (2007a) studied facial thermal signals in
fear-conditioned individuals (Figure 4). Unexpected
sub-painful mild electric stimuli were delivered to
the subject’s median nerve. Results showed a
reduction of temperature and sweating on the
perioral region, forehead as well as the palm.
3.3 Sexual Arousal and Interpersonal
Contact
Sexual arousal has clear and marked
interrelationships with ANS activity.
Merla (2007a) studied the facial thermal
response, in terms of facial cutaneous perfusion
change, to the view of erotic clips in contrast with
the view of sport movies. During the presentation of
the erotic movies, the temperature and the cutaneous
perfusion of the forehead, periorbital regions, nose
and lips increased (Figure 3). Hahn (2012) examined
social contact and sexual arousal during
interpersonal physical contact. The physical contact
was performed on different parts of the body such as
PhyCS2014-InternationalConferenceonPhysiologicalComputingSystems
372
the face, chest (high-intimate), arm and palm (low
intimate) from both male and females experimenters.
It was observed that, when high-intimate regions
were touched, temperature increased. The
temperature augment was higher when the
experimenter was of the opposite sex of the subject.
The temperature increase was localized on the
mouth, nose, and the periorbital regions of the face.
3.4 Social Neuropsychology
Developmental and social neuropsychology is a
particular challenging field where thermal IR
imaging has been introduced with very encouraging
results.
Early infant attachment was studied using
thermal IR imaging in infants exposed to three
different experimental phases: i) separation from the
mother; ii) a short-lived replacement of the mother
by a stranger; and iii) infant in the presence of the
mother and the stranger. By observing negative
temperature changes on the infants’ forehead, the
researchers concluded that infants are aware of
strangers and that infants form a parental attachment
earlier than previously thought, specifically from 2-4
months after birth (Mizukami, 1990).
Mothers’ ability to empathically share
offspring’s emotional feelings is considered integral
to primary affective bonds and a healthy socio-
emotional development. Ebisch (2012) investigated,
in an ecological context, whether maternal empathy
is accompanied by a synchrony in autonomic
responses by assessing simultaneously the facial
thermal imprints of mother and child, while the
former observed the latter when involved in a
distressing situation (Figure 5). The results showed a
situation-specific parallelism between mothers’ and
children’s facial temperature variations, providing
evidence for a direct affective sharing involving
autonomic responding (Figure 6).
An extension of the above study including an
additional group of female participants showed that
mothers-child dyads in contrast to other-women-
child dyads have faster empathic reactions to the
child’s emotional state (Manini, 2013).
The above research paradigms used an
experiment inducing guilt, further explained in
Ioannou et al (2013). All of these studies, once
more, highlighted the peculiar role of the nasal
temperature as indicator of autonomic activity
related to social interaction in children. As for the
adults, fewer studies with thermal IR imaging are
available about social neuropsychology.
In the only study for embarrassment (Merla
(2007a), participants were exposed to the attention
of unknown people, while performing a stroop task.
The study was designed in order to elicit feeling of
embarrassment and mild stress when the participants
wrongly performed the task in the presence of
others. Temperature decreases associated with
emotional sweating were observed on the palm and
the face, especially around the mouth and over the
nose tip.
Given the capability of thermal IR imaging to
capture emotional states, a variety of studies have
examined the potentialities of this technique in the
context of deception detection. Pavlidis (2002)
accurately identified 11 out of 12 subjects as guilty
in a mock scenario experiment through cutaneous
blood flow rate increases on the forehead and in the
periorbital regions. Following the same experimental
approach, Tsiamyrtzis (2006) suggested that
temperature and cutaneous blood flow monitoring of
the periorbital vessel during interrogation provides
87.2% accuracy in detecting deceptive individuals.
Zhu (2008) by focusing on the forehead, and
particularly on the corrugator muscle supplied by
supraorbital vessels, achieved a percentage of 76.3%
accuracy for lie detection. Temperature increases
were accounted as results of flight or fight response
to the sensitive questions and increased blood
perfusion to facial muscles as a result of mental
stress.
Table 1 reports a list of studies applying thermal
IR imaging to psychophysiology.
Figure 5: Thermal IR imaging allows the simultaneous
recording of individuals sharing a social condition or task.
Evidence of the same sudomotor response is found in this
thermal picture of a mother looking at her child
experiencing a distressful situation.
RevealingPsychophysiologyandEmotionsthroughThermalInfraredImaging
373
Figure 6: Facial thermal imprints of a mother-child dyad and nose tip temperature synchronization during distressing
situation (Adapted from Ebisch, 2012).
Table 1: List of some of the studies applying thermal IR Imaging to psychophysiology
Authors Year Subjects Emotion/Response Experimental Paradigm Regions
Mizukami et al.,
1990 34 (pairs) Mother infant
separation-stress
Separation from
mother/stranger exposure
Forehead
Naemura et al.,
1993 52 Startle White Noise (45-100db) Nasal Region
Kistler, et al.,
1998 20 Fear Horror Movie Fingers
Pavlidis, et al.,
2001 6
Startle Loud noise (60dB)
Periorbital area,
Cheeks, Neck area.
Pavlides et al.
2002 12 Lie Detection Mock interrogation Face
Puri et al.,
2005 12 Stress Stroop Test Supraorbital Vessels
Tsiamyrtzis et al.
2006 39 Lie Detection Mock interrogation Periorbital vessels
Kang et al.,
2006 9 Learning process-Stress Alphabet arithmetic task Forehead, Nose
Calvin & Daffy
2007 33 Mental workload-Stress Driving – MLT Forehead, Nose
Merla & Romani
2007 10
Fear of Pain
Electric stimulation &
Trigger
Face, Palm
Nakanishi &
Matsumura
2007 12
Laughter Playing
Nose, Forehead,
cheek
Zhu et al.
2008 38 Lie Detection Mock interrogation Supraorbital vessels
Shastri, et al.,
2009 10
Startle
Natural startling sounds:
glass breaking, phone
ringing
Periorbital,
supraorbital,
maxillary
Gane, et al.,
2011 11 Startle Loud noise (102dB) Periorbital
Ebisch et al.,
2012 12 (dyads) Empathy Toy Mishap Face: Nose, Maxillary
Hahn et al.,
2012 16
Sexual Arousal
Touch on high intimate
regions
Nose, lip, periorbital
Manini et al.
2013 18 (dyads) Empathy Toy Mishap Face: Nose, Maxillary
Ioannou et al.,
2013 15 Guilt Toy Mishap Nose
PhyCS2014-InternationalConferenceonPhysiologicalComputingSystems
374
4 THERMAL IR IMAGING
AND HUMAN-MACHINE
INTERACTION
Thermal IR imaging is widely spreading in
psychophysiology as an adjunct tool for obtaining
information of psychophysiological relevance non-
invasively and ecologically, that is without
interfering with the spontaneous activity of the
person.
Computational physiology based on thermal IR
imaging is possible and reliable and, being this
technique based on digital imaging data, it could be
completely automatized and managed by an artificial
intelligence agent, without human user-assistance.
Even though most of the available literature relays
on measuring or characterizing just one
physiological parameter at once, at least from a
theoretical point of view, there are no problems with
combining together the physiological that can be
recorded all together thermal IR imaging to improve
the performance of classification of
psychophysiological states and emotions (Nhan,
2010).
Facial regions of interest in the thermal video can
be automatically detected and identified basically
adapting the algorithms for visible videos so far
developed for automatic feature extraction
(Dowdall, 2006). Software for automatic tracking of
regions of interest across the time series of the
recorded frames is also available (Dowdall 2006;
Zhou, 2009), thus setting the observed subjects free
from any motion restriction or requirement. The
methodology has been proven to be solid and
reliable in a series of studies dealing with moral
emotions in three years old children engaged in free
activity and games across the experimental room
while being recorded (Ebisch, 2012; Manini, 2013;
Ioannu, 2013). However, a relevant issue related
with automatic tracking is the accurate estimation of
the temperature of the facial regions of interest when
the subject’s face is turned away or rotate from the
orthogonal projection with respect to the camera’s
plane (i.e., out-of-the-plan position), as this may
cause underestimation of cutaneous temperature
(Dowdall, 2006; Ebisch, 2012).
Real time processing of thermal IR imaging
psychophysiological data has been demonstrated
(Buddharaju, 2005). Particularly relevant is the
demonstrated possibility of real-time estimation of
the psychophysiological state of the driver while
engaged in real car driving (Merla, 2011). Patent
claiming the possibility of automatic computation of
the residual efficacy of the man-machine interaction,
based on the real-time estimation of the
psychophysiological state of the human user through
thermal IR imaging, has been issued as well (Merla,
2013).
These results suggest the intriguing possibility of
integrating thermal IR imaging with other existing
technology in the field on human-machine
interaction to provide artificial agents with the
capability of understanding the psychophysiological
state of the human interlocutor. To the best of our
knowledge, no previous studies have analysed such
a possibility, while a very few of pilot applications
have been so far proposed (Buddharaju, 2005;
Merla, 2013).
There are several advantages that could derive
from the use of thermal IR imaging for human-
machine interaction. From the point of view of the
computational physiology, it has to be remarked that
there is the concrete possibility of monitoring, in a
realistic environment, at a distance and
unobtrusively, several physiological parameters and
vital signs like pulse rate, breathing rate, cutaneous
vasomotor control and indirect estimation of electro-
dermal activity. This opens the way for remote
monitoring of the physiological state of individuals
without requiring their collaboration and without
interfering with their usual activities, thus favouring
the use of assistive robots, for example, for elder
people or for monitoring the regular breathing
activity in neonates. Automatic agents devoted to the
control of environmental conditions, for example
within a car or an house, could take advantage from
a biofeedback control of the actuation through the
thermal-based monitoring of vitals signs of the
human user, in order to achieve and maintain
optimal or desired performances of the system user-
agent (i.e., adaptive environment).
Another relevant possibility is to capitalize on
thermal IR imaging to provide artificial agent with
the capability of adopting behavioural or
communicative strategies contingent with the actual
psychophysiological state of the human interface.
This possibility, even though still theoretical, could
be particularly effective for affective robots and
automatic agents designed for improving and
personalizing learning or treatment strategies on the
basis of the measured user’s psychophysiological
feedback.
A major issue that needs to be addressed for a
real use of thermal IR imaging in human-machine
interaction is how much the method could be
specific for identifying specific emotional states at
individual level. There are no specific studies
RevealingPsychophysiologyandEmotionsthroughThermalInfraredImaging
375
available at the moment to answer such an important
question, which remains matter of further research.
A global limitation derives from the fact that
cutaneous thermal activity is intimately linked to the
autonomic activity. The question therefore becomes:
“How much specific and descriptive of each emotion
are the autonomic responses?” No answer
universally accepted is available. Also no extensive
studies are available about the fascinating possibility
of merging together physiological information and
automatic recognition of facial expressions for
providing an atlas of the thermal signatures of
emotions.
5 CONCLUSIONS
Thermal IR imaging is a reliable method for
ubiquitous and automatized monitoring of
psychophysiological activity. It provides a powerful
and ecological tool for studies aimed at assessing
emotional arousal, responses, and affective states. Its
capability of capturing autonomic responses and
psychophysiological states opens the way to
innovative and ecological paradigms for studying
social relationships, emotional charge and
autonomic activity.
The results of the available studies suggest that
specific thermal signatures related to specific
emotional conditions exist, but further studies are
needed to assess the specificity and the sensitivity of
the method.
Affective robots or artificial intelligence systems
could be endowed with this methodology in order to
capitalize on the possibilities offered by thermal IR
imaging for reading, classifying, understanding and
interacting with individuals’ affective and
psychophysiological states, and emotions.
ACKNOWLEDGEMENTS
Figures 2, 3, 4, 6 have been adapted from previous
papers of the author respecting the copyright rights
for their publication in the present form.
REFERENCES
Buddharaju, P., Dowdall, J., Tsiamyrtzis, P., Shastri, D.,
Pavlidis, I., Frank, M.G., 2005. Automatic Thermal
Monitoring System (ATHEMOS) for Deception
Detection, Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition, 2: 1179.
Calvin, K.L., Duffy, V.G., 2007. Development of a facial
skin temperature-based methodology for non-intrusive
mental workload measurement, Occupational
Ergonomics, 7, 83-94.
Coli, M., Fontanella, L., Ippoliti, L., Merla A., 2007.
Multiresolution KLE of psycho-physiological signals,
In: Proceedings of S.Co.2007 Venice, Sept, 6-8,
PADOVA, Book of Short Papers, 116-121.
Dowdall, J., Pavlidis, I., Tsiamyrtzis, P., 2006, Coalitional
tracking in facial infrared imaging and beyond,
Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition 2006, Category numberP2597; Article
number 1640579.
Ebisch, S. J., Aureli, T., Bafunno, D., Cardone, D.,
Romani, G.L., Merla, A., 2012. Mother and child in
synchrony: thermal facial imprints of autonomic
contagion, Biological Psychology, 89, 123-129
Fujimasa, I., 1998. Pathophysiological expression and
analysis of far infrared thermal images, IEEE Eng Med
Biol Mag., 17(4), 34-42.
Gane, L., Power, S., Kushki, A., Chau, T., 2011. Thermal
Imaging of the Periorbital regions during the
Presentation of an Auditory Startle Stimulus. PLOS
One, 11: e27268. doi:10.1371/journal.pone.0027268.
Garbey, M., Sun, N., Merla, A., Pavlidis, I., 2007.
Contact-free measurement of cardiac pulse based on
the analysis of thermal imagery, IEEE Transactions on
Biomedical Engineering, 54(8), 1418-1426.
Hahn, A. C., Whitehead R. D., Albrecht, M., Lefevre C.
E., Perret D. I., 2012. Hot or not? Thermal reactions
to social contact, Biology letters, 1-4. doi:
10.1098/rsbl.2012.0338.
Ioannou, S., Ebisch, S. J., Aureli, T., Bafunno, D.,
Ioannides, H. A., Cardone, D., Manini, B., Romani, G.
L., Gallese, V., Merla, A., 2013. The autonomic
signature of guilt in children: A Thermal Infrared
Imaging Study. Plos One, 1-18 (In Press).
Kang, J., McGinley, J. A., McFadyen G., Babski-Reeves
K., 2006. Determining Learning level and effective
training times, Proceeding of the 25th Army Science
Conference, 27-30.
Kistler, A., Mariazouls, C., Berlepsch, V. K., 1998.
Fingertip temperature as an indicator for sympathetic
responses. International Journal of Psychophysiology,
29, 35-41.
Lewis, G.F., Gatto, R.G., Porges, S.W., 2011. A novel
method for extracting respiration rate and relative tidal
volume from infrared thermography,
Psychophysiology, 48(7):877-87.
Manini B., Cardone D., Ebisch S. J. H., Bafunno D.,
Aureli T., Merla A., 2013. Mom feels what her child
feels: thermal signatures of vicarious autonomic
response while watching children in a stressful
situation. Frontiers in Human Neuroscience, 7 (299),
1-10. doi: 10.3389/fnhum.2013.00299.
Merla, A., Di Donato, L., Rossini, P.M., Romani, G.L.,
2004. Emotion detection through Functional Infrared
PhyCS2014-InternationalConferenceonPhysiologicalComputingSystems
376
Imaging: preliminary results, Biomedizinische
Technick, 48, 284-286.
Merla, A., Romani, G.L., 2007a. Thermal Signatures of
Emotional Arousal: A Functional Infrared Imaging
Study, in Proceedings of the Annual International
Conference of the IEEE Engineering in Medicine and
Biology Society, p. 247-249.
Merla A., 2007b. Computational physiology in a thermal
image setting, Proceedings of S.Co.2007, Fifth
Conference on Complex Models and Computational
Intensive Methods for Estimation and Prediction,
Book of Short Papers, 338 - 343.
Merla, A., Di Donato, L., Romani, G. L., Proietti, M.,
Salsano F., 2008. Comparison of thermal infrared and
laser Doppler imaging in the assessment of cutaneous
tissue perfusion in healthy controls and scleroderma
patients. International Journal of Immunopathology
and Pharmacology, 3, 679-686.
Merla, A., Cardone, D., Di Carlo, L., Di Donato, L.,
Ragnoni, A., Visconti, A., Romani, G.L., 2011.
Noninvasive system for monitoring driver’s physical
state, Proceedings of the 11
th
AITA Advanced Infrared
Technology and Applications, Abstract book, 21.
Merla, A., 2013, Method and system for the control of the
residual efficiency of the interaction man-vehicle,
European Patent EP13425145.
Mizukami, K., Kobayashi, N., Ishii, T., & Iwata, H., 1990.
First selective attachment begins in early infancy: A
study using telethermography, Infant Behavior and
Development, 13, 257–271.
Murthy R., Pavlidis I., 2006. Noncontact measurement of
breathing function, IEEE Engineering in Medicine and
Biology Magazine, 25(3), 57-67.
Murthy, J. N., van Jaarsveld, J., Fei, J., Pavlidis, I.,
Harrykissoon, R., Lucke, J. F., et al., 2009. Thermal
infrared imaging: A novel method to monitor airflow
during polysomnography, Sleep, 32, 1521–1527.
Nhan, B. R., and Chau, T., (2010). Classifying Affective
States Using Thermal Infrared Imaging of the Human
Face. IEEE Transaction on Biomedical Engineeiring,
57 (4), 979-987.
Pavlidis, I., Levine, J., Baukol, P., 2001. Thermal imaging
for anxiety detection. IEEE Engineering in Medicine
and Biology Magazine, 2, 315-318.
Pavlidis, I., Levine, J., 2002. Thermal image analysis for
polygraph testing. Engineering in Medicine and
Biology Magazine, 21(6), 56-64.
Pavlidis, I., Eberhardt, N.L., Levine, J.A., 2002. Seeing
through the face of deception, Nature, 3,
415(6867):35. Erratum in: Nature, 7;415(6872):602.
Pavlidis, I., Dowdall, J., Sun, N., Puri, C., Fei, J., Garbey,
M., 2007, Interacting with human physiology,
Computer Vision and Image Understanding, 108(1-2),
150-70.
Pavlidis, I., Tsiamyrtzis, P. ,Shastri, D., Wesley, A., Zhou,
Y., Lindner, P., Buddharaju, P., Joseph, R.,
Mandapati, A., Dunkin, B., Bas, B., 2012, Fast by
Nature - How Stress Patterns Define Human
Experience and Performance in Dexterous Tasks,
Scientific Reports, 2, 1-9.
Puri, C., Olson, L., Pavlidis, I., Levine, J., and Starren, J.,
2005. Stress-cam: Non-contact measurement of users’
emotional states through thermal imaging. In
Proceedings of the 2005 ACM Conference on Human
Factors in Computing Systems, 1725–1728.
Shastri, D., Merla, A., Tsiamyrtzis, P., and Pavlidis, I.,
2009. Imaging facial signs of neurophysiological
responses, IEEE Transactions on Biomedical
Engineering, 56(2), 477-84.
Sun, N., Garbey, M., Merla, A., Pavlidis, I., 2007. Imaging
the Cardiovascular pulse, IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition, 2, 416-421.
Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I.,
Frank, M., and Ekman, P., 2006. Imaging facial
physiology for the detection of deceit, International
Journal of Computer Vision, 71(2), 197–214.
Zhu, Z., Tsiamyrtzis, P., Pavlidis, I., 2008. Forehead
Thermal Signature Extraction in Lie Detection, IEEE
Engineering in Medicine and Biological Society, 243 –
246.
Zhou, Y., Tsiamyrtzis, P., Pavlidis, I., 2009. Tissue
tracking in thermo-physiological imagery through
spatio-temporal smoothing, Med. Image Comput.
Comput. Assist. Interv., 12(Pt 2):1092-9.
Buddharaju, P., Dowdall, J., Tsiamyrtzis, P., Shastri, D.,
Pavlidis, I., Frank, M.G., 2005. Automatic Thermal
Monitoring System (ATHEMOS) for Deception
Detection, Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition, 2: 1179.
RevealingPsychophysiologyandEmotionsthroughThermalInfraredImaging
377
