![](bg2.png)
Figure 1: McGurk effect. Visual stimuli (a) are combined
with audio stimuli (b). Subjects will expect congruent
stimuli (b
1
), where visual and auditory clues are
concordant (e.g. happy face and happy voice). Non-
congruent stimuli (b
2
), where visual and auditory clues are
discordant (e.g. happy face and angry voice), will induce
distortions in either the visual or auditory perception (this
distortion is termed as a “McGurk effect”.
the brain by multi-modal stimuli, and to control if
such stimuli could induce reproducible changes in
EEG signal. Through the investigation of this
‘abnormal’ perceptual condition, we intend to reveal
the mechanisms of normal emotional judgment (how
one can distinguish the valence of emotions in a
given stimulus). The use of different valence stimuli
(neutral, aggressive, appeasing, etc.) will be
compared.
3 METHOD
We exposed two right handed, male subjects to
auditory, visual, or combined audio-visual stimuli.
Stimuli were presented for 2 sec, the subjects was
asked to answer afterwards within a 3 sec window,
and then had 5 sec of rest (one trial = 10 sec). Audio
stimuli consisted of voice recordings of the word
‘arigato’ (thank you) pronounced with three
different intonations (Angry - A, Happy - H or
Neutral - N). Visual stimuli consisted of faces of
women expressing the same emotional valences (A,
H or N), taken from the JACfee and JACNeuf
Japanese-Caucasian photo databases (Biehl et al.
1997). Audio-visual stimuli were composed using
either congruent combinations of faces and voices
(e.g. HxH) or non-congruent (e.g. AxH). The
experiment consisted in three different sessions:
• In the first session, the subjects were
exposed to visual stimuli only. Their task
was to judge if the face was neutral, angry,
or happy. 60 stimuli were presented in a
pre-decided random order, and so that two
consecutive emotions were always
different.
• In the second session, the subjects were
exposed to visual stimuli only. Their task
was to judge if the voice was neutral,
angry, or happy. 60 stimuli were presented
in a pre-decided random order, and so that
two consecutive emotions were always
different.
• In the third session, the subjects were
exposed to the combined audio-visual
stimuli. Their task was to judge if the
percept was neutral, angry, or happy. 60
stimuli were presented in a pre-decided
random order, and so that two consecutive
emotions were always different, and so that
the same number of trials occurred for all
possible pairs of stimuli.
In each of the three sessions, the task of the subjects
was to judge if the face was neutral, angry, or happy,
and to provide this response with a keyboard.
The data was collected with a 64-channel
Biosemi EEG system with active electrodes in a
shielded room. Sampling rate was fixed at 1024 Hz,
notch filter at 50 Hz and analog bandpass filter
between 0.5 and 100 Hz. The topographic
distributions of EEG signals (relative power) in the
theta (4-8 Hz), apha (8-12 Hz) and beta (12-25 Hz)
ranges was afterwards computed using the Welch
periodogram method (Welch, 1967) on the trials of
the third session.
4 RESULT
We compared the conditions stimuli (A or H) vs.
control (N), and congruent vs. non-congruent.
Topographic maps of EEG power differed between
those conditions on both subjects. Generally, the
difference is maximized for HxH vs. HxA (Figure
2), in other words, the non-congruent stimuli are
“more different” than the neutral stimuli. Significant
changes are observed in the alpha range in the
frontal and right temporal areas; and in the left
parietal area in the theta range (Figure 2, Figure 3).
These changes are specific to the McGurk effect
NCTA 2011 - International Conference on Neural Computation Theory and Applications
462