![](bg5.png)
Concerning head position in space, vertical ro-
tation is described through a pitched sound repe-
atedly generated, with an increasing reproduction
speed (and, consequently, an increasing pitch) as the
face turns towards the zenith, in order to suggest the
sense of elevation. Horizontal rotation is represented
through an impulsive sound which is activated in the
left channel when the face rotates to the left, and in
the right channel vice versa, with a playback rate di-
rectly proportional to the rotation amount. Finally, la-
teral inclination is represented by a sine wave, whose
source is located in the stereo image depending on
the direction detected. All the mapping techniques
concerning head position and orientation are activated
only when the head is not in a frontal “idle” state (i.e.
only when neither tilt nor rotation are null). These
observations, along with the adoption of modulated
sound events for sonification, push the mentioned data
bindings towards the asynchronous area, in some ca-
ses more abstract and “synaesthetic” due to the adop-
tion of panning.
The overall sound result of the sonification beha-
ves much like a model-based one, sometimes resem-
bling a soundscape, sometimes recalling a sonic plot;
nevertheless, it is quite responsive to changes and cha-
racterized by a complex sound texture, which sugge-
sts to place it within the sonification space as shown
in Fig.4.
Finally, it is worth underlining that the present
work focuses on the conceptual framework, without
taking into account the pleasantness of the sonifica-
tion (a non-trivial issue discussed, e.g., in (Hermann
et al., 2015) and (Susini et al., 2012)), nor the is-
sues of information overload and audio channel satu-
ration, particularly relevant for visually impaired pe-
ople. Please note that we are not presenting a mu-
sical instrument nor a gesture-controlled synthesizer,
especially since some facial expressions derive from
autonomic reflexes.
The ultimate goal of the global project will be to
let the final user customize the experience by selecting
his/her own samples as the source material to be mo-
dulated by the sonification engine.
5 MeetMeTonight 2016
An early implementation of the prototype was pre-
sented at the stand of the LIM (Laboratory of Music
Informatics, University of Milan) in occasion of the
4
th
edition of the European Researchers’ Night,
4
spe-
cifically during the Milanese initiative called “Meet-
4
http://ec.europa.eu/research/researchersnight/
MeTonight 2016” held at the Giardini Indro Monta-
nelli of Milan on September 30 - October 1, 2016.
5
This free-entrance public event aimed to bring uni-
versity research outside traditional academic environ-
ments, addressing a wide audience ranging from pri-
mary school children to elder people. It was the oc-
casion to achieve an early validation of our approach
thanks to the presence of thousands of heterogeneous
visitors.
The research theme chosen by the LIM for the
2016 edition was sonification, namely the possibili-
ties offered by sound as an alternative communication
channel to deliver non-audio information. A num-
ber of experiences were proposed to a general au-
dience, including some historical examples of soni-
fication (e.g., meteorological data, global warming,
pollution levels in Milan, etc.) and a step sequencer
driven by the detection of visitors’ position inside the
stand.
In this context, an area was dedicated to our face
sonification experience. This space was equipped
with a front webcam to catch the user’s face, two
loudspeakers for audio output, a background screen
(not visible to the user) to let the audience watch the
ongoing computation, and a computer running the re-
quired software hidden “behind the scenes”.
During this experience, the sonification prototype
described above was tested by a large number of
people, very heterogeneous in age, education level,
technological knowledge, culture and nationality, etc.
(see Figure 5). Moreover, the duration of the event,
spanning over 2 days, 10 hours per day, allowed to
test very different lightning conditions.
Figure 5: People around the installation at MeetMeTonight
2016.
Concerning the video-capture module, the appli-
cation proved to effectively recognize the presence of
a face, without false positives, but it showed three ca-
ses of wrong tracking: the eyebrows of users wea-
5
http://www.meetmetonight.it/
A Proposal for the Interactive Sonification of the Human Face
167