An Ensemble of Tangible User Interfaces to Foster Music Awareness and

Interaction in Vulnerable Learners

Adriano Barat

, Luca A. Ludovico

and Eleonora Oriolo

Laboratory of Music Informatics (LIM), Department of Computer Science, University of Milan,

via G. Celoria 18, Milan, Italy

Keywords:

Music, Tangible User Interfaces, Computer-supported Education, Educational Vulnerability, Social

Disadvantage, Accessibility, Educational Poverty.

Abstract:

This paper describes an educational and creative experience based on tangible user interfaces for making music

together. The learners involved in the initiative, aged 7 to 50, presented various forms of social disadvantage

and, in some cases, also physical and cognitive impairment. The methodology consisted in building a number

of user-tailored experiences to let participants acquire basic musical skills by a hands-on approach imple-

mented through an ensemble of digital instruments. Aspects such as peer-to-peer collaboration, usability, and

accessibility had to be addressed. The achieved results included not only the improvement of music compe-

tences in vulnerable users, but also the acquisition of social and soft skills. After analyzing the state of the art,

this paper aims to investigate all the aspects of the initiative, starting from the design phase (devices, organiza-

tional aspects, learning subjects, etc.), then describing the experimental setting, and, ﬁnally, documenting the

achieved results.

1 INTRODUCTION

In the ﬁeld of sound and music computing, a relevant

research question is how technology can bridge the

gap between musical creativity and expressiveness on

one side, and physical and cognitive user impairment

on the other. Not only disability, but also a condition

of social disadvantage can hamper the development

of musical skills (Harrison, 2013; McAnally, 2013;

Bates, 2018). For example, in a context of educational

poverty due to economic conditions or geographical

isolation, even the availability of a musical instrument

can be a critical issue.

Digital technologies can help in a number of het-

erogeneous ways: just to mention a few examples,

a computer-based system can substitute and/or aug-

ment a standard musical instrument (Gabrielli et al.,

2011; d’Alessandro et al., 2015; Ko and Oehlberg,

2020), can pave the way for unleashing creativity

(Mellor, 2008; Riley et al., 2009; Morreale et al.,

2014), can provide alternative interfaces suitable to

overcome impairments (de Oliveira et al., 2015; Gor-

bunova and Voronov, 2018; Frid, 2019), can encour-

age the development of music-related skills (Avanzini

https://orcid.org/0000-0001-8435-8373

https://orcid.org/0000-0002-8251-2231

et al., 2019b; Barat

e and Ludovico, 2020; Pesek et al.,

2020). According to Li et al. (2019), music teaching

through information technology can also affect be-

havior relating to learners’ online learning attitudes,

music learning motivation, and learning engagement.

The ubiquity of portable devices (notebooks, tablets,

smart phones, etc.), equipped with ad-hoc hardware

accessories, suitable software tools, and easy-to-use

interfaces, can represent a solution even in vulnera-

bility contexts.

As stated by Barton and Riddle (2021), music is

learned and taught in multiple ways depending on the

socio-cultural contexts in which learning occurs, and

musical activities should be culturally responsive and

meaningful so as to respond to diverse learning con-

texts. In the scenario described in this paper, critical

aspects such as user-friendliness, usability, accessibil-

ity, affordability, and a suitable use of multimodality

must be considered.

The experience described in this work makes use

of multiple tangible user interfaces, acting as the mu-

sical instruments, connected via Bluetooth to a mobile

device, acting as both the synthesizer and the conﬁgu-

ration center. The goal is to foster musical expressive-

ness and interaction between young users living in a

condition of social disadvantage, due to educational

Baratè, A., Ludovico, L. and Oriolo, E.

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners.

DOI: 10.5220/0010653400003060

In Proceedings of the 5th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2021), pages 48-57

ISBN: 978-989-758-538-8; ISSN: 2184-3244

poverty or cognitive/physical impairments.

The key research questions we aim to answer are:

RQ1. How can a tangible user interface encourage

the acquisition of basic musical skills in so-

cially distressed subjects?

RQ2. How can an ensemble of tangible user inter-

faces improve socialization and cooperation

among peers?

RQ3. How can this experiment be extended and eas-

ily exported to other contexts?

The remainder of the paper tries to answer such

questions, and is organized as follows: Section 2 will

describe some previous experiences employing mu-

sic therapy to overcome physical and cognitive im-

pairments; Section 3 will introduce tangible user in-

terfaces, focusing, in particular, on their applicability

to music expressiveness; Section 4 will frame the ac-

tivities in the context of the “Note Digitali” project;

Section 5 will report some design choices and de-

scribe the experimental setting; Section 6 will focus

on achieved results, assessed through the analysis of

pre- and post-activity surveys and teacher observa-

tions; ﬁnally, Section 7 will draw the conclusions.

2 BACKGROUND

The current proposal is rooted in some previous

music-therapy experiences conducted by the same

working group with the help of digital technologies

and documented by Barat

e et al. (2018, 2019). Also

in that case, the idea was to employ a computer-based

interface in order to overcome the physical and cog-

nitive impairments that often hamper musical activi-

ties in users with disabilities. Such a goal only par-

tially overlaps the one of the “Note Digitali” project;

in fact, in the scenario described below, not only im-

pairments, but also conditions of social disadvantage

will be considered.

Other key differences with respect to the afore-

mentioned experiences must be remarked. First, from

a technical point of view, human-computer interac-

tion occurred through the Leap Motion controller, an

optical hand tracking module able to detect and cap-

ture the movements of user’s hands with great ac-

curacy (Weichert et al., 2013). The applicability of

such a device to music recalls the concept of “air”

musical instruments, i.e. virtual instruments employ-

ing depth cameras or other sensor systems to imple-

ment an interaction paradigm based on performing

gestures in the air, without touching a physical inter-

face (Godøy et al., 2005). Examples have been docu-

mented by Tarabella (2004), Fan and Essl (2013) and

Dahl (2015). Conversely, the “Note Digitali” project

will employ a speciﬁc tangible user interface called

Kibo and described in Section 3. Needless to say,

this kind of interfaces completely changes the human-

computer interaction paradigm.

Moreover, in previous experiences the musical

performance mainly involved only the impaired user –

interacting through the Leap Motion controller – and

the therapist – playing a traditional instrument. The

social aspects typical of making music together were

limited to the relationship between a single learner

and the instructor. Conversely, the proposal detailed

below will focus on peer-to-peer interaction between

multiple young performers, and the main roles played

by the instructor will be to explain, help, and propose

new musical experiences.

The scientiﬁc literature reports many other expe-

riences of ensemble music based on digital instru-

ments. For example, the aspects of human interaction

and communication in a digital music ensemble have

been addressed by Hattwick (2011) and Hattwick and

Umezaki (2012); Ben-Tal and Salazar (2014) propose

a new model for collaborative learning based on the

connections between the technological tools and the

social frameworks in emerging digital music collec-

tives; Cheng (2019) investigates the development of

musical competency in a laptop ensemble.

With respect to other similar initiatives, our pro-

posal presents novel features regarding the expres-

siveness of the selected digital device, the availability

of a ﬁne–tuned learning environment, and the atten-

tion paid to affective and emotional aspects. These

characteristics will be better clariﬁed in next sections.

3 TANGIBLE USER INTERFACES

FOR MUSIC

In the digital domain, tangible user interfaces (TUIs)

aim to overcome some limitations posed by classic

computer interaction, offering, for example, intuitive

ways to build complex structures, manipulate param-

eters, and connect objects. TUIs use physical forms

that ﬁt seamlessly into a user’s physical environment,

giving physical form to digital information and tak-

ing advantage of haptic-interaction skills (Ishii and

Ullmer, 1997). TUIs make digital information di-

rectly manipulatable with our hands and perceptible

through our peripheral senses through its physical em-

bodiment (Ishii, 2008). Such an approach is partic-

ularly effective for young (Xu, 2005) and disadvan-

taged users (Farr et al., 2010; Aljaam et al., 2011;

Carre

no-Le

on et al., 2020).

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners

Another key concept to clarify in this context is

the one of music embodiment: it can be deﬁned as

a corporeal process that enables the link between mu-

sic as experienced phenomenon and music as physical

energy, or the physical environment in general (Le-

man et al., 2008). This sense-giving process focuses

on the cognitive relationship that ties musical subjects

and objects. Such an idea has been critically analyzed

and reworked by Schiavio and Menin (2013).

Music TUIs are a technological means able to sup-

port and encourage music embodiment, thus break-

ing down the barriers that hinder musical creativity

and expressiveness especially in young people and in

impaired performers. On one side, a tangible inter-

face – implying something “real”, “concrete” – offers

a physical way to interact with music and sound pa-

rameters, somehow recalling the kind of interaction

of traditional musical instruments; on the other side,

it can simplify the process, e.g., making it more ac-

cessible and intuitive.

Many music TUIs are based on ﬁducial markers,

or simply ﬁducials, namely objects placed in the ﬁeld

of view of an image-recognition system with func-

tions of control, user input, reference or measure. Ex-

amples of ﬁducials include:

• 2D markers, e.g. barcode systems or pictograms

(Fiala, 2005);

• basic 3D geometrical shapes, e.g. multi-faceted

cubes (Rabbi and Ullah, 2014);

• 3D printed objects, e.g. diorama models of musi-

cal instruments (Avanzini et al., 2019a).

3.1 Musical Applications of TUIs

Fiducials are the basis of the functioning of the re-

acTable, a digital musical instrument developed by

the Music Technology Group at the Universitat Pom-

peu Fabra in Barcelona, Spain (Jord

a, 2010). The re-

acTable employs ﬁducials to generate and control mu-

sic and sound parameters. This device has a tabletop

tangible user interface formed by a round translucent

table used as a backlit display. Special blocks called

tangibles (see Figure 1) can be placed on the table

and moved by the user according to the intended re-

sult; their geometrical and spatial characteristics are

detected in real time by the image-recognition system,

that, in turn, pilots the virtual modular synthesizer to

create music or sound effects.

Another ﬁducial-based framework for music is D-

Touch (Costanza et al., 2003), deﬁning a class of tan-

gible media applications that can be implemented on

consumer-grade personal computers. D-Touch ﬁdu-

cial markers for music-performance applications are

shown in Figure 2.

Finally, it is worth mentioning the TuneTable

(Xamb

o et al., 2017), a platform based on pro-

grammable ﬁducials for music coding (Figure 3).

This approach was assessed in a computational musi-

cal tabletop exhibit for the young held at the Museum

Figure 1: Fiducial markers in use in the reacTable. Picture

taken from (Dance Music Northwest, 2015).

Figure 2: Fiducial markers in use in D-Touch. Picture taken

from (Costanza et al., 2003).

Figure 3: Fiducial markers in use in TuneTable.

CHIRA 2021 - 5th International Conference on Computer-Human Interaction Research and Applications

Figure 4: Kibo’s body and tangibles.

of Design, Atlanta (MODA). Workshop activities had

the goal of promoting hands-on learning of computa-

tional concepts through music creation.

3.2 Kodaly Kibo

Kibo by Kodaly is a wooden board presenting eight

unique geometric shapes that can be inserted into and

removed from suitable slots. This device, also sensi-

tive to pressure variations on single tangibles, returns

the dynamic response of a polyphonic acoustic instru-

ment.

The main control over music parameters is real-

ized through a set of 8 easily-recognizable tangibles,

shown in Figure 4. Each object has a different shape

ﬁtting in a single slot. Tangibles present symmetry

properties so that they can be rotated and ﬂipped be-

fore being inserted in their slots. They have a mag-

netic core, consequently they can be also stacked one

on top of the other. The body of Kibo contains a multi-

point pressure sensor that allows to detect the inser-

tion and removal of tangibles. The characteristics of

the sensor make the instrument both extremely sensi-

tive and very resistant. Concerning the former aspect,

it is sufﬁcient to bring a tangible closer to the body

to trigger a reaction; similarly, the gentle touch of ﬁn-

gers over an already plugged tangible is recognized as

a pressure variation. Concerning robustness, Kibo has

been designed to tolerate strong physical stresses, like

ﬁsts and bumps. A distinctive feature is the possibility

to detect pressure variations over tangibles.

Kibo can be connected via Bluetooth or USB to

iOS and macOS devices running a proprietary app,

that acts both as a synthesizer and a conﬁguration cen-

ter. Windows and Android operating systems are also

supported via third-party drivers.

The communication between the controller and

the app occurs by exchanging standard MIDI 1.0 mes-

sages. The MIDI engine integrated into the app sup-

ports up to 7 Kibo units simultaneously, without per-

ceivable latency. Being a fully compatible controller,

Kibo can also be integrated into any MIDI setup with-

out the intervention of the app as a mediator.

The reasons why Kibo was chosen for the “Note

digitali” project have been already anticipated by Am-

ico and Ludovico (2020). In addition to the advan-

tages of any music-oriented TUI, this device simpli-

ﬁes the establishment of a network of musical instru-

ments working together like an orchestral ensemble.

Moreover, the app natively embeds three operat-

ing modes that are particularly useful in educational,

rehabilitative, and therapeutic ﬁelds:

1. Musical Instrument Mode — In this scenario,

Kibo’s tangibles are usually mapped onto pitches.

Associations between shapes and notes can be

customized; in this way, the device is not bound

to a ﬁxed association (e.g., a C-major scale), but

it support key changes, other scale models, non-

standard note layouts, etc. Through a suitable

processing of MIDI messages, a single key could

also trigger multiple musical events, e.g. custom

chords or arpeggios. The metaphor of a keyboard

controller is further extended by the aftertouch ef-

fect, namely the possibility to detect pressure vari-

ations over tangibles after note attacks;

2. Beat Mode — In this scenario, tangibles are

mapped onto single percussive instruments. The

pressure sensor, presenting a high level of resis-

tance to strong mechanical stresses but also a no-

ticeable sensitivity, allows effects ranging from

hard mallet beats to delicate brush rubbing. With

respect to the previous one, such an operating

mode greatly simpliﬁes interaction and makes the

performance more intuitive for beginners; for ex-

ample, the melodic and harmonic dimensions of

music are absent, and a number of musical param-

eters (e.g., the release time for notes) are ignored;

3. Song Mode — In this scenario, Kibo is employed

as a controller to trigger already available mu-

sic loops. Tangibles are associated with mutu-

ally synchronized but independent tracks, like in

a multi-track environment. When tangibles are

inserted, the corresponding tracks are activated;

when they are removed, tracks are muted (but they

keep running silently, so as to preserve global syn-

chronization). This type of interaction with mu-

sic content is particularly suitable to engage users

who are not able or do not wish to create their own

music.

The conﬁgurability of Kibo, coupled with the

adoption of standard communication protocols, en-

ables numerous and heterogeneous scenarios. Mul-

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners

tiple Kibo units in an ensemble can be set to cover

distinct note ranges and timbres, or even to work

in different operating modes, thus providing the

teacher with great ﬂexibility. Additional operating

modes could be easily implemented by assigning

other meanings, even extra-musical ones, to the MIDI

messages generated by Kibo via ad-hoc software in-

terfaces.

4 THE “NOTE DIGITALI”

PROJECT

The initiative described in this work was launched in

response to “Bando 57”, a call promoted and funded

by Fondazione di Comunit

a di Milano - Citt

a, Sud

Ovest, Sud Est e Adda Martesana ONLUS. In this

framework, a number of workshop activities under

the common umbrella of the “Note digitali” (i.e. Dig-

ital Notes) project has been proposed. The project

directly involved 3 partners:

1. Laboratorio di Informatica Musicale (Laboratory

of Music Informatics), Department of Computer

Science, University of Milan. Established in

1985, it is one of the most relevant Italian research

centers dealing with sound and music computing;

2. Casa di Redenzione Sociale (House of Social Re-

demption), Milan. Founded in 1927, this insti-

tution has been conducting activities in both the

social and cultural ﬁelds, speciﬁcally addressing

problems linked to the context of the northern sub-

urbs of Milan: fragmentation of the social fabric,

widespread educational poverty, and lack of pub-

lic spaces;

3. Fondazione Luigi Clerici, Milan. In opera-

tion since 1972, this foundation offers vocational

courses and apprenticeship initiatives, also for

adult and impaired students. The mission is to

create a network able to integrate education and

organizational skills in collaboration with public

and private authorities, local institutions, trade as-

sociations, and social organizations.

The project was conceived as an experiment of

cultural citizenship where music turns into a means

of self-empowerment and social cohesion. The goals

included providing basic musical competences and

skills, fostering creativity, and, above all, encourag-

ing interaction and socialization in vulnerable young

students. The expected results included the promotion

of participation in the socio-cultural life of the com-

munity by people with different types of disabilities,

the perception of music as a means of aggregation,

Figure 5: The hardware equipment used during the experi-

mental activity: 5 Kibo units and an Apple iPad.

Figure 6: The classroom for workshop activities.

and self-empowerment, namely the self-discovery for

the participants of their skills and abilities.

5 EXPERIMENTAL SETTING

Workshop activities were conducted in small groups

under the guidance of an experienced tutor in a time

span from December 2020 to May 2021.

In this section we will describe the design choices

made when planning the workshop activities, con-

cerning the type and number of hardware devices, the

expected structure of the course, the characterization

of the subjects involved, and the learning materials

prepared to guide participants towards the acquisition

of both musical and social skills.

The basic hardware equipment used during the ex-

perimental activity included 5 Kibo units connected

to an Apple iPad via Bluetooth Low Energy (BLE),

as shown in Figure 5. The room where most activities

took place (Figure 6) was also equipped with tradi-

CHIRA 2021 - 5th International Conference on Computer-Human Interaction Research and Applications

tional and digital musical instruments, such as drums

and an electric piano. This setting provided the tu-

tor with many options, including the exclusive use of

Kibo units (with or without involving the tutor her-

self) or mixed performances involving also traditional

instruments, speciﬁcally the piano and the ukulele. In

the latter scenario, the tutor was the only performer

enabled to play a non-digital instrument, since most

participants had no previous music knowledge, and

the function was basically to explain musical con-

cepts and guide learners towards an autonomous per-

formance.

The technical limitations of the BLE communica-

tion protocol currently limit to 7 the total amount of

Kibo units simultaneously connected to a single mo-

bile device. In case of an expanded Kibo orchestra,

such a constraint can be easily overcome by employ-

ing a higher number of devices suitably conﬁgured to

communicate with a subset of Kibo units. Moreover,

these TUIs can be used as standard MIDI controllers,

thus operating in conjunction with other compatible

hardware equipment.

As the participants admitted to the workshops

were expected to present different types of impair-

ment or distress conditions, the idea was to create

small and homogeneous groups. Participants were

subdivided into teams made of 4 people, in order to

guarantee a number of peers sufﬁcient to foster so-

cial interaction on one side, and let the tutor easily

supervise and guide the experience on the other. The

tutor had background experiences both in music ther-

apy and in digital music technologies.

Participants belonged to 3 categories: 1. young

students aged 12 to 18 with psycho-social support

needs; 2. adults aged 25 to 50 with cognitive and/or

physical impairment; 3. children with special needs

(in particular due to dyslexia, dyspraxia and dyscal-

culia) aged 7 to 10. In total, participants were 20

(12 males, 8 females) and they formed 5 teams: 2

teams (8 participants, 2 females) for the ﬁrst category,

2 teams (8 participants, 4 females) for the second cat-

egory, and 1 team (4 participants, 2 females) for the

third category.

Each 4-people team attended a complete cycle

made of 4 didactic units; units were administered

once a week and lasted 2 hours each. In this way, any

cycle was completed in the time span of one month.

Table 1 shows the tentative program of each edu-

cational cycle, divided into units and tasks. Depend-

ing on the characteristics of the team (age, type of im-

pairment, previous music knowledge, level of atten-

tion, etc.), some adjustments were made on the ﬂy by

the tutor in order to ﬁnely tune the educational activ-

ities. The basic idea was to drive learners along two

parallel growth paths: on one side, improving their

musical skills by gradually introducing new dimen-

sions (rhythm, melody, harmony, timbre); on the other

side, encouraging their interaction aptitudes through

music (listening to the tutor’s performance, playing

alone, playing with the tutor, playing in an ensemble,

playing together and improvising in front of an audi-

ence). Some tasks implied theoretical investigations,

other tasks focused on practical activities.

The adoption of a TUI was fundamental to break

down the initial barriers (physical impairments, lack

of instrumental practice, sense of insecurity or shame)

and let participants be involved in a musical perfor-

mance in a very limited amount of time. It is worth

underlining the importance and the outcome of some

tasks. Task 2.3 implied the ability to translate a se-

quence of musical events (possibly available in Com-

mon Western Notation format) into a sequence of pic-

tograms referring to ﬁducials. In other words, the

TUI pushed learners to develop soft skills (team work,

problem solving, etc.) and the ability to reason ab-

stractly. Task 3.3 asked participants to perform a mu-

sic piece together by playing different roles: two lead-

ing voices, a rhythmic base and a harmonic accompa-

niment. This task encouraged synchronization abil-

ities, information exchange and peer-to-peer cooper-

ation. Finally, Tasks 4.1 and 4.2 explored the ﬁeld

of music improvisation, both mixing already available

materials and playing freely under the inﬂuence of vi-

sual artworks. In the latter case, the portability of the

system (the Kibo units and the tablet) was a key aspect

to conduct such an experience in a museum with a col-

lection of paintings. Not only the TUI is an enabling

technology for impaired users unable to play a tra-

ditional instrument, but it can also unleash creativity

and support musical expressiveness after a very short

exploration phase.

6 ACHIEVED RESULTS AND

CRITICAL DISCUSSION

6.1 Answering Research Question 1

RQ1 aimed to investigate the applicability of a TUI-

based approach to the acquisition of basic musical

skills in vulnerable learners. This research ques-

tion focused on individual experience, acquisition of

knowledge, and development of music-related skills.

From classroom observations and user feedback,

Kibo proved to be a suitable tool to let users with no

previous knowledge develop “musical intelligence”,

namely abilities in the ﬁeld of perception and au-

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners

Table 1: The program of the workshop, with details about didactic units and tasks.

Unit Task Description

1 1.1 Presentation of participants and pre-test

1.2 Introduction to Kibo and its features

1.3 Kibo and piano interactive performance

1.4 All participants playing the same rhythmic pattern

2 2.1 Theoretical fundamentals of melody and rhythm

2.2 Playing a short piece as an ensemble

2.3 Writing and reading a simpliﬁed score

3 3.1 Theoretical fundamentals of harmony and timbre

3.2 Playing a piece with different musical instruments

3.3 Making music together with a Kibo ensemble

4 4.1 Playing a song mixing Kibo’s operating modes

4.2 Music improvisation inspired by paintings

tonomous production of music. The results achieved

by all participants, including very young as well as

impaired ones, included the ability to understand the

main dimensions of music (melody, rhythm, har-

mony, timbre), recognize variations in some parame-

ters (e.g., dynamics, tempo, instrumentation), and au-

tonomously reproduce a simple tune. These achieve-

ments have been assessed through instructor’s obser-

vations.

If compared with the pre-workshop situation, the

best results have been obtained by those participants

who presented both physical and cognitive impair-

ments (speciﬁcally, the second team of the second cat-

egory). The members of this team had started from a

lower level of knowledge, whereas other participants

had recently studied music at school. Moreover, us-

ing an enabling technology was the only opportunity

for them to make music, and in most cases this work-

shop was their ﬁrst active musical experience. For

these reasons, their reaction to the use of a TUI was

enthusiastic. Figure 7 shows one of the wheel-chaired

participants.

Another observation is more tightly related to the

speciﬁc features of Kibo. The geometric shapes of the

Figure 7: A physically impaired user making music with

Kibo.

ﬁducials proved to be suitable both to overcome vi-

sual impairments (tangibles were easily recognizable

to the touch and pluggable into the slots) and cogni-

tive ones (sequences of shapes were easy to remem-

ber also in case of memory deﬁcit or inability to read

a score).

6.2 Answering Research Question 2

RQ2 measured the achievement of non-musical re-

sults. One of the goals of the workshop was to em-

phasize a series of soft and transversal skills through

the creation of a shared musical performance. Mak-

ing music together as an ensemble requires the devel-

opment of social skills, encourages cooperative apti-

tudes, promotes the ability of listening to the other

and perform in front of an audience.

For most teams cooperation did not represent

a problem, rather it encouraged the relations and

strengthened the ties inside each group. For the ﬁrst

team, conversely, playing together was a real chal-

lenge. Let us recall that the members were chil-

dren aged 12 to 14 with a difﬁcult background, com-

ing from a context of social fragility and educational

poverty. The problems encountered with them were

mostly behavioral: respecting the others, listening

without talking over, playing the instruments at the

right time. Luckily, the engagement due to making

music together and the interest towards the playful in-

terface of Kibo let them overcome internal conﬂicts

and focus on a common goal.

The adoption of a TUI also encouraged problem-

solving and abstraction skills, that are two key aspects

of computational thinking. The problem to solve was

how to reinvent a music score suitable both for people

with no music knowledge and for impaired users. The

solution was to translate music notation into Kibo’s

symbols (see Figure 8).

CHIRA 2021 - 5th International Conference on Computer-Human Interaction Research and Applications

Figure 8: A music sheet for Kibo with ﬁducial symbols

added by hand.

The cooperation of each team member was funda-

mental to complete Task 3.3. Learners had to form a

small musical ensemble where everyone should play

an important part. The tutor guided the process so as

to promote personal abilities without causing frustra-

tion in participants. Each team was able to apply the

principles of self-regulation, also thanks to the dis-

tinguishing features of the TUI in use. For example, a

blind girl who demonstrated a great, unexpected sense

of rhythm could perform her part using Kibo in Beat

Mode; two young students autonomously decided to

share the leading voice of a piece by playing it in turn

in Instrument Mode; and less skilled users were able

to participate taking beneﬁt from the Song Mode.

More time would have been helpful to consolidate

this work, but all the learners understood the meaning

of working together and contributing actively to the

achievement of a shared purpose.

6.3 Answering Research Question 3

RQ3 focused on the extensibility and reproducibility

of this workshop activity in other contexts, for exam-

ple with different types of participants or a different

time schedule. To answer such a question, some crit-

ical aspects must be highlighted.

A ﬁrst consideration concerns the choice of the

device. Kibo proved to be a good solution from

many points of view, from technical aspects (e.g.,

easy device connection and communication) to phys-

ical ones (e.g., user-friendliness and robustness). Un-

fortunately, it is not an affordable product. At the mo-

ment of writing, in Italy this device is sold for 900 to

1000e. Building an ensemble of Kibo units, includ-

ing the need to have an Apple mobile device, is not a

low-cost operation. From this point of view, a mixed

approach that includes other traditional or digital in-

struments can help.

Concerning organizational aspects, a team com-

posed by up to 5 participants was a good compro-

mise. Conversely, the presence of a single tutor in the

classroom did not guarantee a ﬂuid conduct of educa-

tional activities. In fact, she had to explain theoretical

concepts, play an instrument, support impaired users

and, sometimes, even solve technical issues simulta-

neously.

The 2-hour lesson length was adequate and gen-

erally appreciated by participants, but the number of

units per cycle should be increased in order to bet-

ter cover the high number of music-related subjects.

For instance, the intriguing relationship between mu-

sic and visual arts was conﬁned to Task 4.2, but it

could become the focus of a whole educational cycle.

Finally, in our experimentation teams were not

formed according to previous musical knowledge but

considering social conditions and impairments. On

one side, this choice facilitated the cooperation be-

tween users sharing similar problems and the conse-

quent ﬁne-tuning of the program, but, on the other

side, it merged in a single team people with differ-

ent expectations. Kibo, as well as many other music-

oriented TUIs, is a facilitator for people with no mu-

sic knowledge, but its limited possibilities can eas-

ily cause boredom and disengagement in more skilled

users.

In conclusion, answering RQ3 requires to take

some critical issues into consideration. The experi-

ence described in this paper represents a pilot study

that will hopefully guide both the authors and the in-

terested readers in better designing future initiatives.

7 CONCLUSIONS

In this paper, we have described the achievements

in terms of increased musical, social, and soft skills

for vulnerable users obtained by organizing ensemble

music sessions with a TUI.

A not surprising result is that a TUI can help solve

the typical issues of accessibility posed by traditional

musical instruments for physically impaired users. In

the experience documented here, most participants

belonging to the second category presented motor im-

pairments, and some of them had also visual impair-

ments. For this kind of users, a TUI is an enabling

technology fundamental to let them experience musi-

cal performance. Many participants had the opportu-

nity to play music for the ﬁrst time, and they were able

to do it together. From this point of view, the experi-

ence was very positive, and reactions from learners

were enthusiastic.

Moreover, a TUI can bridge the gap between cog-

nitive impairment and a full comprehension and ex-

perience of musical dimensions. In this scenario, the

ability to read and memorize a score is not trivial. Not

only these critical issues were successfully tackled,

but participants were able even to produce new scores,

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners

thanks to a simpliﬁed language and a gamiﬁca-

tion approach.

In conclusion, we realized that engagement can

push the limits of users, making them obtain unprece-

dented results. A music-oriented TUI, employed un-

der the guidance of an experienced tutor, can fos-

ter engagement bringing down the initial barriers and

limiting the sense of frustration that often hinders

music creativity and expressiveness in disadvantaged

users.

ACKNOWLEDGEMENTS

This project has been funded by “Bando 57”, a call

promoted by Fondazione di Comunit

a di Milano -

Citt

a, Sud Ovest, Sud Est e Adda Martesana ONLUS.

The authors wish to acknowledge their project part-

ners: Casa di Redenzione Sociale di Milano (in par-

ticular, Luigi Codemo) and Fondazione Luigi Clerici

di Milano (in particular, Federica Monguzzi). The

authors also thank Cooperativa Sociale Cura e Ri-

abilitazione Onlus for the support offered to impaired

users.

REFERENCES

Aljaam, J., Jaoua, A., AlHazbi, S., Hasnah, A., Karime, A.,

and Elsaddik, A. A. (2011). An assistive computerized

system with tangible user interfaces for children with

moderate intellectual and learning disabilities. Inter-

national Journal of Emerging Technologies in Learn-

ing (iJET), 6(2011).

Amico, M. D. and Ludovico, L. A. (2020). Kibo: A MIDI

controller with a tangible user interface for music ed-

ucation. In Lane, H., Uhomoibhi, J., and Zvacek, S.,

editors, Proceedings of the 12th International Con-

ference on Computer Supported Education (CSEDU

2020) - Volume 1, CSEDU, pages 613–619, Set

ubal.

SCITEPRESS - Science and Technology Publications,

Lda.

Avanzini, F., Barat

e, A., and Ludovico, L. A. (2019a).

3D printing in preschool music education: Opportu-

nities and challenges. Qwerty - Open and Interdis-

ciplinary Journal of Technology, Culture and Educa-

tion, 14(1):71–92.

Avanzini, F., Barat

e, A., Ludovico, L. A., and Mandanici,

M. (2019b). A computer-based approach to teach

tonal harmony to young students. In Lane, H., Uho-

moibhi, J., and Zvacek, S., editors, Proceedings of

the 11th International Conference on Computer Sup-

ported Education (CSEDU 2019), volume 1, pages

271–279. SCITEPRESS - Science and Technology

Publications, Lda.

Barat

e, A., Elia, A., Ludovico, L. A., and Oriolo, E. (2018).

The leap motion controller in clinical music therapy.

a computer-based approach to intellectual and motor

disabilities. In McLaren, B. M., Reilly, R., Uho-

moibhi, J., and Zvacek, S., editors, Proceedings of

the 10th International Conference on Computer Sup-

ported Education (CSEDU 2018), 15-17 March 2018,

Funchal, Madeira, Portugal, pages 461–469, Set

ubal.

SCITEPRESS - Science and Technology Publications,

Lda.

Barat

e, A. and Ludovico, L. A. (2020). An open and multi-

layer web platform for higher music education. Jour-

nal of e-Learning and Knowledge Society, 16(4):29–

37.

Barat

e, A., Ludovico, L. A., and Oriolo, E. (2019). Inves-

tigating embodied music expression through the leap

motion: Experimentations in educational and clinical

contexts. In McLaren, B. M., Reilly, R., Uhomoibhi,

J., and Zvacek, S., editors, Computer Supported Edu-

cation - 10th International Conference, CSEDU 2018,

Funchal, Madeira, Portugal, March 15–17, 2018, Re-

vised Selected Papers, volume 1022 of Communica-

tions in Computer and Information Science, pages

532–548. Springer International Publishing.

Barton, G. and Riddle, S. (2021). Culturally respon-

sive and meaningful music education: Multimodal-

ity, meaning-making, and communication in diverse

learning contexts. Research Studies in Music Educa-

tion, page 1321103X211009323.

Bates, V. C. (2018). Equity in music education: Back to

class: Music education and poverty. Music Educators

Journal, 105(2):72–74.

Ben-Tal, O. and Salazar, D. (2014). Rethinking the musi-

cal ensemble: A model for collaborative learning in

higher education music technology. Journal of Music,

Technology & Education, 7(3):279–294.

Carre

no-Le

on, M., Sandoval-Bringas, J. A., Alvarez-

Robles, T., Cosio-Castro, R., Cota, I. E., and Car-

rillo, A. L. (2020). Designing a tangible user in-

terface for braille teaching. In International Con-

ference on Human-Computer Interaction, pages 197–

207. Springer.

Cheng, L. (2019). Musical competency development in a

laptop ensemble. Research Studies in Music Educa-

tion, 41(1):117–131.

Costanza, E., Shelley, S. B., and Robinson, J. (2003). Intro-

ducing audio D-TOUCH: A tangible user interface for

music composition and performance. In Proc. of the

6th Int. Conference on Digital Audio Effects (DAFX-

03), London, UK, September 8-11, 2003.

Dahl, L. (2015). Comparing the timing of movement events

for air-drumming gestures. In International Sympo-

sium on Computer Music Multidisciplinary Research,

pages 3–21. Springer.

d’Alessandro, N., Tilmanne, J., Moreau, A., and Puleo,

A. (2015). Airpiano: A multi-touch keyboard with

hovering control. In Berdahl, E. and Allison, J., edi-

tors, Proceedings of the International Conference on

New Interfaces for Musical Expression, pages 255–

258, Baton Rouge, Louisiana, USA. Louisiana State

University.

Dance Music Northwest (2015). The awesome in-

strument you’ve never heard of: Reactable.

CHIRA 2021 - 5th International Conference on Computer-Human Interaction Research and Applications

https://www.dancemusicnw.com/awesome-

instrument-youve-never-heard-reactable/.

de Oliveira, P. A., Lotto, E. P., Correa, A. G. D., Taboada,

L. G., Costa, L. C., and Lopes, R. D. (2015). Virtual

stage: an immersive musical game for people with

visual impairment. In 2015 14th Brazilian Sympo-

sium on Computer Games and Digital Entertainment

(SBGames), pages 135–141. IEEE.

Fan, X. and Essl, G. (2013). Air violin: A body-centric

style musical instrument. In NIME, pages 122–123.

Farr, W., Yuill, N., and Rafﬂe, H. (2010). Social beneﬁts

of a tangible user interface for children with autistic

spectrum conditions. Autism, 14(3):237–252.

Fiala, M. (2005). Artag, a ﬁducial marker system using dig-

ital techniques. In 2005 IEEE Computer Society Con-

ference on Computer Vision and Pattern Recognition

(CVPR’05), volume 2, pages 590–596 vol. 2.

Frid, E. (2019). Accessible digital musical instruments—a

review of musical interfaces in inclusive music prac-

tice. Multimodal Technologies and Interaction,

3(3):57.

Gabrielli, L., V

alim

aki, V., and Bilbao, S. (2011). Real-time

emulation of the clavinet. In ICMC.

Godøy, R. I., Haga, E., and Jensenius, A. R. (2005). Playing

“air instruments”: mimicry of sound-producing ges-

tures by novices and experts. In International Gesture

Workshop, pages 256–267. Springer.

Gorbunova, I. and Voronov, A. M. (2018). Music computer

technologies in computer science and music studies

at schools for children with deep visual impairment.

In 16th International Conference on Literature, Lan-

guages, Humanities & Social Sciences (LLHSS-18)

Oct. 2-4, 2018 Budapest (Hungary), pages 15–18.

Harrison, K. (2013). The relationship of poverty to music.

Yearbook for traditional music, 45:1–12.

Hattwick, I. (2011). Face to face, byte to byte: Approaches

to human interaction in a digital music ensemble.

MFA, University of California-Irvine.

Hattwick, I. and Umezaki, K. (2012). Approaches to col-

laboration in a digital music ensemble. In NIME 2012

Proceedings of the International Conference on New

Interfaces for Musical Expression, pages 466–469.

Ishii, H. (2008). The tangible user interface and its evolu-

tion. Communications of the ACM, 51(6):32–36.

Ishii, H. and Ullmer, B. (1997). Tangible bits: towards

seamless interfaces between people, bits and atoms.

In Proceedings of the ACM SIGCHI Conference on

Human factors in computing systems, pages 234–241.

Jord

a, S. (2010). The reactable: tangible and tabletop mu-

sic performance. In CHI’10 Extended Abstracts on

Human Factors in Computing Systems, pages 2989–

2994.

Ko, C. L. and Oehlberg, L. (2020). Touch responsive aug-

mented violin interface system ii: Integrating sen-

sors into a 3d printed ﬁngerboard. In Michon, R.

and Schroeder, F., editors, Proceedings of the Inter-

national Conference on New Interfaces for Musical

Expression, pages 166–171, Birmingham, UK. Birm-

ingham City University.

Leman, M. et al. (2008). Embodied music cognition and

mediation technology. MIT press.

Li, J., Wu, C.-H., Lin, T.-J., and Chang, R.-J. (2019). Deter-

minants affecting learner’s behaviour in music educa-

tion applying information technology. In Proceedings

of the 11th International Conference on Computer

Supported Education - Volume 2: CSEDU,, pages

424–431. INSTICC, SciTePress.

McAnally, E. A. (2013). General music and children living

in poverty. General Music Today, 26(3):25–31.

Mellor, L. (2008). Creativity, originality, identity: Investi-

gating computer-based composition in the secondary

school. Music Education Research, 10(4):451–472.

Morreale, F., De Angeli, A., Masu, R., Rota, P., and Conci,

N. (2014). Collaborative creativity: The music room.

Personal and Ubiquitous Computing, 18(5):1187–

1199.

Pesek, M., Vu

cko,

Z.,

Savli, P., Kav

c, A., and Marolt, M.

(2020). Troubadour: A gamiﬁed e-learning platform

for ear training. IEEE Access, 8:97090–97102.

Rabbi, I. and Ullah, S. (2014). 3d model visualization and

interaction using a cubic ﬁducial marker. In De Pao-

lis, L. T. and Mongelli, A., editors, Augmented and

Virtual Reality, pages 381–393, Cham. Springer Inter-

national Publishing.

Riley, P., Alm, N., and Newell, A. (2009). An interactive

tool to promote musical creativity in people with de-

mentia. Computers in Human Behavior, 25(3):599–

608.

Schiavio, A. and Menin, D. (2013). Embodied music cogni-

tion and mediation technology: A critical review. Psy-

chology of Music, 41(6):804–814.

Tarabella, L. (2004). Handel, a free-hands gesture recog-

nition system. In International Symposium on Com-

puter Music Modeling and Retrieval, pages 139–148.

Springer.

Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D.

(2013). Analysis of the accuracy and robustness of the

Leap Motion controller. Sensors, 13(5):6380–6393.

Xamb

o, A., Drozda, B., Weisling, A., Magerko, B., Huet,

M., Gasque, T., and Freeman, J. (2017). Experience

and ownership with a tangible computational music

installation for informal learning. In Proceedings of

the Eleventh International Conference on Tangible,

Embedded, and Embodied Interaction, pages 351–

360.

Xu, D. (2005). Tangible user interface for children – an

overview. In Proc. of the UCLAN Department of Com-

puting Conference. Citeseer.

An Ensemble of Tangible User Interfaces to Foster Music Awareness and Interaction in Vulnerable Learners