A Music Tangible User Interface for the Cognitive and Motor

Rehabilitation of Elderly People

Adriano Barat

, Helene Korsten and Luca A. Ludovico

Laboratory of Music Informatics, Department of Computer Science, University of Milan, via G. Celoria 18, Milan, Italy

Keywords:

Music, Tangible User Interface, Web, Kibo.

Abstract:

This paper proposes the combined use of tangible user interfaces, digital technologies, and musical expression

in the context of cognitive and motor rehabilitation of elderly people. After analyzing the state of the art about

common age impairments and tangible user interfaces in rehabilitation, we will introduce the Kibo, a MIDI

controller based on the concept of ﬁducials and able to communicate with other MIDI devices via Bluetooth.

The peculiar characteristics of the Kibo will be exploited in a Web framework aiming to foster the development

or recovery of cognitive and motor abilities through 3 specially designed games. The paper will also report

remarks from domain experts (music therapists and physiatrists) and the consequent redesign.

1 INTRODUCTION

According to a 2019 report by the United Nations,

there were 703 million persons aged 65 years or over

in the world in 2019 and the number of older persons

is projected to double to 1.5 billion in 2050. These

changes for individuals are mirrored in society since

older persons represent a growing demographic group

in society. One of the goals for sustainable devel-

opment should be investing in education, health, and

well-being for all, including the elder.

The target audience of the project documented

in this paper is represented by elderly people who

need rehabilitation sessions or are dissatisﬁed or in-

active in their social interactions. Starting from

an analysis of the most common impairments and

changes in cognitive and motor abilities due to age,

we designed and developed a prototype applicable to

music-therapy rehabilitation sessions. Interventions

can address a variety of healthcare and educational

goals (e.g., promoting wellness, managing stress, al-

leviating pain, expressing feelings, enhancing mem-

ory, improving communication, promoting physical

rehabilitation, etc.).

The result is a Web-based prototype that offers in-

tuitive interaction with musical parameters via a spe-

ciﬁc tangible user interface, called the Kibo, so as to

take advantage of the haptic interaction skills with the

https://orcid.org/0000-0001-8435-8373

https://orcid.org/0000-0002-8251-2231

environment. In particular, three browser games have

been developed in order to train rhythm, spatial, and

recognition skills.

2 STATE OF THE ART

In this section, we will review the scientiﬁc literature

that is most relevant for our purposes. Due to the vast-

ness of the subject, we cannot be exhaustive. We will

limit ourselves to a few references of particular im-

portance for our work, addressing: i) common age

impairments, both cognitive and physical; ii) tangi-

ble user interfaces and their applicability to the music

ﬁeld; and iii) experiences of tangible user interfaces

in rehabilitation.

Starting from age impairments, due to an in-

creasing life expectancy, the number of elderly peo-

ple has increased signiﬁcantly worldwide. Thus, so-

ciety will have to offer more and more products and

services that meet the speciﬁc needs and desires of

a geriatric age group that makes up a great percent-

age of the population. These people are seeking solu-

tions to help them cope with daily life, give them the

opportunity to interact socially, and ﬁnd alternative

ways of entertainment and learning. Moreover, health

systems are dealing with the ever-increasing burden

of ﬁnding solutions and cures for age-related and de-

generative conditions (e.g., dementia and Alzheimer’s

disease) and impaired-movement pathologies (e.g.,

Baratè, A., Korsten, H. and Ludovico, L.

A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People.

DOI: 10.5220/0011395900003323

In Proceedings of the 6th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2022), pages 121-128

ISBN: 978-989-758-609-5; ISSN: 2184-3244

121

Parkinson’s).

Everyone experiences changes as an inevitable

part of the natural degeneration associated with ag-

ing (Iversen, 2015). Concerning changes in cognitive

abilities, aging causes a decline in spatial cognition,

which is the ability to represent spatial relationships

among objects. The results of a study conducted in

2008 by Iachini et al. show that some spatial abili-

ties, such as the ability to mentally rotate visual im-

ages and to retrieve spatio-temporal sequences, de-

cline with age (Iachini et al., 2008). Elderly peo-

ple also struggle more with multitasking, especially if

the tasks are complex (Wecker et al., 2005). Another

common age impairment is the decline in ﬂuid intel-

ligence, which refers to the processing and reasoning

components of intelligence and the natural ability to

learn something new (Czaja and Lee, 2007). Because

of reduced processing efﬁciency, the working mem-

ory, which is the ability to keep information active

while processing or using it, declines with age (Salt-

house, 1991). Similarly, prospective memory, which

is the ability to remember to do something in the fu-

ture, also declines with age (Maylor, 1995). Another

issue emerging with age is the ability to select infor-

mation in the environment, e.g., to attend to informa-

tion on a Web page. Attention is the ability to focus on

a speciﬁc task or an object in the environment while

ignoring other things. This ability changes with age

and older people are slower to move their attention

from one thing to another (Czaja and Lee, 2007).

Another category of impairments due to aging is

that of changes in physical abilities. Response time

and accuracy of movement decline with age. Older

persons’ movements, reﬂexes, and reactions to stim-

uli are typically slower than younger persons’ (Farage

et al., 2012). Differently from the above-mentioned

normal age impairments, a disease that limits move-

ment is Parkinson’s, a neurodegenerative brain dis-

order that progresses slowly and worsens with age.

Symptoms are involuntary shaking of the hands, arms,

legs, jaw, chin, and lips. Other main symptoms in-

clude slowness of movement, stiffness of arms and

legs, and trouble with balance (Jankovic, 2008).

Even if not an impairment, another phenomenon

relevant for our work is anxiety towards technol-

ogy. In a study by Czaja and Sharit dating back to

1998, older people saw themselves as having less con-

trol over computers than younger people (Czaja and

Sharit, 1998). They had signiﬁcantly less efﬁcacy

in completing computer tasks, but, surprisingly, they

also perceived computers as being more useful com-

pared to younger people. Another study revealed that

older subjects reported higher levels of computer anx-

iety than younger persons and that the anxiety level

was related to the decision time on the computer when

performing a test (Laguna and Babcock, 1997). The

elderly may have a harder time using new technology,

but it helps with practice (Chu, 2010). These consid-

erations pushed us to develop a solution based on a

computing system but relying on a tangible interface

to ease user interaction.

Focusing on tangible user interfaces (TUIs),

they are intended to replace the graphical user inter-

faces (GUIs), more common in computing systems,

with real physical objects the user can interact with.

The key idea is to give digital information a physical

form and let these physical forms serve as a represen-

tation and control for digital information. A TUI lets

users manipulate digital information with their hands

and perceive it with their senses. One of the pioneers

in tangible user interfaces is Hiroshi Ishii, a profes-

sor at MIT who heads the Tangible Media Group at

the MIT Media Lab. His particular vision for TUIs,

called Tangible Bits, is to give physical form to digi-

tal information, making bits directly manipulable and

perceptible (Ishii, 2008b). Tangible Bits pursues the

seamless coupling between physical objects and vir-

tual data. “TUIs will augment the real physical world

by coupling digital information to everyday physical

objects and environments” (Ishii and Ullmer, 1997).

All physical objects can potentially be a part of a dig-

ital user interface (Ishii, 2008a). For example, if an

object, which is a part of a TUI, is moved or put in

a speciﬁc position, a digital signal will be sent from

either the tangible object itself or from another de-

vice that senses the object. Currently, there are dif-

ferent research areas and applications related to TUIs.

For instance, tangible augmented reality implies that

virtual objects are “attached” to physically manipu-

lated objects; in tangible tabletop interaction, physical

objects are moved upon a multi-touch surface; more-

over, physical objects can be used as ambient displays

or integrated inside embodied user interfaces.

TUIs have been used considerably in musical per-

formances and music-therapy treatments. A tangible

interface – implying something “real”, “concrete” –

offers a physical way to interact with music and sound

parameters. Most traditional musical instruments are

played through this kind of interaction, but the ad-

vent of digital technologies paved the way for innova-

tive and original approaches. Concerning technology-

enhanced TUIs for music, the scientiﬁc literature de-

scribes a number of theoretical approaches, proto-

types, and available products. Music TUIs can play

a number of roles: for example, synthesizers to gen-

erate sound, sequencers that perform audio samples

and mix them together, remote controllers for music

and sound parameters, or interfaces for music-related

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

122

games. To cite but a few references, Paradiso et al.

review TUIs based on magnetic tags (Paradiso et al.,

2001), Newton-Dunn et al. describe a way to control

a dynamic polyrhythmic sequencer using physical ar-

tifacts (Newton-Dunn et al., 2003), and Schiettecatte

and Vanderdonckt present a distributed cube interface

based on interaction range for sound design (Schiette-

catte and Vanderdonckt, 2008). A noticeable example

of commercially available music TUI is the reacTable

(Jord

a et al., 2007), used by renowned artists such as

ork in their live performances. Our solution adopts

a low-cost device called the Kibo, whose features will

be described in detail in Section 3.

Finally, let us address the applicability and role of

tangible user interfaces in rehabilitation. Motiva-

tion is one of the main problems evidenced in tradi-

tional therapy sessions, often hampered by the repet-

itive nature of exercises. Most studies show that an

effective rehabilitation must be early, intensive and

repetitive (Rego et al., 2014; Burke et al., 2010). As

such, these approaches are often considered repeti-

tive and boring by the patients, resulting in difﬁculties

in maintaining their interest and in assuring that they

complete the treatment program (Rego et al., 2014).

On the other hand, due to their nature, games can mo-

tivate and engage the patients’ attention and distract

them from their rehabilitation condition. On one side,

they require some motor and cognitive activity, but,

on the other, they have a story and can offer feed-

back and levels of challenge and difﬁculty that can

be adapted to the patients’ skills.

Serious games are an option that provides learn-

ing combined with entertainment. The locution “se-

rious games” refers to playful activities that provide

training and physical or mental exercise in a fun and

enjoyable way (D

orner et al., 2016). These games

can be not only a way to prevent the feeling of loneli-

ness (De Carvalho et al., 2012), but they can also en-

able social interaction (Fonseca et al., 2021). During

the last decades, digital games have become a popu-

lar leisure activity. Ijsselsteijn et al. claim that digital

games can be considered a promise to improve the

lives of seniors. To this end, it is important to develop

interesting and accessible games which could provide

an option to spend quality time with clear beneﬁts

(Ijsselsteijn et al., 2007). Pearce reports that the stud-

ies of digital games with an emphasis on older peo-

ple are still rare because this need is relatively new

(Pearce, 2008). This perception may be justiﬁed by

the fact that the current elderly population did not

have much access to technology at their earlier age.

Many rehabilitation games based on TUIs are al-

ready available. For example, Handly is an integrated

upper-limb rehabilitation system for persons with a

neurological disorder (Vandermaesen et al., 2016).

Handly consists of tangibles for training four-hand

tasks with speciﬁc functional handgrips and a moti-

vational game. The system consists of four tangible

training boxes, which each present one essential grip

and associated hand task: push-pull, squeezing, knob

turning, and key turning. Handly combines tangibles

speciﬁcally designed for repetitive task-oriented mo-

tor skill training of typical daily activities with serious

gaming, thus offering a comprehensive approach.

Segara is an integrated hand rehabilitation sys-

tem for patients with rheumatoid arthritis (Zhao et al.,

2021). Segara consists of tangibles for training six

tasks with interactive functional handgrips and a mo-

tivational serious game. It shows that a system com-

bining games and tangibles to enhance hand rehabili-

tation is feasible and highly appreciated by patients.

Resonance is an interactive tabletop artwork that

targets upper-limb movement rehabilitation for pa-

tients with an acquired brain injury (Duckworth et al.,

2015). The artwork consists of several interactive

game environments, which enable artistic expres-

sion, exploration, and play. Resonance provides

uni-manual and bi-manual game-like tasks and ex-

ploratory creative environments of varying complex-

ity geared toward reaching, grasping, lifting, moving,

and placing tangible user interfaces on a tabletop dis-

play. Each environment aims to encourage collabora-

tive, cooperative, and competitive modes of interac-

tion for small groups of co-located participants.

NikVision is a tangible tabletop based on a user-

centered design approach for the cognitive stimula-

tion of older people with cognitive impairments and

dementia problems in nursing homes (Cerezo et al.,

2020). The general experiences of the users when

working with the tangible tabletop were assessed and

applied to the design of new cognitive and physical

stimulation activities. Game activities are specially

designed for the elderly, including daily tasks such

as getting dressed, cognitive assignments, and upper-

half motor skills training. Activities present different

levels of difﬁculty and audio feedback.

3 EMPLOYED TECHNOLOGIES

In this paper, we want to take advantage of real phys-

ical objects and the beneﬁts that TUI technologies of-

fer in order to provide the elder with a playful way to

interact in rehabilitation therapies. As mentioned be-

fore, one of the advantages of a TUI is the user expe-

rience, because a physical interaction occurs between

the user and the interface itself. Another advantage is

usability since the user intuitively knows how to ma-

A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People

123

nipulate the interface by knowing the function of the

physical object; consequently, the user does not need

to learn the functionality. That is why TUIs are often

used to make technology more accessible for elderly

people.

In this section, we will address the key technolo-

gies employed in the project: a music TUI called the

Kibo (Amico and Ludovico, 2020), the way it is con-

nected to a Web application, and the Web languages

and formats used to implement the browser games de-

scribed later.

The Kibo is a wooden board produced by Kodaly

that presents eight distinct, easy-to-recognize tangi-

bles.

These geometric shapes can be inserted into

and removed from the corresponding slots, thus trig-

gering events encoded in the form of MIDI messages.

The device is also sensitive to pressure variations on

single tangibles and individual dynamic responses can

be communicated via MIDI messages. As an addi-

tional controller, there is a knob that can be rotated

clockwise and counterclockwise. Finally, the device

is equipped with a gyroscope.

The Kibo can be connected via Bluetooth or USB

to iOS and macOS devices running a proprietary app

that acts as both a synthesizer and a conﬁguration cen-

ter. Windows and Android operating systems are also

supported via third-party drivers. The communication

between the controller and the app occurs by exchang-

ing standard MIDI 1.0 messages. The MIDI engine

integrated into the app supports up to 7 Kibo units

simultaneously, without perceivable latency. This as-

pect is particularly interesting for collaborative expe-

riences (Barat

e et al., 2021). Being a fully compatible

controller, Kibo can also be integrated into any MIDI

setup without the intervention of the app as a media-

tor, as we will do in our proposal.

The control over music parameters is mainly

based on the 8 tangibles shown in Figure 1. Each ob-

ject has a different shape ﬁtting in a single slot and

presents symmetry properties so that it can be rotated

and ﬂipped before being inserted. Tangibles have a

magnetic core; they can be stacked one on top of

the other and interact through magnetic ﬁelds. The

body of the Kibo contains a multi-point pressure sen-

sor that allows for the detection of the insertion and

removal of tangibles. The characteristics of the sen-

sor make the instrument extremely sensitive and, si-

multaneously, very resistant.

Even if our proposal bypasses the native Kibo’s

app, it is interesting to analyze the three original op-

erating modes and shed some light on rehabilitative

and therapeutic scenarios:

https://www.kodaly.app/

Figure 1: The Kibo’s body and tangibles.

1. Musical Instrument Mode — In this scenario,

Kibo’s tangibles are mapped onto pitches. As-

sociations between shapes and notes can be cus-

tomized, even triggering multiple notes through a

single tangible. The device is able to detect after-

touch effects, namely the possibility to track pres-

sure variations over tangibles after note attacks;

2. Beat Mode — In this scenario, tangibles are

mapped onto single percussive instruments. The

pressure sensor, presenting a high level of resis-

tance to strong mechanical stresses but also a no-

ticeable sensitivity, allows effects ranging from

hard mallet beats to delicate brush rubbing;

3. Song Mode — In this scenario, Kibo is employed

as a controller to trigger already available mu-

sic loops. Tangibles are associated with mutu-

ally synchronized but independent tracks, like in

a multi-track environment. When tangibles are

inserted, the corresponding tracks are activated;

when they are removed, tracks are muted.

The conﬁgurability of Kibo, coupled with the adop-

tion of standard communication protocols, enables

numerous and heterogeneous scenarios. Multiple

Kibo units in an ensemble can be conﬁgured to cover

distinct note ranges and timbres, or even to work in

different operating modes, thus providing the ther-

apist with great ﬂexibility. Moreover, the standard

MIDI output of the Kibo allows the implementation

of additional operating modes where other meanings,

even extra-musical ones, can be assigned to user ges-

tures. Our proposal, described in detail in Section 4,

explores this possibility.

Even if originally conceived as a general-purpose

tangible MIDI controller, when used in a suitable sce-

nario the Kibo can also be considered an assistive

technology falling in the category of communication

boards. Under this perspective, it has a therapeutic

function since it encourages upper-limb movements

and challenges cognitive skills. Moreover, it is com-

pensatory both from a motor point of view, being able

to translate even small movements into sound, and

from a cognitive point of view, enabling intuitive mu-

sical expression by lowering the barriers of a tradi-

tional instrument.

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

124

In this project, the Kibo has to be directly con-

nected via Bluetooth to the computing system that

hosts the browser. A speciﬁc protocol – called MIDI

over Bluetooth or, simply, Bluetooth MIDI – is spe-

cially conceived to exchange MIDI messages over

Bluetooth connections (Bartolomeu et al., 2005). The

Kibo adopts Bluetooth Low Energy (BLE), a wireless

personal-area network technology that, compared to

the original Bluetooth protocol, is intended to provide

considerably reduced power consumption and cost

while maintaining a similar communication range.

A BLE MIDI device will transparently operate with

a MIDI-compatible application on most mobile and

desktop platforms with no additions and provides “out

of the box” support for most use cases compared with

any wired or wireless alternative.

The visual feedback of the system is imple-

mented in the form of a Web page by adopting W3C

standard languages and formats. As a result, the

browser games can be experienced using any HTML5

browser. The project was realized using HTML, CSS,

and JavaScript.

Please note that all the mentioned technologies are

client-side, so the platform could also be enjoyed lo-

cally on the user’s client, with no need to connect to

a server. Nevertheless, we have publicly released it

over the Web in order to distribute it easily at no cost

for the user and to keep it up-to-date in the case of a

new release.

4 KIBO WEB GAMES

The goal of linking music therapy, rehabilitation ex-

ercises, and technology through the Kibo brought to

the design and implementation of a Web platform that

proposes three serious games. The name of the plat-

form is Kibo Web Games. All game activities allow

working the association between physical elements

(geometric tangibles) and the concepts they represent

(e.g., notes, tracks, hit buttons). The platform is avail-

able at https://kibogames.lim.di.unimi.it/ and its orig-

inal Web interface is shown in Figure 2. Kibo Web

Games need a Kibo device connected via BLE.

A number of game parameters have been intro-

duced to allow ﬂexibility in usage and adaptation to

gradual improvements without causing frustration in

the player. The user or therapist can set different types

of activities and levels of difﬁculty according to the

following parameters: i) the game to play, that is basi-

cally a rhythm-based activity where given shapes are

proposed to the player and must be timely touched,

inserted, removed, or continuously pressed depend-

ing on the game mode, as explained in the following;

Figure 2: The interface of Kibo Web Games.

ii) the number of shapes (1 to 8 out of the 8 avail-

able) that can randomly be involved during the game

experience; iii) the total length of the game sequence,

i.e. the number of actions a user should carry out in a

game session; and iv) the speed of the shape genera-

tion and scroll animation.

The central part of the interface is taken by the

Game ﬁeld, which displays the moving shapes during

a game session. Shapes move from right to left, and

the perfect timing for user’s actions is when they hit

the black vertical line. Another relevant part of the

interface in Figure 2 is the MIDI message console,

namely the lower rectangle that displays the MIDI

messages received from the Kibo. Finally, the Play

button starts the selected game with the parameters

set by the user.

The second game expects the user to insert or re-

move the shapes from their slots in synchronization

to the game’s graphic scenario is able to train preci-

sion in performing movements. The third game asks

the user to simultaneously press and release the rec-

ognized shapes. The scores, levels of difﬁculty, and

feedback given to the player depend on her/his tem-

poral precision in performing the movement.

Concerning the gameplay, Kibo Web Games focus

on the interactions occurring between the player and

the Kibo, i.e. simple motor movements like hitting,

tapping, grabbing, holding, releasing, placing, and

removing the geometrical tangibles. Furthermore,

recognition and listening skills are trained to help re-

store or keep active cognitive functions. The Kibo

gives tactile and musical feedback to every action that

is performed and allows the user to proceed in small

steps. It also allows users to manipulate objects giving

the clearer image of the connection between physical

interaction and the response that it triggers.

All games share the same game ﬁeld and func-

tional concepts. The system generates shapes

scrolling across the screen from right to left and the

A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People

125

goal is to timely recognize the shape by hitting, press-

ing, inserting, or removing the corresponding tangible

from the Kibo physical body when it reaches the ver-

tical line in the delimiter box (Figure 2). Throughout

the duration of a game session, the arrow, square, and

vertical line in the Game ﬁeld animate to facilitate

the game experience. Arrow and square ﬂash when

the user is supposed to perform a task. The vertical

line turns either red or green according to the accu-

racy of the performance. The score ﬁeld keeps track

of the points gained while playing and remains dis-

played until the next game session is started.

Perfect hits are those performed in a very small

range around the computed timing. Good hits occur

in a slightly wider timing range. The exact values de-

pend on the game speed set by the user, but they are

in the order of tenths of a second. Currently, score

penalties for wrong shape recognition or missed hits

are not implemented to avoid frustration in players,

but this feature could be easily integrated.

Please note that the Kibo Web Games interface

allows users to play with the Kibo instrument as

a simple musical controller, regardless a game ses-

sion is active or not, after connecting it to the Web

page. This provides sound feedback (C-major scale)

and console messages only, but it might help to gain

conﬁdence and increase motivation. As another re-

mark, the three games help gain rhythm awareness

and present musical feedback, but they never imply

a music-education background nor make use of any

kind of music-related concept.

Game A – Tap the Shapes is based on the Kibo’s

Beat Mode (see Section 3). Tangibles are mapped

onto single notes of a C-major scale, one grade per

shape. Pressing, hitting, or tapping a shape sends a

Note On message immediately followed by a Note

Off. MIDI messages are interpreted by the interface

and played back thanks to an embedded audio syn-

thesizer. Difﬁculty varies according to speed, number

of shapes involved, game length, and score precision.

The rhythm game expects the user to simply activate

a midi ON message by pressing the expected tangible

in the right timing range, namely when the arrow and

delimiter box turn yellow. In order to play this game,

all tangibles must be inserted.

Game B – Insert and Remove Shapes recalls the

Kibo’s Song Mode (see Section 3). Tangibles are as-

sociated with independent synchronized tracks within

a multi-track environment realized with Pro Tools

for this purpose. The multi-track starts playing when

the game is initialized but has no volume. Each track

is associated with a shape and is unmuted when the

corresponding tangible is inserted into the device’s

https://www.avid.com/pro-tools

wooden base. Conversely, removing the shape will

mute the track. All tangibles must be removed and

placed in front of the player or at a reachable distance

before starting the game. For a better user experience,

long game sequences must be set. The aim of Game B

is to score points by inserting or extracting the Kibo’s

tangibles once the randomly generated shapes touch

the hit-line or perfectly ﬁt the delimiter box before the

animation ends and the shape disappears. The random

algorithm should assure that, in the case of right user

performance, at the end of the session no shapes are

left inside Kibo’s body. This game mode solicits the

development or recovery of some cognitive abilities,

including memory (shapes have to be quickly recog-

nized, found in the space around the Kibo, and in-

serted into the right slot) and creative reasoning (e.g.,

ﬁnding the best initial layout for the pieces around the

Kibo’s body).

Game C – Hold and Release Shapes recalls the

Kibo’s Musical Instrument Mode and takes beneﬁts

from the polyphonic aftertouch feature. Tangibles

are mapped once again onto the grades of a C-major

scale. Somehow similar to Game A, when a shape

touches the vertical line or enters the delimiter box,

the user is expected not only to tap the correspond-

ing tangible but also to keep it pressed until the same

shape appears again. Pressing a tangible sends a

MIDI Note On message, and releasing it sends the

corresponding Note Off message. This game mode is

probably the most challenging one, from both a mo-

tor and a cognitive point of view. For example, some

combinations of shapes require not only the ability

to have them selected simultaneously but also a good

strategy to have a hand free for the next insertion.

5 EARLY EXPERIMENTATION

Due to the restrictions imposed on society during the

COVID-19 pandemic and the impact it had on the el-

derly and fragile populations, it has not been possi-

ble to test this prototype in a real-life setting so far.

Nonetheless, the games were proposed to elder fam-

ily members, colleagues, and domain experts to get

objective feedback on their beneﬁts and usefulness.

Overall, Kibo Web Games were appreciated and

described as an enjoyable form of entertainment, yet

challenging enough to possess rehabilitative proper-

ties. Differentiating between similarly shaped tangi-

bles (e.g., star and ﬂower) was initially perceived as

demanding by many but overcome with practice. An-

other challenging aspect was maintaining the level of

performance with the speed increment.

An early prototype was presented to music thera-

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

126

pists and physiatrists of Fondazione Don Carlo Gnoc-

chi, Milan. Many useful remarks emerged about the

games’ structure and the user interface. Game A was

particularly appreciated due to its simplicity, but the

interface was considered too complex for users with

impairments, above all cognitive ones. A ﬁrst sugges-

tion was to move the parameter conﬁguration and the

MIDI console away from the game screen, thus intro-

ducing a sort of back-ofﬁce area to set game sessions;

in fact, in a standard scenario, the parameter conﬁg-

uration on one side and the gameplay on the other

involve two distinct actors, and the presence of side

controls and messages can be a source of distraction.

Similarly, the use of colors should be limited as much

as possible, just to differentiate the actions required

(e.g., either insert or remove a tangible). The nu-

meric score, too, should be turned into a more direct

and comprehensible representation, such as a given

number of stars, a growing bar, or something similar.

Games B and C involve persistent actions (i.e. shapes

to keep inserted or pressed for a given time). In this

case, the experts’ objections focused on the graphi-

cal representation itself of the game ﬁeld. In fact, in

their opinion, having 8 scrolling areas corresponding

to the 8 tangibles could help, above all in the case of

cognitive impairment. Actually, a colored multi-track

representation for Game C would recall the piano-roll

visualization typically adopted by MIDI sequencers.

Moreover, experts agreed that the last two games are

more challenging from any point of view and, for this

reason, they suggested a step-by-step process before

playing the ﬁnal version (e.g., ﬁrst locate given shapes

with no time constraint, then test the physical actions

by moving shapes into their slots, etc.). Early experi-

mentation of the revised interface should start at Fon-

dazione Don Carlo Gnocchi in October 2022. Thanks

to the positive feedback received, it is likely that Kibo

Web Games have the requisites to be utilized among

the elderly population. Future testing on this speciﬁc

population will be necessary to ﬁne-tune the develop-

ment of a truly suitable device that can be used in the

rehabilitative space. Connecting performance results

to a database or exporting them into a CSV ﬁle could

provide a more accurate track of progress.

The approach of Kibo Web Games can be easily

generalized and adapted to other scenarios. For ex-

ample, the platform can be proﬁtably used to develop

music parameters and geometric concepts awareness

in young students, even in preschool age, thanks to

TUI-based playful activities. Moreover, Kibo Web

Games can help recover cognitive and motor abilities

not only in elderly people but also in younger users;

some experiences in this sense are reported in (Barat

et al., 2021).

6 CONCLUSIONS

In this project, we investigated the suitability of the

Kibo to compensate for age impairments. The device

proved to be a good solution from a technical and a

physical point of view due to user-friendliness and ro-

bustness. Unfortunately, it is not an affordable prod-

uct. Thanks to the Kibo’s 8-voice polyphony, mul-

tiple participants have the opportunity to access the

interface simultaneously. Collaboration could be ad-

vantageous for seriously disabled individuals needing

assistance from caregivers. However, the described

games are not designed to explore this scenario in full.

The possibility to choose parameters and set dif-

ﬁculty levels allows users to progress at their own

pace without confronting insurmountable obstacles.

A therapist or a caregiver can suitably conﬁgure game

sessions based on the players’ attitudes and abilities.

Moreover, no game penalties were implemented with

the distinct purpose of not discouraging elderly users

who are often not technology savvy and likely to be

afraid of making mistakes or getting stuck.

The inspiration for future work mainly comes

from the observations and remarks by experts re-

ported in Section 5. Short-term future improvements

include the creation of a different game ﬁeld for each

speciﬁc game, keeping the interface’s look as sim-

ple as possible to avoid any distractions during the

gameplay, and the implementation of two different

windows (one for the user/patient who can only play

the game and one for the specialist who can set the

game parameters). In addition, tracking parameters

like pressure variations could test and improve tac-

tile and ﬁne motor skills in action. After on-the-ﬁeld

experimentation, gaming speed will probably need to

be adjusted to accommodate slower apprehension in

aged people. Finally, an aspect to further explore is

the use of proper musical feedback to be applied in

music therapy.

Our expectation is that Kibo Web Games are able

to stimulate declining cognitive abilities and reignite

strained motor skills in elderly people in order to keep

them active and fully integrated into society through

a set of engaging activities.

REFERENCES

Amico, M. D. and Ludovico, L. A. (2020). Kibo: A MIDI

controller with a tangible user interface for music ed-

ucation. In Int. Conf. on Computer Supported Educa-

tion, pages 613–619. SCITEPRESS.

Barat

e, A., Ludovico, L. A., and Oriolo, E. (2021). An

ensemble of tangible user interfaces to foster mu-

sic awareness and interaction in vulnerable learn-

A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People

127

ers. In Proc. 5th Int. Conf. on Computer-Human In-

teraction Research and Applications, pages 48–57.

SCITEPRESS.

Bartolomeu, P., Fonseca, J., Duarte, P., Rodrigues, P., and

Girao, L. (2005). Midi over bluetooth. In 2005 IEEE

Conference on Emerging Technologies and Factory

Automation, volume 1, pages 8 pp.–102.

Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie,

J., and McDonough, S. (2010). Augmented reality

games for upper-limb stroke rehabilitation. In 2010

Second International Conference on Games and Vir-

tual Worlds for Serious Applications, pages 75–78.

Cerezo, E., Bonillo, C., and Baldassarri, S. (2020). Ther-

apeutic activities for elderly people based on tangible

interaction. In ICT4AWE, pages 281–290.

Chu, R. J.-c. (2010). How family support and internet

self-efﬁcacy inﬂuence the effects of e-learning among

higher aged adults–analyses of gender and age differ-

ences. Computers & Education, 55(1):255–264.

Czaja, S. J. and Lee, C. C. (2007). The impact of aging on

access to technology. Universal access in the infor-

mation society, 5(4):341–349.

Czaja, S. J. and Sharit, J. (1998). Age differences in atti-

tudes toward computers. The Journals of Gerontology

Series B: Psychological Sciences and Social Sciences,

53(5):P329–P340.

De Carvalho, R. N. S., Ishitani, L., Nogueira Sales De Car-

valho, R., et al. (2012). Motivational factors for mo-

bile serious games for elderly users. Proceedings of

XI SB games, pages 2–4.

orner, R., G

obel, S., Effelsberg, W., and Wiemeyer, J.

(2016). Serious games. Springer.

Duckworth, J., Mumford, N., Caeyenberghs, K., Eldridge,

R., Mayson, S., Thomas, P. R., Shum, D., Williams,

G., and Wilson, P. H. (2015). Resonance: an in-

teractive tabletop artwork for co-located group reha-

bilitation and play. In Int. Conf. on Universal Ac-

cess in Human-Computer Interaction, pages 420–431.

Springer.

Farage, M. A., Miller, K. W., Ajayi, F., and Hutchins,

D. (2012). Design principles to accommodate older

adults. Global journal of health science, 4(2):2.

Fonseca, X., Slingerland, G., Lukosch, S., and Brazier, F.

(2021). Designing for meaningful social interaction

in digital serious games. Entertainment Computing,

36:100385.

Iachini, T., Borghi, A. M., and Senese, V. P. (2008). Cat-

egorization and sensorimotor interaction with objects.

Brain and Cognition, 67(1):31–43.

Ijsselsteijn, W., Nap, H. H., de Kort, Y., and Poels, K.

(2007). Digital game design for elderly users. In

Proceedings of the 2007 Conference on Future Play,

Future Play ’07, page 17–22, New York, NY, USA.

Association for Computing Machinery.

Ishii, H. (2008a). Tangible bits: Beyond pixels. In Proc.

2nd Int. Conf. on Tangible and Embedded Interac-

tion, TEI ’08, page xv–xxv. Association for Comput-

ing Machinery.

Ishii, H. (2008b). The tangible user interface and its evolu-

tion. Commun. ACM, 51(6):32–36.

Ishii, H. and Ullmer, B. (1997). Tangible bits: towards

seamless interfaces between people, bits and atoms.

In Proceedings of the ACM SIGCHI Conference on

Human factors in computing systems, pages 234–241.

Iversen, T. R. (2015). Exploring tangible interaction: Alter-

native interfaces for assisting elderly users. Master’s

thesis, Department of Informatics, University of Oslo.

Jankovic, J. (2008). Parkinson’s disease: clinical features

and diagnosis. Journal of neurology, neurosurgery &

psychiatry, 79(4):368–376.

Jord

a, S., Geiger, G., Alonso, M., and Kaltenbrunner, M.

(2007). The reacTable: exploring the synergy be-

tween live music performance and tabletop tangible

interfaces. In Proc. 1st Int. Conf. on Tangible and Em-

bedded Interaction, pages 139–146.

Laguna, K. and Babcock, R. L. (1997). Computer anxiety

in young and older adults: Implications for human-

computer interactions in older populations. Comput-

ers in human behavior, 13(3):317–326.

Maylor, E. A. (1995). Prospective memory in normal ageing

and dementia. Neurocase, 1(3):285–289.

Newton-Dunn, H., Nakano, H., and Gibson, J. (2003).

Block jam: A tangible interface for interactive music.

Journal of New Music Research, 32(4):383–393.

Paradiso, J. A., Hsiao, K.-y., and Benbasat, A. (2001). Tan-

gible music interfaces using passive magnetic tags. In

Proceedings of the 2001 Conference on New Inter-

faces for Musical Expression, NIME ’01, page 1–4,

SGP. National University of Singapore.

Pearce, C. (2008). The truth about baby boomer gamers:

A study of over-forty computer game players. Games

and Culture, 3(2):142–174.

Rego, P. A., Moreira, P. M., and Reis, L. P. (2014). Archi-

tecture for serious games in health rehabilitation. In

New Perspectives in Information Systems and Tech-

nologies, volume 2, pages 307–317, Cham. Springer.

Salthouse, T. A. (1991). Mediation of adult age differ-

ences in cognition by reductions in working mem-

ory and speed of processing. Psychological Science,

2(3):179–183.

Schiettecatte, B. and Vanderdonckt, J. (2008). Audiocubes:

A distributed cube tangible interface based on interac-

tion range for sound design. In Proc. 2nd Int. Conf.

on Tangible and Embedded Interaction, TEI ’08, page

3–10, New York, NY, USA. Association for Comput-

ing Machinery.

Vandermaesen, M., De Weyer, T., Feys, P., Luyten, K., and

Coninx, K. (2016). Integrating serious games and tan-

gible objects for functional handgrip training: A user

study of handly in persons with multiple sclerosis. In

Proc. of the 2016 ACM Conf. on Designing Interactive

Systems, pages 924–935.

Wecker, N. S., Kramer, J. H., Hallam, B. J., and Delis, D. C.

(2005). Mental ﬂexibility: age effects on switching.

Neuropsychology, 19(3):345.

Zhao, X., Deng, Z., Jin, M., Li, X., Zhang, W., Jin, M.,

Ying, F., Wang, Q., and Wang, G. (2021). Segara:

Integrating serious games and handgrip for hand reha-

bilitation in rheumatoid arthritis patients. In The 9th

Int. Symp. of Chinese CHI, pages 101–104.

CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications

128