Songs in Music Education: Design and Early Experimentation of a Web

Tool for the Recognition of Harmonic Changes

Federico Avanzini

1 a

, Adriano Barat

1 b

, Luca A. Ludovico

1 c

and Marcella Mandanici

2 d

LIM – Laboratorio di Informatica Musicale, Dipartimento di Informatica “Giovanni Degli Antoni”,

Universit

a degli Studi di Milano, Via Giovanni Celoria 18, Milano, Italy

Conservatorio di Musica “Luca Marenzio”, Piazza Arturo Benedetti Michelangeli 1, Brescia, Italy

Keywords:

Online Music Education, Harmonic Rhythm, Free Web Platform, COVID-19.

Abstract:

This paper deals with Harmonic Touch, a Web platform designed to foster the practice of tonal harmony

also in young children. The work focuses on one of the experiences provided by the framework, namely the

gamiﬁcation of harmonic change recognition in songs. The platform, speciﬁcally equipped with new features

to accommodate the needs of teachers during the COVID-19 pandemic, has been tested in two Italian schools

in February 2021. Early experimental results about the main difﬁculties encountered by the children during

the games are presented and discussed.

1 INTRODUCTION

The Coronavirus disease 2019 (COVID-19) pan-

demic pointed out suddenly and with little room for

doubts the importance of the use of communication

and information technologies in educational contexts.

Teachers of all types and levels have been forced to

learn distance-teaching technologies and change their

working habits overnight, often without any prepara-

tion. As a consequence, learning technologies have

gained much more attention than before, and a grow-

ing number of teachers are eager to discover new

ways to integrate technology into their daily pedagog-

ical practices.

Moreover, the atmosphere of uncertainty that is

characterizing school activities makes the use of on-

line educational platforms extremely useful and func-

tional to various forms of blended learning. As a mat-

ter of fact, web-based tools allow regular school work

to be easily complemented with online activities out-

side the school, and, in case of school closure, they

can ﬁll the gap by ensuring continuity to educational

activities.

In this context, we have released a Web prototype

designed and implemented in a pre-COVID-19 era. It

https://orcid.org/0000-0002-1257-5878

https://orcid.org/0000-0001-8435-8373

https://orcid.org/0000-0002-8251-2231

https://orcid.org/0000-0003-1863-4178

adopts the principles of gamiﬁcation to foster the de-

velopment of a number of tonal-harmony abilities and

competences. The prototype has been tested in two

Italian schools in February 2021, in a period charac-

terized by lessons given partially in presence and par-

tially from remote. Primary-school teachers decided

to focus only on one of the experiences offered by

the platform. In this activity, well-known songs are

employed to make children detect harmonic changes,

i.e. the timed occurrence of new chords. The experi-

mental setup starts from the assumption that popular

music is a vital part of children’ lives. Although the

use of popular music in schools has aroused over the

years a wide debate among educators (Woody, 2007),

the changes in the educational approach, the necessity

of facing more informal education styles and of re-

shaping the contents of music programs are sufﬁcient

reasons to overcome any contraindications (Dunbar-

Hall and Wemyss, 2000; Hebert, 2011). Popular mu-

sic has the power of engaging children in social and

emotional learning, a set of skills that promote chil-

dren’ well-being and enhance cognitive performance

(Campayo–Mu

noz and Cabedo–Mas, 2017). Beyond

these important educational features, songs have a

clear and sound musical structure. Choruses, refrains

and bridges with their standardized functions and du-

rations help the perception of melodic and harmonic

elements, and for this reason are easy to spot and

memorize.

The remainder of the paper is structured as fol-

Avanzini, F., Baratè, A., Ludovico, L. and Mandanici, M.

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes.

DOI: 10.5220/0010540407090720

In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 709-720

ISBN: 978-989-758-502-9

709

lows. Without claiming to be exhaustive, Section 2

mentions other categories of Web tools for music ed-

ucation, and for tonal harmony speciﬁcally. Section 3

shortly describes the milestones that brought to the

current release of Harmonic Touch. Section 4 ex-

plains the role played by harmonic rhythm in the per-

ception of music. Section 5 presents the design princi-

ples and the graphical aspects of the Web interface for

the experience under exam. Section 6 describes the

early results collected in the ﬁrst month of experimen-

tation. As reported in Section 6.3, the platform has

been equipped with new materials and functionalities,

with customized work spaces for teachers and stu-

dents. User performances are shown in Section 6.4.

Section 7 discusses the results collected during the

early experimentation phase. Finally, Section 8 draws

the conclusions and presents the future perspectives

and the expected evolution of the project.

2 RELATED WORKS

The World Wide Web offers many platforms and tools

for music education. Based on their functionalities

and educational objectives, they can be subdivided

into various categories:

1. Online music studios, which provide users with

the functionalities of music sequencers, including

music loops and sounds for song creation (e.g.,

BandLab,

Soundtrap,

Groovy Music Cloud Edi-

tion, and Soundation

);

2. Music performance platforms, which offer some

practical tools to help students learn how to

play an instrument or sight-read music at dis-

tance (e.g., SmartMusic,

Doozzoo,

Practice-

First,

Sight Reading Factory,

and Yousician

);

3. General music education platforms for devel-

oping listening skills, theory, and composition

(e.g., BrainPOP Arts and Music,

PBS KIDS Mu-

sic Games,

Classics for Kids,

and Focus on

Sound

);

https://edu.bandlab.com/

https://www.soundtrap.com/edu/

https://soundation.com/

https://www.smartmusic.com/

https://doozzoo.com/

https://www.musicﬁrst.com/applications/practiceﬁrst/

https://www.sightreadingfactory.com/

https://yousician.com/

https://www.brainpop.com/artsandmusic/

https://pbskids.org/games/music/

https://www.classicsforkids.com/

https://www.focusonsound.com/

4. Music notation and creation platforms (e.g.,

Flat,

Noteﬂight,

and O-Generator

);

5. Music theory and ear training platforms (e.g.,

EarBeater,

Musictheory.net,

and Musition

Even if the present review is far from being exhaus-

tive, it can provide a broad idea of the educational

offer for music education currently available on the

Web.

The ﬁrst observation is that nearly all the men-

tioned platforms are not free, and, consequently, they

require the purchase of a premium access to fully ben-

eﬁt of the services offered. If, on one side, paid ser-

vices can guarantee a good quality of contents and

performances, on the other side they do not respond

to the needs of students and teachers who still need to

be encouraged in the use of learning technologies.

As far as tonal harmony is concerned, theoretical

knowledge is usually embedded inside music theory

programs. The typical approach of web platform is to

provide an online version of traditional music theory

books, possibly augmented through the integration of

interactive tools for chord calculation or identiﬁca-

tion. The targeted audience is made of music-theory

students engaged in formal learning programs. For

this reason, such tools can be proﬁtably applied to the

academic study of harmony, but they are not suitable

for children or adults interested in understanding the

phenomenon rather than the theory behind it.

A less formal approach is employed in the Hook-

pad Musical Sketchpad,

a software where the

chords of a song are displayed under a piano-roll win-

dow by employing traditional chord notation and dif-

ferent color codes for each chord. The piano-roll in-

terface can be modiﬁed by changing pitches, keys,

meter, tempo and chord progressions.

Another interesting tool for the study of harmony

is Mapping Tonal Harmony,

a software which dis-

plays in real time the movement of chords in the har-

monic space. It can also be used for very complex

musical pieces both in classic and jazz style, but it is

suitable for professional rather than amateur use.

https://ﬂat.io/

https://www.noteﬂight.com/

http://www.o-generator.com/

https://www.earbeater.com/

https://www.musictheory.net/

https://www.risingsoftware.com/musition

https://www.hooktheory.com/hookpad

https://mdecks.com/mapharmony.phtml

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

710

3 BACKGROUND WORK

Under the pressure of the dramatic uncertainty and

discontinuity that is characterizing the life of school

institutions in Italy, we publicly released Harmonic

Touch, a Web platform designed and implemented a

few months before COVID-19 pandemic (Avanzini

et al., 2020). The original idea had been presented and

discussed during a workshop titled “Didattica della

musica e linguaggi digitali” (i.e., music education and

digital languages) in occasion of the 2

edition of

Fiera Didacta Italy, Florence, October 18-20, 2018

(Avanzini et al., 2019a). The audience, mainly com-

posed by primary and lower-secondary school teach-

ers, had the opportunity to test the early prototype and

make remarks that guided a later re-implementation.

Harmonic Touch aims to drive the attention of

teachers towards the possibility of using tonal har-

mony features in music education even from a very

early age. In order to achieve this goal, the plat-

form offers a number of activities following the prin-

ciples of gamiﬁcation. The user is not pushed towards

gaining theoretical knowledge, as in formal music

education, but rather to develop harmony awareness

through musical perception.

The platform currently presents 3 different expe-

riences:

1. The ﬁrst experience deals with the perception of

implied harmony, where the user is asked to lis-

ten to a music excerpt and try to pick the implied

harmony - the tonic - from a set of 6 chords repre-

senting the harmonic space. The implied harmony

is the chord that best ﬁts all the melody excerpt;

2. The second experience deals with the detection

of the onsets of the harmonic changes under a

melody while the tune is playing. This kind of ex-

ercise is the focus of the present paper, so it will

be described in detail in Section 5.2;

3. The third experience consists in melody harmo-

nization. After carefully listening to a melody and

exploring the harmonic space, the user is asked

to click on the chord sequence that best ﬁt the

melody. This type of exercise is a combination

of the previous two. In our vision, it represents

the ﬁnal step on the way towards the development

of tonal harmony awareness.

For this experimental session, we decided to focus our

attention on the second experience, namely the recog-

nition of harmonic changes. Being the most playful

thanks to its gamiﬁcation features, it has been pos-

itively accepted by school teachers and enthusiasti-

cally welcomed by students.

4 HARMONIC RHYTHM

Harmonic rhythm plays an important role in rhythmic

and melodic perception, as well as in the segmenta-

tion process that governs the way listeners interpret

and group surface musical events (e.g. the notes of a

melody).

Harmonic rhythm has been deﬁned by as the “un-

derlying changes in harmony” (Burkat, 1944) or “the

perception of rhythm that depends on changes in as-

pects of harmony” (Swain, 2002). Although the lat-

ter deﬁnition implies that the perception of harmonic

changes may depend on various musical elements,

such as timbre, musical style, the leading of the bass

line, chord density and texture, for the purposes of

the educational activities described in this work we

focus only on two main aspects of harmonic rhythm.

As described by Dawe (Dawe et al., 1993), harmonic

rhythm can be considered in two different ways: as

the sequence of the onsets of different harmonies, or

as a “composite rhythm” that is formed by a sequence

of events belonging to the same chord. This is the case

of the melody of the nursery rhyme “London bridge”

which, despite its simplicity, shows an unexpectedly

complex underlying structure. The arcs in Figure

1 embrace the 2 semiphrases that form the melody;

they are perfectly regular and symmetric. The under-

lying structure of the harmonic rhythm, conversely,

presents different durations: 2 bars tonic (T), 1 bar

dominant (D) and, again, 1 bar tonic (T). The same

structure is repeated in the second semiphrase. How-

ever, as the same chord (T) ends the ﬁrst semiphrase

and begins the second, the overall duration of the

third chord is 3 bars. Thus, a regular 4 + 4 struc-

ture is superimposed to an irregular harmonic rhythm

(2 + 1 + 3 + 1 + 1). This is due to the effect of com-

posite rhythm (shown in brackets in Figure 1), which

happens whenever the rhythmic accent expected at

the beginning of a new bar is not matched with a

change in the harmony. Examples are provided in

bars 2, 5, and 6 of the excerpt. The regular structure

of the 2 semiphrases makes the rhythmic accent at

bar 5 particularly important but, in spite of this, there

is no change in the harmonic rhythm. This example

demonstrates how the perception of a simple nursery

rhyme can hide a complex harmonic structure, and

what is the role of harmonic rhythm in building such

a richness of musical events.

As theorized by Lerdahl and Jackendoff (Lerdahl

and Jackendoff, 1996), tonal music is built through

the superimposition of 4 main layers, each contain-

ing different musical elements. The most superﬁ-

cial is the grouping structure, which refers to the

patterns created by the notes of the melody (themes

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes

711

7 8654321

























D T

G C(C)(C)

G(C)

LondonBridge





Figure 1: Harmonic changes of “London bridge”. Slurs embrace 4-bar semiphrases. The lines below chord symbols indicate

the occurrence of chord changes and the extension of the corresponding harmonic areas. Composite rhythm is expressed by

bracketed chord indications at bars 2, 5 and 6.

and phrases). By applying the Gestalt theory prin-

ciples of similarity, proximity and symmetry (Ellis,

1999), listeners are able to partition a melody into

small segments according to the perception of beat

accents and short-term memory (Bigand, 1993). At

lower levels, there are the metrical structure, which

considers the organization of strong and weak beats,

and the time-span reduction and the prolongational

reduction, which represent structural accents of mu-

sical events and tension-relaxation dynamics, respec-

tively (Hansen, 1011).

The process of melody segmentation is not only a

cognitive representation of tonal melody. It has also

been employed as a computational model for the de-

scription of musical activities and for the accomplish-

ment of musical tasks, such as music query and mu-

sic information retrieval (Zhu and Kankanhalli, 2004;

Hirai and Sawada, 2019).

In the case of the second experience of Harmonic

Touch, we provide a segmentation of the melody

based on the occurrence of a new chord instead on the

metrical organization of the melody. As shown by the

case of “London bridge”, this can increase the level

of complexity in the musical perception, forcing the

listener to distinguish between metric and harmonic

accents. This is one of the skills targeted by the sec-

ond experience of Harmonic Touch.

5 THE WEB TOOL

Harmonic Touch is a free and cross-platform tool, cur-

rently supporting Italian and English language. It is

available at the following URL: http://harmonictouch.

lim.di.unimi.it/.

The front-end section and the underlying database

that manages accounts and gathers performance data

have already been designed and released. The back-

end area at the moment of writing is still under devel-

opment, currently supporting basic account options

(such as password change) and, for teacher accounts

only, the conﬁguration of classes and the selection

of exercises. The personal preparation and upload of

new materials through the Web interface, very useful

functions in a crowd-sourcing perspective, have not

been developed yet.

5.1 User Accounts

The landing page lets the user enter his/her creden-

tials. Different user types are supported: an adminis-

tration account reserved for developers, a test account

for authorized beta testers, an anonymous account, a

teacher account, and, ﬁnally, a student account. For

the goals of this paper, only the last three types will

be described.

The teacher account grants access to all the experi-

ences and exercises, so that the educator can listen and

test them in order to select the most suitable ones. The

back-ofﬁce area lets the teacher conﬁgure each class

and assign it a speciﬁc subset of exercises, also deter-

mining the level of difﬁculty for the exercise. Even

if a default difﬁculty level has been assigned by do-

main experts to each piece belonging to the second

experience, such a value is currently under investiga-

tion and should be considered a placeholder; deter-

mining which aspects (e.g., the length of the piece, its

tempo, the regularity in harmonic changes, the type of

chords, etc.) have an impact on the perceived level of

difﬁculty is one of the goals of the experimentation,

as discussed later.

The student account gives access to the experi-

ences and exercises selected by the teacher for the

class the student belongs to. In other words, a student

can see only a speciﬁc subset of the song dataset.

Finally, the anonymous account has been con-

ceived to let a generic user experience the platform in

all its aspects and exercises with no need to be regis-

tered. Performance data from anonymous players are

collected as well, but, in general, they do not allow

to make inferences, since the account is not linked to

a single user, and the aims of accesses can be very

heterogeneous, ranging from curiosity to the explo-

ration of otherwise protected materials. Concerning

the latter aspect, teachers involved in early experi-

mentation discovered that their students had learned

how to overcome the limited piece selection imposed

for their class by using the anonymous account. In

the future, also the anonymous account will present a

small selection of songs for demonstrative purposes.

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

712

5.2 Game Play for Experience No. 2

In order to play Experience No. 2, the user has to com-

plete a 4-step process: i) piece selection, ii) pre-game

listening activity, iii) game play, and iv) game results.

These steps are shown in Figure 2.

After selecting a piece from the list, in the second

step users can listen to it as long as they need in order

to get acquainted with the tune and try to locate chord

changes.

The gaming experience starts in the next step. The

audio track has been segmented into n segments, each

one corresponding to a musical chord; consequently,

Figure 2: From top to bottom, the 4 steps for Experience

No. 2.

harmonic changes are n − 1. The interface presents a

treasure map with a dashed path connecting n nodes.

The ﬁrst node, not numbered, serves as the start but-

ton; the remaining nodes are in correspondence with

timed harmonic changes. When the user clicks over a

node, music is played from that time position.

The ideal behavior, i.e. the one that brings the

player to win with the maximum score, consists in

clicking over nodes step by step, in the right order and

at the right moment (namely when the chord change

occurs). A tolerance of ±0.5 s centered on the cor-

rect timing has been introduced to manage delays and

avoid frustration. Tolerance deﬁnes the range within

which user’s click may fall while still being accept-

able.

When trying to detect the harmonic change, the

user has two possibilities of error: anticipated and

delayed clicking. The former case comes from the

perception of a chord change when the tonal area re-

mains unaltered, and it implies a sudden jump to the

new segment of the song (often, the step right ahead

the current one in the dashed path). The latter scenario

refers to the opposite situation, namely the occurrence

of a harmonic change not detected by the user; in this

case, after a short fade-out effect, music stops waiting

for any user action. Please note that, while anticipated

clicking is a type of error that the platform simply

acknowledges (by playing an error sound, forcing a

jump in playback, and decreasing the score), delayed

clicking becomes evident to the user due to the ab-

sence of sound, thus pushing to take an action.

In a gamiﬁcation approach, score is a typical

means to increase engagement and push players to-

wards better performances. For this reason, each

piece has been associated with a maximum score

s = 100 · (n − 1) points, where n is the total number

of tonal areas in the song. A click over the right node

within the ±0.5 tolerance interval adds a score which

is still positive, but reduced on the base of the abso-

lute distance from the right timing: each 0.1 s, the

100-points original amount is cut by 25 points. Click-

ing outside the tolerance window and/or on a wrong

node implies a 10-point penalty. Values of time toler-

ance and score increments/decrements are parametric,

so, in a future evolution of the platform, they could be

customized by the teacher.

The game board presents buttons to conﬁrm re-

sults, restart the match and go back to the pre-game

listening activity. In all these cases, user perfor-

mances are saved in a database where players are

identiﬁed by their nickname, class and school only,

so as to protect their privacy; no personal data nor any

other sensitive information are saved in the database.

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes

713

6 EXPERIMENTATION

The experimentation started in two Italian primary

schools in February 2021, in the midst of the resur-

gence of the COVID-19 pandemic. The schools in-

volved were “Istituto Comprensivo Lucilio”, Sessa

Aurunca, Caserta, and “Istituto Ancelle della Carit

a”,

Palazzolo sull’Oglio, Brescia. The experimental ac-

tivities started on February 4

at “I.C. Lucilio” and

on February 9

at “Istituto Ancelle della Carit

a”.

Even if educational activities started in presence,

by the end of the month many schools were forced to

switch to remote teaching. This fact necessarily inter-

fered with the experimental plan, which included:

• an introductory lesson where teachers had to ex-

plain basic concepts about melody and harmony

and show students how to use the platform;

• a pre-test session where the initial level of chil-

dren has been recorded through in-class games

with Songs No. 1, 2, and 3.

• a number of lessons where the games on the plat-

form should have been accompanied by explana-

tions by the teachers and face-to face and physical

activities, as described in (Mandanici et al., 2019);

• a ﬁnal post-test, planned at the end of the fourth

week of activity in order to evaluate the advances

obtained.

Due to the forced switch to remote teaching, only a

part of the pre-test phase could be realized in presence

(see Figure 3), while the experimentation – currently

under way at distance – will miss all the planned face-

to-face activities.

Anyway, Harmonic Touch is a Web-based frame-

work, consequently it is able to support different use

modes without changing the interaction paradigm,

thus reducing the differences that exist between su-

pervised in-class activities, supervised out-of-school

activities, and unsupervised home activities.

6.1 Aims

For the purposes of the present paper and in the un-

certainty of being able to continue and control the ex-

perimentation in a remote teaching situation, we con-

sider pre-test results (obtained during the ﬁrst lesson

in each school) and the data gathered during unsuper-

vised experimentation in a 7-day period after the date

of ﬁrst access.

The analysis concerns three main research ques-

tions:

RQ1 the evaluation of user performances for each

song;

Figure 3: In-class activities during the pre-test phase.

RQ2 the evaluation of user performances for each

harmonic change in relation to the musical char-

acteristics of the song excerpt;

RQ3 the evaluation of the efﬁcacy of the platform in

extending school work at home.

RQ1 focuses on both personal and aggregated results

achieved by players song by song. Even if this ques-

tion apparently addresses how well players have per-

formed, the real goal is understanding what principles

can drive the choice of song excerpts to be employed

both in pre-test and in unsupervised game sessions.

Game-play data, gathered for the ﬁrst time in a ﬁeld

experimentation, can help to unveil the musical char-

acteristics that inﬂuence the perception of difﬁculty.

In preparing materials for the second experience, we

aprioristically assumed that the perceived degree of

difﬁculty could depend on the following parameters:

excerpt duration, number of harmonic changes, aver-

age frequency of changes, harmonic complexity, ir-

regularity of the harmonic rhythm. Answering RQ1

implies a better understanding of such mechanisms.

RQ2 focuses, once again, on both personal and ag-

gregated results achieved by players, but, in this case,

at a ﬁner level of detail: user performances are as-

sessed on single chord changes. Understanding the

origin of poor results requires the analysis of chord

models, the regularity in harmonic changes, and the

extension of tonal areas. Harmonic changes where

children encountered higher difﬁculties can give some

useful hints to steer the didactic action and to set the

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

714

1211

1098

1413

1211

1098

543



 



 



 



 















 







 



 



 



 









 



 











 































 























 



















 

 

















 







































































342.11

220.47

(E)

234.00

Fm6

414.70

168.00

Aadd9

343.69

434.36

170.74

B(E)

(E)

151.84

105.21

Fm6

100.58

Am7

408.40

(E)

141.05

B♭m

C♯m7

290.01

196.79

B♭m6

366.00

E(A)

(E)

272.45

174.48

C♯m7

279.54

162.33

Fm6

546.40

(E)E

164.23

440.38

217.35

Fm6

2614.36

112.55

C6/dim5

191.22

Fm6

336.03

340.44

286.93359.96

ADm

329.98

B♭

313.34

390.79

1107.86

791.00

Dm(G)

699.18

701.31

(F)

3350.17

F(Dm)(Dm)(Dm)

1037.94

598.35

726.10

267.76

(A7)A7

356.73

303.90

239.83263.50

B♭G

349.37

FDm

498.62

Dm(G)

549.74

G(F)

753.56

554.41

496.53

(Dm)Dm

606.14

EmC

6.IWanttoBreakFree

5.Comeunfurbotopolino

4.Oceania-HowfarI'llgo(Italianversion)

3.HouseoftheRisingSun

2.Sayonara

1.SomewhereOvertheRainbow







Figure 4: Scores of the songs used in the experimentation. The lines below chord symbols indicate the occurrence of chord

changes and the extension of the corresponding harmonic areas. Composite rhythm is expressed by bracketed chord indica-

tions. Small numbers below indicate the mean of the absolute values of errors (anticipations and delays) expressed in ms. The

ﬁrst chord has no number associated since it corresponds to the initial play event, so the error is always null.

post-test materials accordingly.

Finally, RQ3 refers to the potential of the platform

to extend the school activities also at home. This is

a relevant goal in a period where in-presence activi-

ties are so discontinuous and uncertain. The platform

can play a unifying role between face-to-face teach-

ing and online activities, thus mitigating the incon-

veniences and the difﬁculties of the COVID-19 pan-

demic. In this sense, it is fundamental to measure

also the level of engagement, so as to push students

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes

715

towards personal goals and avoid the sense of frustra-

tion due to repeated failures.

6.2 Subjects

The experimentation has involved so far 46 children

(29 females) from two Italian primary schools: 21

students (15 females) from “Istituto Comprensivo Lu-

cilio” and 25 students (14 females) from “Istituto An-

celle della Carit

a”. The children of the former school

are attending 2 classes of the 4

grade, the students

of the latter are attending 2 classes of the 5

grade.

Their age is comprised between 8 and 11 years.

The two student populations are comparable

concerning age, educational goals, previous music

knowledge, etc., so their performances have been

evaluated without considering their different origin.

6.3 Materials

Music excerpts have been chosen by teachers based

on their educational goals, the characteristics of stu-

dents, and considerations related to amusement and

engagement. The songs selected for the pre-test phase

and for the games to be played at home are listed

in Table 1: they cover popular music (Songs No. 1,

3 and 6), nursery rhymes (Songs No. 2 and 5) and

animation-movie soundtracks (Songs No. 4). Even if

children are expected to be already familiar with most

of them, a pre-game listening phase is available for

each song.

While Songs No. 1, 4, and 5 show a regular pace

in harmonic changes, Songs No. 2, 3, and 6 are char-

acterized by different types of irregularities (see Fig-

ure 4).

Song No. 2 presents harmonic changes every two

bars and at a slow tempo, but introduces a prolonged

Dm chord across the phrase that begins at bar 9.

Moreover, in the ﬁrst beat of bar 9 the audio track

presents a strong percussive sound, which makes it

very difﬁcult for children to resist the temptation to

push the button, even if there is no harmonic change.

Song No. 3 is rather regular except for bar 4,

where the harmonic rhythm stops for a whole bar

on the dominant chord (A7). However this structure

does not conﬂict with the metrical organization of the

melody.

Song No. 6 is characterized by a very unusual

structure, with a very long tonic chord (E) at the be-

ginning, lasting 6 bars. Then a more regular repetition

of 2-bar harmonies (A, E) is followed by bars 11, 12

Please note that music pieces can be listened by en-

tering the platform with the anonymous account, choosing

Experience No. 2 and selecting them from the complete list.

and 13 each with one harmonic change. Moreover

Song No. 6 has the slowest pace of chord changes.

6.4 User Performances

The assessment of user performances is a non-trivial

problem. Which aspects should be measured? What

time offset should be tolerated to consider a real-

time user action as correct? Is the set of parame-

ters presented in Section 6.1 suitable to describe the

perceived level of difﬁculty of a song? These prob-

lems, partially addressed in a pre-experimental phase

(Avanzini et al., 2019b), will be re-evaluated in light

of experimental results.

Detailed data about user performances are avail-

able at http://harmonictouch.lim.di.unimi.it/results/

early/lucilio.html for “Istituto Comprensivo Lu-

cilio”, and at http://harmonictouch.lim.di.unimi.it/

results/early/ancelle.html for “Istituto Ancelle della

Carit

a”. In these pages, results are organized user by

user. Each diagram represents a game, each trial is

one of the attempts made to win in a single game.

At the end of such pages, synoptic tables for each

song are reported, with detailed information about

chord changes: the average error, the variance, the

maximum error due to anticipated (highest negative

value) and delayed (highest positive value) clicking,

the mean of absolute errors, the number of positive

errors, the mean of positive errors only, the number of

negative errors, and the mean of negative errors only.

For the sake of brevity, only the most relevant data

will be presented and discussed in the following.

Concerning RQ1, we analyze the data collected in

both schools, paying particular attention to the aver-

age error obtained during school pre-test and home

activities. In order to obtain a single value for each

song, the process consisted in calculating the average

error measured on all chord changes as detected by

the students of both schools (see Figure 5). The un-

derlying idea was to ﬁnd an aggregate indicator for

the level of difﬁculty experienced by students: abso-

lute values closer to 0 should indicate exercises where

the identiﬁcation of chord changes is simpler; never-

theless, it is worth underlining that the reasons can be

heterogeneous.

The piece that reports the highest error is Song No.

2, followed by Song No. 6, while the piece where chil-

dren performed best is Song No. 5. From the analysis

of these results, difﬁculty does not seem to depend on

the number of harmonic changes (Song No. 2 has the

lowest value) nor on harmonic complexity (Songs No.

2 and 6 employ the smallest number of harmonies),

but, rather, on the average duration of the chords µ

(see Table 1).

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

716

Table 1: The songs employed in the pre-test and for home activities: d is the duration of the song excerpt in seconds, n

is the

number of harmonic changes, µ is the average duration of chords in seconds, n

is the number of different harmonies, and r is

a Boolean value concerning the regularity of harmonic changes. Other columns are self-explanatory.

phase song no. title year d key n

µ n

pre-test 1 Over the Rainbow 1990 25.4 C 8 2.82 5 yes

pre-test 2 Sayonara 2009 31.0 Dm 6 4.43 3 no

pre-test 3 House of the Rising Sun 1964 27.2 Dm 13 1.94 5 no

home 4 Oceania 2016 23.7 E 8 2.63 4 yes

home 5 Come un furbo topolino 2009 18.2 C 8 1.14 6 yes

home 6 I Want to Break Free 1984 30.5 E 8 5.08 3 no

Figure 5: Mean of the (absolute) errors on chord changes measured song by song. The tallest bar is associated with the hardest

exercise (Song No. 2), the shortest one with the easiest exercise (Song No. 5).

Songs No. 2 and 6 present the largest average du-

rations µ. Surprisingly, Song No. 3, whose number

of harmonic changes is signiﬁcantly higher than the

other songs, obtains the second best score with an er-

ror of 51.97 ms. Song No. 3 has also an irregularity

in the harmonic rhythm at bar 4, where the A7 chord

stops for 2 beats; but, in this case, there is no conﬂict

with the metrical organization of the melody. Any-

way, the next change after the irregularity – occurring

at bar 5 – obtains the maximum error of the whole

song: 1107.86 ms.

On the other hand Song No. 5 – although charac-

terized by a regular harmonic rhythm – has the highest

level of harmonic complexity and the smallest aver-

age chord duration (µ =1.14 s). Evidently the regu-

larity combined with the high speed of change plays

a decisive role in the perception harmonic accents.

In order to answer RQ2, we analyze the occur-

rence of the harmonic changes in Songs No. 2 and

6 where children collected the most relevant errors.

For Song No. 2, the highest error is at the 4

har-

monic change, equal to 3350.17 ms (see Figure 4).

This is due to the effect of the superposition of har-

monic rhythm (4 bars of Dm from bar 7 to 10) with

the metric accent of bar 9. For Song No. 6, the most

relevant error, equal to 2614.36 ms, occurs at the 1

harmonic change. This is due to the confusing effect

of the very long tonic chord (6 bars of E from bar 1 to

6) which makes it hard for children to locate the next

change at bar 7.

RQ3 can be addressed by comparing the number

of games played during the pre-test sessions and the

games played at home in the 7 days after the pre-test.

We have limited our investigation to one week since it

was the shortest interval before the following lesson.

The pre-test is also the lesson when children experi-

enced the ﬁrst access to the platform, while, from the

afternoon onward, they were free to play whenever

they wanted.

Table 2 shows the statistics of the games played

during the pre-test and at home. Although there is a

remarkable difference between the two schools in the

number of users who autonomously accessed the plat-

form at home, it is interesting to note that the overall

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes

717

percentage of games played at home is not less than

64% of those played in the pre-test. The percentage

of successful games is much better at home than at

school for the students of ‘Istituto Comprensivo Lu-

cilio” (2% at school, 12% at home), while it is nearly

the same for the students of “Istituto Ancelle della

Carit

a” (15% at school, 14% at home). These results

seem to indicate that the game has greatly engaged

children, encouraging some of them to play even out-

side school hours. The gamiﬁcation approach has

pushed them to experiment autonomously, in some

cases obtaining much better performances.

7 DISCUSSION

While experimental activities are still running and in

the uncertainty of its progress, we can only guess

what the trends will be once the experimentation is

complete. However, from these early results, we can

assume that the real difﬁculty in the detection of har-

monic changes is mainly connected to the average du-

ration of the chords µ, or equivalently to the speed of

changes.

In the choice of the materials for the pre-test we

tried to avoid songs with frequent harmonic changes,

considered too challenging for children. Unexpect-

edly, slow changes seem to be more problematic,

mainly when they conﬂict with the metrical structure

of the melody, as it happens in Song No. 2.

Also irregularities in the changes and in the har-

monic structure do not seem to have a deep impact

on success rate. Actually, looking at the errors of

Song No. 6, there is a peak in the ﬁrst change, but off-

sets dramatically decrease as soon as the song takes a

more regular trend. Song No. 3, even if marked with

a (low) level of irregularity, obtains a good score and

performs better than regular songs such as No. 1 and

On the other hand the successes recorded for Song

No. 5 – with a speed of harmonic changes approach-

ing the second – suggest that the timing is a critical

point for deciding the levels of difﬁculty for the exer-

cises of this second experience of Harmonic Touch.

Table 2: User performances collected during in-class pre-

test and home activities.

Lucilio Ancelle

pre-test home pre-test home

Users 22 7 19 14

Games 92 59 488 384

Won 2 9 59 55

Lost 90 50 429 329

The trend that emerges is that a problematic is-

sue is the inability to predict the occurrence of har-

monic changes when distant in time. This prolonged

standby seems to leave the listener without reference

points, causing errors as soon as the next harmonic

change occurs. Consequently, most teaching efforts

in the 4 weeks of experimentation before the post-test

should address the sensitization of the various cases

of composite harmonic rhythm, mainly when it con-

trasts with the metrical structure of the song. This

goal can be achieved through user-tailored exercises,

listening activities, and games on songs with char-

acteristics similar to those proposed in the pre-test.

Also physical games with different groups of children

or different parts of the body marking metrical and

harmonic accents could help, but these activities are

clearly hampered by the current pandemic emergency.

Finally, data about the use of the platform are very

encouraging. Soon after enabling user accounts, some

students began to play independently outside of class

hours, as revealed by the timestamps of game ses-

sions recorded in the database. Another encourag-

ing aspect is the request made by children to increase

the song dataset available. These facts bode well for

the continuation of educational activities even at dis-

tance, and they are an indicator of the engagement

level. At the moment we are unable to predict if

the accesses outside school time will increase or de-

crease as the experimentation proceeds, also because

teaching at distance breaks the traditional division be-

tween school hours and extracurricular activities. Ac-

tually, what is emerging is a new learning structure

where face-to-face activities are increasingly medi-

ated by virtual learning environments. This allows not

only the overcoming of the traditional organization of

time and space dedicated to educational activities, but

changes their inner nature, helping to design new or-

ganizational models and educational practices.

8 CONCLUSION AND FURTHER

WORK

In this paper we have presented one of the compo-

nents of the Harmonic Touch platform, whose gen-

eral goal is to raise awareness about tonal harmony

in young students through a gamiﬁcation approach.

The tool under exam is an experience to help users to

detect harmonic changes in songs, namely the occur-

rence of new chords. The interface has been designed

as a treasure map, with a number of steps (corre-

sponding to new chords) to complete at the right time

(corresponding to harmonic changes in audio) in or-

der to reach the ﬁnal goal. We have presented and dis-

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

718

cussed the early results gathered with primary school

students aged 8 to 11 and coming from 4 classes of 2

Italian institutes.

Regarding future works, there are multiple di-

rections to take. The ﬁrst one concerns the exten-

sion of experimental activities to a wider audience

of users, potentially presenting different characteris-

tics in terms of age, school grade, geographic ori-

gin, previous musical experience and skills, and so on.

Other school teachers interested in the project have al-

ready been contacted, and only the uncertainty due to

the COVID-19 pandemic has stopped the launch of a

larger-scale experimentation. Gathering more data is

expected to make more statistically signiﬁcant trends

emerge.

Other considerations about the suitability of the

platform for distant education are likely to emerge in

the near future. In fact, so far the platform has been

used either in class, under the direct supervision of

teachers, or at home, in a relaxed out-of-school envi-

ronment. A third way is not only possible, but also

probable: a new lockdown for Italian schools could

foster the adoption of the platform for remote and

synchronous school activities. In this case, it will be

interesting to analyze student performances and com-

pare them to the previously mentioned scenarios.

Another direction for future work concerns plat-

form design and implementation. Field trials high-

lighted a number of minor bugs, promptly exploited

by smarter students to artiﬁcially improve their re-

sults. In this sense, our beta testers demonstrated to

be very motivated to cheat the system. An example

was the extremely quick repetition of mouse clicks

over the current change in order to get additional score

points, an effect obtained by exploiting the very nar-

row tolerance window introduced to manage clicks

not perfectly timed. This bug was solved as soon as it

was detected by reading data; anyway, it did not inval-

idate performance analysis since we did not base our

considerations on score points, which are presented in

the interface only to improve motivation. Concerning

the latter aspect, in a future release we are planning

to further enhance gamiﬁcation aspects, e.g. by in-

troducing hall-of-fame and leaderboard functions in

order to increase user’s engagement. One interesting

remark that came from one teacher is that, in the ab-

sence of such features, students started to share their

scores by taking pictures of their game results and

sending them to the class chat.

An aspect to further investigate concerns the evo-

lution of the engagement level when students, on one

side, get accustomed to the game play, and, on the

other side, have to face harder exercises. The data

collected so far do not allow to make inferences about

the long-term trend and understand to what extent stu-

dent engagement is due to the novelty of the platform

rather than the idea itself.

From the educators’ perspective, the availability

of a web tool to support music-education activities

has been appreciated by the teachers who decided to

take part into this initiative. Anyway, being an early

experimentation that involved 4 classes only, we can-

not make general statements about teacher’s accep-

tance and level of satisfaction. This aspect will be

further investigated in future experimentation, that is

planned to involve a much higher number of schools

and classes.

The song dataset, currently constituted by 44

pieces, will be further extended and customized

thanks to the contribution of educators, based on their

educational goals. New pieces can be prepared ofﬂine

by teachers, but, at present, they can be added to the

platform only by system administrators. The working

group is also planning to implement back-end tools to

let teachers upload their materials independently.

ACKNOWLEDGEMENTS

We wish to thank Giuseppina Verile and Ivana Lan-

zanova, the principals of the “Istituto Comprensivo

Lucilio”, Sessa Aurunca, Caserta (Italy) and of the

“Istituto Ancelle della Carit

a”, Palazzolo sull’Oglio,

Brescia (Italy), respectively. For their active coopera-

tion in the experimental activities, we are also grateful

to Sonia Vezza, teacher, and Maurizio Botti, Maura

Castelnovo, Ottavia Marini and Sara Valente, students

of the Master Degree in Technologies for Music Edu-

cation of the Conservatory “Luca Marenzio”, Brescia

(Italy).

REFERENCES

Avanzini, F., Barat

e, A., Ludovico, L. A., and Mandanici,

M. (2019a). A computer-based approach to teach

tonal harmony to young students. In Proceedings of

the 11

International Conference on Computer Sup-

ported Education (CSEDU 2019), volume 1, pages

271–279. SCITEPRESS - Science and Technology

Publications, Lda.

Avanzini, F., Barat

e, A., Ludovico, L. A., and Mandanici,

M. (2019b). Metrics for the automatic assessment of

music harmony awareness in children. In Proceed-

ings of the 16

Sound & Music Computing Confer-

ence (SMC 2019), pages 372–379. SMC.

Avanzini, F., Barat

e, A., Ludovico, L. A., and Mandanici,

M. (2020). A web platform to foster and assess tonal

harmony awareness. In Computer Supported Edu-

Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes

719

cation. 11

International Conference, CSEDU 2019,

volume 1220 of Communications in Computer and In-

formation Science, pages 398–417. Springer Interna-

tional Publishing, Cham.

Bigand, E. (1993). The inﬂuence of implicit har-

mony, rhythm and musical training on the abstrac-

tion of “tension-relaxation schemas” in tonal musical

phrases. Contemporary Music Review, 9(1-2):123–

137.

Burkat, L. (1944). Harvard dictionary of music.

Campayo–Mu

noz, E. and Cabedo–Mas, A. (2017). The role

of emotional skills in music education. British Journal

of Music Education, 34(3):243–258.

Dawe, L. A., Platf, J. R., and Racine, R. J. (1993). Har-

monic accents in inference of metrical structure and

perception of rhythm patterns. Perception & psy-

chophysics, 54(6):794–807.

Dunbar-Hall, P. and Wemyss, K. (2000). The effects of the

study of popular music on music education. Interna-

tional Journal of Music Education, 36:23–34.

Ellis, W. D. (1999). A source book of Gestalt psychology,

volume 2. Psychology Press.

Hansen, N. C. (2010/11). The legacy of Lerdahl and Jack-

endoff’s ‘A generative theory of tonal music’. Danish

Yearbook of Musicology, 38.

Hebert, D. G. (2011). Originality and institutionalization:

Factors engendering resistance to popular music ped-

agogy in the usa. Music Education Research Interna-

tional, 5:12–21.

Hirai, T. and Sawada, S. (2019). Melody2vec: Distributed

representations of melodic phrases based on melody

segmentation. Journal of Information Processing,

27:278–286.

Lerdahl, F. and Jackendoff, R. S. (1996). A Generative The-

ory of Tonal Music. MIT press.

Mandanici, M., Ludovico, L. A., Avanzini, F., and Barat

A. (2019). Learning tonal harmony through bodily

interactions and gamiﬁcation. In MERYC19. Coun-

terpoints of the senses. Bodily experiences in music

learning, pages 237–246. Ghent University.

Swain, J. P. (2002). Harmonic rhythm: Analysis and inter-

pretation. Oxford University Press.

Woody, R. H. (2007). Popular music in school: Remixing

the issues. Music Educators Journal, 93(4):32–37.

Zhu, Y. and Kankanhalli, M. (2004). Key-based melody

segmentation for popular songs. In Proceedings of the

International Conference on Pattern Recognition,

2004. ICPR 2004., volume 3, pages 862–865. IEEE.

CSME 2021 - 2nd International Special Session on Computer Supported Music Education

720