An IEEE 1599 Framework to Play Music Intuitively

The Metapiano Case Study

Luca Andrea Ludovico

and Mario Malcangi

Laboratorio di Informatica Musicale (LIM), Milan, Italy

Digital Signal Processing & Real-Time Systems (DSPRTS), Dipartimento di Informatica,

Universit

a degli Studi di Milano, Via Comelico 39, Milan, Italy

Keywords:

Education, IEEE 1599, Music Meta-instruments.

Abstract:

This work aims at proposing an innovative way to approach music education. The idea is coupling the power

of IEEE 1599, an XML-based international standard for music description, to the concept of music meta-

instruments, namely new interfaces conceived for a simpliﬁed interaction with music contents. The proposed

framework will provide a tool for music practice, powered by the multiple and heterogeneous contents con-

tained in an IEEE 1599 document. A case study based on Jean Haury’s metapiano will be presented.

1 INTRODUCTION

Music education is evolving in several directions us-

ing electronic, information, and communication as

enabling technologies to make the teaching process

more effective and to enhance the human ability to

learn. Computer-assisted musical learning has sev-

eral approaches. Musical games, a subset of computer

games, are ﬁnalized to enable kids to interact with

music intuitively, stimulating different types of intel-

ligence, such as the instinctive, intuitive, and sensory-

motor one (Wechsler, 1975). The application of com-

puter games to music education has been discussed

in a number of scientiﬁc works - see e.g. (Denis and

Jouvelot, 2005) - and exploited in many experiments

- see e.g. (Kim et al., 2008).

In this paper we want to introduce a new paradigm

for music education, based on the concept of meta-

instrument. A musical meta-instrument (Miranda and

Wanderley, 2006; Malcangi and Castellotti, 2013) can

be either a virtualization of an existing instrument or

a brand new one; in any case, it is conceived to move

the sound generation and texture capabilities into the

instrument itself, and to leave sequencing and timing

under the performer’s control. In this sense, it aims at

being closer to the natural and intuitive ability of the

performer, in order to enable him/her to play music

without a speciﬁc technical skill. In our opinion this

new learning paradigm is extremely innovative since

it could remove any starting barrier between the per-

son and the instrument.

The behaviour of a meta-instrument somehow re-

sembles an orchestra conducted by a director. In fact

such a tool requires a score in input but it has no

knowledge about timing and interpretation in itself:

it acts like an orchestra player waiting for conductor’s

instructions. The gestures required to produce sounds

are demanded to a human player, who - freed by a

number of technical constraints - should be able to

perform music in a more straightforward way.

The activity of playing a meta-instrument, when

instanced in an appropriate context, can become a

kind of music game. It is worth citing the case

study of the popular console video game Guitar Hero

(Miller, 2009), which documents the changing nature

of amateur musicianship in an increasingly techno-

logical world. The framework we will propose in the

following is not oriented to pure entertainment, rather

to edutainment. It aims at constituting an entry point

to music learning, also for handicapped students who

have physical, mental or health impairments.

In our approach, two key paradigms of music

- namely the writing-oriented and the performance-

oriented paradigms - are mixed for educational pur-

poses. This experiment has already been successfully

conducted in other contexts, for instance in electroa-

coustic music composition, as reported in (Desainte-

Catherine et al., 2013).

In order to transform the concept of musical meta-

instrument into a real tool to play intuitively, we pro-

pose a framework that embeds meta-instrument nota-

tion in the IEEE 1599 format. The latter is an XML-

409

Andrea Ludovico L. and Malcangi M..

An IEEE 1599 Framework to Play Music Intuitively - The Metapiano Case Study.

DOI: 10.5220/0004937904090414

In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 409-414

ISBN: 978-989-758-020-8

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

based international standard for the representation of

music in all its aspects. Section 2 will present the key

concept of IEEE 1599, whereas Section 3 will provide

details on the proposed solution.

Finally, Section 4 will introduce a clarifying

example based on the experience of Jean Haury’s

metapiano. This already existing tool has been

adopted in our case study to test the effectiveness of a

“meta-instrumental” approach to music education.

2 AN OVERVIEW OF THE IEEE

1599 FORMAT

IEEE 1599 is a standard internationally recognized by

the IEEE, sponsored by the Computer Society Stan-

dards Activity Board and designed by the Techni-

cal Committee on Computer Generated Music. IEEE

1599 adopts XML (eXtensible Markup Language)

in order to describe a music piece in all its aspects

(Baggi and Haus, 2009).

The innovative contribution of the format is pro-

viding a comprehensive description of music and

music-related materials within a unique framework.

The symbolic score - intended here as a sequence of

music symbols - is only one of the many descriptions

that can be provided for a piece. For instance, all

the graphical and audio instances (scores and perfor-

mances) available for a given music composition are

further descriptions, as well as text elements (e.g. cat-

alogue metadata, lyrics, etc.), still images (e.g. pho-

tos, playbills, etc.), and moving images (e.g. video

clips, movies with a soundtrack, etc.).

Comprehensiveness in music description is real-

ized in IEEE 1599 through a multi-layer environ-

ment. The XML format provides a set of rules to

create strongly structured documents. IEEE 1599 im-

plements this characteristic by arranging music and

music-related contents within six layers:

• General - music-related metadata, i.e. catalogue

information about the piece;

• Logic - the logical description of score in terms of

symbols;

• Structural - identiﬁcation of music objects and

their mutual relationships;

• Notational - graphical representations of the

score;

• Performance - computer-based descriptions and

executions of music according to performance

languages, such as MIDI or MPEG4;

• Audio - digital or digitized recordings of the piece.

Music events are univocally identiﬁed in the en-

coding, so that they can be described in different lay-

ers (e.g. the graphical aspect of a chord and its audio

performance), and multiple times within a single layer

(e.g. many different music performances of the same

event). Consequently, in the multi-layer environment

provided by IEEE 1599, one recognizes two synchro-

nization modes:

1. Inter-layer synchronization, which takes place

among contents described in different layers. Dif-

ferent layers store - by deﬁnition - heterogeneous

information, to allow the enjoyment of heteroge-

neous music contents simultaneously, in a syn-

chronized way. Applications involving multi-

media and multi-modal fruition, such as score fol-

lowing, karaoke, didactic products, and multime-

dia presentations, can be realized thanks to this

kind of synchronization;

2. Intra-layer synchronization, which takes place

among the contents of a single layer. Each layer

contains - by deﬁnition - homogeneous informa-

tion. Thanks to this feature, one can jump from

an instance to another instance of the same type

in real time, without losing synchronization.

Coupling the aforementioned kinds of synchro-

nization, it is possible to design and implement ad-

vanced frameworks for music. For further details

about the format, please refer either to the ofﬁcial

IEEE documentation or to a recent book covering

many speciﬁc aspects of the standard (Baggi and

Haus, 2013).

In this context, the most relevant aspect is the pos-

sibility to integrate and synchronize within an IEEE

1599 document many heterogeneous kinds of descrip-

tion, including any form of meta-instrument notation.

This matter will be discussed in depth in the next sec-

tion.

3 THE PROPOSED

FRAMEWORK

In our approach, two activities can be clearly dis-

tinguished: music encoding and music performance.

Usually they are asynchronous, since the former can

be completed before the performance, and often this

is even required by a number of technical issues. In

fact, for our goals music encoding implies the pro-

duction of a “rich” IEEE 1599 document, namely a

single XML ﬁle containing both the spine and the

meta-instrument notation, mutually linked. Encoding

music in a proper way during a live performance is a

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

410

1 [] 1 63 [< 4 70 [< 3 75 [< 2 79

2 [> 4 70 [> 3 75 [> 2 79 [] 1 75

3 [] 1 74 [< 4 67 [< 3 70 [< 2 79

4 [] 1 70

5 [] 1 72

...

Figure 1: A short example of plain-text notation for J.

Haury’s metapiano. The score contains voice, pitch and

velocity encoding, together with basic information on ar-

ticulations.

hard task, even if theoretically feasible (Baldan et al.,

2009).

Needless to say, an IEEE 1599 document can con-

tain much more, as explained in Section 2. For in-

stance, it could host a number of pre-recorded audio

tracks referring to other performances of the piece,

or conceived as a background for the current perfor-

mance.

Similarly, the Notational layer could host

evocative graphics together with a traditional score

version in common Western notation.

A number of IEEE 1599 applications oriented to

music education has been treated in (Barat

e et al.,

2009) and (Barat

e and Ludovico, 2012). In this con-

text, the novelty is the presence of meta-instrument

notation. Usually it contains basic symbolic informa-

tion (i.e. notes, rests, a few articulation signs, etc.),

namely the input required by the meta-instrument

parser. A simple example is the notation for the

metapiano by Jean Haury, illustrated in Figure 1. It

is worth underlining that the information contained

in a meta-instrument score is potentially redundant

with the contents of the Logic layer, and actually

the knowledge of encoding rules makes an automatic

conversion possible between formats.

Moreover, software tools and plug-ins have been

developed to compile the Logic layer starting from

commonly adopted formats (e.g. MusicXML and

MIDI) as well as score editing software (e.g. Mus-

eScore, MakeMusic Finale



and Sibelius



). Simi-

larly, computer applications could be implemented for

ad hoc meta-instrument scores, too.

IEEE 1599 provides richness in music description,

including multiple audio, video and score digital ob-

jects. Since the format supports any representation

of score symbols, also new notation for music meta-

instruments can be embedded and synchronized with

all the other contents.

Please note that in this case timing information would

be implicitly provided to the human player. Such a result

could be either desirable, e.g. to teach students how to go

in time with the music, or unwanted, e.g. to make children

express themselves during Music Therapy sessions.

After producing the IEEE 1599 document, the

second phase - i.e. music performance - is enabled

to start. Before the design of this framework, two to-

tally independent concepts were available:

• An IEEE 1599 viewer, namely an environment

oriented to a multi-layer and synchronized musi-

cal experience. This software is able to present si-

multaneously information contents from multiple

layers, allowing the user to enjoy them together

and to choose the material to bring to front. The

user is active in the choice of current materials

(scores, audio tracks, video clips, etc.), and he/she

can use standard navigation controls (start, stop,

pause, change current position); however, from

the performance point of view, the user can only

experience already prepared materials.

• A meta-instrument parser, where a symbolic score

is loaded and the user can interact through the in-

terface of the musical instrument. The parser is

not standard, since it is customized for the pecu-

liar meta-instrument. Besides, it usually gets in-

put only from the external controller and from a

digital score representation. Consequently, other

interactions with related materials is demanded to

a posteriori processing of its output, which limits

the expressive possibilities of the framework.

These two environments could be (and actually

have been) implemented under different HW/SW ar-

chitectures, and implementation details are not rele-

vant for our proposal. For instance, IEEE 1599 play-

ers have been developed for multi-platform off-line

fruition as well as embedded in Web portals. Sim-

ilarly, there are some meta-instruments entirely im-

plemented via software and others based on the com-

munication between Arduino and Max/Msp environ-

ment. An example of the latter category will be pro-

vided in Section 4. Our idea is creating a unique

framework where the two contributions can be mixed

and integrated, in order to take advantage by both the

approaches.

As regards the music meta-instrument, it can be

any hardware or software device capable of sending

computer-interpretable messages: MIDI controllers,

external peripherals such as computer keyboards,

graphical interfaces, and so on.

The function of the parser is interpreting both the

IEEE 1599 and the controller input, producing a se-

quence of commands to drive the player. One of the

key roles is disambiguating synchronization. As men-

tioned before, most contents in an IEEE 1599 doc-

ument have intrinsic timing information, such as all

audio and video tracks. On the contrary, in this con-

text metronome is provided by the human player, so

AnIEEE1599FrameworktoPlayMusicIntuitively-TheMetapianoCaseStudy

411

Figure 2: Process ﬂow chart of the proposed framework.

the parser has to match human gesture with meta-

instrument notation, and other contents must be con-

sequently timed.

Figure 2 illustrates the proposed framework. The

upper half corresponds to music encoding, whereas

the lower half is about music performance.

4 CASE STUDY: THE

METAPIANO

The metapiano is a musical meta-instrument made of

only nine piano keys (Haury, 2013). It can be played

with a few ﬁngers, or even with one ﬁnger, as shown

in Figure 3. The metapiano notation stores the notes

to be performed by the musician. In practice, the mu-

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

412

sic is analysed in terms of its melodic, harmonic, and

contrapuntal relations. Only notes’ pitches and their

relations are codiﬁed and stored digitally, according

to pianotechnie rules (Haury, 1987).

This musical structure can produce music and

sound by playing the metapiano’s limited number of

keys. The musician can instantly interpret music with

his/her own style by applying his/her rhythm, tempo,

articulation, accent, dynamic and agogic phrasing.

In this kind of meta-instrument pitch information

is received from the score, and consequently recon-

structed at parser level. The 9-key interface is pro-

vided only to allow more effective gestures. For in-

stance, quick sequences of notes are easier to be ob-

tained using many ﬁngers, independently from the

melodic contour. Similarly, a legato effect can be ob-

tained only using at least two keys.

Experiments have shown that such an interface is

extremely intuitive for complete beginners and inex-

pert players, who are not used to associate keys to

sounds (Haury and Schmutz, 2006). On the contrary,

for skilled piano players this abstraction is harder to

be managed. However, the latter category is not the

typical recipient of our initiative.

In order to apply the IEEE 1599-based framework

to the metapiano case, a meta-instrument oriented

language layer has been designed. This language is

based on the syntactic and semantic encoding deﬁned

by Jean Haury. Starting from an XML encoding of

the score, an integrated IEEE 1599 document is gen-

erated to feed the parser controlled by a musical meta-

instrument interface. In this way, a 3-level hierarchy

of music representation has been realized:

1. A low-complexity encoding for complete begin-

ners, namely people unable to read music scores

and to play any music instrument;

2. A medium-complexity encoding for learners,

namely people who can read scores but with no

instrumental skill;

3. Finally, a high-level encoding for musicians,

namely people interested in improving their mu-

sic abilities and experiencing new kinds of music

interfaces.

Our approach can be easily extended to any other

music meta-instrument, thanks to the extensibility of

IEEE 1599 format.

5 CONCLUSIONS AND FUTURE

WORK

In this work we have introduced at ﬁrst the key fea-

tures of IEEE 1599. Such a format allows a com-

Figure 3: Jean Haury playing F. Chopin’s

Etude Op. 10 No.

4 on his 9-key metapiano.

prehensive description of music in all its facets, sup-

porting multiple media encodings and keeping digi-

tal objects synchronized. Even if commonly adopted

for education purposes, this format was never used

before for music performance. The innovative idea

is introducing a special controller acting as a music

meta-instrument and designing a framework to inte-

grate such a tool and the related notation with an IEEE

1599 parser/player.

Future work will concentrate on a re-engineered

implementation of the framework, extensive tests

conducted on impaired children and piano begin-

ners, and the extension to meta-instruments other than

Haury’s metapiano.

ACKNOWLEDGEMENTS

The authors gratefully wish to acknowledge the re-

searchers who contributed to this work. In particu-

lar, we want to thank for their support: Jean Haury,

the inventor of the metapiano, Pierre Castellotti, an

expert in the ﬁeld of meta-instruments and musician-

to-machine interfaces, and Stefano Bruni, who im-

plemented an automatic generator and translator of

metapiano notation.

REFERENCES

Baggi, D. and Haus, G. (2009). IEEE 1599: Music encod-

ing and interaction. IEEE Computer, 42(3):84–87.

Baggi, D. and Haus, G. (2013). Music Navigation with Sym-

bols and Layers: Toward Content Browsing with IEEE

1599 XML Encoding. Wiley.

Baldan, S., Ludovico, L. A., and Mauro, D. A. (2009).

Puremx: Automatic transcription of midi live music

performances into xml format. In Jensen, K., edi-

tor, Proceedings of the 2009 International Conference

on Auditory Display, pages 195–201. Aalborg Univer-

sitet, Aalborg, Denmark.

AnIEEE1599FrameworktoPlayMusicIntuitively-TheMetapianoCaseStudy

413

Barat

e, A., Haus, G., and Ludovico, L. A. (2009). Ieee

1599: a new standard for music education. In Pro-

ceedings of the International Conference on Elec-

tronic Publishing (ELPUB), pages 29–45.

Barat

e, A. and Ludovico, L. A. (2012). New frontiers in mu-

sic education through the ieee 1599 standard. In Pro-

ceedings of the 4th International Conference on Com-

puter Supported Education (CSEDU 2012), volume 1,

pages 145–151. SciTePress - Science and Technology

Publications.

Denis, G. and Jouvelot, P. (2005). Motivation-driven educa-

tional game design: applying best practices to music

education. In Proceedings of the 2005 ACM SIGCHI

International Conference on Advances in Computer

Entertainment Technology, pages 462–465. ACM.

Desainte-Catherine, M., Allombert, A., and Assayag, G.

(2013). Towards a hybrid temporal paradigm for

musical composition and performance: The case of

musical interpretation. Computer Music Journal,

37(2):61–72.

Haury, J. (1987). La grammaire de l’ex

ecution musicale au

clavier et le mouvement des touches. Analyse musi-

cale: la musique et nous, (7):10–26.

Haury, J. (2013). The metapiano and the instant intepreta-

tion of musical scores. In Proceedings of DSP Appli-

cation Day 2013, pages 143–158.

Haury, J. and Schmutz, J. (2006). D

eveloppement dun in-

strument dinterpr

etation de la musique pour personnes

en situation de d

eﬁcience motrice. Rapport APF.

Kim, Y. E., Schmidt, E. M., and Emelle, L. (2008).

Moodswings: A collaborative game for music mood

label collection. In ISMIR, volume 8, pages 231–236.

Malcangi, M. and Castellotti, P. (2013). Meta-instruments:

the musician-machine interface and gesture sensing

for real-time algorithm control. In Proceedings of

DSP Application Day 2013, pages 135–140.

Miller, K. (2009). Schizophonic performance: Guitar hero,

rock band, and virtual virtuosity. Journal of the Soci-

ety for American Music, 3(4):395–429.

Miranda, E. R. and Wanderley, M. M. (2006). New digital

musical instruments: control and interaction beyond

the keyboard, volume 21. AR Editions, Inc.

Wechsler, D. (1975). Intelligence deﬁned and unde-

ﬁned: A relativistic appraisal. American Psychologist,

30(2):135.

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

414