Novel 3D Game-like Applications Driven by Body Interactions for
Learning Specific Forms of Intangible Cultural Heritage
E. Yilmaz
1
, D. Uğurca
1
, C. Şahin
1
, F. M. Dagnino
2
, M. Ott
2
, F. Pozzi
2
, K. Dimitropoulos
3
,
F. Tsalakanidou
3
, A. Kitsikidis
3
, S. K. Al Kork
4,5
, K. Xu
4,5
, B. Denby
4,5
, P. Roussel
4,5
,
P. Chawah
6,7
,
L. Buchman
6,7
, M. Adda-Decker
6,7
, S. Dupont
8
, B. Picart
8
, J. Tilmanne
8
, M. Alivizatou
9
,
L. Hadjileontiadis
10
, V. Charisis
10
, A. Glushkova
11
, C. Volioti
11
, A. Manitsaris
11
, E. Hemery
12,13
,
F. Moutarde
12,13
and N. Grammalidis
3
1
Argedor Information Technologies, Ankara, Turkey
2
Institute for Educational Technology, National Research Council (ITD-CNR), Palermo, Italy
3
Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece
4
Universite Pierre et Marie Curie - Paris 6, Paris, France
5
University Paris 3 Sorbonne Nouvelle, Paris, France
6
Centre National de la Recherche Scientifique, Laboratoire de Phonétique et Phonologie UMR 7018, Paris, France
7
Phonetics and Phonology Laboratory, LPP-CNRS, Paris, France
8
Universite de Mons, Mons, Belgium
9
University College London, London, U.K.
10
Aristotle University of Thessaloniki, Thessaloniki, Greece
11
MTCG Lab, Dep. Of Ap. Informatics, University of Macedonia, Thessaloniki, Greece
12
Robotics Lab, Dep. Of Mathématiques et Systèmes, MINES ParisTech-ARMINES, Paris, France
13
Signal Processing and Machine Learning Lab ESPCI, Paris, France
Keywords: Human Computer Interaction, Interactive Game, 3D Visualization, Sensorimotor Learning, Game based
Learning.
Abstract: The main objective of the EU FP7 ICT i-Treasures project is to build a public and expandable platform to
enable learning and transmission of rare know-how of intangible cultural heritage. A core part of this platform
consists of game-like applications able to support teaching and learning processes in the ICH field. We have
designed and developed four game-like applications (for Human Beat Box singing, Tsamiko dancing, pottery
making and contemporary music composition), each corresponding to one of the ICH use cases of i-Treasures
project. A first preliminary version of these applications is currently available for further validation,
evaluation and demonstration within the project. We have encountered a number of issues, most of which
derive from the peculiarities of the ICH domains addressed by the project, and many have already been
resolved/ The evaluation results are expected to lead to further optimization of these games.
1 INTRODUCTION
Cultural expression is not limited to architecture,
monuments or collections of artefacts, for which
various systems have been developed for capturing,
analysis and visualization (Kyriakaki, 2014),
(Makantasis, 2014) and (Makantasis, 2013).
Intangible Cultural Heritage” (ICH) is defined as a
part of the cultural heritage of societies, groups or
sometimes individuals and it includes practices,
presentations, expressions, knowledge, skills and
related tools to all of these such as equipment and
cultural sites. This intangible heritage passes from
generation to generation and gives people a sense of
identity and continuity; it is the result of the
continuous interaction of communities and groups
with their nature and history and it promotes respect
for cultural diversity and human creativity.
The main objective of the EU FP7 ICT i-Treasures
project (Dimitropoulos, 2013) is to build a public and
expandable platform to enable learning and
transmission of rare know-how of intangible cultural
heritage. The proposed platform combines lots of
different technologies like multisensory technology,
651
Yilmaz E., U
˘
gurca D., ¸Sahin C., Dagnino F., Ott M., Pozzi F., Dimitropoulos K., Tsalakanidou F., Kitsikidis A., Al Kork S., Xu K., Denby B., Roussel
P., Chawah P., Buchman L., Adda-Decker M., Dupont S., Picart B., Tilmanne J., Alivizatou M., Hadjileontiadis L., Charisis V., Glushkova A., Volioti C.,
Manitsaris A., Hemery E., Moutarde F. and Grammalidis N..
Novel 3D Game-like Applications Driven by Body Interactions for Learning Specific Forms of Intangible Cultural Heritage.
DOI: 10.5220/0005456606510660
In Proceedings of the 10th International Conference on Computer Vision Theory and Applications (MMS-ER3D-2015), pages 651-660
ISBN: 978-989-758-090-1
Copyright
c
2015 SCITEPRESS (Science and Technology Publications, Lda.)
singing voice synthesis and sensorimotor learning to
leave the beaten path in education and ICH
knowledge transmission.
In i-Treasures project, four main ICH use cases
are selected: i) rare traditional songs with the
following sub use cases: a) Byzantine hymns, b)
Corsican “cantu in paghjella”, c) Sardinian “canto a
tenore” and d) beat box; ii) rare dance interactions
with the following sub use cases: a) Calus dance, b)
Tsamiko dance, c) Walloon dance and d)
contemporary dance; iii) traditional craftsmanship
focusing on the art of pottery making, and finally iv)
contemporary music composition based on
Beethoven, Haydn and Mozart musical pieces.
In order to prevent these ICH expressions from
extinction, we aim to provide a tool that will allow
their transmission to new people. Towards this goal,
four novel game-like educational applications are
currently developed, one for each of the four ICH use
cases, to be studies within i-Treasures project: i) rare
traditional songs with a contemporary singing style
namely Human Beat Box (HBB); ii) rare dance
interactions with a popular Greek dance (Tsamiko)
iii) traditional craftsmanship focusing on the art of
pottery making, and finally iv) contemporary music
composition based on Beethoven, Haydn and Mozart
musical pieces. In the current development phase, a
first version of these game-like applications is
available and is planned to be validated and evaluated
for sensorimotor learning using educational scenarios
that have been already been defined within the
project. These applications will be continuously
updated towards fulfilling satisfying the needs of the
project and will also be extended to cover the needs
of other sub-use cases (e.g. Walloon dance, Byzantine
music, etc.).
The rest of this paper is organized in the following
Sections: Section 2 provides an overview of the
related work in similar game-like applications for
learning purposes. Section 3 summarizes the system
architecture and the design/development principles
for these applications and Section 4 presents an
overview of the four ICH games that were developed.
Finally, Section 5 includes discussion in the lessons
learnt, limitations and future work.
2 RELATED WORK IN
GAME-LIKE APPLICATIONS
FOR LEARNING PURPOSES
The adoption of game-like applications in i-Treasures
follows a quite well consolidated trend in the
Technology Enhanced Learning field. The
educational potential of games has been widely
explored and highlighted by researchers within the
wider research area of Game Based Learning (de
Freitas, 2012), (Gee, 2003), (Van Eck, 2006),
(Hainey, 2010). A variety of games originally
developed with entertainment purposes can be used
for educational purposes (Djaouti, 2011); in addition
we recently assisted to the taking off and to the fast
increasing adoption of Serious Games (SGs), those
games that are explicitly designed for educational
purposes Michael, 2006), (Breuer & Bente, 2010),
(Derryberry, 2007).
SGs are more and more employed to sustain
learning and training in a variety of educational fields
(formal and informal education as well as military,
medical training etc.); this is done for a wide range of
target populations, ranging from children to adults
(Charlier, 2012).
The learning potential of Serious Games (SGs)
has been extensively investigated in recent years.
Despite some contrasting voices (Hays, 2005), their
educational effectiveness has been widely recognized
(Facer, 2007), (Mc Farlane, 2002), (Milovanović,
2009). When asserting the effectiveness of game-
based learning, many authors focus on the nature of
interaction with the game environment, citing aspects
like motivation, flow and immersiveness (de Freitas,
2009), (Garris, 2002).
SGs, in fact, proved to support learning in a more
active and engaging way (Gee, 2003) and, from the
pedagogical viewpoint, they offer advanced
interaction such as the possibility of customizing the
learning paths (Bottino, 2009) and of keeping track of
the learners’ behavior and successes/failures and are
more adaptive to meet the specific users’ learning
needs (de Freitas, 2013). Games seem to be able to
support the learning a wide range of skills (spatial
skills, decision-making, problem solving, etc.) (De
Aguilera, 2003) and are also recognized as supporting
the learning of procedures and gestures, reason why
they are widely adopted in professional training
(Martinez-Durà, 2011)
In the field of Cultural Heritage Education ICT
technologies are increasingly being adopted (Ott,
2011), (Gaitatzes, 2001), (Veltman, 2005) and in
particular, Virtual Worlds are often used to broaden
the opportunity to appreciate cultural contents that are
remote in space and or time. Even though they should
be considered very helpful for widening access to
cultural contents, these applications, for example
Virtual Museums, often are not intrinsically engaging
and sometimes fail in supporting active learning, just
giving the opportunity to access information
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
652
(Mortara, 2014).
This is why in the i- Treasures project it was
decided to develop game-like educational
applications (or, in other words, Serious Games,
developed with the specific purpose of sustaining
specific educational interventions)
As to the digital games available in the Cultural
Heritage (CH) area, (Anderson, 2009) and afterwards
(Mortara, 2014) carried out interesting State of the
Art reviews. While the first one is more focused on
technical aspects, the second sketches a panorama of
the actual use of Serious Games in Cultural Heritage
education. According to (Mortara, 2014) in the field
of CH, SGs of different kind are adopted: from trivia,
puzzle and mini-games to mobile applications for
museums or touristic visits, (e.g. Muse-US
i
, Tidy
City
ii
) to simulations (e.g. the battle of Waterloo
iii
) to
adventures and role playing games (the Priory
Undercroft
iv
, Revolution
v
).
As it could be expected, games are more
widespread in the Tangible Cultural Heritage (TCH)
area, where several different examples can be found
(Ott, 2008). Among these, we can mention
ThIATRO, a 3D virtual environment where the player
acts as a museum curator (Froschauer, 2012), or other
digital artifacts such as My Culture Quest
vi
, which
aims at advertising real collections or even the
History of a Place
vii
, which is integral part of a
museum experience at the Archaeological Museum of
Messenia in Greece.
A number of games for smartphones also exist
like Tate Trumps
viii
and YouTell and “G. Averof”
(Cao, 2011) which, for instance, allow museum
visitors to create and share through smart phones their
own media and stories.
Many games also exist in the area of historical
reconstruction for instance the Battle of Thermopylae
(Christopoulos, 2011) or the Playing History
ix
, which
are mainly based on 3D technology so to closely
recreate the environment in which each single event
happened.
A (smaller) number of games have also been
developed in the field of Intangible Cultural Heritage
transmission, where they are also considered very
promising (Mortara, 2014). Some examples are:
Icura (Froschauer, 2010), a 3D realistic
environment in which the player learns about
Japanese culture and etiquette, which can raise
cultural interest and support a real pre-trip planning.
Discover Babylon
x
, Roma Nova
xi
and
Remembering 7th Street
xii
, that are aimed at raising
awareness about ancient Mesopotamia's contribution
to modern culture, ancient Rome and West Oakland
in the time period post-World-War-II.
Africa Trail
xiii
and Real Lives 2010
xiv
simulate
a 12,000 mile travel by bicycle through Africa or a
different life in any country of the world (e.g. a
peasant farmer in Bangladesh, or a computer operator
in Poland), respectively.
Papakwaqa (Huang, 2013), a serious game
about the Atayal minority in Taiwan, particularly
focused on intangible cultural assets like tribal
beliefs, customs, and ceremonies.
3 SYSTEM ARCHITECTURE
The general system architecture of the game-like
applications that have been developed for the four
ICH cases is shown in Figure 1. As shown in this
figure, there is bilateral communication between the
game-like applications (i.e., the 3D Visualization
Module for Sensorimotor Learning - 3DVMSL) and
the ICH capture and analysis modules, which are
different for each use case (e.g. Ultrasound and
optical sensors and microphones for Rare singing,
Kinect sensors for Rare Dancing, Kinect and/or Real
Leap Motion
TMxv
sensors for Pottery and
Contemporary Music Composition). This
communication includes: a) data transfer from ICH
capture sensor/s to the games, and b) control of ICH
capture and analysis modules by the 3DVMSL
module. The 3DVMSL module can start/stop sensor
data capture and trigger the required data
capture/analysis functions. 3DVMSL also has
bilateral communication with a) a Content
Management System (“i-Treasures Web Platform”)
that is responsible for different tasks such as user
profile management, ICH expert recording
management, etc. and b) a Learning Management
System, which contains additional educational
courses and material regarding each use case.
Regarding the internal architecture of the
3DVMSL module, the initializer is responsible of
activating sensor setups and starting the game-like
application. This action is followed by login where
the user credentials are taken from the Learning
Management System (LMS). Thereafter, the game
menu is shown and the user can either see the game
tutorial or select one of the activities offered such as
Activity 1, Activity 2,… or Final Challenge. The
“Observe” screen currently displays pre-captured
expert data (corresponding to the selected Activity) to
the user. In the current version, we used expert data
stored at the local game repository. However, in the
final version of the games, this data will be obtained
from the Content Management System (CMS). This
is why the connection between CMS and 3DVMSL is
Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage
653
displayed with dash lines. The “Practice” functions
allow the user to try to replicate the expert moves and
then receive feedback from the system regarding the
evaluation of his/her performance. In order to capture
the performance, a set of ICH capture modules are
used and there is a bilateral communication between
the 3DVMSL and the in order to transfer sensor data
to the game.
Figure 1: Detailed System Architecture of Game-like
Applications.
All four ICH games are incorporated in a single
game-like application to ensure simplicity and
integrity. Certain game characteristics, e.g. the splash
screen, the authentication and game selection screen,
the observe and practice phases are common in all
ICH games. We used Unity 3D game engine to
develop the games, due to the following reasons: a)
Unity 3D is an industry proven game development
environment, b) It is easy to deploy the games to
various platforms ranging from Desktop PCs to
mobile devices, c) The development team is highly
experienced in Unity. We used 3DMax Studio and
Maya to model the characters and game
environments, namely 3D assets. Those 3D
modelling tools are also very common in game
development industry. We used Photoshop to prepare
2D assets of the games. All of the codes are written
by using C# programming language.
4 GAME-LIKE APPLICATIONS
FOR LEARNING FORMS OF
ICH
4.1 Rare Traditional Singing: The
Human Beatbox (HBB) Game
The Human Beatbox (HBB) is an artistic form of
human sound production in which the vocal organs
are used to imitate percussion instruments, but also
wind and string instruments. Therefore, facial and
intra-oral movements of the tongue, lips and jaw are
involved in sound production.
The game scenario for Human Beatbox (prepared
by the sub-use case leader CNRS together with beat
box expert) is as follows:
The learner will have at its disposal a portable
helmet based system (Jaumard-Hakoun, 2013),
(Chawah, 2014), (Al Kork, 2014). The latter provides
vocal tract sensing techniques developed for speech
production and recognition. It consists of a
lightweight “hyper-helmet” containing an ultrasonic
(US) transducer to capture tongue movement, a video
camera for the lips, and a microphone.
The learner will be guided by some description of
the sound to be produced and will be listening to the
sounds to imitate them. These descriptions can be
related to natural sounds such as imitating animal
sounds or mouth noise.
The learner will also look at the mouth and tongue
of the expert to try to produce different sounds. The
rhythm is an important dimension in HBB and hand
movement or a metronome can be added in the
scenario in the updated future version.
The HBB game will consist initially of a single
lesson, where each isolated sound will be described
and will be demonstrated by the expert (“Observe
mode”) and then imitated by the learner(“Practice
mode”). Later on, the learner will go to the next step
that will be to produce two sounds, then three in a
row.
The learner should practice each activity a number
of times, and finally perform all the activities at once.
The practice screen of the HBB game is illustrated in
Figure 2.
Figure 2: Practice Screen HBB Game.
4.2 Rare Dancing: The Tsamiko Game
Four different sub use cases are examined in the
project: (a) Calus dance (Romania), (b) Tsamiko
dance (Greece), (c) Walloon dance (Belgium) and (d)
contemporary dance. In this first version of the
Activities
#1, #2, …,
Final
Challen
g
e
ICH Capture
modules
3DVMSL
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
654
application, a game for Tsamiko dance learning was
implemented. Tsamiko is a popular traditional folk
dance of Greece, done to music of ¾ meter. The game
focuses on teaching the basic steps of Tsamiko to
learners with minimal or no prior dancing experience.
The game scenario was prepared in close cooperation
with Tsamiko dance experts in Thessaloniki and was
the basis for designing the game.
In order to simplify the learning process of
Tsamiko dance, the experts have defined two
activities comprised of several exercises. The first
activity focuses on the simple single step Tsamiko
style (10 steps) and includes five exercises while the
second activity focuses on the more advanced double
step style (16 steps) and includes four exercises.
The learner has to repeat all of the exercises to
complete the activity objective. Each exercise
consists of several dance steps, which are presented
to the learner one by one. In order to proceed to the
next exercise, the learner must repeat the current
exercise at least 3 to 5 times correctly. Some exercises
of the second activity are more challenging and must
be repeated at least 8 times.
At the beginning of each exercise, a video with
the 3D expert virtual avatar performing the specific
moves is shown to the learner. Afterwards, the learner
is expected to try to imitate the same moves correctly.
The system starts the data capture and gets fused
animation data from an Body motion capture module
that acquires animation data via multiple Kinect v1
sensors, fuses the information properly and provides
to the game a single sequence of animation data
corresponding to the user body movements.
If the imitation is completed properly, the learner
can proceed to the next exercise. Otherwise, the
learner is expected to repeat the same exercise till s/he
does the moves correctly. Although the exercises
progress sequentially, in some cases the learner has to
Figure 3: Practice Screen: Backward View of 3D Expert
Avatar (Big Central Window), Animation Controller
(Bottom Center), Exercise Indicator (Top Center), Expert
Video (Top Right), 3D Learner Avatar (Center Right),
Backward View of Close-up Expert Legs (Bottom Right).
repeat not only the previous exercise but also some
more. A screenshot from the Practice Screen is shown
in Figure 3.
One of the most important aspects of the game is
to evaluate the performance of the learner. In order to
achieve this challenging task, moves of the learner
avatar and expert avatar are compared using a Fuzzy
Inference based algorithm (Kitsikidis, 2014). This
approach uses fuzzy inference to not just compare
absolute joint positions, but to evaluate more
meaningful features.
4.3 Traditional Craftsmanship
The traditional craftsmanship use case focuses on
pottery, which is an art of crafting ceramic. The main
objective of this game is to let the learner observe and
practice the basic moves of wheel-throwing
earthenware pottery by using various sensors and
gaming interface.
The following game scenario, prepared with the
close cooperation of pottery experts in Thessaloniki,
is taken as a basis while designing the traditional
craftsmanship application. There are four activities,
which are different from each other and have different
levels of complexity. The expert demonstrates the
pottery making procedure and it is expected from the
learner to imitate the expert to form an object (Figure
4). The first activity focuses on throwing and
centering the clay on the wheel, the second on how to
make the bottom of the object, the third activity
shows how to shape formation with a tool and the
forth how to cut and remove the final object from the
wheel. In the “Final challenge” activity, the learner
tries to perform all activities in a row in order to make
an object. There is a performance evaluation
threshold value for each activity that allows the user
to proceed to the next activity.
The application communicates with the Body and
Gesture Data Capture and Analysis Module that is
Figure 4: Practice Screen: Angled View of 3D Learner
Avatar (Big Central Window), Animation Controller
(Bottom Center), Expert Video (Top Right), Close-up on
Expert Hands (Center Right).
Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage
655
responsible for capturing the hand and upper body
motions of an expert potter. Specifically, the module
can read, process and capture hands animation data
from a single Leap Motion sensor (multiple Leap
Motion sensors will be supported in the near future)
and combine this information with body movement
from multiple Kinect sensors. In addition,
synchronization and fusion of hand and body
animation streams are performed in order to output a
single animation stream containing skeletal
information of both hands and the body of a person.
Also, there is an optional mode available for
accurately tracking wrist positions in space, based on
blob tracking. This mode significantly improves the
quality of captured data but requires the user to wear
round coloured markers around the wrists.
4.4 Contemporary Music Composition
The Contemporary music composition use case aims
to develop a novel Intangible Musical Instrument
(IMI), which is supposed to map natural gestures
performed in a real-world environment to
music/voice segments. Besides, the emotional status
of the performer is also planned to be part of this
multimodal human interface. Thus, the game-like
application is designed in a way to map not only
gestures but also the emotions of the learner.
The Intangible Musical Instrument aims at
capturing piano-like gestures, which means that the
gestures of the performer resemble piano player
gestures. The new gestures are inspired by piano-like
performances and are transformed into sounds via a
“mapping” phase. Another objective, made possible
with the technology used here, is to create a powerful
pedagogical tool. The player then becomes a learner
and interacts with the system in order to master piano-
like techniques.
The current IMI sensor setup (Figure 5) includes:
a) a “stand” made of wood and Plexiglas under
which a Leap motion sensor is placed. b) Two
inertial sensors for wrists (Animazoo motion capture
suit) and c) an Emotiv sensor for capturing emotions.
This prototype setup is integrated into a game
framework where the user/learner is assessed by the
Figure 5: Detailed description of IMI sensor setup.
system. There are two phases: the first one is the
observing phase, in which the learner observes the
expert’s gesture and listens to the corresponding
sound. The second phase is the practice phase, in
which the learner experiments with the expert’s
gestures. The practice phase of gestures involved in
the contemporary music case could be depicted as
follows (Figure 6): a simple table stands in the middle
of a room. This room looks like a recording studio
with several musical instruments dispatched around.
The user can also see the avatar of the learner/expert
player inside the 3D environment and s/he is able to
manipulate both the studio environment and the
avatar so as to get an “objective view” where s/he sees
his/her own hands plus additional views (perspective,
top view). One main idea conveyed in this first
prototype is the use of the table as a reference plane.
The main approach concerning the learning phase
is “learn by doing”. The learner should basically
attempt to perform the gestures introduced by a
virtual expert on a (real) table. The expert gestures are
driven offline from previously recorded gesture data.
In this first prototype, only the effective gestures were
considered (gestures for Ascending/descending
scales and ascending/ descending arpeggios).
The musical game activities include observe and
practice phases. In the observe phase, the learner can
observe the video of an expert performing a musical
gesture.
Figure 6: Practice Screen: Isometric View of 3D Learner
Avatar (Big Central Window), Animation Controller
(Bottom Center), Expert Video (Top Right), Close-up
Expert Hands (Center Right).
4.4.1 EmoActivity Scenario
A special activity (EmoActivity) of IMI is basically
designed to prompt the user to reach and sustain
certain affective states via a gamified process
involving affective images. The activity is also based
on the characterization of affective states by the
valence-arousal model of affect (Russel, 1980).
Valence denotes whether an emotion is positive or
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
656
negative, while arousal constitutes a measure of the
excitation that accompanies the emotion. Thus, each
affective state can be modeled as a point on the plane
defined by the orthogonal axes of valence and
arousal. The EmoActivity will be part of the novel
intangible musical instrument where the user's
affective state will influence the produced musical
content.
The game consists of three levels with each level
having three difficulty levels. In Level 1 the user is
asked to reach a certain affective state and sustain it,
while in Levels 2 and 3 the user is asked to reach two
and three consecutive states, respectively. The
difficulty in each level (easy, medium, hard) lies in
the target affective states that the user is asked to
reach based on his/her current emotional status, e.g.,
if difficulty level is set to easy and the user is calm,
he/she will be asked to reach a target affective state
of excitement as this transition is more easy than a
state of sadness or anger/fear.
5 DISCUSSION: LESSONS
LEARNT, LIMITATIONS,
FUTURE WORK
Four novel 3D game-like applications driven by body
interactions have been developed of four ICH types
within i-Treasures project. In this first version, we
have encountered a number of problems, most of
which derive from the peculiarities of the ICH
domains addressed by the project. Each ICH has its
own specificities and knowledge domains that need to
be made explicit and formalized, but also there are no
consolidated teaching/learning practices, as the
transmission of these cultural expressions relies on
informal situations based on imitation. Thus, the task
of designing and developing games able to support
teaching and learning processes in the ICH field, is
thus per se a huge challenge.
Starting from the definition of the learning
objectives and the contents to be included in the
games, down to the definition of the game dynamics
and the interface, the whole process was the result of
a complex interaction among many variables, some
of them predictable, others completely unforeseen.
More specifically, a first issue to discuss is the
multi-disciplinary structure of the development
(project) team. The i-Treasures consortium is
comprised of partners with different backgrounds
ranging from commercial video game development to
audio research. Thus, this first version of
visualization module harmonizes not only technical
research and 3D game development, but also
educational aspects. Probably blending technical
research and educational targets under a game design
pipeline at the same time was the toughest challenge,
since the expectations differ significantly. This has
led to accept some compromise, especially from the
educational’ and game dynamics’ point of view.
On the other hand, when we look at it from the
technical perspective, communication of sensors and
the visualization module and conversion of the sensor
output into 3D animation were very challenging
issues. However, any of these technical issues have
no meaning to the end users of the i-Treasures project,
unless it is presented in a coherent way in respect to
the learning objectives and educational design.
Therefore, significant amount of time and resources
have been assigned to blending technical work and
educational aspects and parallelization of
development efforts. At the end of this first version,
the development team has reached a common
understanding and raised the required know-how,
thus facilitating future work.
Regarding to the communication between the
sensors and the visualization module, we encountered
many issues, such as different OSs, different types of
same OS (such as Windows 32 bit, Windows 64 bit)
due to variety of sensors, 3
rd
party controllers,
limitations of research tools etc. At the beginning, we
have tried to define a single data transfer framework
based on TCP. However, at later development phases
many modifications have been made and even some
data transfer is shifted from TCP to UDP considering
the data size, data transfer frequency and even the
limitations of the 3
rd
party controllers. Besides, web
services have also been assigned for low-frequency
communications between remote systems. In order to
minimize the workload, we transferred fused data
when we used different sensors. Besides, we defined
a common structure to visualize both the offline and
online version with same software functions. Thus,
the same data template is used to keep expert data and
to transfer learner data in real-time. Even though we
spent additional resources, this approach will save
additional development time for the future versions.
One last issue regarding the communication
between sensors and visualization module is the
bilateral communication. In typical games, game
controllers and sensors are all controlled by the game
in order to ensure full control. However, in this
research project we have to use more experimental
setups, which have no built-in functionality to be
controlled by a game. Thus, the game must be ensure
that everything is OK and calibrated. Besides, the
game should trigger many events, such as starting a
Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage
657
record, data transfer or even request an evaluation
result. For this first version, this issue was quite
challenging. Even though we achieved some sort of
control, there are still significant issues to be solved
for the next versions.
Another lesson learnt is to have more flexibility
for modifications of the games. Although an initial
design was available, when it came to the realization,
there were many modification requests. In a typical
commercial game or educational application most of
the requirements are determined earlier and the
design is realized by following the initial plan.
However, in this work, there are external participants,
such as domain experts, who have no expertise in
software development and games. Thus, we followed
a more iterative approach to get their feedback during
intermediate development steps and update the design
to reflect their contributions and feedback.
Considering the rich variety of ICH content, the
games started to show more diversity. Many things
have been changed, ranging from activity captions to
practice modes. In order to overcome this issue, the
only way is to design an external entity, which is
capable of changing almost every visual aspect of the
game. Therefore, we designed a configuration XML
that can help us define almost everything about the
games. At the beginning, this took significant time
and caused some delays in the planned work.
However, in the next versions and development
cycles this XML will be a life-saving jacket for us,
allowing everything will be fully configurable
without compiling the games.
In conclusion, many of the issues have been
resolved and four solid game-like applications have
been, able to be integrated in the i-Treasures platform
and further updated and extended in the future.
Besides, the latter outputs will be more robust, since
most of the resources will be used for research and
development rather than trying to solve the technical
problems.
ACKNOWLEDGEMENTS
This work is funded by the European Commission via
the i-Treasures project (Intangible Treasures -
Capturing the Intangible Cultural Heritage and
Learning the Rare Know-How of Living Human
Treasures FP7-ICT-2011-9-600676-i-Treasures). It is
an Integrated Project (IP) of the European Union's 7th
Framework Programme 'ICT for Access to Cultural
Resources.
REFERENCES
Dimitropoulos K., Manitsaris S., Tsalakanidou F.,
Nikolopoulos S., Denby B., Al Kork S., Crevier-
Buchman L., Pillot-Loiseau C., Dupont S., Tilmanne J.,
Ott M., Alivizatou M., Yilmaz E., Hadjileontiadis L.,
Charisis V., Deroo O., Manitsaris A., Kompatsiaris I.,
and Grammalidis N., 2014. Capturing the Intangible:
An Introduction to the i-Treasures Project, in Proc. 9th
International Conference on Computer Vision Theory
and Applications (VISAPP2014), Lisbon, Portugal, 5-8
January.
G. Kyriakaki, A. Doulamis, N. Doulamis, M. Ioannides, K.
Makantasis, E. Protopapadakis, A. Hadjiprocopis et al.,
2014. 4D Reconstruction of Tangible Cultural Heritage
Objects from Web-Retrieved Images, in International
Journal of Heritage in the Digital Era 3, no. 2 (2014):
431-452.
K. Makantasis, A. Doulamis, N. Doulamis, M. Ioannides,
N. Matsatsinis, 2014. Content-Based Filtering for Fast
3D Reconstruction from Unstructured Web-Based
Image Data, Int. Conf. on Cultural Heritage (Euromed
2014), Lemessos, Cyprus, 3-8 November.
K. Makantasis, A. Doulamis, N. Doulamis, 2013. A non-
parametric unsupervised approach for content based
image retrieval and clustering. Proc. of 4th ACM/IEEE
international workshop on Analysis and retrieval of
tracked events and motion in imagery stream, ACM.
de Freitas, S., Earp, J., Ott, M., Kiili, K., Ney, M., Popescu,
M., Stanescu, I., 2012. Hot Issues in Game Enhanced
Learning: the GEL Viewpoint. Procedia Computer
Science, 15, 25-31.
Gee, J.P., 2003. What Video Games Have to Teach Us about
Learning and Literacy. NY: Macmillan.
Van Eck R, 2006. Digital Game-Based Learning: It's Not
Just the Digital Natives Who Are Restless.
EDUCAUSE Review, 41( 2), 17-30.
Hainey, T., 2010. “Using Games-Based Learning to Teach
Requirements Collection and Analysis at Tertiary
Education Level”, Thesis submitted in partial fulfilment
of the requirements of the University of the West of
Scotland for the award of Doctor of Philosophy.
Djaouti D., Alvarez J., Jessel J.-P., 2011. Classifying
Serious Games: The G/P/S Model, Patrick Felicia (ed),
"Handbook of Research on Improving Learning and
Motivation through Educational Games: Multidisciplinary
Approaches", (pp.118-136), IGI Global.
Michael, D., Chen, S., 2006. Serious games: Games that
educate, train, and inform. Boston, MA.: Thomson
Course Technology.
Breuer J., Bente G., 2010. Why so serious? On the Relation
of Serious Games and Learning - Eludamos. Journal for
Computer Game Culture, 4 (1) 7-24.
Derryberry A., 2007. Serious games: online games for
learning. (online), http://www.adobe.com/ resources/
elearning/pdfs/serious_games_wp.pdf.
Charlier, N., Ott, M., Remmele, B., Whitton, N., 2012. Not
Just for Children: Game-Based Learning for Older
Adults. In 6th European Conference on Games Based
Learning, Cork, Ireland (pp. 102-108).
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
658
Hays, R.T., 2005. The effectiveness of instructional games:
a literature review and discussion, Naval Air Warfare
Center Training Systems Division,
http://adlcommunity.net/file.php/ 23/GrooveFiles/
Instr_Game_Review_Tr_2005.pdf.
Facer, K., Ulicsak, M., Sandford, R., 2007. Can Computer
Games Go to School …?, BECTA http://partners.becta.
org.uk/index.php?section=rh&rid=13768.
Mc Farlane, A., Sparrowhawk A., Heald, Y., 2002. Report
on the educational use of games: an exploration by
TEEM of the contribution which games can make to the
education process. Teem, Cambridge.
Milovanović, M., Minović, M., Kovačević, I., Minović, J.,
Starčević, D., 2009. Effectiveness of Game-Based
Learning: Influence of Cognitive Style. In: Lytras,
M.D., Ordonez de Pablos, P., Damiani, E. Avison, D.,
Naeve, A., Horner, D. G. (eds.) Best Practices for the
Knowledge Society. Knowledge, Learning,
Development and Technology for All. Communications
in Computer and Information 49 (pp. 87-96). Science
Springer, Heidelberg.
de Freitas, S., Neumann, T., 2009. The use of ‘exploratory
learning’ for supporting immersive learning in virtual
environments. Computers and Education, 52(2): 343-352.
Garris, R., Ahlers, R., Driskell, J. E., 2002. Games,
motivation, and learning: a research and practice model.
Simulation & Gaming, 33(4), 441–467.
Bottino, R. M., Ott, M., & Benigno, V. (2009). Digital mind
games: experience-based reflections on design and
interface features supporting the development of
reasoning skills. In Proc. of 3rd European Conference
on Game Based Learning (pp. 53-61).
de Freitas, S., Ott, M., Popescu, M. M., Stanescu, I., 2013.
Game-Enhanced- Learning: Preliminary Thoughts on
curriculum integration. New Pedagogical Approaches
in Game Enhanced Learning: Curriculum Integration.
IGI Global.
De Aguilera, M., Mendiz, A., 2003. Video games and
education: (Education in the face of a "parallel school").
ACM Computers in Entertainment, 1(1), 10-10.
Martínez-Durá, M., Arevalillo-Herráez, M.,García-
Fernández, I., Gamón-Giménez, M.A., Rodríguez-
Cerro, A. 2001. Serious Games for Health and Safety
Training. In M.Prensky, (Ed). Digital game-based
learning. New York: McGraw-Hill.
Ott, M., Pozzi, F., 2011. Towards a new era for Cultural
Heritage Education: Discussing the role of ICT.
Computers in Human Behavior, 27(4), 1365-1371.
Gaitatzes A., Christopoulos, D., Roussou, M., 2001.
Reviving the past: cultural heritage meets virtual
reality. In Proc. of Virtual reality, archeology, and
cultural heritage, pp. 103-110.
Veltman K.H., 2005. Challenges for ICT/UCT Applications
in Cultural Heritage. In: Carreras, C. (ed.) ICT and
Heritage, online at http://www.uoc.edu/digithum/
7/dt/eng/dossier.pdf.
Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G.,
Houry-Panchetti, M., Petridis P., 2014. Learning
Cultural Heritage by serious games. Journal of Cultural
Heritage, 15(3), 318–325.
Anderson, E. F. McLoughlin L., Liarokapis F., Peters, C.,
Petridis, P., de Freitas, S., 2009. Serious Games in
Cultural Heritage. M. Ashley and F. Liarokapis
(Editors). In Proc. of the The 10th International
Symposium on Virtual Reality, Archaeology and
Cultural Heritage VAST - State of the Art Reports.
Ott, M., & Pozzi, F., 2008. ICT and Cultural Heritage
Education: Which Added Value?. In Lytras et al (Eds).
Emerging Technologies and Information Systems for the
Knowledge Society. Lecture Notes in Computer Science,
5288 (pp.131-138). Springer Berlin Heidelberg.
Froschauer, J.; Arends, M.; Goldfarb, D., Merkl, D., 2012.
A serious heritage game for art history: Design and
evaluation of ThIATRO. In 'VSMM', 18th International
Conference in Virtual System and
MultimediaConference, IEEE, pp. 283-290.
Cao, Y., Klamma, R., Jarke, M., 2011. The Hero's Journey
template-based storytelling for ubiquitous multimedia
management. Journal Multimedia, 6 (2) 156–169.
Christopoulos, D., Mavridis, P., Andreadis, A.,
Karigiannis, J.N., 2011. Using virtual environments to
tell the story: The battle of Thermopylae. In VS-Games
2011, pp. 84-91.
Froschauer, J., Seidel, I., Gartner, M., Berger, H., Merkl D.,
2010. Design and evaluation of a serious game for
immersive cultural training. In 'VSMM', 16th
International Conference in Virtual System and
MultimediaConference, IEEE, pp. 253–260.
Huang, C., Huang, Y., 2013. Annales school-based serious
game creation framework for Taiwan indigenous
cultural heritage. Journal of Computing in Cultural
Heritage, 6 (2).
Jaumard-Hakoun A., Al Kork, S.K., Adda-Decker M.,
Amelot A., Buchman L., Fux T., Pillot-Loiseau C.,
Stone M., Dreyfus G and Denby B., 2013. «Capturing,
Analyzing, and Transmitting Intangible Cultural
Heritage with the i-Treasures Project» in Ultrafest VI.,
Edinburgh, UK.
Chawah P., Al Kork S. K., Fux T., Adda-Decker M.,
Amelot A., Audibert N., Denby B., Dreyfus G.,
Jaumard-Hakoun A., Pillot-Loiseau C., Stone M., Xu
K. and Buchman L., 2014. «An educational platform to
capture, visualize and analyze rare singing,» in
Interspeech Conference, Singapore.
Al Kork S. K., Jaumard-Hakoun A., Adda-Decker M.,
Amelot A., Buchman L., Chawah P., Dreyfus G., Fux
T., Pillot C., Roussel P., Stone M., Xu K. και Denby B.,
2014. A Multi-Sensor Helmet to Capture Rare Singing,
An Intangible Cultural Heritage Study, 10th
International Seminar on Speech Production, Cologne,
Germany.
Kitsikidis A., Dimitropoulos K., Yilmaz E., Douka S.,
Grammalidis N., 2014. Multi-sensor technology and
fuzzy logic for dancer’s motion analysis and
performance evaluation within a 3D virtual
environment, HCI International 2014, Heraklion,
Greece, 22-27 June.
Russell, J. A., 1980. “A circumflex model of affect,”
Journal of Personality and Social Psychology, vol. 39,
1161-1178.
Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage
659
i
Coenen T., 2013. MuseUs: case study of a pervasive cultural
heritage serious game. Journal on Computing and Cultural
Heritage (JOCCH), 6(2), 8:2-8:19
ii
http://totem.fit.fraunhofer.de/tidycity
iii
http://www.bbc.co.uk//history/british/empire
seapower/launch gms battle waterloo.shtml
iv
A. Doulamis, F. Liarokapis, P. Petridis, G. Miaoulis. 2011.
Serious games for cultural applications. In D. Plemenos, G.
Miaoulis (Eds.), Artificial Intelligence Techniques for
Computer Graphics, Springer. The game is a reconstruction
of the Benedictine monastery in Coventry, dissolved by
Henry VIII
v
Francis R., 2006. Revolution, learning about history through
situated role play in a virtual environment. Proceedings of
the American educational research association conference.
The game is a role-playing game in the town of colonial
Williamsburg during the American Revolution.
vi
http://www.mylearning.org/my-culture-
quest/interactive/587/
vii
http://www.makebelieve.gr/mb/www/en/portfolio/museums
-culture/54-amm.html
viii
http://www.hideandseek.net/tate-trumps/
ix
www.playinghistory.eu
x
http://www.fas.org/babylon/
xi
http://www.seriousgamesinstitute.co.uk/research. aspx?item
=171§ion=14
xii
http://7thstreet.org/
xiii
http://www.mobygames.com/game/africa-trail
xiv
http://www.educationalsimulations.com/products.html
xv
https://www.leapmotion.com/
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
660