PlayMancer

: A European Serious Gaming 3D

Environment

Elias Kalapanidas

, Hikari Watanabe

, Costas Davarakis

, Hannes Kaufmann

Fernando Fernandez Aranda

, Tony Lam

, Todor Ganchev

and Dimitri Konstantas

Systema Technologies, 215 Mesogion Av., GR-11525, Athens, Greece

University of Geneva, Faculty of Social and Economic Sciences

24 rue General Dufour, CH-1204 Geneva, Switzerland

Vienna University of Technology, Interactive Media Systems Group

Favoritenstrasse 9-11/188/2, A-1040 Wien, Austria

Hospital Universitario de Bellvitge, Servicio de Psiquiatría

Feixa Llarga s/n, E-08907 Hospitalet del Llobregat, Barcelona, Spain

NetUnion SARL, Avenue de Villamont, 19, CH-1005 Lausanne, Switzerland

University of Patras, Dept. of Electrical & Computer Engineering

Wire Communications Laboratory, GR-26500 Rion-Patras, Greece

Abstract. Serious games are about to enter the medical sector to give people

with behavioural or addictive disorders the ability to use them as part of health

promotion and disease prevention. The PlayMancer framework will support

physical rehabilitations and psycho-education programs thru a modular

multiplayer networked 3D game based on the Universally Accessible Games

(UA games) guidelines.

1 Introduction

The potential of games for entertainment and learning has been demonstrated

thoroughly from both research and market. Unfortunately, the investments committed

to entertainment dwarfs what is committed for more serious purposes. Furthermore,

game development has become more complex, expensive, and burdened with a long

development cycle. This creates barriers to independent game developers, and inhibits

the introduction of innovative games, or new game genres, i.e. serious games, or

games accessible to communities with special needs. The aim of the PlayMancer

project is to implement a framework and a platform for serious games. PlayMancer

will implement a new Serious Game environment, by augmenting existing 3D gaming

engines with new possibilities. The objectives of the project are four-fold:

The work described in this paper is supported by the Commission of the European Union in the frame of

the ICT FP7 research program under project number FP7-215839.

Kalapanidas E., Watanabe H., Davarakis C., Kaufmann H., Fernandez Aranda F., Lam T., Ganchev T. and Konstantas D. (2008).

PlayMancer : A European Serious Gaming 3D Environment.

In Proceedings of the 2nd International Workshop on e-Health Services and Technologies, pages 51-59

DOI: 10.5220/0001898900510059

 SciTePress

− To construct a next generation networked gaming environment, mainly augmenting

the gaming experience with innovative information and communication

technologies (ICT) modes of interaction between the player and the game world,

− to allow for a shorter and most cost-effective game production chain, by enabling

techniques for procedural content creation based on generative modelling, and thus

reduce the cost of offering a full-fledged pre-designed gaming world,

− to evolve the principles of Universally Accessible Games (UA games) [1] for

application into 3D-based games, following a design for all philosophy, with the

ultimate goal of designing games to be equally challenging to players of different

abilities and

− to evaluate the proposed framework and gaming infrastructure by developing and

testing a series of serious games modules as applied to two application domains:

physical rehabilitation, and therapeutic support and lifestyle management programs

for behavioural and addictive disorders.

The driving applications for the project will be physical rehabilitation and lifestyle

related disorders. Physical rehabilitation will drive platform requirements for

supporting the development of UA games and the integration of low cost player

motion tracking and bio-feedback devices. Games scenarios from the lifestyle related

disorder will implicate platform requirements for emotion recognition of states such

as boredom, depression, anxiety and associated cognitive responses. Due to the

modular nature of the envisioned PlayMancer gaming platform architecture and the

commitment to Design-for-All philosophy, the project results could be generalised to

other serious games applications and user communities.

2 Concept

Serious games (SGs) or persuasive games are computer and video games used as

educational technology or as a vehicle for presenting or promoting a point of view.

They can be similar to educational games, but are often intended for an audience

outside of primary or secondary education. Serious games can be of any genre and

many of them can be considered a kind of edutainment. The serious games are

intended to provide an engaging, self-reinforcing context in which to motivate and

educate the players towards non-game events or processes, including business

operations, training, marketing and advertisement. Serious games can be compelling,

educative, provocative, disruptive and inspirational.

The PlayMancer project is conceived to take advantage of the current market

momentum towards a next generation gaming platform. After the mass-market

adoption of 3D graphics acceleration cards due to the recent game technology

advancement, the PlayMancer consortium anticipates a similar trend to happen with

new interaction modes in the near future. The game core (the PlayMancer platform)

will be built by encompassing an Augmented Reality (AR) 3D game-like world. AR

has been used so far for scientific applications with success, but due to the fact that

these applications are very specialized, AR constitutes a very limited market and as a

consequence the average cost of AR products are exaggeratedly high for an average

game user. However, we believe that reducing the games production cycle will

accelerate create an economy of scale that will drive down the cost and increase the

availability of games, games components and technology. Thus it is expected that the

cost of AR technology will be considerably dropped, if future game platforms

massively exploit them, the same way that 3D games did for 3D acceleration card

costs.

In remote constrained physical spaces (distributed playgrounds), multiple players

will use an accessible navigation interface through as many information channels as

possible: speech, emotional cues, simple gestures, tactile devices, haptic sensitive

devices, 3D shapes and volumes. Using a full-scale spoken dialogue system that

includes speech recognition and understanding, speaker recognition, emotion

detection, user modelling, context awareness, dialogue flow modelling, natural

language generation, text-to-speech (TTS), etc, would enable advanced human-

machine interaction and eventually higher level of user satisfaction. According to the

game plot, the players will have to navigate through the virtual universe, interact with

AI-motivated digital characters, with each other, make plans, or just watch the

narration of the story. The games will be structured around micro-goals, which will

form missions and quests. By combining different interaction patterns, the players

will be able to achieve these goals. An indicative scenario including such a micro-

goal follows: A player’s avatar has dropped off his wheel chair. In order to sit again

on the chair, the player should: be calm (self-controlled), move to the chair, grab the

chair and then lift himself and put himself on the chair. The game will judge calmness

by getting and processing input from bio-feedback regarding pulse and heart rate, and

then each of the motion actions based on gait monitoring and motion tracking.

3 General Description

The project will follow a user-centred design cycle throughout its development

engaging the different user groups in all the stages of design. For determining the

proper interaction mechanisms that users would like, two iterations will conduct

definition, specification, implementation, integration and utilization on a set of

different interface configurations, enabling different setups of the available modes and

information sources: spoken dialogue interaction, recognition of motion gestures,

recognition of emotional states based on speech, bio-feedback. User-centred design

implies:

− Early focus on users, tasks and environment,

− active involvement of users,

− an appropriate allocation of function between user and system,

− incorporation of user-derived feedback into system design,

− iterative design whereby a prototype is designed tested and modified.

Project iterations (lifecycles) have been designed according to ISO 13407 (Human-

Centred Design Processes for Interactive Systems).

3.1 PlayMancer System Architecture and System Components

PlayMancer will exploit open-source 3D game engines, such as OGRE [2] and

CrystalSpace [3], and develop code and tools in order to deliver a new advanced

computer game platform. The overall architecture will follow the layers in figure 1:

Fig. 1. Deployment of PlayMancer components over the network.

3.2 Motion Capture Systems

In order to enable full body interaction in networked environments, an affordable

motion capture system will be developed. Marker-based optical motion capture has

become a de facto standard capture technology in the movie and entertainment

industry in the previous ten years. Thereby one or more actors wearing retro-reflective

markers attached to them, are tracked by a number of cameras (minimum six). All

motions are computed in real time (with millimetre accuracy) and are available for

further processing e.g. recording, analyzing, motion transfer to a virtual character and

more.

The hardware platform will be based on TUW’s work [4] of an existing accurate,

fast and affordable infrared-optical tracking system. Choosing commodity hardware

over custom-built components has always been a reliable cost-minimization strategy.

Due to lower costs there are a number of fields that would dramatically benefit

from affordable motion-capture including rehabilitation clinics (e.g. gait analysis,

stroke patient therapy and many more) and independent biomedical researchers in

many fields. Even veterinary clinics could use accessible motion-tracking systems to

examine animal gaits and behaviours for diagnosis.

3.3 Bio-feedback Interaction

In recent years bio-feedback has become increasingly important as a non-classical

user interface. Especially in medical applications where users’ biosignals are of vital

importance various sensors have been integrated and used for feedback to patients and

medical personnel.

Within the PlayMancer game biosignals will give important indications on a patient’s

medical condition, his motivation, excitement and engagement. These user input

signals will be integrated into the platform, providing user interaction in a non-

classical way. The game itself will respond to these signals and provide feedback

accordingly. In order to analyze multiple biosignals and to research which sensor

modalities can best be utilized within the project, a mobile biosignal acquisition

device g.MOBIlab (from g.tec) will be acquired. It comes with four EEG/EOG, two

ECG/EMG channels, four digital channels and two analog inputs which can be used

for other sensors. This allows the investigation of brain-, heart-, muscle-activity, eye

movement, respiration, galvanic skin response, pulse and other body signals.

Wireless connection to the game client provides sufficient flexibility to analyze

biosignals in the different PlayMancer game scenarios.

3.4 Gait Rehabilitation Infrastructure

Gait therapy might be necessary after neurological injuries of patients with movement

disorders caused by stroke, spinal cord injury and traumatic brain injury, multiple

sclerosis or Parkinson’s disease. Other causes can be orthopaedic injuries such as

fractures in the leg or foot. In Germany for instance 250.000 people suffer from stroke

every year. Modern concepts of motor learning favour a task specific training, i.e. to

relearn walking, the patient has to walk repetitively in a correct manner.

Up to date a small number of robotic treadmill devices are available on the market

(e.g. Lokomat, HapticWalker,…) which are used for gait therapy in rehabilitation

clinics throughout Europe. One disadvantage of existing devices is that they provide

no kind of visual feedback to the patient. Feedback, especially positive feedback is

very important to encourage and motivate patients to continue training. The motion

capture module of the PlayMancer platform will be used to capture and conduct gait

motion analysis. A PlayMancer game will be developed to provide important positive

visual feedback to the patient. Bio-feedback monitoring will be used in addition to

interface with the game.

3.5 QoS Packet Sniffing and Adaptive State Synchronisation

The traditional game topology of networked games is client-server, with most to all

sensitive data kept server-side. An architecture commonly used for such applications

is a client-server object replication system. In this system, specific objects and data

members are transmitted on creation or changes. This can happen explicitly, with a

call to serialize and replicate, or implicitly, where data is polled every update cycle

and compared to the value it had the last update cycle. This is a good approach that is

both straightforward and easy to understand. It fits into existing single player systems.

With implementations that replicate objects automatically, scripts can modify data

without knowledge of the underlying system and still achieve network

synchronization. However, it is less efficient than a hand-tweaked system, sending

data independent of context and often redundantly. It is also less secure, creating a

strong coupling between game-code and network activity. As game-code changes are

made, especially by programmers that aren't familiar with network security,

vulnerabilities arise. Aiming at improving the efficiency of network usage in

networked games, PlayMancer will introduce a technique for adaptive game state

synchronisation that exploits a quality-of-service packet sniffing component.

3.6 Dialogue Systems

The increasing use of spoken dialogue systems raises the need for more effective and

user friendly interaction between human and machine. Most of the dialogue systems

implemented in the past years do not take advantage of the knowledge of the

emotional state of the user. However, detection of the emotional state of the user can

be proved valuable for the dialogue manager. For instance, retrieving information

about the user’s emotion can provide feedback to the dialogue flow manager so as to

resolve problematic situations.

Further to exploiting knowledge and experience from past research results [5], [6],

[7], PlayMancer is going to develop a full-scale multimodal dialogue system that is

aware about the identity of speakers, their emotional state, the specific context, and

that is capable of natural spoken speech understanding and generation. Specifically,

we will investigate new research directions related to the multimodal dialogue

interaction and to develop novel algorithms for fusion of heterogeneous data streams.

We will invest efforts in designing a next-generation dialogue flow management

algorithms which would enable dynamic compilation of the dialogue flow depending

on various indicators. Finally, an advanced user modelling, context awareness and

behaviour analysis techniques will be employed to enhance the intelligence of the

PlayMancer game platform. The final multimodal dialogue-enabled interaction

manager will interface the PlayMancer game engine and their complementary use

would enable advanced human-machine interaction and eventually higher level of

user satisfaction.

Due the multidisciplinary effort that is required for creation of the PlayMancer

multimodal interaction interface and to the new application area, we anticipate that we

will push beyond state in the area of multimodal interaction, multimodal dialogue

systems, intelligent computing and learning. Specifically, employing novel techniques

such as argumentation based reasoning and hierarchical temporal memory [8] we will

address game-strategy and human-behaviour analysis problems, which up to now

have not been solved.

4 Game Modules

While designing the game scenarios, PlayMancer will consume effort into creating

the main characteristics of flow experience [9]. This is a state where the following

components are identified relative to the player interaction with the game:

− A challenging activity requiring skill,

− a merging of action and awareness,

− clear goals,

− direct, immediate feedback,

− concentration on the task at hand,

− a sense of control,

− a loss of self-consciousness,

− an altered sense of time.

Development of the PlayMancer platform will be driven by serious games

scenario from two application domains: therapeutic support programs for lifestyle

related disorders, and physical rehabilitation. These domains provide some unique

opportunities and challenges to move the networking and platform requirements

beyond the current state of the art and provide a basis for developing a generic

platform that could support application of serious games in other application domains.

Specifically, the focus on physical rehabilitation will drive platform requirements for

supporting the development of UA games and the integration of low cost player

motion tracking and gesture recognition devices. Games scenarios from the lifestyle

related disorder could implicate a platform requirements for multi-modal emotion

measurement, domain rather or task-oriented dialogue, measuring emotion states such

as boredom, depression, anxiety and associated cognitive responses. While guided

online therapeutic support programs are developing at a rapid rate [10], [11] with

promising initial results, several major problems remain: The difficulty of

communicating and working with important cognitive concepts, and the problem of

motivation and compliance, i.e. working with the program consistently. The goal of

PlayMancer is to develop serious games to illustrate difficult to explain concepts, to

provide sensory and emotional feedback, and to provide more enjoyable exercises that

would increase compliance. These games are a replacement. They should be used

initially as a tool for health care professionals. Specific game scenario will be defined

in PlayMancer, to deal with problems, cognitive traits and risk factors common to a

variety of lifestyle related behavioural disorders. These scenarios will be developed

within the PlayMancer platform and will be evaluated in pilot trials to determine user

acceptance and efficacy in improving cognitive believes and lifestyle improvement.

The user requirements and game play scenarios will be developed to insure

anonymity, privacy and confidentiality. These requirements will no doubt require that

no personal data of any kind will be stored on the platform, in some cases, the games

scenario would be only be played by an individual interacting with a game objects, or

with a restricted group of authorised users.

These games are intended for use as part of the health promotion and disease

prevention process, and not as a replacement. The games modules could be evaluated

as supplementary modules (seen as vertical components in the modular framework in

figure 2) based on the SALUT platform for delivery of guided self-management

programs for behavioural and addictive disorders. This platform was developed by

NetUnion within the FP5 research project SALUT and evaluated by hospitals and

clinics in Sweden, Germany, Switzerland, Spain (University Hospital of Bellvitge),

Holland, and Austria. All evaluations were conducted with approval from the ethical

committee of the partner institution.

The games modules could also be integrated, dynamically, based on user need and

interest using a dynamic composition of service model for games delivery. Special

configurations will be tailored for each user group. Scoring and feedback will also be

configured according to needs and requirements for different user groups. All players

will receive some standard scoring, while end user with special needs will be also be

provided additional information, available only to the user, such as charts showing

progress towards a therapeutic objective over time. After the implementation of these

game modules in the 3D platform of PlayMancer, experimental field trials will be

conducted in order to evaluate the effectiveness of applying these modules to adult

players. Evaluation of the games module will be conducted by the University Hospital

Fig. 2. Game modules for gambling addiction and eating disorders treatment.

of Bellvitge, in Barcelona, supported by NetUnion in Lausanne. All evaluation studies

will be conducted in compliance with all national and European ethical standards and

guidelines. Both partners have extensive experience in working with online

therapeutic support programs with populations with special needs within the SALUT

project (cited above) and other national and international research projects. Approval

from ethical committee, and informed consent for patients will be part of any studied

protocol within PlayMancer. The consortium will conform to all applicable laws and

regulations regarding experiments with human subjects.

− The subjects will be selected among the adult population who can give consent.

Most eating disorder (bulimia and binge eating disorders) and pathological

gamblers are adult patients.

− The serious games will only be made available to an adult population having been

fully informed about the purpose of the games and the application of certain rules

of conduct.

− Small populations or closed communities will be selected to take part into the

evaluation field trials. Most of the participants will have to be screened or will

require individual interviews.

The project started in November 2007 and at the time of the writing of this paper it

has reached the state of user requirement identification and a first definition of

functional requirements. We expect to have a first fast prototype by January 2009 and

start trials in May 2009.

References

1. Grammenos, D., Savidis, A., Stephanidis C.: UA-Chess: A Universally Accessible Board

Game. Proceedings of the 3rd International Conference on Universal Access in Human-

Computer Interaction. G. Salvendy (ed.). Las Vegas, USA (2005). Lawrence Erlbaum

2. OGRE (Object-Oriented Graphics Rendering Engine), http://www.ogre3d.org

3. CrystalSpace realtime 3D graphics SDK, http://www.crystalspace3d.org

4. Pintaric T., Kaufmann H.: Affordable Infrared-Optical Pose-Tracking for Virtual and

Augmented Reality. Proceedings of Trends and Issues in Tracking for Virtual

Environments workshop, IEEE VR 2007, Charlotte, USA (2007)

5. Lee, C. M., Narayanan, S.S.: Towards detecting emotions in spoken dialogs. IEEE

Transactions on Speech and Audio Processing, (2005) Vol 13, No. 2, 293–303

6. Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A.: Prosody based automatic

detection of annoyance and frustration in human computer dialog. Proceedings of ICSLP,

Denver (2002) 2037–2040

7. Liscombe, J., Riccardi, G., Hakkani-Tür, D.: Using context to improve emotion detection in

spoken dialog systems. Proceedings of Interspeech (2005), 1845–1848

8. Hawkins, J. and George, D.: Hierarchical Temporal Memory: Concepts, Theory, and

Terminology. Numenta Inc, (2006)

9. Csikszentmihalayi, M.: Flow: The Psychology of Optimal Experience. Harper Perennial,

London (1990)

10. Fernández-Aranda, F., Martínez, C., Núñez,A., Jiménez-Murcia, S.: Nuevas tecnologías en

el tratamiento de los trastornos de la alimentación. En J. Vallejo (Ed.). Update Psiquiatría.

Ed. Masson (2005) 105–118

11. Carrard, I., Rouget, P., Fernandez-Aranda, F., Volkart, A.C., Damoiseau, M., Lam, T.:

Evaluation and deployment of evidence based patient self-management support program for

bulimia nervosa. International Journal of Medical Informatics (2006) 75, 101–109.