DEVELOPMENT OF THE VIRTUAL ELA

-HOUSE

Milán Horváth

, Csaba Dániel

, Jacqueline Stark

and Cecília Sik Lanyi

1, 2, 4

Department of Image Processing and Neurocomputing, University of Pannonia, Veszprém, Hungary

Austrian Academy of Sciences, Department of Linguistics and Communication Research, Vienna, Austria

Keywords: Virtual reality, aphasia, language therapy.

Abstract: In this paper, the rationale for the development and the process of creating the Virtual ELA

(Everyday Life

Activities)-House are described. The Virtual ELA

-House is an innovative therapy program designed for

use with clients with language and speech disorders and/or with other cognitive neuropsychological

disorders, which result from brain damage, e.g. aphasia, apraxia of speech, neglect, etc. The Virtual Reality

setting is chosen as a modern and relevant therapy setting which imitates real everyday life scenarios.

Computer supported cognitive and language therapy allows for repetitive application in the clinical and

home setting which is necessary for learning to take place. The advantages of employing a software

program based on a Virtual Environment, in particular the Virtual ELA

-House, are discussed.

1 INTRODUCTION

Virtual Reality offers a wide range of applications in

the field of cognitive neuropsychology, both in

diagnosing cognitive deficits and in treating them.

Virtual Reality is also a useful tool for skill-building

and training by creating a virtual setting, which

imitates the real environment including the attributes

to be trained. The virtual world is a computer-based,

simulated environment intended for its users to

inhabit and interact via avatars. This habitation is

usually represented in the form of two-or three-

dimensional graphical representations of humanoids

(or other graphical or text-based avatars) (Stark,

2005).

A Virtual ELA®-House was created to provide

an alternative means of language and cognitive

rehabilitation for use with clients suffering from

various disorders of higher cortical functioning

resulting from brain damage (The National Aphasia

Association). The decision to create a Virtual

ELA®-House was based on the impressive language

therapy results from single case studies of aphasia

based on the analogue version of the ELA- Photo

Series (Stark, 1992-1998), (Stark, 2003).

One possible reason for the significant

improvement in language skills following provision

of ELA®-based therapy protocols is that the picture

stimuli are realistic and relevant for everyday life.

Thus, an important goal in this project was to attain

a high degree of correspondence with the analogue

ELA®-picture stimuli, which is also required for an

authentic virtual world. In a virtual world the

designer of a task attempts to capture all aspects

which a particular task requires, but it is very crucial

for it to be true to nature. If the virtual world

established reaches this level, the user will feel that

he/she is part of the Virtual Environment. The term

Virtual Environment can refer to a room, a place, a

house, etc.

During the design process of a Virtual World, the

designer should be aware of the available

possibilities. To design or model a virtual world, a

personal computer is needed with 3D modeling

software and a programming environment.

2 FIRST STEPS OF THE

DEVELOPMENT – DESIGN,

MODELING AND

CONVERTING

2.1 Design and Modeling

Before modeling the Virtual World, the entire

project must be designed. In our case the layout for

the ELA®-Virtual House was designed by

519

Horváth M., Dániel C., Stark J. and Sik Lanyi C. (2008).

DEVELOPMENT OF THE VIRTUAL ELA

 -HOUSE.

In Proceedings of the Third International Conference on Computer Graphics Theory and Applications, pages 519-523

DOI: 10.5220/0001099205190523

 SciTePress

Jacqueline Stark. Figure 1 depicts the layout of the

house, which consists of various rooms located on

two floors and a basement:

Figure 1: The floor layout for the ELA®-Virtual House.

After completion of the room layout for ELA®-

Virtual House, modeling the components for the

ELA®-Virtual House was initiated. For this

operation, of the several available programs, we

chose the Autodesk Maya(Autodesk Maya) program

for the ELA®-Virtual House.

2.1.1 Criteria for Selecting the Autodesk

Maya

3D modeling is the process of creating three-

dimensional surfaces using a computer for the

purpose of rendering them into a picture or a

sequence of pictures (Harovas et. al, 2000). Maya is

a manifold 3D modeling program, which

encompasses textures, thus allowing for animating

software. This program is very versatile due to its

wide range of applications. The Maya project has a

scene where the house is modeled and built by

means of a wide range of tools. To model a building

- such as the ELA®-Virtual House – the first step

was to create a polygon cube. This cube can be

transformed, rotated or scaled to achieve the desired

position and size. The cube must then be divided for

the next modeling steps. Every polygon object has

faces, vertices and edges. In the case of a cube there

are 8 vertices, 6 faces and 12 edges. It is possible to

transform, rotate or scale these vertices, faces and

edges. The subdividing process generates more

vertices, faces and edges, which allows the building

of more exact and realistic objects.

Figure 2 shows the basic form of the cube after its

creation. In Figure 3 the result of the subdividing

process is shown, namely, that there are more

vertices, faces and edges which are necessary to

model the task.

Figure 2: A simple cube.

Figure 3: A cube with rotation, translation and scale.

In Maya there are eight fundamental polygon

objects: sphere, cube, cylinder, cone, plane, torus,

pyramid and pipe. With Maya it is possible to build

any objects from these polygons with the

aforementioned transformations, rotation and scales.

An object can consist of one polygon object, or

several polygon objects. In the second case, objects

which belong together can be combined by using the

Combine polygons tool.

2.2 Providing Realistic Textures and

Materials

In the modeling process, after the object has been

completed, the next step is to give it texture. For this

operation texture mapping was used. Texture

mapping is a method for adding detail, surface

texture, or color to a computer-generated graphic or

3D model (Texture mapping). A virtual house has

many textures, mostly in JPEG format. Maya can

work easily with this format as well as with almost

all of the other picture formats. The ELA

-project

requires the use of precise and realistic images,

therefore the analogue photos from the original

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

520

ELA

-Photo Series were scanned and additional

digital photos were made of components of the real

rooms. These images provide the basis for the

modeling of the entire house and the individual

rooms. The important parts of the pictures were

copied and used to texture the objects in Maya.

There are several methods for texturing an object,

but the simplest way is to assign a material to an

object. First of all, the type of material is chosen. In

this project, lambert material was used for the wall

material. This is a material (shader) that represents

lusterless or dull, mat surfaces (such as chalk,

sheenless paint, unpolished surfaces) without

mirroring qualities. For metal-like materials, blinn

material is used. It is a material (shader), that is

particularly effective at simulating metallic surfaces

(for example, brass or aluminum), which typically

give soft specular, i.e., mirroring , highlighting. It

must be stressed that Maya supports many types of

materials. With Maya it is even possible to design

one’s own material by adding special colors,

transparency, ambient color or incandescence. Maya

saves the configurations and these can be used for

other models.

In the process of modeling and texturing,

numerous objects were created, including walls,

windows, floors, doors, etc. It is very useful to use

layers in Maya, especially if the project consists of

many individual objects. The project is more

comprehensible, if the individual parts are

categorized. Working with layers makes it easier to

maintain an overview of the whole project. The

procedure is simply to add a new layer to the project,

and select the objects which belong to the same

category, and then add selected objects to the layer.

First, the layer must be named. When the modeling

and texturing part of the layer’s objects has been

completed, it can simply be hidden, if necessary.

Proper layers represent categories. In each

category there are objects which have their own

texture map or color, and which belong together.

The result of modeling and texturing is shown in

Figure 4, where the layers present the individual

parts of the house, such as the roof, the inside walls,

the furniture, etc.

2.3 Converting: An Advantage of

Software Independence

This project was developed in the programming

environment Eclipse (Eclipse), where the

programming language is C++. Use of the engine

Irrlicht (Irrlicht Engine) enables the user to walk

around in the Virtual House, click on several objects

Figure 4: The ELA®-House in Maya.

around in the Virtual House, click on several objects

and respond interactively. Irrlicht is a free open

source 3D engine. However, one problem involved

in using Irrlicht is that it does not support the Maya

binary files. It was a challenge to find a way to

export the model to a file type supported by Irrlicht.

Maya can export its models to object format, but this

format contains only geometry, and textures cannot

be saved in this format. The solution to this problem

was the program Right Hemisphere’s Deep

Exploration (Deep Exploration).

2.3.1 The Advantages of using Deep

Exploration

Deep Exploration supports almost all 3D or 2D file

types, including Maya’s scenes. When the program

runs, a scene, a file browser, and a little viewer

appears. It is possible to open a saved Maya file in

the scene field of Deep Exploration. Deep

Exploration is not merely a file converter. When the

file is loaded, the user is able to turn or fly around

the object, zoom in or zoom out, but it is not

possible to modify the object. Deep Exploration can

assign textures or colors to the objects, but this

operation and the possibilities are very far from

those which one is able to accomplish with Maya.

Once the scene is loaded, the software is ready to

export the result to all file types that Deep

Exploration supports. A very popular format is the

DirectX file format. The DirectX file format is an

architecture- and context-free file format. It is

template-driven and is free of any user-knowledge or

experience. The file format may be used by any

client application and currently is used by Direct3D

Retained Mode to describe geometry data, frame

hierarchies, and animations (DirectX). If the

exporting process is done correctly, the resulting file

is an .x file consisting of the whole geometry and

inside the file the paths of the texture files are also

DEVELOPMENT OF THE VIRTUAL ELA®-HOUSE

521

included. As of this point the resulting .x file is

ready to be imported into the Irrlicht.

3 THE SECOND STEP OF THE

DEVELOPMENT: MAKING

THE MODEL INTERACTIVE

BY USING IRRLICHT

The Irrlicht Engine is an open source, high

performance, real time 3D engine written and usable

in C++ and also available for .NET languages. It is

completely cross-platform, using D3D, OpenGL and

its own software renderer, and it integrates all the

state-of-the-art features for visual representation

such as dynamic shadows, particle systems,

character animation, indoor and outdoor technology,

and collision detection, which can be found in

commercial 3D engines. It is a powerful,

high level

API for creating complete 3D and 2D applications

required in games or scientific visualizations. It

comes with an excellent documentation, which is

very useful in the development of software.

This engine is being used in many projects.

There are enhancements for Irrlicht all over the web,

including alternative terrain renderers, portal

renderers, exporters, world layers, tutorials, editors,

language bindings for java, perl, ruby, basic, python,

lua, and so on.

Irrlicht has its own graphical file manager,

named irrEdit (Ambierra: irrEdit), to load, rebuild

and test our C++ project every time. In irrEdit it is

possible to load many .x files, position, rotate or

scale the object, but it cannot be modified. After the

house model was loaded, some furniture models

were added, so it became possible to furnish the

whole house according to the layout. The irrEdit

loader can save the objects, transformations,

rotations and scales in its own file format, which at

the moment is best for Irrlicht. These saved scenes,

which can be modified for the specific exercises, are

loaded into the program.

4 THE MAIN FEATURES OF THE

SOFTWARE

The Virtual ELA

-House program can be executed

on Windows following installation. The project

supports two languages for the clients: German and

English. Before beginning the program, the user can

choose between a male and a female voice. After

that the user selects an exercise from the various

types of exercises. The picture stimuli used for all of

the language therapy tasks are taken from the ELA

Photo Series. A constant blue-red figure shows the

user where he/she is looking at the moment, when

he/she is walking in the house.

4.1 Types of Exercises for the Virtual

House

4.1.1 Discovery Task

In this task, the avatar wanders through the house on

his/her own to learn the vocabulary of the objects

and activities in each room of the house. Two

different symbols are used for showing the active

object. A blue cube is for an object, as shown in

Figure 5 and a yellow sphere is for an activity. When

the client clicks on an object by means of the right

mouse button, a window appears. In the window an

image of an object or an activity and four buttons are

shown. With three of these buttons the user can read

or hear separately, or see and hear the description of

the image simultaneously. With the fourth button

he/she can close the window, and continue

discovering the components of the house.

Figure 5: A virtual table with chairs.

4.1.2 Structured Discovery Task

A variation of the first task is that the avatar is

standing outside the house. The avatar receives

either an auditory command or one in written

language to which it must respond. The following

questions are included in this task:

• ‘Where does activity ‘x’ usually take

place?’ or, ‘Where is activity ‘x’ usually

performed in the house?’ A sentence is heard or

is written on the screen requesting the avatar to

GRAPP 2008 - International Conference on Computer Graphics Theory and Applications

522

move to that location in the house, where

he/she thinks the activity could be performed.

For one level of difficulty, when he/she clicks

on the correct room, a certain symbol blinks.

At a more difficult level, the avatar has to be

more specific and click on the exact location,

where the activity takes place. For example, if

it is a picture of a boy brushing his teeth, for the

first level the avatar would move into the

bathroom, To be more specific, the avatar must

click on or near the sink in the bathroom. If the

avatar makes a mistake in the selection of the

room, some response is heard or seen, signaling

to him/her to continue the search. If the avatar

is in the right room and makes a mistake

pertaining to the exact position, then he/she

also hears/sees a response: ‘You are getting

closer, keep trying/looking’.

• ‘Where does ‘x’ object belong in the

house?’ The avatar clicks on an icon on the

screen and an object appears. He/She should

then look for the location where that object

should be placed in the house. The same types

of responses are used as in the previous

exercise.

4.1.3 Memory Task

Several items (i.e. objects) are shown on the screen

and the program then puts these pictures in different

places in the house, one at a time. (Range of

difficulty: the number can be set to include 3 to 5

objects). The avatar’s task is to find the hidden items

in the house in the same order as they were shown.

The items can be hidden in different places.

These are a few of the exercises included in the

present version of the Virtual ELA

-House.

5 CONCLUSIONS

The software development and the designing and

modeling of a Virtual ELA

-House is a very

complex task. The project has two significant parts,

which can be completely separated from each other.

The first part is the software development, and the

second is the modeling and texturing. During the

modeling, the most difficult task was to produce a

house which gives the users a feeling that the house

exists, and that they are actually walking in the

house. For this task digital photographs were used

from a real house. Difficulties arose during the

texturing part of the project. New images were

created based on the actual photos, with different

resolutions and compilation. In this project, the

programming task was completed after being tested

only once in a simple building. This was necessary

because the software development and the modeling

tasks were performed at the same time. After the

second task was completed, the two tasks were

integrated into the software environment, where the

finished project was built and tested.

In the future, other exercise types will be added

to the program which will be based on feedback

from the clients and an analysis of the data collected

while clients are using the ELA

-virtual house

program. An author system will also be developed

for the therapists working with individual clients to

adapt the program according to the individual needs

of each client.

ACKNOWLEDGEMENTS

The authors would like thank the Austrian Science

and Research Liaison Office (project number:

2007.ASO-N/4/5) for their support in the

development of the ELA®-Virtual House.

REFERENCES

Stark, J. A., Analysing the language therapy process: The

implicit role of learning and memory. Aphasiology,

19, 10/11 (2005), 1074-1089.

The National Aphasia Association, which is available at:

http://www.aphasia.org/

Stark, J. A., Everyday Life Activities Photo Series,

Manuals to Set 1 to Set 3. Bösmüller/ Jentzsch, Vienna

(1992-1998)

Stark, J. A., Everyday Life Activities Object Photo Series

Set 1, Jentzsch, Vienna, (2003)

Autodesk Maya, available at: http://usa.autodesk.com/

adsk/servlet/index?siteID=123112&id=7635018#

Perry Harovas, John Kundert-Gibbs and Peter Lee,

Mastering MAYA Complete 2, ISBN: 0-7821-2521-2,

2000 SYBEX, Inc.

Texture mapping, which is available at: http://

en.wikipedia.org/wiki/Texture_%28computer_graphics%29

Eclipse – an open development platform, which is

available at: http://www.eclipse.org

Irrlicht Engine, Available at: http://irrlicht.sourceforge.net/

Deep Exploration, which is available at:

http://www.righthemisphere.com/products/dexp/

DirectX file format, which is available at: http://

local.wasp.uwa.edu.au/~pbourke/dataformats/directx/

Ambierra: irrEdit, which is available at:

http://www.ambiera.com/irredit

DEVELOPMENT OF THE VIRTUAL ELA®-HOUSE

523