http://www.sologicolibre.org/projects/atomic/en/index.php

http://alpha.zooburst.com/

A NOVEL TUTOR-GUIDED PLATFORM FOR INTERACTIVE

AUGMENTED REALITY LEARNING

Héctor Martínez, David Abadía, Luis Miguel Sanagustín, Isabelle Hupont, Rafael Del-Hoyo

Instituto Tecnológico de Aragón, P. T. Walqa Ctra. Zaragoza, N-330a, Km 566, Cuarte, Huesca, Spain

Carlos Sagüés

Departamento de Informática e Ingeniería de Sistemas, Universidad de Zaragoza, C/Maria de Luna 1, Zaragoza, Spain

Keywords: Augmented Reality, Virtual Agents, Interactive learning system, Intelligent Tutoring Systems.

Abstract: Modern education is continuously incorporating new technologies in the learning process. Some of these

technologies involve Augmented Reality applications and virtual agents. The proposed architecture aims to

offer a novel tutor-guided platform for non-programming experienced users to develop intelligent

Augmented Reality e-learning applications. The platform has been used to create a bakery tutorial and some

children games with learning purposes as examples of its capabilities. A pilot experience has been carried

out, and the feedback has shown good results concerning the usefulness and usability of the platform.

1 INTRODUCTION

There is a recent interest on including new emerging

technologies in e-learning systems in order to

enhance the learning process. In particular,

Augmented Reality has been proved to be a useful

tool (Balog, Pribeanu and Iordache, 2007; Chen, Su,

Lee and Wu, 2007; Kaufmann and Dünser, 2007).

The concept of Augmented Reality refers to the

representation of virtual elements over a real scene

captured by a camera. Students find the concept

acquisition more attractive and fun when a virtual

environment is mixed with the reality. Contrary to

other new technologies, Augmented Reality

usability has a fast learning process. Even users who

have never used any Augmented Reality application

before have reported a good feedback in the use of

this technology for education purposes (Sumadio

and Rambli, 2010). Some examples of Augmented

Reality for e-learning can be the MagicBook where

a traditional book is augmented to offer virtual

content (Billinghurst, Kato and Poupyrev, 2001), a

book with finger marker used to enhance the

contents (Hwa Lee, Choi and Park, 2009) or an

application to learn concepts related with the human

body (Juan, Beatrice and Cano, 2008). However, the

interaction in those applications is very limited. The

purposes of those works are mainly focused to show

virtual 3D contents to the users who can see some

objects under different angles and dimensions to

better understand how they work. The main gap

between Augmented Reality applications and

educators is the lack of programming skills of the

educators. Thus, the creation process involves

computer science experts and pedagogic

professionals. Some user-friendly authoring tools

have arisen to help those people who don’t have

programming skills to make some simple but

powerful Augmented Reality applications. Some

authoring tools examples can be ATOMIC

and

ZooBurst

. However, the created applications are

limited to show contents and the logic of the

program is fixed by the software creators.

The use of 3D environments enables the use of

intelligent virtual agents. In the field of e-learning,

the benefits of using virtual humans capable to adapt

the transmission of knowledge to each student have

been proved (Sklar and Richards, 2006).

It is important to point out that the virtual agent

shows intelligent behaviours that respond

accordingly to the evolution of the interaction, like,

for instance, offering help when needed.

The proposed system is a novel tutor-guided

platform for interactive Augmented Reality learning.

Martínez H., Abadía D., Miguel Sanagustín L., Hupont I., Del-Hoyo R. and Sagüés C..

A NOVEL TUTOR-GUIDED PLATFORM FOR INTERACTIVE AUGMENTED REALITY LEARNING.

DOI: 10.5220/0003337600880093

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 88-93

ISBN: 978-989-8425-49-2

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

The presented system enables to create intelligent

Augmented Reality applications for learning

purposes. Instead of just showing contents, the final

applications are able to offer a rich variety of

interactive actions. The system uses an intelligent

framework that enables to define application logic

using natural language. Therefore, any non-

programming expert person is able to create

interactive Augmented Reality exercises for e-

learning with his/her imagination as the only

limitation.

One of the main features of the proposed system

is the introduction of a virtual tutor, as an intelligent

interactive virtual agent who guides the user through

the learning process and evolves his behaviour in

function of the user’s actions, achieving more

interactivity with the student making the exercises.

Thanks to the applications created with the

proposed system, the traditionally acquired learning

concepts can be moved into practical exercises. Due

to the attractiveness of some of the technological

elements included in the system, such as Augmented

Reality exercises and virtual tutors, the learning

process becomes more interesting for students.

The rest of the text is organized as follows:

section 2 describes an overview of the proposed

platform architecture. In section 3, a practical

example is explained to show the potential of the

platform. Finally, in section 4 some conclusions and

future work are discussed.

2 PLATFORM ARCHITECTURE

OVERVIEW

The proposed platform is a powerful authoring tool

for teachers and educators. The system enables the

users to create a great variety of Augmented Reality

tutorials, practices or games for learning purposes.

The system also offers the possibility of including an

intelligent virtual tutor who guides the student

through the learning process. Figure 1 shows the

proposed architecture. The system has three

modules: Perception module, Artificial Intelligence

module and 3D Multimedia Scene Manager.

2.1 Perception Module

The Perception module allows the student to

communicate with the system. The communication

can be established by different ways. The main

feature of the Perception module is the Augmented

Reality system, which is the core of the interaction.

The Augmented Reality system detects some

markers in the image captured by the camera and

calculates their 3D coordinates. Once those

coordinates have been calculated, the virtual

elements will be mixed with the real video motion

and displayed to the student according to the

markers position and orientation. The fact of

manipulating the virtual elements makes the student

to feel like manipulating real objects.

Apart from the Augmented Reality system, the

user can communicate with the system through a

keyboard, a mouse and a microphone. These system

inputs are sent to the Artificial Intelligence module,

being the system logic completely transparent to the

student.

2.2 Artificial Intelligence Module

This module is the engine where the system logic is

defined. It is also the inference engine that makes

the virtual tutor to react to the student inputs. In this

module, the educator can define the instructions for

the student to carry out the exercises. The system

Figure 1: Proposed system’s architecture.

A NOVEL TUTOR-GUIDED PLATFORM FOR INTERACTIVE AUGMENTED REALITY LEARNING

is controlled by an intelligent rules-based framework

called ISIS: Intelligent Support Interaction System

(Martínez, del-Hoyo, Sanagustín, Hupont, Abadía

and Sagüés, 2011). Any non-programmer user can

easily implement the system behaviour by defining

natural language rules. Two kind of rules can be

established: crisp and fuzzy rules (Zadeh, 1965).

Figure 2 shows an example of fuzzy rules definition

using the ISIS’ interface. Furthermore, the language

interaction between the virtual tutor and the student

is described using the Artificial Intelligence Markup

Language (AIML).

Figure 2: Screen of the ISIS tool for Fuzzy rules

definition. The screen is divided in two parts. The first one

is used to define the attributes (the definition of fuzzy

attributes is accompanied of their fuzzy sets). The second

part is used to define the system rules.

2.3 3D Multimedia Scene Manager

The 3D Multimedia Scene Manager is the module

that creates a virtual environment to be mixed with

the real video motion. This module communicates

with the Artificial Intelligence module and evolves

according to the data received from it. It allows to

include any kind of multimedia content (3D models,

audio, video, still images or webpages) in the e-

learning application.

One important figure in the system is the virtual

tutor, which is the entity in charge of guiding the

user through the learning process. The virtual tutor

can be selected according to the context of the

application and the end user. For example, a human

look-like virtual tutor can be used for training

formative applications while some toons can be used

as virtual tutors for children learning applications

(using different toons depending on the age of the

target students). The fact of personalizing the virtual

tutor gives an added value to the system because the

student gets more empathy with the virtual tutor and

he/she makes so in a faster way. The student is able

to interact with the virtual tutor who responds in an

intelligent way to the actions made by the user as

wrong answers, help questions and so on. The

virtual tutor intelligent behaviour is controlled by

ISIS so it is able to interact with the user in an

intelligent natural way. The virtual tutor figure has

been created with the purpose of making the student

to feel accompanied at every moment. The virtual

tutor communicates with the student through chat or

voice but also with emotions and gestures. It is also

in charge of offering help to the student both when

the virtual tutor thinks that it is necessary or when

the student asks for it. The offered help may be a

simple comment but also can be a more complex

element such as a video or a web browser. When the

conversation student-virtual tutor is established, the

latter will search in a question-answer engine and

will answer as accurately as possible attending to the

student’s needs.

3 PRACTICAL APPLICATIONS

In order to prove the potential of the system, two

different application examples have been created.

Firstly, a tutorial for making bread has been

developed. This tutorial is oriented to train future

bakers, helping them to acquire and reinforce the

concepts needed to cook bread. The tutorial consists

of some exercises oriented to different difficulty

levels. The second developed application consists of

a series of educational games for children to learn

basic concepts, such as learning vowels or

increasing their creativity.

3.1 Bakery Tutorial

The bakery tutorial is an example created to show

the capabilities of the system oriented to formative

training applications. The tutorial is guided by a

human look-like virtual 3D tutor. The tutorial

instructions are offered in two formats

simultaneously (audio and text), in order to achieve

a better comprehension of the information from the

student. The system also shows some videos when

some tasks are successfully completed. Thanks to

the combination of different exercises and rules, the

educator is able to offer different applications for

different students, adapting the level for each case.

The tutorial begins with a welcome message and the

student is asked to put the virtual tutor in the scene.

Once the virtual tutor is visible, it introduces himself

and the tutorial begins. The tutorial consists of a

CSEDU 2011 - 3rd International Conference on Computer Supported Education

variety of consecutive exercises with different levels

of difficulty.

Depending on the exercise, the student is

instantiated to select, between the different tools,

those needed to cook the bread (Figure 3.a). He/she

can also be asked to locate the available ingredients

on the scene (Figure 3.b).

Another example of interaction in the exercises

is the possibility of modifying the 3D objects

properties. Thanks to some useful controls (such as

buttons and selectors), the student is able to set the

quantity of every ingredient needed to prepare the

bread. For example, Figure 3.c shows how the

quantity of the water needed to properly cook the

bread can be fixed.

Figure 3.d shows an example of a tutorial

exercise that has been successfully completed. The

goal of the exercise is to select the right ingredients

and take them next to the oven (leaving apart those

which are wrong). When the right ingredients have

been selected, those ingredients and the oven

disappear and a piece of bread appears instead.

Finally, an explanatory video is displayed to

enhance the concepts that have been learned.

Figure 3: Some examples of exercises. (a) and (b) are

choosing exercises (ingredients (a) and tools (b)). (c) is an

example of virtual buttons usage. The user can change the

quantity of water needed to cook the bread using the

virtual controls. (d) Another practical exercise. The tutor

reacts to the user’s actions and a video is displayed when

the exercise has been successfully solved.

The virtual tutor accompanies the student

through the different exercises. It explains the

concepts, asks the student to perform some tasks and

congratulates the student when some task has been

achieved. The virtual tutor does not only

communicate with the student through his voice but

also with nonverbal communication. The tutor can

react with some emotions, such as smiling or

showing a sad expression, according to the different

actions of the student. As it has been mentioned, the

tutor may help the student to accomplish the

different tasks. Apart from explaining the different

goals, the virtual tutor offers some help when it

detects that the student has some problems. A chat

help mode is also available, in order to allow the

student to ask for help using the keyboard.

As it has been explained, the system logic can be

completely defined by the educator through natural

language rules. The tutorial has been implemented

with those rules. An example of fuzzy rule can be

the following:

if ((Distance_Oven_Salt is near) and

(Distance_Oven Yeast is near) and

(Distance_oven_water is near) and

(Distance_oven_Egg is far)) then

Success_Degree is success

As it can be seen, the rule defines the condition

to successfully complete the exercise showed in

Figure 3. Every distance has been defined as a fuzzy

variable. The quantity of every ingredient has also

been defined as a fuzzy set, so it is easy to define

another rule to establish the condition of a right

quantity for any ingredient, as, for example:

if (Water_Quantity is high) then

Success_Degree is failure

3.2 Children Learning Games

The second application is made up of some

educational games for children. The goal of those

games is to learn some basic concepts while playing.

The games are hosted by a virtual tutor. The virtual

tutor is here a ginger bread toon. Two examples of

those games are introduced in the next lines.

The first example is a puzzle game. The child is

asked by the virtual tutor to complete a basic puzzle.

If the virtual tutor detects that the child is having

some problems to achieve the goal, it encourages

him/her. A help chat mode is also available if

needed. When the puzzle is completed, the virtual

tutor congratulates the child and remarks the

congratulation with some nonverbal communication

such as smiling and dancing. Figure 4.a and 4.b

show an example of the game. The game logic has

been implemented in ISIS with fuzzy attributes for

distances and angles between the different pieces of

the puzzle.

Another example of learning-oriented child game

is a vowel-learning game. The five vowels are

shown to the child, and the virtual tutor asks him/her

to press one of them. If the wrong vowel has been

selected, the virtual tutor shows a sad face and asks

A NOVEL TUTOR-GUIDED PLATFORM FOR INTERACTIVE AUGMENTED REALITY LEARNING

the vowel again. On the other hand, if the right

vowel has been pressed, the virtual tutor smiles and

dances. In order to enhance the learning process,

when the right vowel has been selected, the virtual

tutor says some sentences relative to some words

that begin with that letter. After that, another random

vowel is asked to continue with the game. Figure 4.c

and 4.d show the proposed game.

Figure 4: Examples of children learning games. Some

puzzle pieces (a) and vowels (c) are offered to the child.

When the puzzle is successfully completed (b) or the right

vowel has been selected (d), the virtual tutor congratulates

the child, smiles and dances.

3.3 Evaluation with Users

The bakery tutorial has been used in a pilot

experience. Twenty students between 20 and 50

years old have tested the tutorial and have been

asked to evaluate the system through a poll. Some

basic Augmented Reality examples have been

introduced to the students using the system. Once

the concept has been introduced to the students, they

have tried the different exercises.

Firstly, the students have performed the exercises

without the presence of the virtual tutor. After that,

the same exercises have been carried out with the

help of the virtual tutor.

When the tutorial has finished, the students have

been asked to evaluate the system. The main ideas

obtained from the polls have been that the

interactivity based on Augmented Reality and the

virtual tutor are two elements that improve the

concept acquisition. Moreover, the system has

proved to be very usable even in the cases of

students who have never used an Augmented Reality

application before. The fact of having the possibility

to interact has been also pointed out by the students.

Figure 5 shows the results of the acceptance levels

of some aspects.

Figure 5: Remarkable aspects of the bakery tutorial. The

interactivity, usability and the help offered by the tutor

have been the most valuable aspects for the students.

It is also remarkable that 3D models and tutor

pitch are a crucial issue for the student’s perception

and credibility towards such interactive systems.

4 CONCLUSIONS

AND FUTURE WORK

An intelligent interactive e-learning authoring tool

based on Augmented Reality has been presented.

The system allows teachers to create interactive

learning applications including exercises, tutorials

and games. The applications can be adapted to

different environments, ranging from training

courses to school concepts acquisition for children.

The application of an artificial intelligence rules-

based engine with Augmented Reality as user

interface supposes a new approach respect to

existent work, as it enables the creation of formative

applications based on rules, which define the

interaction between the virtual elements (3D models,

images, sounds and videos) and the student in order

to enable the improvement of the interactivity in the

learning process.

The proposed Augmented Reality applications

consist of a work area (typically a desk) where some

markers are shown to a camera connected to a

computer. The system detects the markers with

pattern recognition and offers the markers position

and orientation to an intelligent virtual environment.

Then, the virtual environment is shown over the real

video in real-time, according to the markers position

and orientation.

One important feature offered by the system is

the ability of interaction. The user can manipulate

the markers, which at the end supposes to interact

with the virtual elements (3D models, images,

CSEDU 2011 - 3rd International Conference on Computer Supported Education

browsers…). Some of those virtual elements are

buttons or selectors what enables the user to change

some properties of the other virtual elements. Other

features of the virtual elements, such as distances

and angles, are also properties that evolve the

system, according to how the student manipulates

them.

The complete learning process is guided by a

virtual tutor. The virtual tutor can be selected from

the available ones according to the context and the

end user. For example, a human look-like model has

been used to develop a training tutorial while some

toons have been used to create children learning

games. The virtual tutor teaches the student and

offers him/her some exercises to carry out. The

student can interact with the virtual tutor in a variety

of ways (listening to the instructions, chatting,

talking to it or receiving nonverbal communication).

The virtual tutor acts according to intelligent

framework, which is also responsible of the system

logic.

The bakery tutorial has been tested in a pilot

experience and has reported good results. The

students have been able to use the application

without any problems even if they had not used an

Augmented Reality application before. The help of

the virtual tutor has shown to be enough to start

using the application. The results of the experience

agree with the results of other similar studies (Balog

et al., 2007; Chen et al., 2007; Kaufmann and

Dünser, 2007).

Besides, the virtual 3D representation of

complex objects may be a help for the student to

assimilate the concepts because sometimes it is

difficult to visually image the objects.

In the future, the system is expected to

automatically analyze the learning indicators

obtained in order to adapt the contents for each

student in real-time, instead of the mediation of the

real tutor and so to provide a personalized learning

process. Emotional detection for content adaptation

is also a desirable feature to obtain in the future.

In order to track the student learning process by

the educator, currently the system is being integrated

in a standard Learning Management System.

ACKNOWLEDGEMENTS

This work has been partly financed by the CETVI

(PAV-100000-2007-307) and the RA-IA Learning

(TSI-020302-2010-155) projects funded by the

Spanish Ministry of Industry and the Grupo de

Ingeniería Avanzada (GIA-SISTRONIC) of the

Instituto Tecnológico de Aragón.

REFERENCES

Balog, A., Pribeanu, C., Iordache, D., 2007. Augmented

reality in schools: Preliminary evaluation results from

a summer school. International Journal of Social

Sciences, 2(3), pp. 163–166.

Billinghurst, M., Kato, H., Poupyrev, I., 2001. The

MagicBook — Moving Seamlessly between Reality

and Virtuality. IEEE Computer Graphics and

Applications, vol. 21, no. 3, pp. 6-8.

Chen, C. H., Su, C. C., Lee, P. Y., Wu, F. G., 2007.

Augmented Interface for Children Chinese Learning.

7th IEEE International Conference on Advanced

Learning Technologies, pp. 268–270.

Hwa Lee, S., Choi, J., Park, J., 2009. Interactive e-learning

system using pattern recognition and Augmented

Reality. IEEE Transactions on Consumer Electronics,

vol.55, no.2, pp. 883-890.

Juan, C., Beatrice, F., Cano, J., 2008. An Augmented

Reality System for Learning the Interior of the Human

Body. 8th IEEE International Conference on

Advanced Learning Technologies, ICALT. 186-188.

Kaufmann, H., Dünser, A., 2007. Summary of usability

evaluations of an educational augmented reality

application. Virtual Reality, pp. 660–669.

Lin, C. T., Lee, C. S., 1996. Neural fuzzy systems: a

neuro-fuzzy synergism to intelligent systems,

Prentice-Hall, Inc. Upper Saddle.

Martínez, H., del-Hoyo, R., Sanagustín, L. M., Hupont, I.,

Abadía, D., Sagüés, C., 2011. Augmented Reality

Based Intelligent Interactive e-Learning Platform. 3rd

International Conference on Agents and Artificial

Intelligence (ICAART).

Sklar, E., Richards, D., 2006. The use of agents in human

learning systems. Proceedings of the 5th international

Joint Conference on Autonomous Agents and

Multiagent Systems AAMAS '06. ACM, New York,

NY, pp. 767-774.

Sumadio, D. D., Rambli, D. R. A, 2010. Preliminary

Evaluation on User Acceptance of the Augmented

Reality Use for Education. Second International

Conference on Computer Engineering and

Applications. ICCEA, vol. 2, pp. 461-465.

Zadeh, L. A., 1965. Fuzzy sets. Information and control,

8(3), pp. 338–353.

A NOVEL TUTOR-GUIDED PLATFORM FOR INTERACTIVE AUGMENTED REALITY LEARNING