AN EMBODIED CONVERSATIONAL AGENT FOR

COUNSELLING ABORIGINES

Mr. Warnanggal

Manolya Kavakli, Tarashankar Rudra and Manning Li

Department of Computing, Macquarie University, NSW 2109, Sydney, Australia

Keywords: Agent based modelling, Virtual sociologist, Expert system, Natural language processing.

Abstract: Aboriginal people are the most neglected community in Australia. Although the trauma induced through

oppression and genocide by the settlers is long gone, the deep scar still manifests through the consequences

that are painfully apparent in their community. One of the consequences of past neglect and torture is the

excessive consumption of alcohol and use of illegal substances. This has compounded the agony of

aboriginal people to incomprehensible proportions and forced a sizable population into a grinding cycle of

poverty and disease. Our paper proposes a novel approach to provide personalized counselling services to

the aboriginal people by developing an interactive virtual sociologist as an embodied conversational agent.

The system will simulate the role of a real sociologist in advising on strategies to overcome their addiction

to alcohol and substance use and hence enjoy the fruits of prosperity with the rest of the Australian

community.

1 INTRODUCTION

Australoids (Elkin, 1979) or Australian aborigines

were food originally gatherers and hunters and are

the original inhabitants of Australia who migrated to

the mainland about 40,000 years ago (Elkin, 1979).

Prior to colonization, the Aboriginals enjoyed an

ideal lifestyle with roles of individual members set

according to their position in the tribe; families

would live in a communal environment with

responsibilities being shared throughout the family

(Walker, 1993). The men were usually hunters while

women gathered nuts, berries and roots for the

family (Sam, 1992)

Today, Aboriginals are the most socially and

economically backward group in the Australian

society and are vulnerable to substance (alcohol,

tobacco, illicit drug and volatile substance) (AI

HealtInfoNet) abuse. According to the Australian

Bureau of Statistics (ABS), there are around 517,200

indigenous people in Australia (see ABS).

Government report on drug abuse in Australia shows

52% of aborigines have consumed substance in one

form or the other (Editor, 2007). The fact sheet from

the office of aboriginal and Torres Strait islander

affairs quotes the life expectancy of indigenous

Australians is 20 years less than other Australians

(Editor, 2003). One of the four primary factors

attributed to this incongruity is alcohol and tobacco

misuse; the other three being lower socio-economic

status, location and environmental factors and

historical factors (Healey, 2004).

Post colonization, aboriginal people were

subjected to oppression and genocide that left the

society in tatters. Racial discrimination over the

years have caused great disadvantage in their

employment, housing, health, education and training

thus resulting in stress and anger within the family

(Walker, 1993). According to Walker (Walker,

1993), the present addiction to alcohol and substance

misuse which were apparently introduced in the

Aboriginal society by the colonists, is a consequence

of guilt and anger of past removal policies of the

government of Aboriginal children from their

parents.

This project models and develops a virtual

sociologist, Mr Warnanggal to counsel aborigines

and deter them from substance abuse for their social,

economic, health and emotional wellbeing thus

encouraging them to join the main stream Australian

community. Warnanggal is a term in Wagiman

aboriginal language, meaning doctor (Harvey &

371

Kavakli M., Rudra T. and Li M..

AN EMBODIED CONVERSATIONAL AGENT FOR COUNSELLING ABORIGINES - Mr. Warnanggal.

DOI: 10.5220/0003496103710376

In Proceedings of the 6th International Conference on Software and Database Technologies (ICSOFT-2011), pages 371-376

ISBN: 978-989-8425-77-5

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

Wilson, 1999). The system will be personalized to

individual needs and will also guide the user on

various educational and professional pathways to

motivate them to lead a better quality of life.

The paper is organized as follows, in section 2

we explain the motivation for developing Mr.

Warnanggal, in section 3 we describe the intricate

implementation details of the proposed system and

finally we present the conclusion.

2 SIGNIFICANCE

The primary motivation for developing the project is

to endeavor to integrate the aboriginal community

with the rest of the Australian society in enjoying the

fruits of prosperity. Our research will also attempt to

bridge the socio-economic gap created in the past by

providing social support through counselling in

resurrecting the Aboriginal community so that they

can enjoy equal status in the modern Australian

multi-cultural society. Aboriginal people have the

concerns while accessing disability and social

services from support services (DSA, 2006). Our

research aims to address the above issues by

providing counselling service through a virtual

counsellor that can be installed and customized in a

home or community computer; the services of which

can then be used at the convenience of the user.

The popularity of computer games justifies that

in highly immersive environments people tend to

interact with virtual avatars as if with real human

beings. Slater and colleagues redesigned a famous

yet controversial 1960 experiment (Milgram, 1963)

in a virtual environment. They found that although

all participants knew that they were interacting with

virtual characters and environment, the participants’

intend to respond to the situation at the subjective,

behavioural and physiological levels as if they were

real (Slater, 2006). This is demonstratedvia concrete

metrics such as increases in the participant’s heart

rate. This finding in literature justifies our strategy

towards aborigines substance abuse using a virtual

sociologist, Mr Warnanggal. Further, there are

supporting studies suggesting that people feel more

comfortable when interviewed by media-mediated

electronic doctors and are more likely to release

their flinched mind during the consultation process

in contrast to human doctors (Yoshida et al., 2002).

In this study, in order for Mr Warnanggal to

effectively conveya feeling of social presence for its

user, we focus on two aspects, namely “behavioural

realism” and “conversational realism”. To enhance

behavioural realism, scientists in the University of

Southern California’s Institute for Creative

Technologies managed to create virtual humans that

can exhibit non-verbal human behaviours such as

emotions and gestures (Rickel et al., 2002).

Advancements in “Chatbot” technology, which can

simulate conversations of an intelligent human

being, complements the above features of virtual

human to make this idea of a high-realism virtual

adviser possible. For example, a virtual advisor

called “Franco”, developed by Defence Science and

Technology Organisation (DSTO) at Edinburgh,

South Australia, would answer questions related to

military aircraft, ships and geographic information

(Broughton et al., 2002). Our research aims at

incorporating both of these advancements into the

design of Mr Warnanggal.

3 MR. WARNANGGAL

Figure 1 illustrates the interactions between the

various components of Mr Warnanggal expert

System. The interface between the user and the

system is through an avatar of Mr. Warnanggal

which is personified using an artificial embodied

conversational agent. To overcome the impediments

of automatic speech recognition, we will simplify

the language of communication by limiting the

grammar and vocabulary from English and focus on

providing a high-quality counselling service with

sophisticated animation.

The user module stores the personal data of the

user for more personalized diagnosis and immersive

experience during the counselling session. The

emotion unit captures the emotion of the user for

fine tuning the inference process of the expert

system and giving a human touch to the avatar of

Mr. Warnanggal by facilitating the generation of an

appropriate non-verbal expression (both facial and

gesture). The diagnosis history stores the diagnosis

and interactions of the current and previous

counselling (if any) sessions of the user with Mr.

Warnanggal. It is updated by the experience unit of

the learning module of the system. The information

from the diagnosis history may serve as an aid to

Mr. Warnanggal and begin the process of initial

dialog with reference to earlier counselling sessions.

The user profile stores the data about individual

characteristics for better understanding of socio-

economic and cultural background of the user to

assist in diagnosis.

The intelligence and behavior of the avatar

comes from the context and behavioral modules of

the interactive drama engine. The context

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

372

Figure 1: Model of Mr. Warnanggal.

module implements the advice for the coping

process outlined in our research model and consists

of an expert system for Mr Warnanggal. The

decisions made by the expert system during the

course of the conversation is drawn from the

application of inference rules on the knowledgebase

developed by consulting existing literature, clinical

documents and experts in aboriginal counselling.

The ontology provides a persona to the avatar of Mr.

Warnanggal and guides in the reasoning process by

cross checking the diagnosis history, experience

from the current session and emotion state of the

user. It governs the behavior of the avatar of Mr.

Warnanggal during its conversation with the user.

The experience unit in the learning module stores

vital conversation details between the user and Mr.

Warnanggal for reference in current session and

future consultations. Post diagnosis, data from the

experience unit is documented in the diagnosis

history database for future consultation with the

same user. Pragmatics governs how the avatar of the

virtual sociologist comprehends a situation and

produce conversation with the user. Any new

experience of the agent is stored as knowledge for

updating the expert system knowledge base after

consultation with a sociologist. The context module

sends diagnosis related questions and decisions to

the behavioral module which codes the speech and

gesture (action) mark-up files for emoting the avatar

of Mr. Warnanggal. The behavioral module is core

to the realization of one of our research goals -

believability of the user on the system.

3.1 Language and Grammar of the

System

There were approximately 200 aboriginal languages

(AL) in Australia, out of which nearly 20 are in use

(Nathan, 2002). The characteristics of Aboriginal

languages are defined by Horton (1994). English

Grammar has primarily four types of sentences they

are simple, compound, complex and compound

complex; falling into the categories of declarative,

interrogative, imperative and exclaimative

(Williams, 1999). In our proposed system, dialogue

between the user and Mr. Warnanggal will be

through the use of simple sentences adhering to the

categories of declarative, interrogative and

imperative. There are distinctive features in

aboriginal English in grammar, words and meanings

as well as language usage that shows continuities

with the traditional aboriginal languages (Eades,

2011). To overcome the impediments of speech

recognition systems, incorporate the first five

characteristics of aboriginal languages, address the

complexity of English grammar and enhance the

perpetuity in conversation with our system, we have

developed a Computer Pidgin Language (CPL) [20]

with a limited set of grammatical rules. The rules for

interacting with Mr Warnanggal consists of 10

grammatical constraints adapted from Rudra (2008).

3.2 Database of the System

One of our design objectives is to isolate the

knowledgebase from the code in the inference

module and the conversation processor. The

knowledgebase of the expert system is derived from

this database along with user characteristics and

vocabulary (for speech recognition during

conversation). To realize our goal, we have

identified 32 tables to implement the expert system

knowledgebase. The tables along with their

attributes are grouped under two broad categories:

User Module and AI Module. We adopted the

relational model from (Silberschatz, 2010) for the

database.

3.3 Expert System for the Project

The system involves complex level of interactions

with the user to advice on techniques to assist in

their rehabilitation process. Our expert system

implements a clinical decision support system

(CDSS) (Berner, 2007) that incorporates the

characteristics associated with CDSS as suggested

by (Kawamato et al., 2005).

To implement Mr. Warnanggal, we have designed

an expert system (Weiss and Kulikowski, 1984) that

will simulate the reasoning and diagnosis strategy of

a human sociologist. The knowledgebase of the

system is derived from the database described in

section 3.2. The input to the system is affective

speech and the output is an emoted embodied

conversational agent of Mr. Warnanggal. The model

for our decision support system is shown in Fig. 2.

Context

Knowledge

base

Ontology

ExpertSystem

EmotionUnit

Diagnosis

History

User

Profile

BehaviouralModule

Pragmatics

Experience

ResponseSystem

Speech Action

Knowledge

UserModule

LearningModule

InteractiveDramaEn

ine

AN EMBODIED CONVERSATIONAL AGENT FOR COUNSELLING ABORIGINES - Mr. Warnanggal

373

Figure 2: Expert system model for Mr. Warnanggal.

The expert system consists of inference engine

that applies the rules of diagnosis on the

knowledgebase based on the spoken input from the

user to produce a verbal output that animates Mr.

Warnanggal. The knowledgebase has two parts; the

first stores the characteristics of the user and the

second stores the knowledge of an expert

sociologist. The user knowledgebase helps in

creating a mind map (Buzan, 2010) of the user

which supports the expert system knowledgebase in

diagnosing affective symptoms. We develop our

inference engine based on chained rules, adapted

from causal network model by Lauritzen, 1988).

The steps from symptoms to advice are recursive

and are repeated until the counselling session is

complete. The issues (identified as either mental or

physical) being spoken by the user determines

his/her symptoms of stress and are based on the

questions being asked by the system. If the issue

under investigation is physical, the system generates

and empathetic advice for relaxation, exercise, food

etc. If the issue is social, the system generates

empathetic advice on social activities, education and

work. If the issue pertains to mental stress, the

system attempts to diagnose the problem by

applying probabilistic rules. The probabilities for

diagnosis of different mental stress symptoms are

developed by consulting expert aboriginal

psychologists. Based on the certainty factor (CF) of

diagnosis of the stress symptoms, an appropriate

statement of advice and empathy (if any) is

generated. The CF is calculated using the formulae

by Castillo & Alvarez (1991).

The reliability of our expert system is totally

based on the effective implementation of the

inference engine. Although research suggests that

transcribing medical knowledge with inference

engines underpinned by probabilities may be

affected with variations in clinical judgment of

experts (Bar-hillel, 1980); we will ensure that this

uncertainty does not influence the final outcome of

the diagnosis. We intend to customize the system to

individual user requirement by segregating the

inference mechanism on user’s demographic

attributes by consulting expert psychologists

specializing in counselling respective communities.

We will also consider personality attributes of users

for personalization of the system to cater for

individual user category and needs.

3.4 Behavioral Modeling and

Animation

We will rely on behavioral protocols in the

development of the behavioral module. Behavioral

protocols require the presentation and recognition of

cues that express social relations between humans.

Although behavioral protocols are defined in

guidelines for human ethics applications, as well as

social policies associated with emergency situations,

these are not integrated with the physical simulation

of human behavior; physical simulations (e.g.,

(Gavrila and Davis, 1996), (Kavakli, 2005);

(Newby, 1994); (Roy et al., 1994); (Semwal, 1996))

(Gavrila and Davis, 1996) treat the problem in

isolation from the social studies. We believe that any

intelligent agent must be able to detect and process

the social cues to be able to operate in a social

context in a virtual environment. While presentation

of physical and social cues is a topic of animation,

their recognition is an active research area in

cognitive science. In modeling Mr. Warnanggal,

first, we investigate social cues and produce realistic

representations of avatars simulating natural motions

based on behavioral protocols, and then, develop

personality and emotion models in order to use them

in user-adviser interaction.

In this project we will use a hierarchical model of

body actions based on fine-grained action primitives

(Emering et al., 1997), Video analysis (Sandrine et

al., 2004), motion capture, and qualitative and

quantitative analysis. The capture of the participants’

motion is normally limited to a small body part, as

one hand (Kavakli, 2005), (Newby, 1994), or one

arm Roy (Roy et al., 1994). (Gavrila and Davis,

1996) attempted to identify the full body posture by

analyzing multiple view frames. (Semwal, 1996)

used a motion capture approach based on magnetic

sensors that allows the identification of full body

movement. We take advantage of the motion capture

technology to provide the joint value input required

for our approach to behavioral modeling that

integrates social and physiological characteristics of

animations. We have explored ways, to interactively,

realistically, and efficiently create and animate

virtual humans (Kavakli & Kartiko, 2007). We have

collected face and body motion capture data to

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

374

develop face and body models to make use of in the

project. We have produced animations using a

motion capture suit, face trackers, Motion Builder,

Softimage, Vizard, etc., at the VR Lab (see Fig. 3).

Figure 3: Inference engine of the system.

4 CONCLUSIONS

This study is a critical and first stage of an on-going

long-term project carried out in a well-established

Virtual Reality Centre by a group of

interdisciplinary researchers and practitioners. We

reviewed the aboriginal, social, psychological, and

computer graphics literature underpinning the design

of this intelligent virtual advisor and suggested an

optimal system framework.

In the next stage of the project we will proceed to

enhancing our initial prototype of Mr. Warnanggal

and then move on to full-scale development of the

system. We plan to use Greta (Mancini &

Pelachaud, 2009, Pelachaud, 2010) a multi-lingual

embodied conversational agent as a visual interface

with the user to test our prototype. The narrative

engine of the interactive drama engine (Szilas et al.,

2007) developed by our team will be used to control

the dialog between the user and the system. A

critical part of our system development involves the

creation of a comprehensive knowledgebase for the

expert system through existing literature, clinical

documents and interviewing aboriginal

psychologists. Next, we will be able to develop and

implement a speech based interactive narrative

engine. Finally, we will develop an emotive talking

head that will interface the patient and the expert

system. However, it is also important to note that the

knowledge base for the virtual sociologist needs to

be verified by experts to ensure not to have any

negative impacts on aborigines.

In this paper, we examine how to utilize the

strengths of virtual interfaces to effectively mimic

the knowledge, diagnosis and behavior of

psychologists through an intelligent agent

personified as an avatar – Mr. Warnanggal, to help

aborigines to better cope with substance addiction.

Besides, the self-learning expert system for

counselling aborigines will have significant

contribution in the area of social science and

psychology. Further, the agent-based behavioral

modeling of the avatar of Mr. Warnanggal will have

implications in the field of machine learning for

human computer interaction.

The current scope of the research is limited to

aborigine psychology but it can be further extended

to the application areas of stress diagnosis and

management for people who are going through post-

traumatic stress disorders,domestic violence, ,

breakups and depression. Our research model and

design can be applied in the above application areas

with modifications to the expert system database and

behavioral modeling of the virtual avatar.

ACKNOWLEDGEMENTS

This project is sponsored by the Australian Research

Council Discovery Grant (DP0988088) titled “A

Gesture-Based Interface for Designing in Virtual

Reality”.

REFERENCES

Elkin, A. P., The Australian Aborigines. 1979 ed. 1979,

Australia: Angus & Robertson.

Walker, Y., Aboriginal family issues, S. f. N. A. a. I. C. C.

(SNAICC), Editor. 1993. p. 51-53.

Sam, M., ed. Through Black Eyes: A Handbook of Family

Violence in Aboriginal and Torres Strait Islander

Communities. 1992, SNAICC, Victoria.

Australian Indigenous HealthInfoNet. Substance use;

2011: http://www.healthinfonet.ecu.edu.au/health-

risks/substance-use.

Aboriginal and Torres Strait Islander Population. 2008:

http://www.abs.gov.au/ausstats/abs@.nsf/0/68AE74E

D632E17A6CA2573D200110075?opendocument.

Editor, 2007: Statistics on drug use in Australia, A. s. n. a.

f. h. a. w. s. a. information, Editor. 2007: Australia.

Editor. 2003: Indigenous Issues fact sheet series, D. o. I. a.

M. a. I. Affairs, Editor. 2003, Office of Aboriginal and

Torres Strait Islander Affairs: Australia.

AN EMBODIED CONVERSATIONAL AGENT FOR COUNSELLING ABORIGINES - Mr. Warnanggal

375

Healey, J., Indegenous Health, in Indigenous Health.

2004, Spinney Press: Australia. p. 10.

Harvey, M. and S. Wilson. The Wagiman online

dictionary. 1999;

http://sydney.edu.au/

arts/linguistics/research/wagiman/dict/dict.html

DSA, 2006: Aboriginal People with Disabilities Getting

Services Right. 2006, Disability Service Commission

WA.

Milgram, S., Behavioral study of obedience./. abnorm.

soc. PsychoJ, 1963. 67: p. 371-378.

Slater, M., et al., A virtual reprise of the Stanley Milgram

obedience experiments. PLoS One, 2006. 1(1).

Yoshida, A., et al. Which do you feel comfortable,

interview by a real doctor or by a virtual doctor? A

comparative study of responses to inquiries with

various psychological intensities, for the development

of the Hyper Hospital. 2002: IEEE.

Rickel, J., et al., Toward a new generation of virtual

humans for interactive experiences. IEEE Intelligent

Systems, 2002: p. 32-38.

Broughton, M., et al. Conversing with Franco, FOCAL's

virtual adviser. 2002.

Nathan, D. Aboriginal languages of Australia. 2002;

http://www.dnathan.com/VL/austLang.htm.

Horton, D., in The Encyclopaedia of Aboriginal Australia.

1994, Australian Institute of Aboriginal and Torres

Strait Islander Studies: Australia.

Williams, J. D., The Teachers Grammar Book. 1999:

Lawrence Erlbaum Associates, USA.

Eades, D. Aboriginal english. Available from:

http://www.hawaii.edu/satocenter/langnet/definitions/a

boriginal.html, 2011.

Hinde, S. and G. Belrose (2010) Computer Pidgin

Language: A new Language to talk to your Computer?

Rudra, T., Emotion Classification and Analysis in the

Design and Implementation of a Game Pidgin

Language, PhD Thesis, Department of Computing.

2008, Macquarie University: Sydney.

Silberschatz, A., H. Korth, and S. Sudarshan, Database

System Concepts, 6th Edition2010: McGraw Hill,

USA.

Berner, E. S., Clinical Decision support Systems Theory

and Practice. 2007, Springer.

Kawamoto, K., et al., Improving clinical practice using

clinical decision support systems: a systematic review

of trials to identify features critical to success. BMJ,

2005. 330.

Weiss, S. M. and C. A. Kulikowski, A practical guide to

designing expert systems. 1984: Knowman and

Allanheld.

Buzan, T., The Mind Map book. 2010: Active Educational

Publishers, Pearson Edu. Group, England.

Lauritzen, S. L. and D. J. Spiegelhalter, Local

computations with probabilities on graphical structures

and their applications to Expert Systems. Journal of

the Royal Society, 1988. Series B, Vol 50(2): p. 157-

224.

Castillo, E. and E. Alvarez, Expert Systems: Uncertainty

and Learning. 1991: Computational Mechanics

Publications, UK.

Bar-hillel, M., The base rate fallacy in probability

judgments. Acta Psychologica, 1980. 44: p. 211-233.

Gavrila, D. M. and L. S. Davis. 3D Model based Tracking

of Humans in action: A multi-view approach. in

Proceedings of IEEE Conference on Computer Vision

and Pattern recognition. 1996. San Fransisco, USA,

June.

Kavakli, M., Speaking Hands of an Artist & Requirements

for an Interface Design, in INISTA 2005, Innovations

in Intelligent Systems and Applications, IEEE

Computational Intelligence Society Turkey Chapter.

2005: YILDIZ Technical University, Istanbul, Turkey.

p. 1-4.

Newby, G., Gesture Recognition Based Upon Statistical

Similarity. Presence, 1994. 3(3): p. 234-243.

Roy, D. M., et al., The Enhancement of Interaction for

People with severe speech and physical impairment

through the computer recognition of Gesture and

Manipulation. Presence, 1994. 3(3): p. 227-235.

Semwal, S. K., R. Hightower, and S. Stansfield. Closed

Form and geometric Algorithms for Real Time

Control of an Avatar. Proc. of IEEE VRAIS’96. 1996:

IEEE Press.

Emering, L., R. Boulic, and D. Thalmann, Live

Participant’s Action Recognition for Virtual Reality

Interactions, in PACIFIC 97. 1997.

Sandrine, D., et al. Virtual Story Telling: A methodology

for developing believable communication skills in

virtual actors. 2004; Available from:

http://www.irit.fr/ACTIVITIES/GRIC.

Kavakli, M. and I. Kartiko. Avatar Construction for

Believable Agents. in 3IA'2007 Conference, The 10th

International conference on computer graphics and

artificial intelligence 2007. Athens (GREECE).

Mancini, M. and C. Pelachaud, Implementing distinctive

behavior for conversational agents: Gesture-Based

Human-Computer Interaction and Simulation. Lecture

Notes in Computer Science, 2009. 5085: p. 163-174.

Pelachaud, C., et al., GRETA: Embodied Conversational

Agent. 2010.

Szilas, N., J. Barles, and M. Kavakli, An Implementation

of Real-Time 3D Interactive Drama,. ACM Journal of

Computers in Entertainment, Special Issue:

Interactive Entertainment, 2007. 5(1): p. 1-2

ICSOFT 2011 - 6th International Conference on Software and Data Technologies

376