AN EMBODIED CONVERSATIONAL AGENT

WITH ASPERGER SYNDROME

Lynette van Zijl and Wessel Venter

Department of Computer Science, Stellenbosch University, Stellenbosch, South Africa

Keywords:

Autism, Assistive technologies, Computerised therapy tools, Embodied conversational agent, Virtual environ-

ments.

Abstract:

We discuss the development of an embodied conversational agent with Asperger-like communication skills.

The agent was developed for use in educational software in a virtual environment speciﬁcally aimed at autism

spectrum disorder software. We describe the design and implementation of the agent, and pay particular

attention to the interaction between emotion, personality and social context. A 3D demonstration shows the

typical output to conform to Asperger-like answers, with corresponding emotional responses.

1 INTRODUCTION

The prevalence of reported cases of autism spectrum

disorders worldwide has increased dramatically over

the past two decades. The latest CDC report puts the

number of people with autism at 1 in every 110 (Cen-

ter for Disease Control, 2009). Autism spectrum dis-

orders (ASDs) display as a moderate to severe lack in

language, communication and social skills. As lan-

guage and communication are the essential compo-

nents of conversational agents, this clearly has a ma-

jor impact on the way that such agents communicate

in educational software developed for children with

ASDs.

There is clear evidence that children with autism

ﬁnd beneﬁt in computer supported education and ther-

apy (Parsons et al., 2006; Tartaro and Cassell, 2007).

Likewise, the advantages of embodied conversational

agents (ECAs) in educational software are also well-

known (see for example (L

opez-Menc

ıa et al., 2010)).

However, it is important to note that such agents actu-

ally perform a social role (Doyle, 1999), and we argue

that an ECA could be distracting rather than helpful

to a person with autism. Weighing the advantages of

an ECA in educational software versus the potential

problems it could cause for persons with autism, we

decided to implement both a standard ECA and an

ECA with autistic traits. We believe that children on

the autism spectrum would ﬁnd better rapport with an

ECA with autistic traits.

This work describes our design and development

of an ECA with autistic traits, and its social affect.

The reader may note that the aim of the article is to de-

scribe the technical design, rather than the educational

or psychological evaluation of the ECA. We describe

some related work in Section 2, followed by our de-

sign is given in Section 3. We discuss results in Sec-

tion 4, and future work in Section 5. We conclude in

Section 6.

2 RELATED WORK

An ECA encompasses all aspects involved in a con-

versation – from speech to facial expressions and

body language, to communication and interaction.

People with ASDs, however, have huge difﬁculties

following the unspoken parts of communication such

as body language and facial expressions. As our aim

is to build an ECA to assist people with ASDs, we in-

tentionally ignored these graphical aspects of ECAs,

and rather concentrated on the conversational lan-

guage structure of the ECA. Note that the results of

using an ECA with autistic traits versus using a neu-

rotypical ECA is to be compared in the ﬁnal stages of

our project. We do take note of the work of Fabri et

al in this regard (Fabri et al., 2007).

The idea of an ECA with emotion and personality

is not new (Allbeck and Badler, 2002), and the sem-

inal works on this topic are those of Badler (Allbeck

and Badler, 2002; Cassell et al., 1994). Badler pro-

posed parameterised action representations (PARs)

to describe the interaction of an agent with its en-

vironment, and his EMOTE system combines PARs

153

van Zijl L. and Venter W..

AN EMBODIED CONVERSATIONAL AGENT WITH ASPERGER SYNDROME.

DOI: 10.5220/0003274101530158

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 153-158

ISBN: 978-989-8425-49-2

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

with movement analysis and psychological theory

to represent emotion and personality in the agent.

The standard ﬁve factor model for personalities (also

known as the OCEAN model (John and Srivastava,

1999)) is used to describe personalities, while emo-

tions are based on the Ortony-Clore-Collings (OCC)

model (Ortony et al., 1988).

Gebhard extended this work by combining emo-

tion and personality with mood (Gebhard, 2005),

in the ALMA (A Layered Model of Affect) sys-

tem. ALMA also uses OCEAN and OCC for

personality and emotions, respectively, while mood

is based on the pleasure-arousal-dominance (PAD)

model (Mehrabian, 1996). Data ﬁles for each agent

record how moods and emotions change over time

and with external stimuli.

In order to conduct a conversation, an ECA

can follow a set of rules, or use a artiﬁcial intelli-

gence reasoning engine to calculate reasonable re-

sponses. Rule-based systems can produce surpris-

ingly convincing conversations, as the well-known

Eliza proved (Weizenbaum, 1966). Nowadays, there

are a myriad of conversational agents (also known

as chatterbots) available (for example, see (L’Abbate

et al., 2005; Bass, 2005)), each with their own advan-

tages and disadvantages. We based our work on the

well-known A.L.I.C.E. (Bush and Wallace, 2005), as

it serves as an international standard on chatterbots.

It also provides an open standard (called AIML) for

interaction.

In the next section, we highlight the interaction

of the different components of our system, and show

how it links to external systems.

3 DESIGN

In the design of any ECA, one has to design both the

agent itself, as well as its supporting graphical and

language processing environments. We ﬁrst consider

the design of our ECA in isolation, and then describe

its supporting underlying graphical and language pro-

cessing environments.

Our agent is intended to be an ECA with autis-

tic communication behaviour. The reader should note

that ASDs are on a continious spectrum, from individ-

uals who are totally isolated and non-communicative,

to individuals who are high-functioning, participa-

tive members of society, and just have some degree

of communication and social difﬁculties. Since our

ECA is intended to communicate, we modeled his

behaviour on that of a person with high-functioning

autism (also called Asperger’s syndrome).

Personality is generally accepted to be indepen-

dent of autism (Austin, 2005), and our ﬁrst step was

to build a background character proﬁle and person-

ality for our ECA. This was necessary to improve

the believability of the ECA (Dryer, 1999). We also

consulted a clinical expert in Asperger’s syndrome

(AS) to verify our ﬁnal character proﬁle and person-

ality (Forrester, 2010).

Our ECA is an 11 year old boy with AS. He is

almost exclusively interested in dinosaurs and Star

Trek. He converses at length about these topics. His

other conversation skills are low, particularly in non-

factual topics. Physically, he does not like to be

touch unexpectedly, and has a distinctly large per-

sonal space. He dislikes crowds, loud noises, and

sharp smells. He shows little interest in competitive

sport, although he does like swimming and horserid-

ing. He spends many hours on the computer, brows-

ing the internet on his special topics of interest. He

attends a mainstream school, but has few friends. He

struggles socially, but does well in some academic

subjects. He has two siblings – an older brother and

a younger sister. He has a typical relationship with

both, although his brother is quite protective, whereas

his sister has little patience with him. He loves ani-

mals, especially the family Labrador.

Given the character proﬁle, we could develop the

OCEAN values of his personality. Note that, although

there is no evidence for an ‘autistic personality’, some

OCEAN values seem typical of a person with a high

autistic quotient (Wakabayashi et al., 2006). There-

fore, in this character design, we tried to follow the

common correlation identiﬁed by (Austin, 2005) and

(Wakabayashi et al., 2006). They found that people

with autism typically have low scores for extraver-

sion and high scores for neuroticism. The former also

identiﬁed low agreeableness, while the latter identi-

ﬁed low conscientiousness as being typical of people

with autism. We now look at how we can derive our

ECA’s OCEAN values from the above proﬁle

• While our character’s ability to absorb and vigour

for seeking out information about his special in-

terests may be thought to be an indication of a

high score for “openness”, it actually is not. A

more careful examination will show that he actu-

ally greatly lacks in active imagination, has little

regard for aesthetics and he has difﬁculty iden-

tifying his own inner feelings, let alone phrase

them for other people. This is an indication of

a medium-negative “openness” value. We select a

Note that, in reality, OCEAN values aren’t “guessed”

based on brief personality descriptions, but rather are ob-

tained from reputable tests, such as the NEO Personality In-

ventory (NEO PI-R) or NEO Five-Factor Inventory (NEO-

FFI) test (Costa and McCrae, 1992).

CSEDU 2011 - 3rd International Conference on Computer Supported Education

154

value of −0.6.

• Like many people who have AS, our agent is dili-

gent in his work (which he comprehends). But

for him, working on something which he under-

stands is more than a pleasant escape from a mis-

understanding world. He feels pride and satisfac-

tion in a job well done. This is an indication of a

medium-positive “conscientiousness” value. We

select a value of 0.5.

• Although our agent likes having friends—

sometimes actively seeking new friends out—and

can be talkative at times, one might assume that he

is fairly extraverted for someone with AS. The ba-

sic human need for social interaction is, however,

virtually universal. Upon closer examination, one

sees that he typically prefers to be alone for a few

hours a day, although he is not totally reclusive.

This is an indication of a low to medium negative

“extraversion” value. We select a value of −0.4.

• Our agent has difﬁculty to be accommodating of

others. Although he recognises the necessity of

being accommodating, he struggles with a natural

tendency not to be so. This is an indication of a

low-negative “agreeableness” value. We select a

value of −0.2.

• While our agent is often angry and feels isolated,

this in itself is not necessarily an indication of

neuroticism, but are side-effects of his AS: feeling

unable to communicate his feelings in a “proper”

way and having constant strain to understand and

decipher a world built around people who are

neurotypical are taxing on him. However, he is

more susceptible to these negative feelings than

the norm (regardless of the measure), but is re-

silient to long periods of anger and depression.

This is an indication of a low-positive “neuroti-

cism” value. We select a value of 0.3.

Following the formula given in (Gebhard, 2005),

we are now able to calculate the agent’s base mood

traits as

Pleasure = 0.21 × E + 0.59 ×A + 0.19 × N

= 0.09

Agreeableness = 0.15 × O + 0.30 × A − 0.57 × N

= −0.32

Dominance = 0.25 × O + 0.17 ×C + 0.60

×E − 0.23 × A

= −0.35

The score of +P−A−D indicates that the agent

has a default mood of being docile, as described in

(Gebhard, 2005).

Given the background proﬁle, personality and

mood indicators for the agent, we now consider our

conversation data set. For our ﬁrst prototype demon-

stration program, we ﬁxed two conversation top-

ics (dinosaurs and Star Trek). Our expert AS psy-

chologist worked with a 15 year old boy with AS

to construct several representative prototype conver-

sations which demonstrate different emotional re-

sponses. These were then used to build the neces-

sary AIML template matching response ﬁles. We are

currently in the process of extending our conversation

data set. Our current demonstration program there-

fore restricts the types of questions, and the content

of the questions rather severely. This will clearly be

remedied as the data set grows.

Now, given an agent and its accompanying con-

versation data sets, we need to let the agent interact

with its environment. This interaction is quite sim-

ple – in a typical conversation, the user can type in a

question. If the question is a factual specialized ques-

tion from the knowledge base of the ECA (in our case,

dinosaurs or Star Trek), the answer can be extracted

from the knowledge base. Here, no emotion or per-

sonality is involved. Otherwise, if the user question

is a generalized question not involving the knowledge

base, it needs to be answered in a chatterbot fashion,

but with personality, emotion and mood taken into

account. Hence, our system in essence provides a

connection between various external modules, as il-

lustrated in Figure 1. Of interest here is the use of

the Natural Language ToolKit (NLTK) (Bird et al.,

2009), the ALMA system (Gebhard, 2005), and the

use of an SQL database. Note that the virtual envi-

ronment we use is the Myoushu engine (Chamberlain,

2009; Van Zijl and Chamberlain, 2010). Myoushu is

a language-independent quick-development platform

for 3D ASD educational and therapy tools.

Figure 1: Design overview.

In a typical scenario, the system receives some

question from the virtual environment (VE). This

question is then sent to the cognitive task module,

AN EMBODIED CONVERSATIONAL AGENT WITH ASPERGER SYNDROME

155

which parses the question and decides whether it is

a factual specialized question or not. If it is a special-

ized question, the question is sent to the query builder

task which transforms it into an SQL query. The

answer to the query is returned from the knowledge

base. This answer is used to construct a response,

which is sent back to the virtual environment. On the

other hand, if the original question was not a special-

ized question, it is sent to the AIML task, which uses

pattern template matching to produce an answer code.

This answer code is then sent to the AffectTask mod-

ule, which produces different responses based on the

code, personality, current emotion and current mood.

The more interesting part of the system is of

course the AffectTask module, which enables the

autistic traits to be incorporated into the answer gen-

erated by the AIML chatterbot. Based on our conver-

sational dataset, we constructed the AIML patterns in

the AIML XML ﬁles. For the personality, emotion

and mood ﬁles, we constructed an XML-like data ﬁle,

which speciﬁes the conditions and their inﬂuence on

the resultant answer.

4 EXPERIMENTAL RESULTS

To demonstrate the working of our ECA system, we

developed a prototype 3D demonstration, which we

describe below.

4.1 3D Demonstration

In our demonstration, the user takes on the role of a

person who is about to conduct a short, informal in-

terview with a boy who has AS. As the demonstration

starts, the user ﬁnds him-/herself in a waiting room

outside a therapist’s ofﬁce. The user sees the child’s

“therapist”, who effectively provides the user with a

brief tutorial of how the demonstration works. When

she is ﬁnished speaking, she leaves the waiting area.

The user must now open the door which leads into

the ofﬁce and speak with the agent. The user can dis-

cuss different topics with the agent. Two of these top-

ics will inﬂuence the agent’s demeanour in a positive

way, two topics will inﬂuence it in a negative way and

one topic is neutral, having no dramatic effect on the

demeanour of the agent. When the user is ﬁnished

with the conversation, he/she can exit the demonstra-

tion by leaving through the same door through which

the therapist left.

To illustrate one of the sensory integration prob-

lems which people with AS normally have, a cou-

ple of scenarios were built into the demonstration.

The ﬁrst one is a radio inside the ofﬁce which plays

Figure 2: Demonstration program screenshots.

baroque music. While the radio is on, the agent’s

mood will deteriorate steadily. This demonstrates

the fact that most, if not all, people with AS have

to strain to distinguish between competing sensory

sources (Attwood, 2006) (in our example, the mu-

sic and the conversation with the user). If the user

switches the radio off, then the agent’s mood will im-

prove. At some point, a dog will start barking in the

background. This second scenario will have the same

negative effect on the agent as the ﬁrst one, but this

time the user can do nothing to stop the source of the

problem. If both the radio and dog are audible, then

agent’s mood will deteriorate rapidly.

4.2 Results from 3D Demonstration

As representative examples of the working of our sys-

tem, we follow three different inputs to the system via

the demonstration program. We discuss the data ﬂow

as the inputs are processed, and show the output to the

questions. In all the examples below, we assume that

the ECA is in a neutral emotional state when the ques-

tion is posed. The reader should note the appropriate

“Asperger-like” answers, depending on the mood and

emotion of the ECA.

Example 1. Input: “What are your parents’

names?” This is a general statement rather than one

from the knowledge base on dinosaurs or Star Trek.

Hence, the cognitive task identiﬁes it as such, and this

input is sent to the AIMLTask.

The pattern template in AIML matches against

“WHAT * YOUR PARENTS NAMES” (with inter-

nal code ID A008) and sends the code to AffectTask

to evaluate against emotion and mood. No speciﬁc

conditions match in this neutral case, so the default

response is sent to the OutputTask.

The OutputTask formats “Robert and Alison.” as

CSEDU 2011 - 3rd International Conference on Computer Supported Education

156

the response with no animations and sends no up-

date signal to the affect engine (this is a neutral ques-

tion/response). The output is sent to the VE.

For another example, we consider a factual ques-

tion to illustrate another data ﬂow through the system.

Example 2. Input: “Where did Triceratops live?”

The cognitive task, recognizing this question as a spe-

cial interest topic, sends the question to the Query-

BuilderTask. Here, the input is matched against

the Dinosaurs feature based grammar, and the query

“SELECT ‘location’ FROM ‘DinoSpecies’ WHERE

‘name’ = ’Triceratops’ ” is constructed via NLTK.

The query is executed, the result retrieved from the

database, and set to the OutputTask. The OutputTask

sends “Triceratops lived in North America.” to the

VE.

As the last example, we consider an emotional

question of the kind that is typically difﬁcult to handle

for a person with AS.

Example 3. Input: “How do you feel when your

sister is nasty to you?” Again, the cognitive task

recognizes the question as a general statement, and

sends it to the AIMLTask. AIML matches the ques-

tion against “HOW DO YOU FEEL WHEN * IS *

TO YOU” (with internal code ID C005) and sends it

to the AffectTask. The AffectTask checks the ECA

affect state and matches the condition where the dom-

inant emotion is “disliking”. The associated response

is sent to the OutputTask. The OutputTask formats

“I don’t know!” as the response. While the re-

sponse is being given, a slight rocking animation is

played, indicating a high level of distress in the ECA.

The “NastyThing” signal is sent to the affect engine,

which negatively inﬂuences the agent’s affect state.

The output is sent to the VE.

5 FUTURE WORK

In the previous sections, we described the design and

implementation of an ECA with autistic communica-

tion traits. This is interesting in itself, but we want

to investigate the use of such an ECA in educational

software for children with AS.

We are currently in the process of implementing

the ECA in an educational game. This is a simple

game reinforcing arithmetic and grammar skills. It

is currently in use in a local school for children with

learning disabilities, but without any ECA. In previ-

ous experiments, we noted that children with AS pre-

fer not to have any help from human teaching assis-

tants when playing the game. However, this some-

times caused frustration, when they could not ﬁnd a

Figure 3: Educational game screenshot.

correct answer. The role of the ECA would be to alle-

viate such frustrations in an autism-friendly manner.

The evaluation of the reaction of children with AS

versus children without AS will be assessed, with re-

gards to our ECA, in future work.

6 CONCLUSIONS

In this paper we presented the motivation for and the

design and implementation of an ECA with autistic

traits. At the heart of the implemented system lies a

collection of hand-constructed, Asperger-like conver-

sations. These conversations were used to build pat-

tern templates for Asperger-like chatterbot answers.

A 3D demonstration prototype illustrates the inﬂu-

ence of emotion on expected answers. We are cur-

rently building our ECA into an educational game for

children with AS. We intend to evaluate the effective-

ness of using the ECA with autistic traits in such soft-

ware, as opposed to a standard ECA or human assis-

tance.

ACKNOWLEDGEMENTS

The ﬁnancial assistance of the National Research

Foundation under research grant number 65856 to-

wards this research is hereby acknowledged. Opin-

ions expressed in this paper and conclusions arrived

at are those of the authors and not necessarily to be

attributed to the National Research Foundation.

REFERENCES

Allbeck, J. and Badler, N. (2002). Toward Representing

Agent Behaviors Modiﬁed by Personality and Emo-

AN EMBODIED CONVERSATIONAL AGENT WITH ASPERGER SYNDROME

157

tion. In Workshop Embodied Conversational Agents—

Let’s Specify and Evaluate Them! at AAMAS 2002,

Bologna, Italy.

Attwood, T. (2006). The Complete Guide to Asperger’s Syn-

drome. Jessica Kingsley Publishers, 116 Pentonville

Road, London N1 9JB, UK.

Austin, E. J. (2005). Personality correlates of the broader

autism phenotype as assessed by the Autism Spec-

trum Quotient (AQ). Personality and Individual Dif-

ferences, 38(2):451 – 460.

Bass, T. (2005). StaffBots: Commercial Bot Technologies.

In DARPA Workshop on Virtual Intelligent Agents, Ar-

lington,USA.

Bird, S., Klein, E., and Loper, E. (2009). Natural Language

Processing with Python. O’Reilly Media, ﬁrst edition.

Bush, N. and Wallace, R. (2005). Artiﬁcial Intelligence

Markup Language (AIML) Version 1.0.1. Working

draft, A.L.I.C.E. AI Foundation.

Cassell, J., Pelachaud, C., Badler, N., Steedman, M.,

Achorn, B., Becket, T., Douville, B., Prevost, S.,

and Stone, M. (1994). Animated Conversation:

Rule-Based Generation of Facial Expression, Ges-

ture & Spoken Intonation for Multiple Conversational

Agents. Computer Graphics, 28(Annual Conference

Series):413–420.

Center for Disease Control (2009). http://www.cdc.gov/

ncbddd/autism/.

Chamberlain, M. (2009). A 3D Virtual Environment De-

velopment Platform for ASD Therapy Tools. Master’s

thesis, Stellenbosch University.

Costa, P. T. and McCrae, R. R. (1992). NEO-PI-R profes-

sional manual: Revised NEO personality and NEO

Five-Factor Inventory (NEO-FFI). Psycological As-

sessment Resources, Odessa, FL.

Doyle, P. (1999). When is a communicative agent a good

idea? In Proceedings of the Third International Con-

ference on Autonomous Agents, Seattle, USA.

Dryer, D. C. (1999). Getting Personal with Computers:

How to Design Personalities for Agents. Applied Ar-

tiﬁcial Intelligence, 13(3):273–295.

Fabri, M., Elzouki, S., and Moore, D. (2007). Emo-

tionally Expressive Avatars for Chatting, Learning

and Therapeutic Intervention. In Jacko, J., editor,

Human-Computer Interaction. HCI Intelligent Mul-

timodal Interaction Environments, volume 4552 of

Lecture Notes in Computer Science, pages 275–285.

Springer Berlin / Heidelberg.

Forrester, J. (2010). Personal communication.

Gebhard, P. (2005). ALMA: A Layered Model of Affect. In

AAMAS ’05: Proceedings of the Fourth International

Joint Conference on Autonomous Agents and Multi-

agent Systems, pages 29–36, New York, NY, USA.

ACM.

John, O. P. and Srivastava, S. (1999). The Big-Five Trait

Taxonomy: History, Measurement, and Theoretical

Perspectives. In Pervin, L. A. and John, O. P., edi-

tors, Handbook of Personality: Theory and Research,

pages 102–138. Guilford Press, New York, second

edition.

L’Abbate, M., Thiel, U., and Kamps, T. (2005). Can

proactive behavior turn chatterbots into conversa-

tional agents? In IAT ’05: Proceedings of the

IEEE/WIC/ACM International Conference on Intelli-

gent Agent Technology, pages 173–179, Washington,

DC, USA. IEEE Computer Society.

opez-Menc

ıa, B., Pardo, D., Hern

andez-Trapote, A.,

Hern

andez, L., and Rela no, J. (2010). A collabo-

rative approach to the design and evaluation of an in-

teractive learning tool for children with special educa-

tional needs. In IDC ’10: Proceedings of the 9th Inter-

national Conference on Interaction Design and Chil-

dren, pages 226–229, New York, NY, USA. ACM.

Mehrabian, A. (1996). Pleasure-Arousal-Dominance: A

General Framework for Describing and Measuring In-

dividual Differences in Temperament. Current Psy-

chology, 14(4):261–292.

Ortony, A., Clore, G. L., and Collins, A. (1988). The Cog-

nitive Structure of Emotions. Cambridge University

Press, The Edinburgh Building, Cambridge CB2 2RU,

UK.

Parsons, S., Leonard, A., and Mitchell, P. (2006). Virtual

Environments for Social Skills Training: Comments

from Two Adolescents with Autistic Spectrum Disor-

der. Computers & Education, 47(2):186 – 206.

Tartaro, A. and Cassell, J. (2007). Using virtual peer tech-

nology as an intervention for children with autism.

In Lazar, J., editor, Toward Universal Usability: De-

signing Computer Interfaces for Diverse User Popu-

lations. John Wiley and Sons.

Van Zijl, L. and Chamberlain, M. (2010). A generic de-

velopment platform for ASD therapy tools. In Inter-

national Conference on Computer Science Education,

pages 84–91.

Wakabayashi, A., Baron-Cohen, S., and Wheelwright, S.

(2006). Are autistic traits an independent personality

dimension? A study of the Autism-Spectrum Quotient

(AQ) and the NEO-PI-R. Personality and Individual

Differences, 41(5):873 – 883.

Weizenbaum, J. (1966). ELIZA—a computer program for

the study of natural language communication between

man and machine. Communications of the ACM,

9(1):36–45.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

158