Toward Gamiﬁed Knowledge Contents Reﬁnement

Case Study of a Conversation Partner Agent

Takayuki Iwamae

, Kazuhiro Kuwabara

and Hung-Hsuan Huang

Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga 525-8577, Japan

College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga 525-8577, Japan

Keywords:

Crowdsourcing, Gamiﬁcation, Knowledge Reﬁnement, Conversation Partner Agent.

Abstract:

Rich knowledge contents are necessary to develop an intelligent agent that interacts with people and supports

their communication or their activities. In this paper, we choose a conversation partner agent for people

with aphasia as an example and propose a method that interactively acquires and reﬁnes knowledge contents

for the agent. The proposed method is invoked when a problem is found in the knowledge contents and

utilizes the concept of gamiﬁed crowdsourcing. Gamiﬁed tasks verify the data input by a user. By utilizing a

crowdsourcing approach, we strive for more accurate knowledge contents. The paper presents its game design

and an example scenario.

1 INTRODUCTION

A large amount of knowledge contents is required to

develop an intelligent agent to support human com-

munication or activities. Knowledge contents may be

constructed from data available on the Internet using

machine learning technologies, for example. On the

other hand, the concept of crowdsourcing can also

be applied to construct various knowledge contents

by harnessing the power of hundreds or thousands of

people. In such crowdsourcing platforms as Amazon

Mechanical Turk (MTurk)

, many microtasks, often

called Human Intelligence Tasks (HITs), are shared

with many workers over networks. An example of

such a task is labeling an image.

In crowdsourcing, giving proper incentives to

workers is important for eliciting both qualitative and

quantitative better performances. A typical reward is

monetary, which can be considered an extrinsic mo-

tivation. The importance of intrinsic motivation has

also been pointed out (Ryan and Deci, 2000). Gami-

ﬁcation, which is deﬁned as the use of game design

elements in non-gaming contexts (Deterding et al.,

2011), is a popular method to provide intrinsic mo-

tivation. In crowdsourcing, the idea of gamiﬁcation is

widely utilized (Morschheuser et al., 2016).

In this paper, we propose a method that utilizes a

gamiﬁed crowdsourcing approach for knowledge con-

http://www.mturk.com/

tents reﬁnement and focuses on a case of ﬁxing prob-

lems in knowledge contents. The proposed method is

intended to be applied when a problem is found while

using the target system. Users provide the initial data

for revising the knowledge contents, and through a

gamiﬁed crowdsourcing process, many users can con-

tribute to its reﬁnement.

As an example application, we use a conversa-

tion partner agent for people with aphasia (Kuwabara

et al., 2016). One of the conversation partner agent’s

functions is to assist the retrieval of a word that a per-

son with aphasia is having difﬁculty recalling. This

assisting process uses knowledge contents to present a

series of questions for the person with aphasia. From

his answers, the conversation partner agent suggests

the word he wants to express.

The rest of this paper is organized as follows. The

next section describes related work, followed by a

description of a conversation partner agent, which is

the target domain of this work. Section 4 explains

our proposed method that reﬁnes knowledge contents,

and Section 5 describes an application of the gamiﬁ-

cation concept. We conclude this paper in Section 6.

2 RELATED WORK

Many applications of the gamiﬁcation concept exist

in crowdsourcing tasks. One early application was

Games with a purpose (GWAPs), where a game ele-

302

Iwamae T., Kuwabara K. and Huang H.

Toward Gamiﬁed Knowledge Contents Reﬁnement - Case Study of a Conversation Partner Agent.

DOI: 10.5220/0006243103020307

In Proceedings of the 9th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2017), pages 302-307

ISBN: 978-989-758-219-6

ment was introduced in a task (von Ahn and Dabbish,

2008). Introducing a proper game rule makes it possi-

ble to produce meaningful results from game-playing.

GWAPs are applied in various ﬁelds including seman-

tic knowledge acquisition (Siorpaes and Hepp, 2008).

The gamiﬁcation idea is also used with paid

crowdsourced microtasks. An incentive model for

such a case was proposed (Feyisetan et al., 2015).

Quizz is another gamiﬁed crowdsourcing system for

knowledge curation (Ipeirotis and Gabrilovich, 2014).

To maintain the quality of the knowledge, it uses a

calibration quiz whose answers are known before-

hand to estimate a worker’s competence.

In addition, a web-based game called Common

Consensus (Lieberman et al., 2007) collects and val-

idates commonsense knowledge for understanding

goals in everyday human life. Similarly, in Robot

Trainer (Rodosthenous and Michael, 2016), not only

factual knowledge is collected but also knowledge in

rule form.

In this paper, we target the knowledge contents

that are used in the domain of word retrieval assis-

tance for people with aphasia and focus on the reﬁne-

ment of the pieces of collected knowledge contents.

In the reﬁnement process, we handle not only con-

ventional answers but also obscure ones and let work-

ers input their own level of conﬁdence to maintain the

quality of the ﬁnal results.

3 CONVERSATION PARTNER

AGENT

Conversation partner agents are intended to be used as

a kind of interpreter for people with aphasia. One of

their main functions is to support the word retrieval

process for such sufferers. That is, a person with

aphasia often has a problem recalling the proper word

even though he knows what he wants to say. In such

a situation, a human supporter (conversation partner)

acts as an interpreter to elicit the word. For example,

when a person with aphasia wants a particular type of

food, the human conversation partner might ask, Is it a

fruit? or Is it a snack?. Depending on the answer, the

human conversation partner narrows down to identify

the word the person with aphasia is thinking about.

The conversation partner agent is designed to

present an appropriate question for the person with

aphasia, and based on the answer, candidates are nar-

rowed down (Fig. 1). When a word with a certain pos-

sibility is identiﬁed, the agent presents it. The knowl-

edge contents are utilized that contain possible words

to be guessed and questions to ask. Note that the con-

versation partner agent is not meant to completely re-

Person with

aphasia

Select the next

question to ask

Update word

probabilities

Question

Answer

Conversation

Partner Agent

Word

suggestion

Determine the word

to suggest

Knowledge

Contents

Figure 1: Conversation partner agent overview.

place human conversation partners; it can also give

assistance by providing proper questions to ask and

reduce his burden.

This word retrieval assistance process resembles

a popular game called Akinator

, which guesses the

identity of a character being thought of through a se-

ries of questions and answers. It also allows a player

to input the correct answer when the system fails to

ﬁnd an answer. This game, however, focuses on char-

acters that are either ﬁctional or real, whereas con-

versation partner agents must handle a variety of top-

ics. In addition, in the Akinator game, some questions

such as Does the name begin with ’A’? assume that the

player already knows the word (name) of the answer

(character). This kind of question is not appropriate

for word retrieval assistance.

3.1 Knowledge Contents

The knowledge contents for the word retrieval assis-

tant consists of words, questions, and the answers to

the given question for the word the person with apha-

sia is thinking of. Let w (∈ D) denote a target word of

the word retrieval assistant, where D denotes a set of

possible words. Let q

(1 ≤ i ≤ n) denote a question to

ask to identify the word. We assume n multiple choice

type questions. Let N

denote the number of answer

choices for question q

, and let c

(1 ≤ k ≤ N

) denote

an answer choice for question q

We also assume that multiple answers are possible

for word w. As we will explain below, we exploit the

concept of information gain to select the next ques-

tion. The knowledge contents contain the probability

that answer choice c

is selected when question q

asked and w is the correct answer. This probability is

denoted by p(c

|w). If there is no possibility that c

will be chosen, p(c

|w) = 0.

http://en.akinator.com

Toward Gamiﬁed Knowledge Contents Reﬁnement - Case Study of a Conversation Partner Agent

303

3.2 Interaction with Users

The word retrieval assistant’s goal is to guess the word

the user is thinking about. In this regard, we calcu-

late the probability that word w is the correct answer,

which is denoted by p(w). Based on the answers to

the questions, the values of p(w) are updated. When

the probability of the word exceeds a certain thresh-

old, it is selected as the one the user has in mind.

The next question is determined based on infor-

mation gain (Arima et al., 2015; Kuwabara et al.,

2016). To calculate the expected information gain

of asking question q

, ﬁrst we set the probabilities of

words, p(w). The average information entropy, H,

can be deﬁned as follows:

H = −

∑

w∈D

p(w)log p(w) .

The probability of w when answer choice c

selected for question q

is represented using the

Bayesian theorem:

p(w|c

) =

p(c

|w)

p(c

)

p(w) .

The probability that c

is selected for question q

calculated as follows:

p(c

) =

∑

w∈D

p(c

|w)p(w) .

The values of p(c

|w) are taken from the knowledge

contents. The expected entropy after c

is selected

when question q

is asked, H

, is given as:

= −

∑

w∈D

p(w|c

)log p(w|c

) ,

and the expected entropy after question q

is asked is

calculated as:

∑

k=1

p(c

Thus, the information gain of question q

, IG

, is

given as:

= H − H

To determine the next question, we calculate the

priority of question q

, PR

. The priority value takes

into consideration the number of answer choices for

question q

in addition to its information gain. Gener-

ally the information gain becomes higher when there

are many answer choices, but it is inadvisable to ask a

question with too many answer choices from the start,

especially for people with aphasia who might have

difﬁculty understanding too many answer choices.

is deﬁned as follows:

− β

Table 1: Example knowledge contents.

: Is it a

fruit?

: What

color is it?

: What

does it taste

like?

strawberry Yes red sweet

banana Yes yellow sweet

orange Yes orange sweet,sour

lettuce No green —

carrot No orange sweet,bitter

tomato No red —

where β denotes the parameter that represents the ef-

fect of the number of answer choices of the question

and is set to 1.5 in the prototype.

The question with the highest priority value is se-

lected and asked. When c

is selected as its answer,

p(w) is updated with p(w|c

). This process contin-

ues until the probability of a certain word exceeds a

predeﬁned threshold. If all the probabilities of words

become 0, the system reports that no answer is found.

If the questions are exhausted before it narrows down

the options to a single word, a list of possible answers

is returned.

3.3 Example Scenario

As an example, consider the simple data shown in Ta-

ble 1. In this example, there are three questions and

six words. Each cell in the table deﬁnes an answer

for the word when a question is asked. When there is

only one answer in the cell, 1 is the probability that

that answer will be selected. For example, an answer

to question q

(What color is it?) for strawberry is red,

and p(red|strawberry ) = 1 for q

. If the cell contains

two possibilities, such as the answer to q

(What does

it taste like?) for orange, they are treated as if the

possibility of each answer is 0.5. If there is no deﬁ-

nite answer or an answer is unknown, “—” is inserted

in the table.

Assume that the user (person with aphasia) is

thinking about strawberry. The system ﬁrst calcu-

lates the priorities of the three questions and selects

: Is it a fruit?. In reply, answer Yes is given, and

then q

is asked next: What color is it?. If answer red

is given, strawberry will be chosen as the word the

user is thinking of since only strawberry remains as a

possibility in the example knowledge contents.

4 KNOWLEDGE CONTENTS

REFINEMENT

An overview of our proposed method of knowledge

contents reﬁnement is shown in Fig. 2. When a word

ICAART 2017 - 9th International Conference on Agents and Artiﬁcial Intelligence

304

Game

Materials

Game

Play Log

Game

Scores

Game Generation

Game Execution Manager

Score

Calculation

Conversation Partner Agent

Word Retrieval

Assistance

Knowledge

Contents

Game Players

(Workers)

[start] Correct answer

cannot be obtained

Data for

Refinement

Data

Collection

New word/question

Revised data

Knowledge Contents Extraction

Person with

aphasia

(Human) Conversation Partner / Caregiver

Figure 2: Knowledge contents reﬁnement overview.

retrieval assistant system fails to produce a correct an-

swer, it asks for data with which to revise the knowl-

edge contents. We assume that a human caregiver

such as a conversation partner who uses the system

with a person with aphasia will input the correct an-

swer. Using the input data, gamiﬁed tasks are gen-

erated and executed by other users. Based on their

results, the knowledge contents are updated.

4.1 Invocation of a Reﬁnement Process

When the word retrieval assistant system fails to ﬁnd a

word with high probability or to narrow a list of words

to one, it is treated as a failure. The followings are the

possible causes of failures: 1) The target word is not

in the knowledge contents; 2) The target word is in

the knowledge contents, but its data are not correct; 3)

There are not enough questions to narrow the possible

words to one.

In our proposed method, the data for correction

are input by the user. When a word cannot be found

or a word comes out that is different from what the

user had in mind, the correct answer (word) is asked

to be entered. When the word cannot be narrowed to

one, another question must be entered.

The data input for revision can be represented as

a triple hw, q

, c

i. The system contains the history

of the user’s responses, which can be used as part of

the data when a new word is input. However, with

this information, we cannot determine the value of

the probability of p(c

|w) required by the knowledge

contents. The gamiﬁed tasks are generated from these

triples to determine p(c

|w) values.

4.2 Example Scenario (cont’d)

Continuing the example scenario explained above, as-

sume that apple is the word being thought of by the

user. As before, based on a question’s priority value,

is asked ﬁrst: Is it a fruit?. The user answers with

Yes. Then the next question (What color is it?) is an-

New word is entered

Answers for each question are shown

Figure 3: Entering a new word.

Table 2: Data added for a new word.

Word Question Answer

apple q

: Is it a fruit? Yes

apple q

: What color is it? red

swered with red, and strawberry is suggested by the

system, which is not correct in this case. This invokes

the revising mode. The correct answer apple is en-

tered as shown in Fig. 3. By utilizing the history of

a user’s responses, additional data for apple are also

obtained, as shown in Table 2.

Assume that these additional data are added to the

knowledge contents. In this case, after the responses

are obtained for q

and q

, the possible words that re-

main are strawberry and apple. Question q

is asked

next: What does it taste like?. However, apple re-

mains one possibility since it has no answer data for

question q

, and consequently, it cannot be eliminated

from the possible word list.

In such a case, a new question is needed to distin-

guish between apple and strawberry. Suppose that a

new question is input with the answers for these two

words: How big is it?. Then the data shown in Table 3

are obtained.

Further, assume that the word the user has in mind

is a speciﬁc kind of apple, for example, a green apple.

In this case, for the ﬁrst question q

(Is it a fruit?), Yes

is the answer, and the next question q

(What color is

it?) is presented with the following possible answer

choices: red, orange, and yellow. Since the answer

choices do not contain the correct response, the user

must choose none of the above. In this case, the cor-

rect answer, apple, is entered in the revising mode,

Toward Gamiﬁed Knowledge Contents Reﬁnement - Case Study of a Conversation Partner Agent

305

Table 3: Data of new question.

Word Question Answer

strawberry q

: How big is it? small

apple q

: How big is it? medium

and the answer to question q

(What color is it?) is

entered as green.

At this state, question q

(What color is it?) will

have two possible answers for apple: red and green.

The possibilities of these answers are determined with

the gamiﬁed tasks explained in the next section.

5 GAMIFIED REFINEMENT

PROCESS

5.1 Task Description

We designed a task to determine the possibilities of

an answer for a given question. The task presents

a statement that can be answered by either Yes or

No. A worker inputs his degree of agreement with

the presented statement and also her conﬁdence value.

The statement of the task, denoted by s

w,ik

, is gener-

ated from the data to revise the knowledge contents:

hw, q

, c

i. Worker u enters the degree of agreement

to statement s

w,ik

, denoted by A

w,ik

) and the conﬁ-

dence value, denoted by B

w,ik

). Both values range

between 0 and 1.

Let M denote a set of workers of the task. We

ﬁrst calculate the weighted average of the degree of

agreement regarding statement s

w,ik

, where the weight

is determined based on the conﬁdence value. Let v

w,ik

denote this weighted average. We determine the value

of p(c

|w) by considering every possible c

for ques-

tion q

. By denoting a set of possible values of c

, we have the following:

w,ik

∑

u∈M

w,ik

)

∑

u∈M

w,ik

)

p(c

|w) =

w,ik

∑

∈C

w,ik

Continuing the above example, for the triple of

word apple, question What color is it?, and answer

red, the generated statement is The color of the ap-

ple is red. A worker enters his degree of agreement to

this statement and his conﬁdence value (Fig. 4).

Assume that after the tasks are executed by differ-

ent workers, the results shown in the three most left

columns of Table 4 are obtained. Note that the num-

ber of entries is kept small for the example’s clarity.

From these inputs we get v

w,ik

= 0.816 for red and

w,ik

= 0.177 for green. From these values, we get

p(red|apple) = 0.821 and p(green|apple) = 0.178.

The color of the apple

is red.

Player’s input

(confidence)

Figure 4: Screenshot of playing the game.

5.2 Scoring Rules

Next we introduce the concept of gamiﬁcation into

the above tasks. Gamiﬁcation’s main objective is to

motive workers (or game players) to execute tasks.

We present a set of tasks to a worker under a time

limit. For example, one game round might consist of

12 tasks that must be executed in 60 seconds. By set-

ting a time limit, a worker is expected to make more

intuitive answers. Points are basically given based on

the number of executed tasks. However, if the points

given are only determined by the number of executed

tasks, workers are not motivated to input appropriate

answers.

To address this problem, we also give bonus points

to workers as an incentive to deliver more plausible

answers. Bonus points, which are calculated after the

value of p(c

|w) is determined, increase when degree

of agreement A

w,ik

) is closer to p(c

|w). When

w,ik

) and p(c

|w) are the same, the bonus points

are the highest; a worker who submits the most dif-

ferent values from p(c

|w) receives the fewest bonus

points. In addition, we consider conﬁdence value

w,ik

) input by a worker.

More speciﬁcally, we calculate the number of

bonus points given to a worker, denoted by R

, as fol-

lows. Let err

denote the absolute difference between

w,ik

) and p(c

|w). If err

is smaller than half of

max

u∈M

err

, the bonus points are increased. Other-

wise, they are decreased. The amount of increase or

decrease is determined by conﬁdence value B

w,ik

err

= |A

w,ik

) − p(c

|w)|

= BP(1 − K(err

−

max

u∈M

err

))B

w,ik

) ,

where K denotes a parameter that represents the effect

of the conﬁdence value to the bonus points and BP

denotes the ratio of the internal value to them.

Continuing the earlier example, a worker’s bonus

points are calculated as shown in the right most col-

umn of Table 4, where K is set to 10 and BP is set to

100. A worker who submitted a degree of agreement

value closer to p(c

|w) is given more bonus points,

and a worker who submitted the value farthest from

p(c

|w) receives fewer bonus points.

The worker with higher conﬁdence values re-

ceives more bonus points even if he enters the same

ICAART 2017 - 9th International Conference on Agents and Artiﬁcial Intelligence

306

Table 4: Example of worker inputs and bonus points.

Worker input Bonus points

Worker Degree of

agreement

Conﬁdence

value

Difference

with

p(c

|w)

Bonus

points R

The color of the apple is red.

0.9 0.9 0.084 77

0.8 0.3 -0.016 113

0.8 0.8 -0.016 134

0.7 0.5 -0.116 71

The color of the apple is green.

0.2 0.4 0.022 116

0.3 0.3 0.122 82

0.1 0.6 -0.078 90

degree of agreement values (see u

and u

in Table 4).

Since the conﬁdence values are reﬂected in determin-

ing both p(c

|w) and bonus points R

, if we assume

that a worker wants to gain more points, he will be

more motivated to input more plausible answers.

6 CONCLUSIONS AND FUTURE

WORK

This paper presented a method of reﬁning knowledge

contents to be used with a conversation partner agent

for people with aphasia. Our proposed method deals

with such problems of knowledge contents as missing

words or questions. The information that revise the

knowledge contents is requested to be entered by a

user, and then the input data are reﬁned by applying

the concept of gamiﬁed crowdsourcing.

Currently we are implementing a prototype as a

web application with conventional gamiﬁcation ele-

ments such as a leader board or a badge system. We

plan to conduct evaluation experiments to show the

effectiveness of our proposed approach, especially

where a gamiﬁed approach can effectively provide

better incentives to workers.

In this paper, the conversation partner agent is

used as the target of our case study. The word retrieval

assistance process can be viewed as guessing an item

a user is consciously or unconsciously thinking of.

It can be viewed as recommending an item a user

wants through a series of questions and answers. The

proposed knowledge contents reﬁning method can be

used for such kinds of applications. We also plan to

apply the proposed method to other application do-

mains.

ACKNOWLEDGEMENTS

This work was partially supported by JSPS KAK-

ENHI Grant Number 15K00324.

REFERENCES

Arima, S., Kuroiwa, S., Horiuchi, Y., and Furukawa, D.

(2015). Question-asking strategy for people with

aphasia to remember food names. Journal on Tech-

nology & Persons with Disabilities, 3:10–19.

Deterding, S., Dixon, D., Khaled, R., and Nacke, L. (2011).

From game design elements to gamefulness: Deﬁning

”gamiﬁcation”. In Proceedings of the 15th Interna-

tional Academic MindTrek Conference: Envisioning

Future Media Environments, pages 9–15. ACM.

Feyisetan, O., Simperl, E., Van Kleek, M., and Shadbolt, N.

(2015). Improving paid microtasks through gamiﬁca-

tion and adaptive furtherance incentives. In Proceed-

ings of the 24th International Conference on World

Wide Web, WWW ’15, pages 333–343. International

World Wide Web Conferences Steering Committee.

Ipeirotis, P. G. and Gabrilovich, E. (2014). Quizz: Targeted

crowdsourcing with a billion (potential) users. In

Proceedings of the 23rd International Conference on

World Wide Web, WWW ’14, pages 143–154. ACM.

Kuwabara, K., Iwamae, T., Wada, Y., Huang, H.-H., and

Takenaka, K. (2016). Toward a conversation part-

ner agent for people with aphasia: Assisting word

retrieval. In Intelligent Decision Technologies 2016:

Proceedings of the 8th KES International Conference

on Intelligent Decision Technologies (KES-IDT 2016)

– Part I, pages 203–213. Springer International Pub-

lishing.

Lieberman, H., Smith, D., and Teeters, A. (2007). Common

consensus: a web-based game for collecting common-

sense goals. In ACM Workshop on Common Sense for

Intelligent Interfaces.

Morschheuser, B., Hamari, J., and Koivisto, J. (2016).

Gamiﬁcation in crowdsourcing: A review. In 49th

Hawaii International Conference on System Sciences

(HICSS), pages 4375–4384.

Rodosthenous, C. and Michael, L. (2016). A hybrid ap-

proach to commonsense knowledge acquisition. In

proceedings of the 8th European Starting AI Re-

searcher Symposium (STAIRS 2016), pages 111–122.

IOS Press.

Ryan, R. M. and Deci, E. L. (2000). Self-determination the-

ory and the facilitation of intrinsic motivation, social

development, and well-being. American psychologist,

55(1):68–78.

Siorpaes, K. and Hepp, M. (2008). OntoGame: Weaving the

semantic web by online games. In The Semantic Web:

Research and Applications, volume 5021 of Lecture

Notes in Computer Science, pages 751–766. Springer

Berlin Heidelberg.

von Ahn, L. and Dabbish, L. (2008). Designing games with

a purpose. Commun. ACM, 51(8):58–67.

Toward Gamiﬁed Knowledge Contents Reﬁnement - Case Study of a Conversation Partner Agent

307