Evaluation of User Engagement for Augmented Reality Educational

Game using PEEM and Characteristics of Good Model

Riwinoto, Nawar Safura

Multimedia and Networking, Politeknik Negeri Batam, Batam, Indonesia

Keywords: User Engagement, Augmented Reality, Puzzle AR, Game.

Abstract: Puzzle AR Game is an educational game based on augmented reality. In general, the game is an application

that has a calculation of user experience to pay attention to the experience gained by users of the use of the

application. User experience itself is known to have four components, that are, utility, usability, appeal, and

engagement. This research focused on user engagement. The assessment of user engagement on mobile-based

augmented reality games must combine two aspects, namely from the game side and the augmented reality

side. This research elaborates on an evaluation of the Puzzle AR game was carried out using a model related

to augmented reality namely PEEM (Positive Engagement Evaluation Model) and related to the classic game

Characteristics of Good Game. The models refer to a conceptual model of user engagement on android games

with augmented reality. A total of 15 respondents in this research were children aged 6 to 10 years and were

in kindergarten to elementary school education grade 4. The value results obtained were user engagement in

the Puzzle AR game is sufficient with an average value of net experience 10.756 after testing the reliability

with a score of 0.548 which means quite reliable.

1 INTRODUCTION

Augmented reality is a technology that combines

unreal or virtual 3D objects generated from a

computer with objects or environments that exist in

the real world in real or real time (Azuma, 1997). Not

only presenting technological advances but

augmented reality which is also being developed at

this time has a unique appeal for modern society

because of its immersive that can make users feel real

interacting with virtual objects brought by the

technology. Digital games are also experiencing the

same development, especially on smartphone media.

Education is one of the fields to implement

educational game. It has a positive thing in the form

of being able to be a medium of learning a topic along

with its entertaining nature. According to Rosa and

Shalahuddin (2011), educational games are digital

games designed for educational enrichment

(supporting teaching and learning) using interactive

multimedia technology (Widiastuti, 2012).

Puzzle AR is an educational application game

based on augmented reality. This game presents

information and simple interactions on animal

animations to be played in a single-player with the

target of children under ten years. This game was

made in 2018 on behalf of a game developer company

named Float Indonesia. This game is still in the

development stage. Trials have been held but have not

touched all the targets.

The sample of this study is children with

educational levels ranging from kindergarten to

elementary school, year one to four. The range of

education is taken by considering two factors. First,

based on the ESRB (Entertainment Software Rating

Board) (ESRB, 2019). ESRB is a brief guide to the

appropriateness of content in video games and

applications to make it easier for customers to choose

a game (ESRB, 2019). There are five ESRB

categories, namely E means everyone matches any

age range, E 10+ means everyone with an age greater

than 10 years, T means teen with 13 years of age, M

means mature, the adult with an age over 17 years,

and AO which means Adults Only 18+ with an age

over 18 years. The research sample of this study falls

into the E category. Within ESRB (2019), it is

explained that the content in this category contains at

least cartoons, fantasies or minor violence, and mild

and even rare language. While the next category is E

10+.Therefore the lowest level category was chosen,

which is a sample of children ranging from those who

had attended kindergarten to grade four of elementary

Riwinoto, . and Safura, N.

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model.

DOI: 10.5220/0010351500480057

In Proceedings of the 3rd International Conference on Applied Engineering (ICAE 2020), pages 48-57

ISBN: 978-989-758-520-3

school. Generally, children enter elementary school

education when they reach seven years old. The

second factor is based on the IPA syllabus (Natural

Sciences) material. It is known from Ministry of

Education and Culture (Kementerian Pendidikan dan

Kebudayaan, 2016), in the scope of things about

creatures and living systems, the focus of the material

on animals begins in the grade four while below the

material on how to care for living things around is still

a simple introduction in general only. So that this

game is considered suitable as a medium of learning

and introduction to animals in a simple way to

children with kindergarten education level up to grade

four elementary school.

Games also have the value of user experience as

applications in general, but the user experience is a

broad discussion. There are tons of journals that

discuss various components of user experience and

evaluation models. Regarding differences in user

experience and user engagement, quoting from UX

Designer (2015) that user experience has several

components, namely utility, usability, appeal, and

engagement. Engagement becomes one of the

components of the user experience that is related to

the convenience of users so that it attracts the user's

appetite for repeated use (2015). According to Ganot

(2015), user engagement refers to how often and for

how long users interact with websites, applications,

or other products and take action in them. O'Brien and

Toms (2008) also argue that successful technology is

not only usable, but they are engaged users (getting

users involved). Referring to this understanding, the

measurement of user engagement becomes

something interesting to study. It can be formulated

that user engagement is part of the user experience

that focuses on how involved the user is with the

application by paying attention to the user's behavior

while using the application.

Permadi and Rafi (2015) used 8 user engagement

models to be formulated into 8 potential attributes

that can be used to measure user engagement in an

augmented reality game on Android. The models

used to consist of four user engagement models

related to digital games including ES for Video

Games, Characteristics of Good Game, UE in Games,

Game Flow, and two user engagement models related

to augmented reality technology including Mixed

Fantasy Triad, and PEEM.

From this formulation, 8 potential attributes were

formed to measure user engagement in an augmented

reality game on android, namely clear goals,

satisfaction, focused attention, mixed fantasy,

perceived usability, challenge, interaction, and social.

The reason Permadi and Rafi conceptualize a user

engagement model by combining user engagement

models related to digital games and augmented reality

technology is referring to Wetzel [9] that in designing

an augmented reality game, designers need to cover

all aspects by combining elements in the game digital

classic with AR technology to enhance user

experience. This method is taken because there is still

a lack of studies that discuss user engagement aimed

at mobile games based on augmented reality. The

model that was brought in Wetzel (2008) also had not

yet been tested because it was only a concept. To do

the trial, it takes a long time so two models are chosen

that are considered sufficient to represent each aspect

Positive Engagement Evaluation Model (PEEM)

(Rutledge & Neal, 2012) related to augmented reality

technology and Characteristics of Good Games

(Malone & Lepper, 1987) related to digital games.

2 THEORETICAL FRAMEWORK

2.1 User Engagement

User engagement is often equated with user

experience even though these two things have their

respective meanings. The core aspects that affect the

success of user experience are utility, usability,

appealing, and engaging (What is UX Designer?

2015. So if the user experience discusses user

satisfaction and feelings when using the product

widely, user engagement is more about measuring the

likelihood of the user to reopen the application, the

repeated use, and perform a series of actions in the

application. User engagement refers to how often and

how long users interact with websites, applications,

or other products and take action in them (Ganot,

2015). So it can be concluded that user engagement is

part of the user experience that focuses on discussing

how involved the user is with the application by

paying attention to how the understanding and

behavior of the user while using the application

2.2 Positive Engagement Evaluation

Model (PEEM)

PEEM is a matrix model formulated based on

psychological theory, narrative transport theory, and

several neurological concepts to evaluate user

engagement and the effectiveness of interactive

immersive media such as augmented reality

(Rutledge and Neal, 2012). PEEM developed by

Rutledge and Neal (2012). The 9 evaluation elements

of user engagement in PEEM are goals, attention,

concentration, interaction, content, identity,

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model

collaboration, and attitudes, enjoyment &

satisfaction.

The goal is an element that acts as motivators of

human behavior and these goals continue to be

inherent in the human mind when carrying out any

interaction. The progress of the process experienced

by players to achieve goals is the dominant source of

value in optimal user engagement. The goal of this

element is a clear, task-oriented component so that the

game becomes productive while still having

narrative-based experience.

Attention is an element to measure how well the

player's attention to the game and this is the most

important component in augmented reality user

engagement. This relates to how the game creates a

focus that involves the physical and mind of the

player so that it affects the subconscious that makes it

focus. This element also investigates the tasks in the

application and their sequence and whether the

controls are easy to understand or not.

Concentration is talking about the player's

attention to the game on an ongoing basis. Optimal

engagement can occur when all the energy and skills

are mobilized by the player to complete a challenge

or task.

Interaction is an element that investigates whether

the application presents a good and clear development

from one task to another or from one display to

another display, does not have significant

interruptions such as error messages, and has content

that suits user needs.

The content discusses the media objects, images,

or videos in the game running smoothly and relevant

to the objectives to be achieved or not. Emotions

cannot be ignored in this element. Content must have

a target of how users' emotions to be achieved

towards the content presented, such as triggering a

sense of adventure, curiosity, and pleasure.

Identity talks about building the skills of players.

Skills can be built through effective interactive design

and responsive progress feedback as challenges

become increasingly difficult. When a player realizes

his success in completing a challenge, he will have

positive confidence in his competence and trigger

intrinsic motivation to continue playing.

The collaboration deals with the social

connections of players. Social needs are one of the

most powerful forces of persuasion. Humans are

neurologically connected to look for social bonds.

Social perception, competition, collaboration, and

other social activities can arouse the motivation of

players to play again and increase user satisfaction.

The results of emotional Attitudes, Enjoyment,

and Satisfaction are good. it will provide a positive

experience for repeated use and motivation to share

their experiences with others. The reward system will

encourage players' intrinsic motivation. The

achievement will increase the positive attitude and

emotions of the player. Starting from clear progress

markers such as scores, leader boards, and social

validation such as connectivity and social comparison

are needed in the game so that the experience of using

the internet in the game becomes important in this

element.

2.3 Characteristic of Good Game

The second model is the Characteristics of Good

Game, developed by Malone and Lepper (1987),

focuses on evaluating classic digital games (Permadi

& Rafi, 2015). Elements to evaluate user engagement

on classic digital games are challenge, curiosity,

control, and fantasy.

Challenge is the main principle in intrinsic

motivation. Challenges that are too easy or too

difficult will get low intrinsic. While challenges that

are difficult in the middle will make the challenges

interesting. To make activities in the game feel

challenging, the game must provide goals that go up

and down. Not always high or not always low.

Feedback is also needed to increase the player's

individual confidence.

Curiosity becomes an element that is related to the

curiosity of players who are divided into sensory and

cognitive curiosity. Sensory curiosity stimulates

curiosity that involves the five senses such as

textbook examples that are full of color, thus

stimulating the eye to want to pay attention to the next

pages. While cognitive curiosity is the existence of

cognitive impulse to realize the "perfect size" in the

game. For example, players see the object of plants in

the game with a garden background behind the house.

Players will have the desire to water because

knowledge in general plants needs water to live.

Then there are controls related to the player's

ability to control what he does and determine his

destiny in the game. The biggest strength of a game is

its control. A high number of control values can be

obtained from players who have a learning process

during the use of controls. With these controls, the

player understands which one is used to walk and

perform other actions.

Fantasy becomes an element that can evoke

mental images or thoughts based on physical or social

situations that are not real. In the aspect of

endogeneity, the game must make players able to

imagine and think about something seriously first

which can improve their skills before achieving what

ICAE 2020 - The International Conference on Applied Engineering

they want. To attract the emotional aspects, the game

also needs to give players satisfaction with success,

feeling in charge of the game, and something that

does not exist in real life.

3 METHODOLOGY

This research method uses a quantitative approach.

The formulation of user engagement evaluation

elements is done first, by exploring the understanding

of each element in the two evaluation models. The

results obtained that all elements have different

definitions so that the number of evaluation elements

is obtained by 12 elements with total assessment

items are 38 items.

The evaluation process refers to Enrique (2012).

THE puzzle AR game was reviewed and rated by 15

respondents. These fifteen people consist of five

levels of education so that there are three respondents

at each level, from kindergarten to elementary school

grade four. Data obtained from filling out the

questionnaire. Each respondent was left playing the

Puzzle AR game then asked to complete an

evaluation. Interviews and observations are also

conducted to ensure that respondents who are

children age 6 to 10 years can understand and assess

each item.

Reliability tests used to measure how reliable the

results of the research are. Reliability test using the

ReCal OIR online tool from the website of Dr.

Freelon http://dfreelon.org/utils/recalfront/recal-oir/.

The range of reliability scores based on Guilford

criteria (Guilford, 1956). For assessment using the

Likert scale method. Each evaluation element

consists of two to four statement items. The final

result of this calculation formula is called net

experience. The criteria of the net experience will be

the data analyzed and the evaluation reference.

For each respondent, the values per element are

added and divided by 12 (number of elements) to get

the net experience value. The 12 elements consist of

8 elements from the PEEM model and 4 elements

from the Characteristics of Good Game. Based on this

formula, the lowest net experience value that can be

obtained is 3.17 with a total score of 38. While the

highest net experience is 15.84 with a total score of

190. To determine the criteria, the Umar [15] scale

range formula is used to obtain the results presented

in Table 1 while formulation of user engagement

elements from the PEEM and Characteristics of Good

Game model can be seen in Table 2.

Table 1: User engagement criteria scores

Score range

Criteria

1 3.16

– 5.696

Very Less

5.697 – 8.232

Less

3 8.233

– 10.768

Sufficient

4 10.767 – 13.304

Good

5 13.305 – 15.84

Very Good

4 RESULT AND DISCUSSION

4.1 Result

The reliability coefficient of the Puzzle AR game is

0.548. That means the research instruments are

reliable enough to be analyzed. In the assessment,

there are averages to display the average value of the

score in each element, and ST DEV (standard

deviation) is the consistent value. Standard deviation

is a statistical valuation technique used to determine

the distribution of data in a sample, as well as how

close the individual data points are to the mean or

average [16]. The more the standard deviation is close

to 0, the more consistent the ratings given by the

respondent to the element. Consistency in value

becomes important as a reliable measurement of

whether or not the score on an element. An element

can be said to be good if the high average value

obtained is also accompanied by a standard deviation

value close to 0. The comparison chart attached in

Table 3.

The net experience category gained from

kindergarten age trainers is good. Kindergarten got

the second-highest grade after grade three. The

highest value with good value consistency is in the

Content, Curiosity, and Control elements. The

content talks about the richness of images, video, and

sound. Kindergarten respondents have the same high

rating of the experience of realizing the many

variations of images and sounds in the game. Overall,

the ratings given between the respondent were

classified as consistent.

Kindergarten respondents have a fairly uniform

assessment. Even so, there are still elements with the

lowest value, namely the elements of Collaboration

and Attitudes, Enjoyment, and Satisfaction. This

shows that players feel the lack of collaboration

features and the feeling of wanting to play again is

due to the rewards given in the game.

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model

Table 2: Evaluation instrument for user engagement from PEEM and characteristics of good game model.

Evaluation

Model

Indicator User Experience Explanation

PEEM

Goals I feel the purpose of this game is to display animals in the

form of augmented reality (appears after confronting the

camera)

The activities and objectives are clear

The tasks and activities given make sense with the aim of

animal recognition

The given task directs the player to reach

the goal

I understand how to achieve my goals and feel I can do it (Ex:

I can explain how to hear animal sounds by following

instructions)

Steps that lead to easy goals with a

solution that can be managed

Attentions

I can play it with smooth stages Seamless task sequence

I understand the use of UI controls (2D buttons and joystick) UI controls are easy to understand and

Pictures and sounds helped me understand the Puzzle AR

game

Visual or hearing support improves

understanding

Concentration

Playing Puzzle AR makes me interested and feel challenged

The UI maintains the player's attention

and the appropriate level of challenge

I can say what I have to do and what I do it for The task is easy to understand and the

purpose is clear

I got helpful feedback for directions to reach the goal

Feedback can provide learning through

task completion

Interaction

I thought of a strategy for playing the Puzzle AR game

Players have many ways to feel control,

such as personalization, activity choices,

or filtering

I don't feel disturbed if there are error messages in the game

There are no disturbances such as

configuration, error messages, or

irrelevant data.

Help by sending messages to others through the application

makes the game feel personal.

Interaction, assistance, and delivery of

messages from the use of social behavior

applications (first-person

communication).

The tasks in the game according to my needs and skills

Content and tasks adjust to the needs and

skills of users

Content

Pictures and videos are many and run smoothly

Sound, touch and rich media (imaging,

overlays, video, enhanced display) are

seamless

I feel adventure and pleasure

Content designed to target certain

emotions (wonder, adventure, fun,

intrigue)

The game runs well and no images, videos or audio interferes

with my goal

Content designed to match costs to

eliminate task interruptions

Pictures, videos or audio make sense with their activity and

purpose

Content is relevant to the task and

supports the logic and purpose of the

activity

Identity

It was fun playing the Puzzle AR game and made me imagine

a lot

Activities provide the integration or

imaginative projection of users into

experiences

Puzzle AR Games improve my skills and knowledge

Development of skills and structured

mastery

There is evidence of progress (appreciation/reward) as the

achievement

Responsive feedback from progress and

achievement

Collaboration

I can compare Puzzle AR games through links or social media

Integrated social connections or

Comparisons (social network link)

I received other people's responses through the Puzzle AR

game

Validation, strengthening feedback from

social elements

I can change the content as I wish

The ability to create, participate or

personalize content

Attitudes,

Enjoyment,

Satisfaction

Gifts and awards make me want to play again

The inherent motivation or reinforcement

to repeat or repeat an activity (emotional,

reward, or social)

ICAE 2020 - The International Conference on Applied Engineering

I can collaborate or share my experience playing Puzzle AR

about the scores or awards that I get

Opportunities for comparison or

competition (scores, prizes, badges)

There is a feature where I can rate, comment, or vote

There are ratings, comments or 'send to

friends' features

Characteristic

of Good

Game

Challenge

I know the Puzzle AR game is a game to find information

about animals

Objectives: Activities must be clear,

some goals have been set

I felt the score I got was uncertain

Uncertain results: varying degrees of

difficulty, selectively revealed,

randomness

I get a lot of feedback in the form of words of encouragement

Feedback must be frequent, clear,

constructive, and encouraging

My self-esteem is increasing, because I always get scores and

positive feedback

Self-esteem: activities must have a level

of difficulty that is assessed, and feedback

techniques to enhance a sense of

competence

Curiosity

I want to know what animals and animal sounds are in the

Puzzle AR game

Sensory curiosity increases because of

the variability of audio and visual effects

I want to know all the actions in the game by running all the

action buttons on the animal.

Cognitive curiosity: Curiosity can be

enhanced by the existence of teaching

techniques that make players feel

surprised, interested because they are

aware of the incompleteness that must be

complete

Control

Puzzle AR Games give me a hint when I'm playing a game.

Contingency: activities must provide a

responsive learning environment

I can control the game if I want to achieve a certain goal. Ex:

if I want a crocodile to dive, then I must first walk it into the

water

Choice: the activity must provide and

emphasize moderate level choices on

various aspects of the learning

environment

I gained enthusiasm by successfully acting like animals

Power: the activity must allow the player

to gain strength

Fantasy

I also felt what the animal I was playing was feeling

Emotional aspects: fantasy must be

designed to attract students' emotional

needs

I understand all the explanations given in the Puzzle AR

game.

Cognitive aspects: fantasy must provide

an analogy for the material presented for

learning

Playing the Puzzle AR game made me have to think and learn

something from the game to get the desired results.

Endogeneity: fantasy must have an

integral (overall) relationship,

endogenous (derived from the deepest

instincts), and material learne

The average value of net experience in this grade

has the lowest level among others. The value of user

engagement obtains sufficient criteria. The highest

average value is found in the Content element with a

consistency value of 1.15 which is close to

uniformity. Respondents averaged the wealth and

smoothness of graphics and audio in the game. It's

just that respondent 6 gives the lowest value than

another respondent.

This is because the respondent is not actively

playing an android game so that it has found some

difficulties when playing it. The consistent element

values are in the Collaboration, Attitudes, Enjoyment,

and Satisfaction and Interaction elements. But the

value given is low. All respondent agreed to disagree

with statements on the elements of Collaboration,

Attitudes, Enjoyment, and Satisfaction, and

Challenge. respondent both felt they had not found

collaboration, feedback reward, and feeling

challenged in playing the Puzzle AR game with

satisfactory value. The element with the most

inconsistent value is Goals. respondent 6 claimed not

to be able to understand the purpose of augmented

reality in the Puzzle AR game so he gave a low rating.

For children who rarely have experience playing

games or are not interested in playing third or first-

person games will find it difficult because they need

to adapt to the UI control layout in the game.

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model

Table 3: Comparison of user engagement evaluation scores

Model

Indi-

cator

Kindergarten 1st grade ES 2nd grade ES 3rd grade ES 4th grade ES

Res

. 1

Res

. 2

Res

. 3

ST DEV

Resp. 4

Resp. 5

Resp. 6

Avg.

ST DEV

Resp. 7

Resp. 8

Resp. 9

Avg.

ST DEV

Resp. 10

Resp. 11

Resp. 12

Avg.

ST DEV

Res

. 13

Res

. 14

Res

. 15

Avg.

ST DEV

PEEM

Goals

13.33

0.58

13.00

3.46

13.33

2.08

13.67

1.15

14.33

0.58

Atten-

tions

13.00

0.00

12.67

3.21

13.00

1.00

13.67

0.58

12.00

1.73

Con-

cen-

tra-

tion

12.67

0.58

10.67

3.21

9.00

3.61

12.00

1.73

13.00

1.00

Kindergarten respondents have a fairly uniform

assessment. Even so, there are still elements with the

lowest value, namely the elements of Collaboration

and Attitudes, Enjoyment, and Satisfaction. This

shows that players feel the lack of collaboration

features and the feeling of wanting to play again is

due to the rewards given in the game.

The average value of net experience in this grade

has the lowest level among others. The value of user

engagement obtains sufficient criteria. The highest

average value is found in the Content element with a

consistency value of 1.15 which is close to

uniformity. Respondents averaged the wealth and

smoothness of graphics and audio in the game. It's

just that respondent 6 gives the lowest value than

another respondent.

This is because the respondent is not actively

playing an android game so that it has found some

difficulties when playing it. The consistent element

values are in the Collaboration, Attitudes, Enjoyment,

and Satisfaction and Interaction elements. But the

value given is low. All respondent agreed to disagree

with statements on the elements of Collaboration,

Attitudes, Enjoyment, and Satisfaction, and

Challenge. respondent both felt they had not found

collaboration, feedback reward, and feeling

challenged in playing the Puzzle AR game with

satisfactory value. The element with the most

inconsistent value is Goals. respondent 6 claimed not

to be able to understand the purpose of augmented

reality in the Puzzle AR game so he gave a low rating.

For children who rarely have experience playing

games or are not interested in playing third or first-

person games will find it difficult because they need

to adapt to the UI control layout in the game.

In grade two elementary players, the average

value of net experience falls into the moderate

category. They gave the second-lowest net experience

rating after grade one. There was an increase in scores

on Goals, Attentions, Interaction, Collaboration,

Attitudes, Enjoyment, and Satisfaction, Challenge,

Control, and Fantasy scores compared to the grades

given by grade one respondent, Control and Fantasy

become the elements with the best value because high

scores are also consistent. Fantasy explains the

intrinsic motivation of the player based on the player's

ability to guess and imagine what he can experience

as a Puzzle AR game player. This means that players

feel the same intrinsic motivation when playing the

Puzzle AR game, likewise experience in

understanding the controls of the Puzzle AR game. At

this level, respondent begins to feel understands what

animal is being played and how it is explained.

Interaction has a low consistency value. One

respondent (respondent 9) gave the lowest rating on

items that discussed strategies, interruption of error

messages, and help features to others. Puzzle AR

Games are considered capable of fulfilling the

characteristic elements of being responsive to their

activities while playing. The lowest score is in the

Attitudes, Enjoyment, and Satisfaction and

Collaboration elements which means that the grade

two respondents also feel the same deficiency as the

previous grade.

The net experience value at the elementary

school level of grade three is the highest and the score

is classified as good criteria. The assessment given

was quite positive especially for respondent 11 who

felt the highest involvement. The highest value is in

the Curiosity and Attentions elements which also

ICAE 2020 - The International Conference on Applied Engineering

have high consistency value. This shows that grade

three elementary school players have the same value

in terms of playing the Puzzle AR game smoothly,

understanding what to do next, and curiosity about

what the players can do in this game. At this level of

education, players begin to be curious about what

they can find in the Puzzle AR game. Even so, the

value given by the respondent is less uniform. The

lowest consistency score is in the Attitudes,

Enjoyment, and Satisfaction element reaches a score

of 5.69 then followed by the Identity and Challenge

elements. That is, there is an imbalance in the

experience of respondent regarding feeling satisfied

with appreciation, and feeling challenging when

playing the Puzzle AR game. respondent 11 was

satisfied with the rewards and challenges in the game.

But for another respondent the game only provides

the experience of playing the game without getting a

satisfying reward to the player.

In the level of education with the oldest players,

the average score of net experience is good. High and

consistent average values are found in the Goals,

Control, and Fantasy elements with a consistency

value of 0.58. Departing from the focus of Control,

Goals and Fantasy elements, it can be analyzed that

grade four elementary school players uniformly

assume the game can provide clear play direction to

feel the augmented reality features, understand the

purpose of the Puzzle AR game, be able to use

controls, and increase motivation intrinsic to imagine

what could happen in this game. The Interaction is

based on explaining the criteria, discussing clear

steps, few interruptions, and responsive games to the

user. This element has a high average value but is not

consistent. respondent 13 gave a fairly low rating on

this element because it had experienced a slight error

while playing. The Challenge element has also

decreased in value compared to other education

classes. For grade four respondents, the Puzzle AR

game is less challenging and the score doesn't raise

the player's self-esteem. The respondent began to

understand that this game was just a simple animal

recognition game that didn't have a difficult mission.

The lowest value is in the Attitudes, Enjoyment and

Satisfaction, and Collaboration aspects. At all levels

of education, these two elements have the lowest

value, which means that the respondent feels

dissatisfied with the collaboration and reward

features of the game.

Evaluation scores per element obtained from the

total of all samples can be useful to provide more

detailed information about which elements are of

good value and need improvement. Goals have good

grades but are slightly volatile between levels of

education. Had experienced a decrease in grade one

respondent because there was one respondent who did

not understand how to play android games. Then the

scores go up and down until the grade four elementary

respondent gives the highest rating which means they

understand the purpose of this game the most.

Initially, Attentions got grades that were

ascending and stable, slightly increasing in grade

three but later scores dropped. Grade three

respondents have a little higher attention but then

grade four respondents give lower value. Even though

it is still in the good category, it indicates that the level

of grade four respondent attention to the Puzzle AR

game is not as enthusiastic as the other respondent

groups. Significant value fluctuations also occur in

the Challenge element. Feeling challenged in this

game is considered insufficient. Had a high value in

the grade three respondents but dropped back the

same as kindergarten respondents. For concentration

overall, it's good value. Starting with a high enough

value but then dropped on the grade two respondents.

This is because some respondents feel less interested

and less attention to instructions. Then the value

increases again and the value is slightly higher than

the kindergarten respondent. The Interaction element

also experienced the same thing, initially the high-

value element but dropped in the grade one and then

back up with a value that is not too different from the

initial value. One respondent is enough to influence

the average rating of the elements. One respondent

feels that the game doesn't match his skills so a lot of

low marks are given.

Content gets a very high value, then decreases

and then rises again but not as high as the value of the

kindergarten respondents. This might be due to that

for the kindergarten respondents; the content is still

very interesting and is something they have just met

so that interest in the content is still very high and

enthusiastic. Value is still in good criteria. For

Identity, it has an ascending graph and gets high

marks on grades three and four. The Puzzle AR Game

can show its identity as an animal introduction game

in augmented reality and the respondent can absorb

the information provided in the game well.

Collaboration and Attitudes, Enjoyment, and

Satisfaction get the lowest scores that fall into the bad

category among all elements even though there is an

increase for grade three respondents. Both of these

elements talk about the experience of the player's

feelings in the form of repeated use, telling others,

ease in share Puzzle AR game information, and

collaborate. The point of most concern is

collaboration, where almost all respondent feel they

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model

disagree with the collaboration capabilities of the

Puzzle AR game feature.

Then there are the Curiosity, Control, and

Fantasy elements that get good grades. In the

Curiosity element, the results are obtained that the

grade four respondents gave a lower value than the

kindergarten respondent even though it had risen in

the grade three respondents. It appears that the

kindergarten and grade three that were sampled in this

study had the same high curiosity towards the Puzzle

AR game. In the Control element, there is a decrease

in grades in elementary grade one respondent but then

experiences ups and downs that end with a value that

is not much different from the kindergarten value.

The Fantasy element becomes the most stable

element after Attentions. The highest score is in grade

three respondents. This means that the Fantasy felt by

the respondent is quite good and does not have

differences between respondent even with different

age ranges.

4.2 Discussion

Evaluation of user engagement using the PEEM

model and Characteristics of Good Game in an

educational game based on augmented reality

produces an evaluation element of 12 with different

characteristics. These elements are Goals, Attention,

Concentration, Interaction, Content, Identity,

Collaboration, and Attitudes, Enjoyment &

Satisfaction, Challenge, Curiosity, Control, and

Fantasy.

The evaluation value of user engagement on the

Puzzle AR game using a combination of the PEEM

model and the Characteristics of Good Game is

10.756 which means it is sufficient. Some results

were divided into evaluation results based on the level

of education and assessment per element. Overall

education level, respondent grade one elementary

school experience the lowest user engagement

experience due to the influence of one respondent

who is not familiar with android games. But then

followed by respondent grade two, then grade four,

kindergarten, and the highest is in grade three. Grade

four respondents, scores start to decline due to simple

gameplay, not having too challenging missions is

considered to be inappropriate for children of this

level of education.

The element with the highest consistency is

Curiosity. Almost all respondent agrees that Curiosity

is very good in the Puzzle AR game. Curiosity itself

discusses the desire of players to know about content

and controls in the game. Control has the highest

average value. This means that the controls on the

Puzzle AR game are judged to be understood even by

kindergarten respondent. The elements with the

lowest values are Attitudes, Enjoyment and

Satisfaction, and Collaboration. This element talks

about games that can tell activities to others, ease in

sharing information about the Puzzle AR game to

others, and can collaborate. Dominantly, the

respondent considered that the game has not been

able to provide feedback in the form of satisfying

rewards and collaboration capabilities as expected.

From the findings of this evaluation the

following technical recommendations to increase the

value of user engagement in the Puzzle AR game:

1. To increase the value of attitudes,

enjoyment, and satisfaction, the game needs

to add simple missions by utilizing actions

that can be done by each animal and then

provide a scoring feature to increase the

sense of challenge and enjoy the

appreciation in the form of assessment.

2. To increase the value of collaboration, add

multiplayer features where between

Androids can play and interact with one

another in the same scene and at the same

time to improve the assessment of

collaboration.

3. Add comments or share the results of

playing to social media to increase

confidence and open opportunities for

indirect game recognition for those who do

not know the Puzzle AR game.

4. Add features giving encouraging words and

responsive hints when players are detected

by the game less able to complete the

mission and achieve its goals which can

increase the value of interaction.

5. Increase the stock of animal animations in

the game along with clear and valid

educative information so that the game

experiences become more professional and

the content value increases.

4.3 Limitation

Characteristics of Good Game model has a very

classic rating indicator based on the source which

gives examples of very simple games such as darts

and mathematics games while the current game has

more complex gameplay. It is expected that the user

engagement evaluation research on android games

based on augmented reality can use a combination of

PEEM evaluation models with the latest evaluation

models so that the elements are more suitable and by

ICAE 2020 - The International Conference on Applied Engineering

the gameplay of existing android games at present. It

is also expected that the subsequent studies will have

higher number of respondents with a more diverse

range of age and education level, hence, a wider

Puzzle AR game can be studied.

5 CONCLUSION

The PEEM and Characteristics of Good Game models

provide user engagement score results in the Puzzle

AR game within sufficient criteria. But the Puzzle AR

game also requires some improvement such as adding

simple missions, scoring, multiplayer, comments, and

sharing, encouraging words, hints, and animal

animations stock.

REFERENCES

Azuma, R T., 1997. A Survey of Augmented Reality.

Teleoperators and Virtual Environments, 6(4), 355-

385.

Designer, UX., 2015. What is a UX Designer? Retrieved

November 1st, 2019, from UXDesigner:

http://www.userexperiencedesigner.co.uk/new-what-

is-ux-designer-ia.htm.

Enrique, L., 2012. Assessing User Experience in

Augmented Reality Applications Using the Positive

Engagement Evaluation Model, Universitas Fielding

Graduate..Santa Barbara.

ESRB., 2019. ESRB Ratings Process. Retrieved January

, 2019, from ESRB:

https://www.esrb.org/ratings/ratings_process.aspx

Fitra., 2019. Retrieved May 14

, 2020, from Rumus.co.id:

https://rumus.co.id/standar-deviasi/

Ganot, R., 2015. What is User Engagement? Retrieved

November 2nd, 2019, from CodeFuel:

https://www.codefuel.com/blog/what-is-user-

engagement/

Guilford, J.P., 1956. Fundamental Statistic in Psychology

and Education, McGraw-Hill Book Company, Inc.

New York, 3rd Edition.

Kementrian Pendidikan dan Kebudayaan, 2016. Silabus

Mata Pelajaran Sekolah Dasar/Madrasah

Ibtidaiyah(SD/MI) Mata Pelajaran Ilmu Pengetahuan

Alam (IPA).

Malone, T. W., Lepper, M. R., 1987. Making Learning Fun:

A Taxonomy of Intrinsic Motivations for Learning

(Vol.3). (M. J. Richard E Snow, Ed.). Mahwah, NJ:

Lawrence Erlbaum Associates.

O’Brien, H. L., Toms, E. G., 2008. What is User

Engagement? A conceptual framework for defining

user engagement with technology. Journal of the

American Society for Information Science &

Technology, 59(6), 938 – 955.

Permadi, D., Rafi, A., 2015. Developing a Conceptual

Model of User Engagement for Mobile-based

Augmented Reality Games. Jurnal Teknologi (Science

& Engineering), 77(29), 9-13.

Rosa A.S, M. Shalahuddin. (2011). Modul Pembelajaran

Rekayasa Perangkat Lunak (Terstruktur dan

Berorientasi Objek). Bandung: Modula.

Rutledge, P., Neal, M., 2012. Positive Engagement

Evaluation Model for Interactive and Mobile

Technologies. In 2012 EEE International Conference

on e-Learning, e-Business, Enterprise Information

Systems, and e-Government. Universitas Fielding

Graduate.

Umar, H., 2005. Metode Penelitian Untuk Skripsi dan Tesis

Bisnis, PT. Raja Grafindo Persada. Jakarta.

Wetzel, R, et al., 2008. Guidelines for Designing

Augmented Reality Games. Future Play 2008. Toronto.

Widiastuti, N. I., 2012. Membangun Game Edukasi Sejarah

Walisongo. Jurnal Ilmiah Komputer dan Informatika,

1(2).

Evaluation of User Engagement for Augmented Reality Educational Game using PEEM and Characteristics of Good Model