Effectiveness of Gamiﬁcation in Undergraduate Education

M. E. Sousa-Vieira, O. Ferreira-Pires, J. C. L

opez-Ardao and M. Fern

andez-Veiga

atlanTTic, Universidade de Vigo. 36310 Vigo, Spain

Keywords:

Gamiﬁcation, Social Learning, Learning Analytics.

Abstract:

What is the quantitative effect of gamiﬁed learning/teaching on the performance of students? This basic

question has not received as much attention as it deserves in the related literature, despite the many works that

analyze the beneﬁts and design principles of gamiﬁed activities across many domains. We present in this paper

an analysis of this question focused on identifying the network structure variables that quantitatively explain

the improvements reported when using gamiﬁcation integrated in the course design. We compare the results

with those from a group that does not follow gamiﬁcation at all, hence we are able to ascertain the strength

and impact that gamiﬁcation has on numerical grades or scores.

1 INTRODUCTION

This paper is an attempt to clarify the following re-

search question: how effective is the use of gam-

iﬁed activities and learning strategies for students?

As an educational approach, gamiﬁed learning (GL)

exhibits a number of clear advantages related to the

dynamics of the learning process (Aldemir et al.,

2018; Cakirogli et al., 2017), for instance in cre-

ating stronger and longer engagement from the stu-

dents, and in improving their perceived learning ex-

perience. However, when in comes to a quantita-

tive analysis of the real beneﬁts of using technology-

mediated gamiﬁed activities in a course, the picture

extracted from the related research literature is less

clear or systematic. Whereas some studies (Buckley

and Doyle, 2017; Davis et al., 2018; Kyewski and

Kramer, 2018) adopt the point of view of students,

their attitudes and personal traits as one of the factors

that may impact on the effectiveness of GL, another

line of work is focused on less subjective experiments,

and attempts to determine how much improvement

in academic achievement is possible through a com-

prehensive embedding of GL into traditional teaching

styles (Yildirim, 2017).

Our previous work in this area has been focused

on identifying the network properties of the dynam-

ics induced by the GL activities designed for under-

graduates in engineering (Sousa et al., 2017). The

main ﬁnding has been not only that the networked

relations formed amongst the students in the online

social realm can be exploited as robust predictors of

ﬁnal learning success (Sousa et al., 2018), but also

that there exist signiﬁcant statistical correlation be-

tween basic structural properties in the network and

the graded performance of students (Ferreira et al.,

2020a). In this paper, we take one step further and

make a statistical comparison between groups of stu-

dents exposed to GL and students taught without GL

with respect to different types of GL activities. Our

aim is to estimate the quantitative measure that a

given GL teaching tool may have on the academic

performance, so as to guide in the selection of the GL

tools suitable for a given course.

The rest of the paper is organized as follows. Sec-

tion 2 summarizes some recent related work. The

methodology employed in the course under study is

reported in Section 3. Section 4 contains the main re-

sults of the analysis applied to the datasets. Finally,

some concluding remarks are included in Section 5.

2 RELATED WORK

Gamiﬁcation is deﬁned as the use of game design el-

ements in non-game contexts (Kapp, 2012). It can be

applied in several situations to inﬂuence the behav-

iors of individuals, mainly to increase engagement, to

motivate action or to promote learning. Due to the

fact that all these are major issues faced by teachers

of all educational levels, in recent years multiple im-

plementations of gamiﬁcation in educational contexts

have emerged. In this section, we present a literature

review of those focused on higher education. Again, a

more extensive compilation can be found in (Subhash

and Cudney, 2018; Zainuddin et al., 2020). Table 1

summarizes, of the game design elements used in our

case study, those mentioned in each paper.

376

Sousa-Vieira, M., Ferreira-Pires, O., López-Ardao, J. and Fernández-Veiga, M.

Effectiveness of Gamiﬁcation in Undergraduate Education.

DOI: 10.5220/0010495603760385

In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 376-385

ISBN: 978-989-758-502-9

The work (Landers and Landers, 2014) studies

the effect of a gamiﬁed version of an online wiki-

based project in an industrial/organizational psychol-

ogy course, showing an increase of the interactions.

Moreover, results indicate that time-on-tasks predicts

learning outcomes. The results explained in (Ib

nez

et al., 2014) show positive effects on the engagement

of C-programming students toward gamiﬁed learn-

ing activities and a moderate improvement in learn-

ing outcomes. In (Strmecki et al., 2015), authors

conduct an experimental study to investigate the ef-

fectiveness of gamiﬁcation of an online informatics

course on computer graphics. Results show that stu-

dents enrolled in the gamiﬁed version of the course

achieved greater learning success. The article (Kuo

and Chuang, 2016), reports the application of gami-

ﬁcation to an online context for academic promotion

and dissemination. Both quantitative and qualitative

data were collected and analyzed, revealing that gam-

iﬁcation has the potential to attract, motivate, engage

and retain users. The research addressed in (Buck-

ley and Doyle, 2017) examines the impact of different

learning styles and personality traits on students’ per-

ceptions of engagement and overall performance in a

gamiﬁed business course. Findings suggest that stu-

dents who are oriented towards active or global learn-

ing as well as extroverted students have a positive im-

pression of gamiﬁcation. The effect of gamiﬁed in-

structional process to ICT students engagement and

academic performance is studied in (Cakirogli et al.,

2017). Conclusions show that using the proposed

combination of elements provided a positive motiva-

tional impact on engagement and indirectly affected

the academic results. Similar results are observed

in (de Marcos et al., 2017), where authors found

evidence that gamiﬁcation can be used to improve

the overall academic performance in practical assign-

ments and to promote social interaction in a qualiﬁ-

cation for ICT users course. The work (Dias, 2017)

describes the application of gamiﬁcation in an oper-

ations research/management science course, where it

was possible to observe an increase of participation

in class, better results and a good assessment of the

course made by the students. In the study (Sailer

et al., 2017) authors vary different conﬁgurations of

game design elements and analyze them in regard to

their effect on the fulﬁllment of basic psychological

needs. The article (S

anchez et al., 2017) presents a

gamiﬁcation experience within prospective primary

teachers in a general science course. In an effort for

promoting collaborative dynamics rather than com-

petitive ones, a new variable called game index, that

takes into account the scoring of the whole class, was

introduced. A positive correlation among scoring and

academic marks was conﬁrmed. The experiment de-

scribed in (Yildirim, 2017) was conducted to deter-

mine the effect of gamiﬁcation-based teaching prac-

tices on achievements and attitudes towards learning,

in a course about teaching principles and methods.

The results show positive attitudes towards the lessons

and a moderate effect on achievements. Although

there was not difference between the ﬁnal grades of

the gamiﬁed and the control groups, students regard

wiki and gamiﬁed activities positively.

Recently, the article (Aldemir et al., 2018)

presents the gamiﬁcation process, iterations made into

the game elements and their features and students’

perceptions in a gamiﬁed teacher education course.

The study (Davis et al., 2018) investigates college stu-

dents’ experiences of a gamiﬁed informatics course,

showing positive trends with respect to students’ per-

ceptions of gamiﬁcation’s impact on their learning,

achievement, and engagement in the course material.

The authors of (Ding et al., 2018) explore the ef-

fects of gamiﬁcation on students’ engagement in on-

line discussions. Conclusions and interviews with

students and teachers suggest a positive effect of the

game-related features of the platform. The focus of

the study reported in (Huang and Hew, 2018) is to

explore whether gamiﬁcation could be a good strat-

egy to motivate students to participate in more out-

of-class activities without decreasing the quality of

work. Results from two experiments conducted in

two master level courses on statistics reveal that the

gamiﬁed classes completed signiﬁcantly more activi-

ties and produced higher quality work. In (Jo et al.,

2018), authors test the effectiveness of adding edu-

cational gaming elements into the online lecture sys-

tem of a ﬂipped classroom, as a method to increase

interest in online preparation before class, obtain-

ing good results and better academic achievements of

mid-upper level students. The work (Jurgelaitis et al.,

2018) analyses the effect of using gamiﬁcation ele-

ments in a course related to software development.

The study conﬁrms that students’ grades and moti-

vation can increase as a result of applying gamiﬁca-

tion to their learning process. In the experiment de-

scribed in (Kyewski and Kramer, 2018) students were

randomly assigned to three different conditions: no

badges, badges visible to peers and badges only vis-

ible to students themselves. Contrarily to expecta-

tions, the last one was evaluated more positively than

the second one. The effects of using gamiﬁcation el-

ements in courses that make use of a wiki environ-

ment on the participation rates as well as on student

academic success is addressed in (Ozdener, 2018).

In (Roy and Zaman, 2018), authors measure the pos-

sible evolution of motivational levels in response to

Effectiveness of Gamiﬁcation in Undergraduate Education

377

the interaction with the game elements used in a uni-

versity course. The ﬁndings illustrate the signiﬁcance

of the individual nature of motivational processes,

the importance of sensitive longitudinal motivation

measurements and the relevance of the implemented

game elements’ design characteristics. The work ad-

dressed in (Tsay et al., 2018) indicates, from a cohort

of undergraduate business students, that course per-

formance was signiﬁcantly higher among those stu-

dents who participated in the proposed gamiﬁed sys-

tem than in those who engaged with the traditional

delivery. The article (Zatarain et al., 2018) describes

an advanced learning environment that detects and re-

sponds to computer programming students’ emotions

by using ML techniques, and incorporates motivation

strategies by using gamiﬁcation methods. In (Bay-

das and Cicek, 2019), authors develop a scale to mea-

sure the factors that may affect the gamiﬁcation pro-

cess via kahoot in a pre-service teachers undergradu-

ate course. The study presented in (Ortiz et al., 2019)

describes the positive effect of gamiﬁcation, based on

leaderboards, on learning performance in an introduc-

tory computer programming course. Finally, in (Toda

et al., 2019) authors propose a solution to help instruc-

tors to plan and deploy gamiﬁcation concepts with

social network features in learning environments. A

case study over a programming course reveals that the

implemented gamiﬁed strategies achieved positive ac-

ceptance among teachers and students.

Related to our prior work in this area, (Sousa et al.,

2017) focused on the quantitative characterization of

non-formal learning methodologies. To this end, we

used one custom software platform for discovering

what factors or variables have statistically signiﬁcant

correlation with the students’ academic achievements.

The dataset was collected along several consecutive

editions of an undergraduate course. Next, in (Sousa

et al., 2018) we compare and combine the power of

different classiﬁers for success/failure learning pre-

diction, using as inputs some of the features discov-

ered in previous works that have measurable correla-

tion with the students’ performance. Finally, in (Fer-

reira et al., 2020b) we focused on the analysis of fo-

rums engagement, modeling forums’ interactions as

social graphs. It was the ﬁrst time that we encourage

and reward quality participation in this activity in the

undergraduate and master level courses under study.

In (Ferreira et al., 2020a) we extended this analysis

and we showed the power of some of the graphs prop-

erties for success/failure learning prediction. In this

work, as the last step of this longitudinal study, we

allow the following-up of the gamiﬁcation method-

ology to be optional and we compare the results ob-

tained with each one of the two learning paths.

3 EDUCATIONAL CONTEXT &

DATASET

We have taken as our educational environment the

2019/2020 edition of a course on Computer Networks

directed to undergraduates of the second year of the

Telecommunications Technologies Engineering de-

gree. This course has a weekly schedule that spans

14 weeks (between January and May). The classroom

activities are organized as follows:

• Lectures, that blend the presentation of concepts,

techniques and algorithms with the practice of

problem-solving skills and discussion of theoreti-

cal questions.

• Laboratory sessions, where the students design

and analyze different network scenarios with dif-

ferent protocols, using real or simulated network-

ing equipment. Moreover, in some of these ses-

sions students make a programming assignment.

In both editions the activities are supported by a tai-

lored Moodle site to which the students and teachers

belong, and wherein general communication about

the topics covered takes place. To encourage net-

worked learning and collaborative work, different ac-

tivities are planned and carried out in the platform.

The students may gain different kinds of recognition

by completing or participating in these activities. In

the editions analyzed in this work, these online activ-

ities were proposed:

1. Homework tasks, to be worked out before the in-

class or the laboratory sessions. With this activity

teachers encourage the students to prepare some

of the material in advance.

2. Quizzes, proposed before the midterm exams for

self-training.

3. Collaborative participation in forums. Several fo-

rums were created in Moodle to allow the students

to post questions, doubts or puzzles related to the

organization of the course, the content of the in-

class lectures or the laboratory sessions and the

programming assignments.

4. Optional activities, such as games, peer assess-

ment of tasks, etc.

The score of tasks (and their peer assessment) and

quizzes is measured in so-called merit points, and rep-

resents the total score gained for accomplishment of

these activities in the modality B of the continuous

assessment (a 10% of the ﬁnal grade). It is possible

to obtain extra merit points by doing the optional ac-

tivities in order to compensate for low scores or late

CSEDU 2021 - 13th International Conference on Computer Supported Education

378

Table 1: Game design elements used in the selected papers.

Badges Boards Levels Points Rewards

(Landers and Landers, 2014) X

(Ib

nez et al., 2014) X X X X

(Strmecki et al., 2015) X X X X

(Kuo and Chuang, 2016) X X X X X

(Buckley and Doyle, 2017) X X X X X

(Cakirogli et al., 2017) X X X

(de Marcos et al., 2017) X X X

(Dias, 2017) X X X

(Sailer et al., 2017) X X X

anchez et al., 2017) X

(Yildirim, 2017) X X X X

(Aldemir et al., 2018) X X X X X

(Davis et al., 2018) X X X X

(Ding et al., 2018) X X X X X

(Huang and Hew, 2018) X X X X X

(Jo et al., 2018) X X

(Jurgelaitis et al., 2018) X X X X X

(Kyewski and Kramer, 2018) X X

(Ozdener, 2018) X X X X X

(Roy and Zaman, 2018) X X

(Tsay et al., 2018) X X

(Zatarain et al., 2018) X X X

(Baydas and Cicek, 2019) X X X X X

(Ortiz et al., 2019) X

(Toda et al., 2019) X X X

submissions of some of the tasks or quizzes. Well-

done peer assessments and the best scores in tasks and

quizzess are rewarded with coins and badges.

Participation in forums, solving doubts or sharing

resources, is also valued with points or votes granted

by the teachers or the classmates. As new points or

votes are obtained, the so-called karma level of each

student increases, depending on different factors that

take into account the quality of the student’s actions

and the comparison with that of his/her classmates.

As the karma level increases, students can get coins.

The use of the virtual classroom is also rewarded

by the automatic scoring of different actions carried

out in the platform related to the normal activity un-

folded along the term, like posting or viewing re-

sources, posting new threads, replying to posts, com-

pleting tasks, etc. The so-called experience points

are organized into levels and are awarded in a con-

trolled environment with maximum values and their

frequency set by the teachers. When students level

up, they get coins.

At any time, a student can check his/her accumu-

lated merit points, karma level and accumulated expe-

rience points and level. Moreover, students can check

their positions in the global rankings and the averages

values of the course. And occasionally, the best stu-

dents of a ranking can be made public to the group.

The coins accumulated at the end of the course

can be converted into the following beneﬁts helpful to

pass the subject (the ﬁnal exam consist of 6 exercises;

each exercise scores 2 points and the maximun score

is 10 points).

- 32 coins can be changed by the extra exercise

wildcard: the 6 exercises of the exam are cor-

rected and their mark is added up to a maximum

of 10 points.

- 22 coins can be changed by the remove worse ex-

ercise wildcard: the 6 exercises of the exam are

corrected and the worse is not taken into account.

- 16 coins can be changed by the remove exer-

cise wildcard: students choose the exercise whose

score is not taken into account.

- 4 or 6 coins can be converted into one or two

pages of notes for the ﬁnal exam.

- 3 coins can be changed by 5 bonus merit points up

to a maximum of 25.

For students who do not have a wildcard, the 6 exer-

cises are corrected and the score of each one is scaled

by 10/6. It is clear that the students that follow the

modality B of the continuous assessment can get more

beneﬁt from the gamiﬁcation strategy.

Students may pass the course after a single ﬁ-

nal examination covering all the material (provided

the programming assignment meets the minimum re-

quirements), but they are encouraged to follow the

continuous assessment. In continuous assessment we

Effectiveness of Gamiﬁcation in Undergraduate Education

379

allow two modalities, A and B. We weigh 40% the

ﬁnal exam, but the rest is split as follows: 36% in

modality A and 24% in modality B from the midterm

exams, 24% from the programming assignment and

12% (only in modality B) coming out from the merit

points obtained by accomplishing the online activi-

ties (task, quizzes and optional tasks) described pre-

viously, devised as a tool to increase the level of par-

ticipation. Students have two opportunities to pass the

exam (non-exclusive), May and July.

To ﬁnish our description, among all students en-

rolled in this course, 121 students did not drop out

(i.e. they attendend the ﬁnal exam). Among these

121 students, of the 114 students who followed the

continuous assessment (18 in modality A and 96 in

modality B), 84 ﬁnally passed the course (13 (72%) in

modality A, almost all second-taking students, and 71

(74%) in modality B). And none of the 7 students not

engaged in continuous assessment was able to pass.

Related to the rate of success in the ﬁnal exam, it was

0% for the single ﬁnal examination modality, 22% for

the continuous assessment modality A and 32% for

the continuous assessment modality B.

At this point, it is important to highlight that the

adaptation of this subject to the lockdown caused by

the pandemic, that affected the second part of the

term, was fast and without incidents for the vast ma-

jority of students, because they were already used to

the platform and the blended learning methodology

since January. In fact, among the students who fol-

lowed the continuous assessment, the dropout rate of

the students in modality A was 1.5 times higher than

that of the students modality B.

4 ANALYSIS OF THE DATASETS

4.1 MERIT POINTS

Figure 1 displays the points obtained by some rep-

resentative students that followed modality B of the

continuous assessment. These points include the

merit points obtained in the tasks and tests proposed,

the extra merit points and the bonus points. In these

ﬁgures we represent the accumulated number of merit

points earned by each student along the term. Stu-

dents are identiﬁed by his/her position in the ordered

list of ﬁnal grades (we represent the ﬁrst 8 students

with the best ﬁnal grades and the 8 students with the

worst ﬁnal grades among the 96 students that fol-

lowed the modality B of the continuous assessment

and did not drop out the course). As we can see, the

pattern of accomplishment of all the students repre-

sented in the ﬁgure on the top is similar, reaching all

100

120

0 20 40 60 80 100 120 140

100

120

0 20 40 60 80 100 120 140

120

119

118

117

116

115

114

113

Figure 1: Accumulated merit points. First positions in the

ordered list of ﬁnal grades (top) and last positions (bottom).

or almost all the possible points. And although the ac-

tivity of the students represented in the ﬁgure on the

bottom is more irregular, almost all of them reached

more than 80% of all the possible points.

Table 2: Individual merit points.

ˆµ

tqmp 58.5253 18.1286

emp 13.8074 6.1131

tmp 75.8786 21.5459

tmps 0.7023 0.2013

Table 2 shows the estimated mean value and stan-

dard deviation of the individual merit points (merit

points form tasks and quizzes(tqmp), extra merit

points (emp) and total merit points (tmp)) and the

slope of total merit points (tmps), that is, the coef-

CSEDU 2021 - 13th International Conference on Computer Supported Education

380

ﬁcient of a linear regression model of the total merit

points per time graph, of all the students that followed

the modality B of the continuous assessment.

Table 3: Correlation between individual merit points and

student’s performance in the ﬁnal exam.

ρ (

β,t, P(> |t|))

tqmp 0.4783 (0.0689,5.3112, 7.21 · 10

−7

)

emp 0.1677 (0.0559,1.6591, 1.01 · 10

−1

)

tmp 0.3654 (0.0483,3.8261, 2.33 · 10

−4

)

tmps 0.3944 (5.2706,4.1831, 6.41 · 10

−5

)

In order to check the relationship among the patterns

of engagement along the term and knowledge acquisi-

tion, we have measured the statistical correlations be-

tween the individual merit points and the performance

in the ﬁnal exam (taking into account the sum of the

scores of the 6 exercises before applying wildcards) of

the students that followed the continuous assessment

and did not drop out the course. For this purpose, the

sample correlations

ρ were computed and the linear

regression statistical test was used to quantify such

correlations. The estimated linear coefﬁcient is de-

noted by

β. Under the null hypothesis (meaning that

there is not such linear dependence) the test statistic

follows a t-distribution and high values are very un-

likely to be observed empirically (James et al., 2013).

In Table 3 we can see statistically signiﬁcant positive

dependences.

4.2 Forums Activity

We have applied standard SNA techniques (Newman,

2018) to mine the data collected in forums. For such

purpose, we have recorded the events that took place

in each forum, users who posted new threads, users

who replied and the average valuations they received.

This information is represented as a graph where two

nodes, the users, are connected by an edge if one has

given a reply to an entry posted by the other. More-

over, self-edges represent new threads. The weight of

each edge is related to the points or votes obtained by

the reply or the new thread post. Orange edges iden-

tify replies marked as useful by the owner of the ques-

tion and green edges identify the best replies based

on the opinion of the owner of the question and/or the

teachers. An illustration of the graphs related to each

forum is given in Figure 2, where every node is a stu-

dent identiﬁed by his/her position in the ordered list

of ﬁnal grades. The node with label 0 corresponds to

the instructors.

Figure 2: Forums activity graphs. Lessons graph (top), pro-

gramming graph (middle) and organization graph (bottom).

4.3 Measures

Next, we report some of the typical measures of a

graph that can be obtained globally or individually for

each node, and their values in our datasets. Notice

that for some measures we consider simpliﬁed ver-

sions of the graphs, where the weight of each edge

is the sum of the weights of all the edges between

the underlying pair of nodes. Moreover, including

self-edges means including the opening of new forum

threads in the analysis.

For the case of degree centrality, we considered

separately the in-degree and out-degree centralities.

In this application, considering the simpliﬁed version

of the graphs, the in-degree centrality is the number

of neighbors whose replies a student receives, and the

out-degree centrality is the number of neighbors that

receive the replies given by a student. The results in

Tables 4-6 reveal that the in-degree centrality values

Effectiveness of Gamiﬁcation in Undergraduate Education

381

Table 4: Summary of parameters of the lessons graph.

degree

in 0.2136

out 0.6244

closeness 0.6026

betweenness 0.6716

eigenvector 0.8454

# cliques

Size

2 110

3 68

4 15

number new threads (µ-σ) 0.4608 0.9485

number new threads (mod. B) (µ-σ) 0.5 0.9765

number replies (µ-σ) 0.9043 2.0475

number replies (mod. B) (µ-σ) 1.0408 2.1821

points new threads (µ-σ) 9.0591 27.5874

points new threads (mod. B) (µ-σ) 14.2755 27.9704

points replies (µ-σ) 20.5652 46.1802

points replies (mod. B) (µ-σ) 24 49.2351

Table 5: Summary of parameters of the programming

graph.

degree

in 0.3298

out 0.7161

closeness 0.6827

betweenness 0.7489

eigenvector 0.8425

# cliques

Size

2 84

3 43

4 4

number new threads (µ-σ) 0.3217 0.7199

number new threads (mod. B) (µ-σ) 0.3673 0.7651

number replies (µ-σ) 0.7304 1.8746

number replies (mod. B) (µ-σ) 0.8163 2.0017

points new threads (µ-σ) 8.5478 19.1532

points new threads (mod. B) (µ-σ) 9.7857 20.3734

points replies (µ-σ) 14.2434 33.9929

points replies (mod. B) (µ-σ) 15.6531 35.9713

are moderate, but the out-degree centrality is notice-

able, indicating a non-homogeneous distribution of

the number of neighbors that receive the replies sub-

mitted by the participants. A subset of few nodes act

as very active participants in forums (among them the

teachers). Nevertheless, more nodes act as generators

of new threads and recipients of information.

For the closeness centrality, that measures how

easily a node can reach other nodes computing the in-

verse of the average length of the shortest paths to all

the other nodes in the graph, the high values shown in

Tables 4-6 are again indicative of the existence of few

very active contributors.

In the case of the betweenness centrality, that tries

to capture the importance of a node in terms of its

role in connecting other nodes, computing the ratio

between the number of shortest paths that a node lies

on and the total number of possible shortest paths

between two nodes, the high values observed in Ta-

bles 4-6 suggest that few nodes act as bridges between

different parts of the graph.

Eigenvector centrality is a measure based on the

premise that a node’s importance is determined by

how important or inﬂuential its neighbors are. The

scores arise from a reciprocal process in which the

centrality of each node is proportional to the sum of

the centralities of the nodes it is connected to. Con-

sidering the version of the graph with self-edges, Ta-

Table 6: Summary of parameters of the organization graph.

degree

in 0.2185

out 0.5832

closeness 0.6071

betweenness 0.6417

eigenvector 0.8721

# cliques

Size

2 141

3 99

4 42

5 12

6 1

number new threads (µ-σ) 0.6086 1.2333

number new threads (mod. B) (µ-σ) 0.6428 1.2701

number replies (µ-σ) 1.2696 2.7859

number replies (mod. B) (µ-σ) 1.3061 2.6879

points new threads (µ-σ) 16.1391 32.3066

points new threads (mod. B) (µ-σ) 16.7959 33.0713

points replies (µ-σ) 27.9304 58.5248

points replies (mod. B) (µ-σ) 28.1428 54.4638

Table 7: Correlation between individual features and stu-

dent’s performance in the ﬁnal exam (lessons graph).

ρ (

β,t, P(> |t|))

in degree 0.2218 (0.2441,2.4192, 1.72 · 10

−2

)

out degree 0.1181 (0.1298,1.2643, 2.09 · 10

−1

)

betweenness 0.1151 (0.0082,1.2332, 2.22 · 10

−1

)

closeness 0.1593 (1.4924,1.7153, 8.91 · 10

−2

)

eigenvector 0.1918 (2.9671, 2.0719,3.99 · 10

−2

)

crossclique number 0.1473 (0.0401, 1.5831,1.16 · 10

−1

)

number new threads 0.1728 (0.3498, 1.8652,6.47 · 10

−2

)

number replies 0.0974 (0.0913,1.0404, 3.01 · 10

−1

)

points new threads 0.1427 (0.0099,1.5332, 1.28 · 10

−1

)

points replies 0.1156 (0.0048, 1.2376,1.77 · 10

−1

)

bles 4-6 show that the measured eigenvector centrality

values are noticeable. Again, this clearly means that

there are substantial differences among the nodes in

their patterns of participation in this activity.

A clique is a completely connected subgraph of a

given graph. So, cliques represent strongly tied sub-

communities where each member interacts with any

other member. And the crossclique number accounts

for the number of cliques a node belongs to. Tables 4-

6 list the number of cliques in the graphs by their size.

Finally, if we consider the non-simpliﬁed version

of the graphs, the in-degree centrality is the number of

replies a student receives, and the out-degree central-

ity is the number of replies given by a student. More-

over, the number of self-edges accounts for the num-

ber of new threads opened by each student. In addi-

tion to the intensity of interactions, another important

factor is their quality that can be measured taking into

account the weights of the edges. The results in Ta-

bles 4-6 show the mean value and the standard devi-

ation of this measures for all the students that did not

drop out the course and for the students that followed

the modality B of the continuous assessment. We can

observe higher values for these, mainly in the lessons

forum.

CSEDU 2021 - 13th International Conference on Computer Supported Education

382

Table 8: Correlation between individual features and stu-

dent’s performance in the course (lessons graph).

ρ (

β,t, P(> |t|))

in degree 0.1654 (0.1951,1.7832, 7.72 · 10

−2

)

out degree 0.1244 (0.1466,1.3331, 1.85 · 10

−1

)

betweenness 0.1526 (0.0116,1.6424, 1.03 · 10

−1

)

closeness 0.1072 (1.0054,1.3943, 1.66 · 10

−1

)

eigenvector 0.1201 (1.9893, 1.2851,2.01 · 10

−1

)

crossclique number 0.1249 (0.0363, 1.3381,1.83 · 10

−1

)

number new threads 0.1286 (0.2791, 1.3791,6.47 · 10

−2

)

number replies 0.1064 (0.1071,1.1385, 2.57 · 10

−1

)

points new threads 0.1279 (0.0095,1.3711, 1.73 · 10

−1

)

points replies 0.1268 (0.0056, 1.3591,1.77 · 10

−1

)

Table 9: Correlation between individual features and stu-

dent’s performance in the ﬁnal exam (programming graph).

ρ (

β,t, P(> |t|))

in degree 0.0461 (0.0575,0.4908, 6.25 · 10

−1

)

out degree 0.0233 (0.1101,0.2481, 8.05 · 10

−1

)

betweenness −0.0005 (−0.0004,−0.0001, 9.99 · 10

−1

closeness 0.1162 (1.0895, 1.2442,2.16 · 10

−1

)

eigenvector 0.0567 (0.9791,0.6041, 0.54 · 10

−1

)

crossclique number 0.0598 (0.0242,0.6381, 5.25 · 10

−1

)

number new threads 0.1042 (0.2781,1.1142, 2.68 · 10

−1

)

number replies 0.0408 (0.0417,0.4342,6.65 · 10

−1

)

points new threads 0.0644 (0.0064, 0.6861,4.94 · 10

−1

)

points replies 0.0055 (0.0003,0.0591, 9.53 · 10

−1

)

Table 10: Correlation between individual features and stu-

dent’s performance in the course (programming graph).

ρ (

β,t, P(> |t|))

in degree 0.0921 (0.1232,0.9832, 3.27 · 10

−1

)

out degree 0.0785 (0.1101,0.8381, 4.04 · 10

−1

)

betweenness 0.0361 (0.0037,0.3857, 7.01 · 10

−1

)

closeness 0.1012 (1.5546,1.6656, 9.88 · 10

−2

)

eigenvector 0.1098 (2.0321, 1.1742,0.24 · 10

−1

)

crossclique number 0.1835 (0.0435, 1.0751,2.85 · 10

−1

)

number new threads 0.1555 (0.4446, 1.6743,9.69 · 10

−2

)

number replies 0.0948 (0.1041,1.0132, 3.13 · 10

−1

)

points new threads 0.1211 (0.0231,1.2976, 1.97 · 10

−1

)

points replies 0.0794 (0.0048, 0.8472,3.99 · 10

−1

)

4.4 Correlations with Final Results

In order to check the relationship among the patterns

of participation in the forums and the achievements of

the course, we have measured the statistical correla-

tions between the features under study in this section

and the ﬁnal grades in the ﬁnal exam (taking into ac-

count the sum of the scores of the 6 exercises before

applying wildcards) and in the course of the students

that followed the continuous assessment and did not

drop out the course.

The results in Tables 7-12 show a statistically sig-

niﬁcant positive dependence between many of the

considered factors and the students’ performance in

the lessons graph.

Table 11: Correlation between individual features and stu-

dent’s performance in the ﬁnal exam (organization graph).

ρ (

β,t, P(> |t|))

in degree −0.0203 (−0.0156,−0.2176, 8.29 · 10

−1

)

out degree −0.0279 (−0.0252,−0.2971,7.67 · 10

−1

)

betweenness −0.0044 (−0.0002,−0.0487, 9.62 · 10

−1

)

closeness 0.0362 (0.3258,0.3861, 7.01 · 10

−1

)

eigenvector −0.0139 (−0.2144, −1.4801,8.83 · 10

−1

)

crossclique number −0.0308 (−0.0039, −0.3281,7.43 · 10

−1

)

number new threads −0.0347 (−0.0541, −0.3691,7.13 · 10

−1

)

number replies −0.0795 (−0.0547,−0.8487, 3.98 · 10

−1

)

points new threads 0.0017 (0.0001,0.0183,9.85 · 10

−1

)

points replies −0.0686 (−0.0022, −0.7324,7.95 · 10

−1

)

Table 12: Correlation between individual features and stu-

dent’s performance in the course (organization graph).

ρ (

β,t, P(> |t|))

in degree −0.0065 (−0.0053,−0.0692, 9.45 · 10

−1

)

out degree 0.0095 (0.0092, 0.1021,9.19 · 10

−1

)

betweenness 0.0301 (0.0014, 0.3221,7.56 · 10

−1

)

closeness 0.0432 (0.8162,0.9044, 3.68 · 10

−1

)

eigenvector −0.0068 (−0.1128, −0.0732,9.42 · 10

−1

)

crossclique number −0.0225 (−0.0031, −0.3281,8.11 · 10

−1

)

number new threads −0.0129 (−0.0216, −0.1381,7.13 · 10

−1

)

number replies −0.0519 (−0.0383,−0.5522, 5.82 · 10

−1

)

points new threads 0.0175 (0.0011,0.1871,8.52 · 10

−1

)

points replies −0.0245 (−0.0008, −0.2611,7.95 · 10

−1

)

5 CONCLUSIONS

We presented in this paper a broad correlation study

between individual features pertaining to the structu-

tal properties of the network relations graph formed in

an online social environment and ﬁnal performance of

students, both for individuals following the GL-based

style and those who refuse this option. Consistently,

our results show that positive correlations are present

always between individuals in the GL group and bet-

ter academic achievement, almost irrespective of the

particular network feature studied. Thus, these stu-

dents attain better grades and better success probabil-

ity in the ﬁnal exams, are much less prone to quit the

course, and provide more frequent contributions and

information to the forums, usually of high quality and

better cr itical thinking.

A possible limitation of our study is that the split

between the GL group and the non-GL group is not

totally random nor blind, which might introduce some

bias in our results. Nevertheless, we conjecture that,

since for most students this is their ﬁrst exposure to

GL, their prior attitude toward gamiﬁcation is incon-

sequential and little relevant as to the ﬁnal outcomes.

Therefore, our study contributes to the identiﬁcation

of the measurable variables that correlate positively

and signiﬁcantly with the students performance.

Effectiveness of Gamiﬁcation in Undergraduate Education

383

REFERENCES

Aldemir, T., Celik, B., and Kaplan, G. (2018). A qualitative

investigation of student perceptions of game elements

in a gamiﬁed course. Computers in Human Behavior,

78:235–254.

Baydas, O. and Cicek, M. (2019). The examination of the

gamiﬁcation process in undergraduate education: A

scale development study. Technology, Pedagogy and

Education, 18(3):269–285.

Buckley, P. and Doyle, E. (2017). Individualising gamiﬁca-

tion: An investigation of the impact of learning styles

and personality traits on the efﬁcacy of gamiﬁcation

using a prediction market. Computers & Education,

69:43–55.

Cakirogli, U., Basibuyuk, B., Guter, M., and Atabay, M.

(2017). Gamifying an ICT course: Inﬂuences on en-

gagement and academic performance. Computers in

Human Behavior, 69:98–107.

Davis, K., Sridharan, H., Koepke, L., Singh, S., and Boiko,

R. (2018). Learning and engagement in a gamiﬁed

course: Investigating the effects of student charac-

teristics. Journal of Computer Assisted Learning,

34(5):492–503.

de Marcos, L., Garc

ıa, A., and Garc

ıa, E. (2017). Towards

the social gamiﬁcation of e-learning: A practical ex-

periment. International Journal of Engineering Edu-

cation, 33(1):66–73.

Dias, J. (2017). Teaching operations research to undergrad-

uate management students: The role of gamiﬁcation.

The International Journal of Management Education,

15:98–111.

Ding, L., Er, E., and Orey, M. (2018). An exploratory study

of student engagement in gamiﬁed online discussions.

Computers & Education, 120:213–226.

Ferreira, O., Sousa, M., L

opez, J., and Fern

andez, M.

(2020a). An assessment of statistically classiﬁca-

tion for socially oriented learning methodologies. In

CSEDU’20, 12

International Conference on Com-

puter Supported Education.

Ferreira, O., Sousa, M., L

opez, J., and Fern

andez, M.

(2020b). Studying relationships between network

structure in educational forums and students’ perfor-

mance. In Communications in Computer and In-

formation Science. Computer Supported Education.

Springer.

Huang, B. and Hew, K. (2018). Implementing a theory-

driven gamiﬁcation model in higher education ﬂipped

courses: Effects on out-of-class activity completion

and quality of artifacts. Computers & Education,

125:254–272.

nez, M., Di-Serio, A., and Delgado, C. (2014). Gam-

iﬁcation for engaging computer science students in

learning activities: A case study. IEEE Transactions

on Learning Technologies, 7(3):291–301.

James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013).

An introduction to statistical learning with applica-

tions in R. Springer.

Jo, J., Jun, H., and Lim, H. (2018). A comparative

study on gamiﬁcation of the ﬂipped classroom in en-

gineering education to enhance the effects of learn-

ing. Computer Application on Engineering Education,

26(5):1626–1640.

Jurgelaitis, M., Ceponien

e, L., Ceponis, J., and Drungilas,

V. (2018). Implementing gamiﬁcation in a university-

level uml modeling course: A case study. Computer

Applications in Engineering Education, 27(2):332–

343.

Kapp, F. (2012). The gamiﬁcation of learning and instruc-

tion: Game-based methods and strategies for training

and education. Pfeiffer.

Kuo, M. and Chuang, T. (2016). How gamiﬁcation moti-

vates visits and engagement for online academic dis-

semination: An empirical study. Computers in Human

Behavior, 55:16–27.

Kyewski, E. and Kramer, N. (2018). To gamify or not to

gamify? an experimental ﬁeld study of the inﬂuence

of badges on motivation, activity and performance in

an online learning course. Computers & Education,

118:25–37.

Landers, R. and Landers, A. (2014). An empirical test of

the theory of gamiﬁed learning: The effect of leader-

boards on time-on-task and academic performance.

Simulation & Gaming, 45(6):769–785.

Newman, M. (2018). Networks. Oxford University Press.

Ortiz, M., Chiluiza, K., and Valcke, M. (2019). Gamiﬁ-

cation through leaderboards: An empirical study in

engineering education. Computer Applications in En-

gineering Education, 27(4):777–788.

Ozdener, N. (2018). Gamiﬁcation for enhancing web 2.0

based educational activities: The case of pre-service

grade school teachers using educational wiki pages.

Telematics and Informatics, 35(3):564–578.

Roy, R. and Zaman, B. (2018). Need-supporting gamiﬁ-

cation in education: An assessment of motivational

effects over time. Computers & Education, 127:283–

297.

Sailer, M., Ulrich, J., Mayr, S., and Mandl, H. (2017). How

gamiﬁcation motivates: An experimental study of the

effects of speciﬁc game design elements on psycho-

logical need satisfaction. Computers in Human Be-

havior, 69:371–380.

anchez, J., Ca

nada, F., and D

avila, M. (2017). Just a

game? gamifying a general science class at univer-

sity. collaborative and competitive work implications.

Thinking Skills and Creativity, 26:51–59.

Sousa, M., L

opez, J., Fern

andez, M., Ferreira, O.,

Rodr

ıguez, M., and Rodr

ıguez, R. (2018). Predic-

tion of learning success/failure via pace of events in

a social learning network platform. Computer Appli-

cations in Engineering Education, 26(6):2047–2057.

Sousa, M., L

opez, J., Fern

andez, M., Rodr

ıguez, M., and

opez, C. (2017). Mining relations in learning-

oriented social networks. Computer Applications in

Engineering Education, 25(5):769–784.

Strmecki, D., Bernik, A., and Radosevic, D. (2015). Gami-

ﬁcation in e-learning: Introducing gamiﬁed design el-

ements into e-learning systems. Journal of Computer

Science, 11(12):1108–1117.

Subhash, S. and Cudney, E. (2018). Gamiﬁed learning in

higher education: A systematic review of the litera-

ture. Computers in Human Behavior, 87:192–206.

CSEDU 2021 - 13th International Conference on Computer Supported Education

384

Toda, A., Carmo, R., Silva, A., Bittencourt, I., and Isotani,

S. (2019). An approach for planning and deploying

gamiﬁcation concepts with social networks within ed-

ucational contexts. International Journal of Informa-

tion Management, 46:294–303.

Tsay, C., Koﬁnas, A., and Luo, J. (2018). Enhancing student

learning experience with technology-mediated gamiﬁ-

cation: An empirical study. Computers & Education,

121:1–17.

Yildirim, I. (2017). The effects of gamiﬁcation-based teach-

ing practices on student achievement and students’ at-

titudes toward lessons. Internet and Higher Educa-

tion, 32:86–92.

Zainuddin, Z., Wah-Chu, S., Shujahat, M., and Perera, C.

(2020). The impact of gamiﬁcation on learning and in-

struction: A systematic review of empirical evidence.

Educational Research Review, 30.

Zatarain, R., Barr

on, M., R

ıos, J., and Alor, G. (2018). A

virtual environment for learning computer coding us-

ing gamiﬁcation and emotion recognition. Interactive

Learning Environments.

Effectiveness of Gamiﬁcation in Undergraduate Education

385