Gamebrics: Integrating Analytical Rubrics into Serious Games
to Teach Analytical Skills
Hugo Huurdeman
1 a
, Hans Hummel
1 b
, Rob Nadolski
1 c
, Giel van Lankveld
1 d
,
Konstantinos Georgiadis
1 e
, Johan van den Boomen
2
, Hub Kurvers
4
, Petra Neessen
2 f
,
Ron Pat-El
3 g
and Aad Slootmaker
4 h
1
Faculty of Educational Sciences, Open Universiteit, Valkenburgerweg 177, Heerlen, The Netherlands
2
Faculty of Management, Open Universiteit, Valkenburgerweg 177, Heerlen, The Netherlands
3
Faculty of Psychology, Open Universiteit, Valkenburgerweg 177, Heerlen, The Netherlands
4
Expertisecentrum Onderwijs (ECO), Open Universiteit, Valkenburgerweg 177, Heerlen, The Netherlands
Keywords:
Serious Games, Analytical Rubrics, Formative Assessment, Monitoring Dashboard, Complex Skills,
Workplace-Based Online Learning.
Abstract:
Complex skills, such as analytical thinking, are essential in the rapidly changing society of the 21st century.
An ongoing question is how to teach these complex skills in higher education. Serious games hold potential
to stimulate the acquisition of analytical skills. Rubrics are proven feedback and evaluation instruments, but
have never been directly integrated into the gameplay of serious games. This position paper discusses how
the novel integration of analytical rubrics into serious games may fill this gap. We discuss our approach
for this integration and illustrate it using an implementation of analytical rubrics into an existing scenario-
based serious game. The discussed approach involves creating theory-informed rubrics, performing validated
mappings of rubric elements to game activities and the formulation of appropriate reflective feedback. In
addition, we outline the design and implementation of a player-facing dashboard to allow players to track
their progress, in-game performance in terms of analytical skills and to receive reflective feedback. Finally,
we provide a brief outlook to an ongoing evaluation study examining the effectiveness of the integration of
rubrics into serious games.
1 INTRODUCTION
The acquisition of complex skills via active learn-
ing, especially in online settings, is a challenge for
higher education as evidenced during the COVID
pandemic. A way to stimulate learning these skills
is via well-designed serious games. These kinds of
games for ‘serious’ purposes (Michael & Chen, 2006)
provide the opportunity to offer students activating
tasks in authentic settings. The use of serious games
may have several advantages: increasing the motiva-
a
https://orcid.org/0000-0002-3027-9597
b
https://orcid.org/0000-0002-3579-749X
c
https://orcid.org/0000-0002-6585-0888
d
https://orcid.org/0000-0001-8319-2244
e
https://orcid.org/0000-0003-2277-5256
f
https://orcid.org/0000-0002-2712-5116
g
https://orcid.org/0000-0002-4742-0163
h
https://orcid.org/0000-0002-4882-0679
tion of students (Connolly et al., 2012), but also low-
ering the workload of the involved educational staff.
This position paper focuses on the use of serious
games to teach vocational (professional) competences
in context. Analytical skills are an important pre-
requisite for such professional learning. To this end,
rubrics are directly integrated into the gameplay of se-
rious games. Rubrics are proven feedback and eval-
uation instruments, which link criteria to proficiency
levels (Van den Bos et al., 2017; Arter & Chappuis,
2010). In this paper, we focus on analytical rubrics,
containing not just scores but also an analysis of a
certain proficiency level, and on rubrics related to an-
alytical skills. While rubrics focusing on complex
skills exist, and can be used to assess the acquisition
of these skills, they have never been integrated into
the actual gameplay of serious games. The Game-
brics project
1
aims to fill this gap by looking at for-
1
https://www.gamebrics.nl
Huurdeman, H., Hummel, H., Nadolski, R., van Lankveld, G., Georgiadis, K., van den Boomen, J., Kurvers, H., Neessen, P., Pat-El, R. and Slootmaker, A.
Gamebrics: Integrating Analytical Rubrics into Serious Games to Teach Analytical Skills.
DOI: 10.5220/0011974900003470
In Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023) - Volume 2, pages 403-409
ISBN: 978-989-758-641-5; ISSN: 2184-5026
Copyright
c
2023 by SCITEPRESS Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)
403
mative assessment of complex skills via the integra-
tion of analytical rubrics into the gameplay of three
serious games.
Achieving this goal involves theory-informed de-
sign, development and evaluation of the integration
of rubrics into existing serious games. For this, ana-
lytical rubrics have to be created, as well as validated
mappings of game activities to these rubrics via a con-
tent validation method (Hummel et al., 2017). Fur-
thermore, guidelines for in-game feedback have to be
formulated. To allow players to track their progress
and to receive feedback, the design of an in-game
dashboard is required.
This paper first describes the backgrounds and
previous literature on serious games, rubrics, think-
ing skills and feedback (Section 2). This is followed
by an outline of the approach of the project, and the
utilized serious games (Section 3). Next, we discuss
the initial results of our work (Section 4), followed by
the conclusions and future work (Section 5). We will
provide an outlook to ongoing work: an evaluation
study with students in which the effectiveness of the
integration of analytical rubrics is being examined.
2 BACKGROUND
This section introduces the concepts related to serious
games, formative assessment, analytical rubrics, ana-
lytical skills and reflective feedback, which are used
in the design and development of the Gamebrics tools.
2.1 Serious Games and Formative
Assessment
Serious games have been defined as games for which
the prime purpose is not just entertainment, but the
achievement of serious goals, ranging from learning
to behavioral change (Michael & Chen, 2006; Gee,
2007). Previous meta-analyses (e.g. Connolly et al.
(2012); Wouters et al. (2013)) have shown that these
games can contribute to knowledge acquisition and
content understanding. However, these analyses have
also shown that complex skills such as problem solv-
ing and analytical thinking may be developed using
serious games.
One important element of serious games is the as-
sessment of learning by the players. This can take the
shape of formative assessment, but potentially also of
summative assessment resulting in a grade. An open
issue of assessment in serious games how to carefully
integrate this into the gameplay, to avoid interrupting
the state of ‘flow’ a player may be in (Csikszentmiha-
lyi, 2008; Shute et al., 2009). At the same time, the
assessment should have a high reliability and valid-
ity. Hummel et al. (2017) have developed a way to
integrate formative assessment in serious games in a
way assuring content validity, involving the mapping
of game activities on performance indicators and pro-
ficiency levels.
Serious games cover a wide diversity of game gen-
res, for instance simulations, strategy games or adven-
ture games (Connolly et al., 2012). The focal point of
this paper is on scenario-based games, which can be
created using platforms such as uAdventure (P
´
erez-
Colado et al., 2019) and EMERGO (Nadolski et al.,
2008). EMERGO has been described as “a method-
ology and generic toolkit for developing and deliver-
ing scenario-based serious games”. Using EMERGO,
more than 70 serious games have been developed. Of-
ten, these games have focused on complex (thinking)
skills, such as analyzing information & problem solv-
ing in professional workplace learning settings.
2.2 Analytical Rubrics and Skills
A rubric is a feedback and evaluation instrument.
It is usually structured as a matrix of two dimen-
sions (Van den Bos et al., 2017). The first dimen-
sion, presented on the vertical axis, contains crite-
ria for evaluation. The second dimension, presented
on the horizontal axis, contains proficiency levels.
Two types of rubrics are commonly distinguished:
holistic rubrics, in which complex skills are assessed
as a whole and analytical rubrics, in which indi-
vidual, singular criteria are specified. Van den Bos
et al. (2017) make a further distinction between task-
specific rubrics (which can be only applied to one con-
text) and generic, domain-independent rubrics, which
can be utilized in a broader context, but at a less spe-
cific level of detail.
As Arter & Chappuis (2010) indicate, rubrics can
be used for assessment in various settings. They can
support assessment goals at the level of understand-
ing and application of knowledge, performance skills
(e.g. presenting), creation of products (e.g. reports)
and higher-order cognitive skills such as analytical
skills. The latter entail sub-skills like analysis, syn-
thesis, critical thinking and problem solving.
The New Taxonomy of Educational Objectives by
Marzano & Kendall (2007, 2008) also defines vari-
ous ‘operations’ in relation to critical thinking skills,
which include Matching (“identify important similar-
ities and differences”), Classifying (“identify superor-
dinate and subordinate categories”), Analyzing Errors
(“identify errors”), Generalizing (“construct new gen-
eralizations or principles”) and Specifying (“identify
(...) logical consequences”).
CSEDU 2023 - 15th International Conference on Computer Supported Education
404
2.3 Reflective Feedback
Hattie & Timperley (2007) have defined feedback as
all the information an educator provides about the
progress of learners, with the goal to increase learn-
ing and achievement. Effective feedback decreases
the gap between the current and the desired knowl-
edge and understanding of a learner. Further, Hattie
& Timperley (2007) distinguish three types of feed-
back: feed-up (“where am I going?”), feedback (“how
am I doing?”) and feed-forward (“what should I do
next?”).
In addition, it is possible to look at the ways
to effectively give feedback in interactive tutoring
systems. Narciss & Huth (2004) created a concep-
tual framework for the design of this kind of feed-
back. They distinguish between the feedback content
(evaluative and informative components), presenta-
tion (e.g. timing) and function (cognitive, metacog-
nitive and motivational). Reflective feedback pro-
vides information on both the progress achieved and
on ways forward to further improve the mastery of
skills.
3 GAMEBRICS APPROACH
Within our Gamebrics approach, formative assess-
ment via analytical rubrics is designed, developed
and evaluated, based on previously validated meth-
ods. We implemented this tooling in three serious
games which are in different stages of development,
but will focus our illustrations on one of them.
3.1 Development Steps
The project consists of four main phases. In the first
phase, a domain-independent, analytical rubric for
analytical skills is developed. This rubric is mapped
to game activities of two existing serious games, via
a validated method for content validation (Hummel
et al., 2017). The second phase involves the cre-
ation and integration of existing and enhanced tool-
ing into the EMERGO framework (Nadolski et al.,
2008). This includes a student dashboard and an au-
thoring environment for teachers. Within the third
phase, an experimental comparison between game-
play with and without rubrics is conducted. Finally,
the fourth phase involves the reporting of findings, the
documentation of created tools and the depositing of
underlying source code.
This position paper focuses on the first and second
phase of the Gamebrics project, and describes find-
ings and its achievements thus far.
3.2 Exemplary Game Implementation
The support for formative evaluation via rubrics is in-
tegrated into three serious games at different stages of
development: Kastanjehoeve (already in exploitation
over a year, domain management science), Junior Sci-
entist (launched in Feb. 2023, domain psychological
research) and EduMythbusters (under development,
domain educational science). However, we limit our
illustration of the implementation to the first game.
The Kastanjehoeve game (Figure 1) constitutes an
introduction to the field of business administration
(within the field of management sciences). Within the
game, students perform a virtual internship in an el-
derly home (named Kastanjehoeve) and have to solve
authentic management problems. The game already
forms an integral part of Management & Organiza-
tion, a first-year Bachelor course in Management Sci-
ences at the Open Universiteit (OUNL).
These games already support online active learn-
ing in higher education, but are expanded to include
rubrics on analytical thinking.
Figure 1: Screenshot Kastanjehoeve game: question from
supervisor. Note: the original game is in the Dutch lan-
guage. The screenshots in this paper have been translated
to English.
4 INITIAL RESULTS
This section describes the initial results of the first two
phases of the Gamebrics project. We focus on the cre-
ated rubrics, the mapping of game activities, guide-
lines for in-game feedback, the design of a dashboard
and the implementation so far.
4.1 Creation of Rubrics
Essential to the integration of formative assessment
of complex skills into the gameplay of the selected
serious games is the creation of the rubrics them-
selves. The aim was that these would cover various
aspects of analytical thinking and would be domain-
independent.
Gamebrics: Integrating Analytical Rubrics into Serious Games to Teach Analytical Skills
405
Table 1: Analytical thinking skills defined within the Gamebrics project, and corresponding skills in previous literature.
# Sub-skill Marzano et al. (1993) Marzano & Kendall (2008)
1 Comparing & selecting Comparison Matching
2 Identifying errors in reliability Error Analysis Analyzing Errors
3 Inducing Induction Generalizing
4 Deducing Deduction Specifying
5 Decomposing information Classification Classifying
6 Structuring information Classification Classifying
7 Making decisions Constructing support Decision making
8 Analyzing perspectives Analyzing perspectives Investigating
Based on an analysis of previous EMERGO
games as well as previous literature (Marzano et al.,
1993; Marzano & Kendall, 2007, 2008), eight sub-
skills of analytical thinking were defined: Comparing
and selecting, Identifying errors, Inducing, Deducing,
Decomposing, Structuring, Making decisions and An-
alyzing perspectives. These sub-skills are further de-
fined and related to previous literature in Table 1. For
each defined skill, a rubric was created. The con-
tents of these rubrics were adapted from operational-
izations of skills by in terms of learning objectives by
Marzano et al. (1993) and Marzano & Kendall (2008).
The final rubrics contained four proficiency levels, ex-
pressed in stars. These levels were inspired by the first
four levels of Krathwohl (2002)’s revision of Bloom
et al. (1956)’s taxonomy, ranging from one star (“rec-
ognize”), to two stars (“understand”), three stars (“an-
alyze”) and four stars (“apply”).
4.2 Mapping of Game Activities to
Rubric
While the created rubrics provided a detailed
overview of different sub-skills of analytical thinking,
there was no direct connection to the game activities
in the serious games used in our project. Therefore,
we created a mapping of the different rubrics and their
proficiency levels to the existing game activities.
The method for this mapping was inspired by a
previously validated method for content validation by
Hummel et al. (2017). Specifically, this method in-
volves the mapping of learning activities (from the
scenario of the serious game) to performance indi-
cators (from the rubrics for each analytical thinking
subskill created in the previous step). This process
was done by the subject matter experts: the educators
involved in creating the original serious games and
using these in their courses.
A challenge occurring in this process was that
Kastanjehoeve, the existing serious game used in the
mapping, contained important elements of analytical
thinking, but was not originally designed with a spe-
cific focus on the eight analytical sub-skills defined
in the previous step. Therefore, each separate game
Table 2: Number of measurement points of analytical think-
ing skills after mapping to game activities Kastanjehoeve.
# Sub-skill Measurement points
1 Comparing/selecting 3
2 Identifying errors 1
3 Inducing 1
4 Deducing 6
5 Decomposing 1
6 Structuring 4
7 Making decisions 6
8 Analyzing persp. 10
Total 32
activity (denoted as “micro-challenge” within Game-
brics), had to be classified in terms of the analytical
thinking skill it captured, the proficiency level which
applied to it, the weight factor for the total score in
the game, and the observables which could be used to
observe the performance of a player. For this process,
a structured mapping table was constructed, by which
the subject matter experts could systematically note
down these elements.
Table 2 summarizes the results of this process for
the Kastanjehoeve serious game. In total 32 mea-
surement points for the eight analytical skills were
found. For some sub-skills, relatively few measure-
ment points were identified (e.g. identifying errors),
while for other sub-skills, many points were identi-
fied (e.g. analyzing perspectives). The observables
ranged from relatively simple measurements, for ex-
ample whether a game activity has been completed,
to more complex ones, for instance how many man-
agement techniques a player could identify in video
fragments and how many attempts she needed.
4.3 Guidelines for In-Game Feedback
An important element of in-game support for stu-
dents to potentially improve their analytical thinking
skills is the feedback they receive during the game-
play. Within Gamebrics, we distinguish two types of
feedback. First of all, there is natural feedback, which
is part of the game scenario itself. For instance, the
player’s supervisor in the Kastanjehoeve game may
indicate why a provided answer to a multiple-choice
CSEDU 2023 - 15th International Conference on Computer Supported Education
406
question is wrong. Here, we take this type of feed-
back “as is”. A second type of feedback is generic
reflection feedback. This type of reflective feedback,
in our case pertaining to the analytical skills, is added
after playing parts of the game, but should not disturb
the game experience itself (in other words: should be
unobtrusive).
To formulate the feedback and to decide when to
display it, we took into account two of Narciss & Huth
(2004)’s factors contributing to the informative value
of feedback: content and presentation.
In terms of evaluative and informative content, in-
game feedback was formulated based on Narciss &
Huth (2004)’s guidelines and an inventarisation of the
necessary conditions for game-independent feedback.
The feedback consists of a generic part, the base, for-
mulated on the basis of the proficiency level, and a
content remediating part, which is also coupled to the
proficiency level.
2
The games within Gamebrics vary
considerably, as they originate from Management Sci-
ences, Psychology and Educational Sciences. There-
fore, similar to the rubrics, the feedback is formulated
in a domain-independent way, so they can be applied
to all games in question. A challenge in this process
is that a balance has to be struck to avoid that the
feedback is either too broad or too narrow. There-
fore, the base of the feedback only gives an indication
of the performance (the score compared to the maxi-
mum score). In the remediating part of the feedback, a
textual description of the desired behavior is included
to improve the mastery of the analytical thinking sub-
skill.
A challenge with regards to the presentation of the
feedback is the timing. It should be shown at mo-
ments which should not interrupt the ‘flow’ of the
gameplay. Therefore, we implemented the display
upon completing parts of the game. For instance,
in the Kastanjehoeve game, the dashboard containing
feedback is shown for the first time when completing
the introductory part of the game.
4.4 Dashboard Design
Via the Gamebrics dashboard, the player can in-
spect her progress in terms of the gathered points in
the game, the performance with regards to the ana-
lytical skills and receive reflective feedback. Dur-
ing the design process, design elements and learned
lessons were incorporated from the Pe(e)rfectly
Skilled project
3
, such as the skills wheel, a circular
2
Since our eight rubrics each contained four proficiency
levels, we had to create 24 feedback texts for each serious
game.
3
https://www.surf.nl/peerfect-vaardig
Figure 2: Gamebrics Dashboard: Progress screen (a), Kas-
tanjehoeve game.
Figure 3: Gamebrics Dashboard: Skills Wheel (b), Kastan-
jehoeve game.
Figure 4: Gamebrics Dashboard: Advice screen (c), Kas-
tanjehoeve game.
visual representation of a player’s performance. A
design challenge during the creation of the dashboard
was to make it generic enough to be usable in existing
serious games.
The dashboard shows a player’s progress (a), the
proficiency level for analytical skills via the ‘skills
wheel’ (b), and feedback for improving the analytical
skills (c).
Figure 2 depicts the progress screen within the
dashboard (a). It includes the scores and maximum
scores on the eight sub-skills, based on the currently
completed challenges. Via two buttons, a player can
switch to a screen with the skills wheel, or the advice
screen. Figure 3 contains the skills wheel (b). It dis-
plays via different colors and tooltips whether a player
has improved, stayed at the same level, or has wors-
Gamebrics: Integrating Analytical Rubrics into Serious Games to Teach Analytical Skills
407
ened on each sub-skill in the last part of the game. Via
additional functionality it is possible to compare dif-
ferent moments in time in the game. Figure 4 shows
the feedback screen (c). All eight sub-skills are dis-
played, with for each skill an advice how the player
could improve herself. This advice is structured based
on the description in Section 4.3. Finally, there is
an indication which game challenges have been com-
pleted and which will still follow.
5 CONCLUSIONS & FUTURE
WORK
With the continuous advent of online learning, seri-
ous games hold potential to enhance the acquisition
of complex skills, such as analytical thinking skills.
Rubrics are proven feedback and evaluation instru-
ments, but these have never been directly integrated
into the gameplay of serious games. This position
paper has shown how the novel combination of an-
alytical rubrics into the gameplay of serious games
(hence ‘gamebrics’) may fill this gap. We have high-
lighted our approach, involving the creation of theory-
informed rubrics, the validated mapping of rubric el-
ements to game activities and the formulation of re-
flective feedback based on existing guidelines. More-
over, we have outlined the player-facing implemented
dashboards in Gamebrics, where players can track
their progress without interrupting the ‘flow’ of the
game.
At the moment of writing this paper, the effec-
tiveness of integrating rubrics is being examined in
a large-scale evaluation with students, for the Kastan-
jehoeve and Junior Scientist serious games. In both
evaluations, an experimental group is given the game
without formative evaluation through the rubric, and
another group with the formative evaluation of the
rubric. This will be combined with validated ques-
tionnaires. The analysis of the experimental data will
shed more light on the effectiveness of directly inte-
grating rubrics within the gameplay of serious games
for formative assessment.
ACKNOWLEDGEMENTS
The authors wish to thank Mick Hummel, who de-
signed the GUI of the Gamebrics tooling. This project
was financed by a SURF grant, within the call “Open
& Online Higher Education 2021”, project # 2021-01.
REFERENCES
Arter, J. A. & Chappuis, J. (2010). Creating & Recognizing
Quality Rubrics. Boston: Prentice Hall.
Bloom, B., Engelhart, M., Furst, E., Hill, W., & Krathwohl,
D. (1956). Taxonomy of educational objectives. 1:
Cognitive domain. London: Longmans, Green and
Co Ltd.
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T.,
& Boyle, J. M. (2012). A systematic literature review
of empirical evidence on computer games and serious
games. Computers & Education, 59(2), 661–686.
Csikszentmihalyi, M. (2008). Flow: The Psychology of Op-
timal Experience. New York: Harper Perennial Mod-
ern Classics, 1st edition.
Gee, J. P. (2007). What Video Games Have to Teach Us
About Learning and Literacy. New York: Palgrave
Macmillan.
Hattie, J. & Timperley, H. (2007). The Power of Feed-
back. Review of Educational Research, 77(1), 81–112.
American Educational Research Association.
Hummel, H. G. K., Joosten-ten Brinke, D., Nadolski, R. J.,
& Baartman, L. K. J. (2017). Content validity of
game-based assessment: case study of a serious game
for ICT managers in training. Technology, Pedagogy
and Education, 26(2), 225–240. Routledge.
Krathwohl, D. R. (2002). A Revision of Bloom’s Taxon-
omy: An Overview. Theory Into Practice, 41(4), 212.
Taylor & Francis Ltd.
Marzano, R. J. & Kendall, J. S. (2007). The New Taxonomy
of Educational Objectives. Thousand Oaks: Corwin
Press, second edition.
Marzano, R. J. & Kendall, J. S. (2008). Designing & as-
sessing educational objectives: applying the new tax-
onomy. Thousand Oaks: Corwin Press.
Marzano, R. J., Pickering, D., & McTighe, J. (1993). As-
sessing Student Outcomes: Performance Assessment
Using the Dimensions of Learning Model. Associa-
tion for Supervision and Curriculum Development.
Michael, D. & Chen, S. (2006). Serious games: games that
educate, train and inform. Boston, Mass: Thomson
Course Technology.
Nadolski, R. J., Hummel, H. G. K., van den Brink, H. J.,
Hoefakker, R. E., Slootmaker, A., Kurvers, H. J., &
Storm, J. (2008). EMERGO: A methodology and
toolkit for developing serious games in higher educa-
tion. Simulation & Gaming, 39(3), 338–352. SAGE
Publications Inc.
Narciss, S. & Huth, K. (2004). How to design informa-
tive tutoring feedback for multimedia learning. In In-
structional design for multimedia learning. Muenster:
Waxmann.
P
´
erez-Colado, V. M., P
´
erez-Colado, I. J., Freire-Mor
´
an, M.,
Mart
´
ınez-Ortiz, I., & Fern
´
andez-Manj
´
on, B. (2019).
uAdventure: Simplifying Narrative Serious Games
Development. In 2019 IEEE 19th International Con-
ference on Advanced Learning Technologies (ICALT),
volume 2161-377X (pp. 119–123). ISSN: 2161-377X.
Shute, V. J., Ventura, M., & Bauer, M. (2009). Melding the
Power of Serious Games and Embedded Assessment
CSEDU 2023 - 15th International Conference on Computer Supported Education
408
to Monitor and Foster Learning: Flow and Grow. In
U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious
Games (pp. 317–343). Routledge.
van den Bos, P., Burghout, C., & Brinke, D. J.-t. (2017).
Toetsen met rubrics. In H. van Berkel, A. Bax, &
D. Joosten-ten Brinke (Eds.), Toetsen in het hoger
onderwijs (pp. 201–214). Houten: Bohn Stafleu van
Loghum.
Wouters, P., van Nimwegen, C., van Oostendorp, H., &
van der Spek, E. D. (2013). A meta-analysis of the
cognitive and motivational effects of serious games.
Journal of Educational Psychology, 105(2), 249–265.
American Psychological Association.
Gamebrics: Integrating Analytical Rubrics into Serious Games to Teach Analytical Skills
409