Analysing the Use of Worked Examples and Tutored and Untutored
Problem-Solving in a Dispositional Learning Analytics Context
Dirk T. Tempelaar
1a
, Bart Rienties
2b
and Quan Nguyen
2c
1
Maastricht University, School of Business and Economics, Tongersestraat 53, 6211 MD Maastricht, The Netherlands
2
Open University UK, Institute of Educational Technology, Walton Hal, Milton Keynes, MK7 6AA, U.K.
Keywords: Blended Learning, Dispositional Learning Analytics, Learning Strategies, Multi-modal Data, Prediction
Models, Tutored Problem-Solving, Untutored Problem-Solving, Worked Examples.
Abstract: The identification of students’ learning strategies by using multi-modal data that combine trace data with self-
report data is the prime aim of this study. Our context is an application of dispositional learning analytics in
a large introductory course mathematics and statistics, based on blended learning. Building on previous
studies in which we found marked differences in how students use worked examples as a learning strategy,
we compare different profiles of learning strategies on learning dispositions and learning outcome. Our results
cast a new light on the issue of efficiency of learning by worked examples, tutored and untutored problem-
solving: in contexts where students can apply their own preferred learning strategy, we find that learning
strategies depend on learning dispositions. As a result, learning dispositions will have a confounding effect
when studying the efficiency of worked examples as a learning strategy in an ecologically valid context.
1 INTRODUCTION
For many decades, research into student learning
tactics and strategies has primarily relied on self-
reports or think-aloud protocols, open to the bias
often present in self-reported perceptions, or
excluding naturalistic contexts from the analysis
(Azevedo et al., 2013; Gašević et al., 2017a; Gašević
et al., 2017b). The increasing use of blended learning
and other forms of technology-enhanced education
gave way to measure revealed learning strategies by
collecting traces of students’ learning behaviours in
the digital learning platforms. This new opportunity
of combining trace data with self-report data has
boosted empirical research in learning tactics and
strategies. Examples of such are Azevedo et al.
(2013), and research by Gašević and co-authors
(Gašević et al., 2017a; 2017b).
This type of research aims to investigate
relationships between learning strategies measured
by trace data, learning approaches measured by self-
reports, and academic performance as learning
outcomes. For instance, Gašević et al. (2017a) finds
a
https://orcid.org/0000-0001-8156-4614
b
https://orcid.org/0000-0003-3749-9629
c
https://orcid.org/0000-0001-8937-5121
that learning strategies are related to deep learning
approaches, but not to surface learning approaches. In
the experimental study Gašević et al. (2017b), the role
of instructional conditions and prior experience with
technology-enhanced education is investigated.
However, most of these studies do not take individual
differences into account, as expressed in Gašević et
al. (2017b, p. 216): ‘Future studies should also
account for the effects of individual differences -e.g.,
motivation to use technology, self-efcacy about the
subject matter and/or technology, achievement goal
orientation, approaches to learning, and
metacognitive awareness’.
Our paper aims to contribute to this lack of
empirical work incorporating individual differences,
by addressing students’ learning strategies within a
dispositional learning analytics context. The
Dispositional Learning Analytics (DLA)
infrastructure, introduced by Buckingham Shum and
Crick (2012), combines learning data (generated in
learning activities through technology-enhanced
systems) with learner data (student dispositions,
values, and attitudes measured through self-report
38
Tempelaar, D., Rienties, B. and Nguyen, Q.
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context.
DOI: 10.5220/0007674900380047
In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), pages 38-47
ISBN: 978-989-758-367-4
Copyright
c
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
surveys). Learning dispositions represent individual
difference characteristics that impact all learning
processes and include affective, behavioural and
cognitive facets (Rienties et al., 2017). Student’s
preferred learning approaches are examples of such
dispositions of both cognitive and behavioural type.
The current study builds on our previous DLA-
based research (Nguyen et al., 2016; Tempelaar et al.,
2013; 2015; 2017a; 2017b; 2017c; 2018). One of our
empirical findings in these studies was that traces of
student learning in digital platforms show marked
differences in the use of worked examples (Nguyen et
al., 2016; Tempelaar et al., 2017a; 2017b; 2018). The
merits of the worked examples principle Renkl (2014)
in the initial acquisition of cognitive skills are well
documented. The use of worked solutions in multi-
media based learning environments stimulates
gaining deep understanding (Renkl, 2014). When
compared to the use of erroneous examples, tutored
problem-solving, and problem-solving in computer-
based environments, the use of worked examples may
be more efficient as it reaches similar learning
outcomes in less time and with less learning efforts.
The mechanism responsible for this outcome is
disclosed in Renkl (2014, p. 400): ‘examples relieve
learners of problem-solving that – in initial cognitive
skill acquisition when learners still lack
understanding – is typically slow, error-prone, and
driven by superficial strategies. When beginning
learners solve problems, the corresponding demands
may burden working memory capacities or even
overload them, which strengthens learners’ surface
orientation. … When learning from examples,
learners have enough working memory capacity for
self-explaining and comparing examples by which
abstract principles can be considered, and those
principles are then related to concrete exemplars. In
this way, learners gain an understanding of how to
apply principles in problem-solving and how to relate
problem cases to underlying principles’.
However, empirical research based on measured
learning behaviour suggests that students may abuse
help facilities available in digital learning
environments through bypassing more abstract hints
and going straightforwardly to concrete solutions
(Shih et al., 2008). Analysing log behaviour of
students, distinguishing proper use and abuse of help
facilities, would allow creating profiles of adaptive
and maladaptive learning behaviours (Shih et al.,
2008; see also Papamitsiou and Economides, 2014).
Following research by McLaren and co-authors
(McLaren et al., 2014; 2016), we extend the range of
preferred learning strategies taken into account to
include, beyond worked-examples, the tutored and
untutored problem-solving strategies. In the tutored
problem-solving strategy, students receive feedback
in the form of hints and an evaluation of provided
answers, both during and at the end of the problem-
solving steps. In untutored problem-solving,
feedback is restricted to the evaluation of provided
answers at the end of the problem-solving steps
(McLaren et al., 2014; 2016).
Evidence for the worked examples principle is
typically based on laboratory-based experimental
studies, in which the effectiveness of different
instructional designs is compared (Renkl, 2014).
McLaren and co-authors take the research into the
effectiveness of several learning strategies a step into
the direction of ecological validity, by choosing for
an experimental design in a classroom context,
assigning the alternative learning approaches
worked-examples, tutored and untutored problem-
solving, and erroneous examples as the conditions of
the experiment (McLaren et al., 2014; 2016). In our
research, we increase ecological validity one more
step by offering a digital learning environment that
encompasses all learning strategies of worked-
examples, tutored and untutored problem-solving,
and observing the revealed preference of the students
in terms of learning strategy they apply. In this
naturalistic context, the potential contribution of LA-
based investigations is that we can observe students’
revealed preferences for a specific learning strategy,
how these preferences depend on the learning task at
hand, and how these preferences link to other
observations, such an individual difference
characteristics. By doing so, we aim to derive a
characterization of students who actively apply
worked examples or tutored problem-solving, and
those not doing so. In line with contemporary
research into learning strategies applying trace data
(Gašević et al., 2017a; 2017b), we adopt two research
questions: 1) how does the choice for learning
strategy relate to learning dispositions? and 2) how
does the learning strategy of using worked examples
or tutored problem-solving relate to learning
outcomes?
2 METHODS
2.1 Context of the Empirical Study
This study takes place in a large-scale introductory
mathematics and statistics course for first-year
undergraduate students in a business and economics
programme in the Netherlands. The educational
system is best described as ‘blended’ or ‘hybrid’. The
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context
39
main component is face-to-face: Problem-Based
Learning (PBL), in small groups (14 students),
coached by content expert tutors (in 78 parallel
tutorial groups). Participation in tutorial groups is
required. Optional is the online component of the
blend: the use of the two e-tutorials SOWISO
(https://sowiso.nl/) and MyStatLab (MSL) (Nguyen
et al., 2016; Tempelaar et al., 2013; 2015; 2017a;
2017c). This design is based on the philosophy of
student-centred education, placing the responsibility
for making educational choices primarily on the
student. Since most of the learning takes place during
self-study outside class through the e-tutorials or
other learning materials, class time is used to discuss
solving advanced problems. Thus, the instructional
format shares most characteristics of the flipped-
classroom design. Using and achieving good scores
in the e-tutorial practice modes is incentivized by
providing bonus points for good performance in
quizzes that are taken every two weeks and consist of
items that are drawn from the same item pools applied
in the practising mode. This approach was chosen to
encourage students with limited prior knowledge to
make intensive use of the e-tutorials.
The subject of this study is the full 2016/2017
cohort of students (1093 students). A large diversity
of the student population was present: only 19% were
educated in the Dutch high school system, against
81% being international students, with 50
nationalities present. A large share of students was of
European nationality, with only 3.9% of students
from outside Europe. High school systems in Europe
differ strongly, most particularly in the teaching of
mathematics and statistics. Therefore, it is crucial that
this introductory module is flexible and allows for
individual learning paths. Students spend on average
24 hours in SOWISO and 32 hours in MSL, which is
30% to 40% of the available time of 80 hours for
learning both topics.
2.2 Instruments and Procedure
Both e-tutorial systems SOWISO and MSL follow a
test-directed learning and practising approach. Each
step in the learning process is initiated by a question,
and students are encouraged to (attempt to) answer
each question. If a student does not master a question
(completely), she/he can either ask for hints to solve
the problem step-by-step, or ask for a fully worked
example. After receiving feedback, a new version of
the problem loads (parameter based) to allow the
student to demonstrate his/her newly acquired
mastery. Students’ revealed preferences for learning
strategies are related to their learning dispositions, as
we demonstrated in previous research (Nguyen et al.,
2016; Tempelaar et al., 2017a; 2017c) for the use of
worked-examples in SOWISO), and the use of
worked-examples in MSL (Tempelaar, 2017b). This
study extends Nguyen et al. (2016) and Tempelaar et
al. (2017a; 2017c) by investigating three learning
strategies in the SOWISO tool: worked examples, and
supported and tutored problem-solving.
Figure 1 demonstrates the implementation of the
alternative learning strategies students can opt for a
sample exercise:
Check: the untutored problem-solving
approach, offering only correctness
feedback after problem-solving;
Hint: the tutored problem-solving approach,
offering feedback and hints to assist the
student in the several problem-solving steps;
Solution: the worked-examples approach;
Theory: asking for a short explanation of the
mathematical principle.
Our study combines trace data of the SOWISO e-
tutorial with self-report survey data measuring
learning dispositions. Clicks in the two e-tutorial
systems represent an important part of that trace data,
and in that respect, our research design is aligned with
the research by Amo-Filvà and co-authors (Amo et
al., 2018; Amo-Filvà et al., 2019) who use a tool
called Clickstream to describe click behaviour in the
digital learning environment. But trace data can
include more than click data only. Azevedo (Azevedo
et al., 2013) distinguishes between trace data of
product and process type, where click data is part of
the category of process data. In this study, we will
combine both process data, as, e.g. the clicks to
initiate the above-mentioned learning supports of
Check, Hint, Solution and Theory, but also product
data, as, e.g. mastery in the tool, as discussed below.
SOWISO reporting options of trace data are very
broad, requiring making selections from the data.
First, all dynamic trace data were aggregated over
time, to arrive at static, full course period accounts of
trace data. Second, from the large array of trace
variables, a selection was made by focusing on
process variables most strongly connected to
alternative learning strategies.
In total, five trace variables were selected:
Mastery in the tool, the proportion of
exercises successfully solved as product
indicator;
#Attempts: the total number of attempts of
individual exercises;
CSEDU 2019 - 11th International Conference on Computer Supported Education
40
Figure 1: sample of SOWISO exercise with the options Check, Theory, Solution, and Hint.
#Hints: the total number of Hints called for.
#Examples: the total number of worked-out
examples called.
To disentangle the effects of learning intensity from
learning strategy, we restricted the sample to those
students who have been very active in the e-tutorial
and achieved at least a 70% mastery level (that is,
successfully solved at least 162 of the 231 exercises):
860 of the 1080 students. The next step in the analysis
is to create profiles of learning behaviours by
distinguishing different patterns of student learning in
the e-tutorials, as in Amo et al. (2018) or Amo-Filvà
et al. (2019). Rather than applying advanced
statistical techniques to create different profiles of
using worked examples, as in Gašević and co-authors
(Gašević et al., 2017a; 2017b) or our previous
research (Nguyen et al., 2016; Tempelaar, et al.,
2017a; 2017c), we applied quartile splits: the selected
students were split into four equal-sized groups
according to intensity of using worked examples, as
well the intensity of using hints. Table 1 provides
descriptive statistics of these four times four sub-
samples.
The operationalization of revealed preferences for
learning strategies follow these quartile splits. The
revealed preference for the worked-examples strategy
is operationalized as students calling for many
examples, ending up in the higher quarters of the
quartile-split for examples. The revealed preference
for the tutored problem-solving strategy is
operationalized as calling for a large number of hints,
thus ending in the higher quarters of the quartile-split
for hints. As is clear from Table 1, revealed
preferences for learning strategies are not disjunct. A
student can combine the strategies of worked-
examples and tutored problem-solving, calling for an
above-average number of hints as well as above
average number of examples.
The strategy of untutored problem-solving is a
necessary component of any of the revealed
preferences, since students can only build mastery
through untutored problem-solving, and the students
included in this analysis all obtained high mastery
levels.
Mastery level is indeed invariant over groups, and
never below 96%: there is a large majority of students
in all sub-samples that reached full mastery. There is
considerable variation in the use of hints, and the use
of examples. The use of hints and examples seems
only weakly associated, except for the quarter of
students using most hints: HintsQ4. In that quarter,
the use of hints and examples is positively correlated
(in HintsQ4, the correlation of hints and examples
equals 0.23, against 0.07 in all four quarters).
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context
41
Table 1: Descriptive statistics of four times four sub-samples of students achieving at least 70% mastery level.
Group N Mastery #Attempts #Hints #Examples
HintsQ1&ExamplesQ1 72 96.2% 340 4.3 38.7
HintsQ1&ExamplesQ2 41 97.8% 418 4.8 88.7
HintsQ1&ExamplesQ3 59 98.2% 534 4.2 139.9
HintsQ1&ExamplesQ4 59 97.7% 729 4.5 157.7
HintsQ2&ExamplesQ1 64 97.5% 367 20.8 41.9
HintsQ2&ExamplesQ2 55 99.2% 432 21.8 85.3
HintsQ2&ExamplesQ3 39 99.7% 520 21.0 134.5
HintsQ2&ExamplesQ4 45 98.8% 758 20.3 269.9
HintsQ3&ExamplesQ1 53 97.0% 370 57.4 48.5
HintsQ3&ExamplesQ2 53 98.8% 433 53.8 88.6
HintsQ3&ExamplesQ3 56 99.3% 528 56.7 136.1
HintsQ3&ExamplesQ4 52 99.5% 786 59.5 275.4
HintsQ4&ExamplesQ1 33 99.2% 486 135.8 49.1
HintsQ4&ExamplesQ2 60 98.4% 476 137.9 89.7
HintsQ4&ExamplesQ3 61 99.3% 587 157.2 140.8
HintsQ4&ExamplesQ4 58 99.4% 764 174.9 249.1
Total 860 98.4% 530 58.1 135.5
In this study, we will focus on a selection of the
self-report surveys measuring student learning
dispositions. More than a dozen were administered,
ranging from affective learning emotions to cognitive
learning processing strategies:
Epistemological self-theories of
intelligence;
Epistemological views on role effort in
learning;
Epistemic learning emotions;
Cognitive learning processing strategies;
Metacognitive learning regulation
strategies;
Subject-specific (math & stats) learning
attitudes;
Academic motivations;
Achievement goals;
Achievement orientations;
Learning activity emotions;
Motivation & Engagement wheel;
Cultural intelligence;
National cultural values; and
Help-seeking behaviour
Main self-report instruments measuring learning
dispositions used in this study are shortly described in
the following subsections. For more extensive
coverage, please see previous studies by the authors
(Nguyen et al., 2016; Tempelaar et al., 2015; 2017a,
2017c; 2018). The description of the research
outcomes will focus on specific aspects of learning
dispositions: learning approaches, anxiety and
uncertainty as aspects of students’ attitudes and
learning emotions.
Course performance data is based on the final
written exam, as well as the three intermediate
quizzes. Quiz scores are averaged, and both exam and
quiz are decomposed into two topic scores, resulting
in MathExam, StatsExam, MathQuiz and StatsQuiz.
2.2.1 Learning Approaches
Students’ learning approaches are based on
Vermunt’s Inventory of Learning Styles (ILS)
instrument (Vermunt, 1996). Our study focused on
two of four domains of the ILS: cognitive processing
strategies and metacognitive regulation strategies.
The instrument distinguishes three different
processing strategies: deep approaches to learning,
stepwise or surface approaches to learning and
concrete or strategic approaches to learning, as well
as three regulations strategies: self-regulation,
external regulation and lack of regulation.
2.2.2 Dispositional Attitudes Data
Attitudes towards learning of mathematics and
statistics were assessed with the SATS instrument
(Tempelaar et al., 2007). The instrument contains six
quantitative methods-related learning attitudes:
Affect: students’ feelings concerning
mathematics and statistics,
CognComp: students’ self-perceptions of
their intellectual knowledge and skills when
applied to mathematics and statistics,
CSEDU 2019 - 11th International Conference on Computer Supported Education
42
Value: students’ attitudes about the
usefulness, relevance, and worth of
mathematics and statistics in their personal
and professional life,
NoDifficulty: students’ perceptions that
mathematics and statistics as subjects are not
difficult to learn,
Interest: students’ level of individual interest
in learning mathematics and statistics,
Effort: the amount of work students are
willing to undertake to learn the subjects.
2.2.3 Dispositional Epistemic Emotions Data
Epistemic emotions are related to the cognitive
aspects of a learning task. Prototypical epistemic
emotions are curiosity and confusion. In this study,
epistemic emotions were measured with the
Epistemic Emotion Scales (EES; Pekrun et al., 2017).
That instrument includes the scales:
Surprise: neutral epistemic emotion,
Curiosity: positive, activating epistemic
emotion,
Confusion: negative, deactivating epistemic
emotion,
Anxiety: negative, activating epistemic
emotion,
Frustration: negative, deactivating epistemic
emotion,
Enjoyment: positive, activating epistemic
emotion,
Boredom: negative, deactivating epistemic
emotion.
3 RESULTS
3.1 Previous Research
In previous research (Nguyen, 2016; Tempelaar et al.,
2015; 2017a; 2017c; 2018), we investigated the role
of worked examples in LA applications and found
that a range of dispositions predict the use of worked
examples as a learning strategy. Demographic
variables, student-learning approaches, learning
attitudes and learning emotions influenced the use of
worked examples, with effect sizes up to 7% for
individual dispositions. In our profiling study
(Tempelaar et al., 2017a) we found that the use of
worked examples and the total number of attempts to
be the two variables shaping most of the characteristic
differences between different profiles in the use of the
e-tutorial. The use of hints did not strongly contribute
to the creation of the student use profiles. As a
consequence, we expect dispositions to play a less
strong role in the explanation of the use of hints as a
learning strategy than it has in the explanation of the
use of worked examples. This expectation does
indeed come true, and in the reporting of the
empirical outcomes in the next sections, we will focus
on the cases where dispositions matter in the
explanation of both learning strategies, leaving out
the cases where the impact is primarily on the use of
worked examples, that are described in previous
research (Nguyen et al., 2016; Tempelaar et al., 2015;
2017a; 2017c; 2018).
3.2 Demographics
Demographic variables have no practical significance
in the explanation of the use of hints: gender and
international status have statistically non-significant
relationships with the intensity of use of hints. Math
prior education has a marginally significant effect
with limited size (p-value=.04, eta squared=1.2%).
Differences in national cultural values follow this
pattern, with the single exception that students from
cultures that assign greater value to long-term
orientation tend to apply the learning strategy of using
hints more often than students from other cultures (p-
value=.004, eta squared=1.2%).
3.3 Learning Approaches
Although the use of learning strategies is the explicit
focus of learning approaches frameworks, the
learning strategy of supported problem solving by
using hints was not adequately captured in our
learning approaches instrument. Hence, we found no
differences in learning approaches for the use of hints.
In the use of worked examples, there were significant
differences for both the deep and stepwise processing
strategies and self-regulation of learning.
3.4 Prior Knowledge
Differences induced by different levels of prior math
schooling are enlarged in the first measurement of
cognitive type: the math entry test, administered at the
very start of the course. The score of diagnostic test
was strongly associated with the intensity of using
hints, and the use of worked examples. Significance
levels are .006, <.001 and .018 for the hints effect, the
examples effect, and the interaction effect,
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context
43
respectively, with a total effect size of eta
squared=11.9%. Figure 2 provides a graphical
illustration of the effects in the several quarters,
where we applied reversed scaling to the several
quarters to facilitate readability. Students with the
highest prior knowledge levels tend to use fewer hints
and fewer examples than the other students. However,
there was more consistency in the pattern for the use
of examples than that for the use of hints: in the
quarter of students with the highest intensity of using
examples, both the Q1 and Q4 quarters of hint use
demonstrate low levels of prior knowledge.
Figure 2: Quarter differences for prior math knowledge, as
measured by a diagnostic test (reversed scaling).
3.5 Learning Attitudes
Since learning attitudes as Affect and Cognitive
Competence are associated with levels of prior
knowledge, it is to be expected that the intensity of use
of both learning strategies is associated with learning
attitudes. That was indeed the case: Affect (p-value
hints<.001, p-value examples<.001, total eta
squared=8.9%), Cognitive competence (p-value
hints<.001, p-value examples<.001, total eta
squared=8.8%) demonstrated clear linear effects in the
absence of interaction effects. Value and Interest had
no role in explaining difference in strategy use,
whereas the NoDifficulty variable was only weakly
associated with both strategies (p-value hints=.034, p-
value examples=.008, total eta squared=2.9%), and the
Effort variable is associated with only the examples
strategy (p-value hints=.461, p-value examples=.002,
total eta squared=3.5%). Figure 3 provides a graphical
presentation for the case of Affect. As in the previous
figure, we see that the highest levels of Affect are to be
found in the group of students who use both hints and
examples least frequently and that intensive use of both
strategies is associated with low levels of Affect.
Figure 3: Quarter differences for learning attitude Affect
(reversed scaling).
3.6 Epistemic Emotions
Epistemic emotions demonstrated group differences
for the negative emotions Confusion (p-value
hints=.004, p-value examples<.001, total eta
squared=7.0%) and Frustration (p-value hints<.001,
p-value examples<.001, total eta squared=6.4%).
Frustration was one of the few disposition variables
that was associated with the use of hints (partial eta-
squared=2.7%) more than with the use of examples
(partial eta-squared=2.4%). Epistemic Enjoyment
makes an even stronger case: here the only significant
relationship is with the use of hints (p-value
hints=.001, p-value examples=.083, total eta
squared=4.8%). Figure 4 demonstrates the
association for Epistemic Frustration, Figure 5 that
for Epistemic Enjoyment.
Figure 4: Quarter differences for Epistemic Frustration
(non-reversed scaling).
CSEDU 2019 - 11th International Conference on Computer Supported Education
44
Figure 5: Quarter differences for Epistemic Enjoyment
(reversed scaling).
3.7 Learning Outcomes
In the main outcome variable of the learning process,
Math Exam or the achievement in the math section of
the final written exam, only associations with the
learning strategy of using examples can be found, be
it that the interaction term is significant (p-value
hints=.106, p-value examples<.001, p-value
interaction=.013, total eta squared=10.9%). Figure 6
provides a graphical description: math exam score is
generally increasing for less intensive use of
examples, but the pattern is not identical for all
quarters of hint use intensity. Specifically, in students
in the third quarter of hint use intensity, the use of
examples and performance seem fairly unrelated.
Figure 6: Quarter differences for Math Exam score
(reversed scaling).
4 DISCUSSION AND
CONCLUSIONS
Existing studies into the efficiency of alternative
learning strategies, both in lab settings (Renkl, 2014)
and in classroom settings (McLaren et al., 2014;
2016), point in the direction of worked-examples
being superior to tutored and untutored problem-
solving. These are generic conclusions, which do not
differentiate between types of academic tasks or types
of students. The main contribution of this study was
the emphasis on the individual differences: learning
dispositions make a difference, academic tasks make
a difference. Allowing for individual differences and
task differences also changes the first order
conclusions.
Regarding the first research question, we found
that students who had less prior knowledge sought
more support from both worked-examples and hints.
Similarly: students who experienced more negative
epistemic emotions such as confusion and frustration,
examples of mal-adaptive dispositions, sought more
support from both worked-examples and hints.
Students who scored higher in the prior knowledge
test usually took on the task by themselves without
seeking help from hints or examples. At the same
time, students who used fewer hints and worked
examples scored higher on the math exam (second
research question). This implies that worked-
examples are only superior to tutored and untutored
problem-solving when the latter two learning
strategies are not sufficient to achieve proficiency.
The initial acquisition of complex knowledge is an
example of such a context. In cases the cognitive
challenges of the learning tasks are less, this
superiority may break down, and worked-examples
may be less efficient learning strategies than problem-
solving approaches.
Transferring the findings of the Renkl (2014) and
the McLaren et al. (2014; 2016) studies to our
context, suggests that the superiority of the worked-
examples strategy may be the result of the tasks
offered to the participants of these studies to be of
such type that students in their studies had little or no
prior knowledge. Our context has been different:
given the wide variety of the tasks and the large
diversity in prior knowledge of students, there exists
a wide range of relevant prior knowledge levels for
any task at hand. In such a context, where students are
expected to demonstrate mastery, a mastery that can
only be acquired in the untutored problem-solving
mode, the use of examples and hints is inevitably a
roundabout route, adding inefficiency to the most
direct way to mastery. That route of using tutored
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context
45
problem-solving and worked-examples is taken by
students who assessed the direct way of untutored
problem-solving to be -still- impassable, explaining
the relationship with prior knowledge.
Our study is based on creating a taxonomy of
learning behaviours by measuring trace data
generated by student activity in e-tutorials. That
taxonomy corroborates the concept of ‘help abuse’
developed by Shih et al. (2008). Rather than trying to
solve problems by asking for hints, some students
bypass these hints and directly call for complete
solutions. Table 1 makes clear that there exist huge
differences in the ratio of hints called for and
solutions called for between the several categories
generated on the quartile splits. That finding is in line
with the hypothesis of help abuse. However, we
cannot easily characterize the extreme categories of
few hints and many solutions versus many hints and
few solutions in terms of the learning dispositions
included in this study. That is: although we find
categories that might represent help abuse, they are
not easily connected with the notions of good and bad
student use as introduced in Shih et al. (2008).
We also corroborate the findings of, e.g. Amo et
al. (2018) and Amo-Filvà et al. (2019) that traces of
learning processes represent useful sources of data for
profiling learning behaviour. At the same time: these
data do capture only part of the learning process. That
is: the main limitation of this research approach is that
all learning that takes place outside the traced e-
tutorials remains unobserved.
The current study has focussed on individual
differences between students in their preference for
learning strategies, and the relationship with learning
dispositions. In future research, we intend to
additionally include the task dimension, by
investigating student preference for learning
strategies as a function of both individual differences
in learning dispositions and task characteristics.
REFERENCES
Amo, D., Alier, M., García-Peñalvo, F. J., Fonseca, D. and
Casañ, M. J. (2018). Learning Analytics to Assess
Students’ Behavior With Scratch Through Clickstream.
In M. Á.Conde, C. Fernández-Llamas, Á. M. Guerrero-
Higueras, F. J. Rodríguez-Sedano, Á. Hernández-
García and F. J. García-Peñalvo (Eds.), Proceedings of
the Learning Analytics Summer Institute Spain 2018 –
LASI-SPAIN 2018, pp. 74-82. Aachen, Germany:
CEUR-WS.org.
Amo-Filvà, D. A., Alier Forment, M., García-Peñalvo, F.
J., Fonseca-Escudero, D. and Casañ, M. J. (2019).
Clickstream for learning analytics to assess students’
behaviour with Scratch. Future Generation Computer
Systems, 93, 673-686. DOI: 10.1016/j.future.2018.
10.057. 2019.
Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-
Behnagh, R., Bouchet, F., et al. (2013). Using trace data
to examine the complex roles of cognitive,
metacognitive, and emotional self-regulatory processes
during learning with multi-agents systems. In R.
Azevedo and V. Aleven (Eds.), International handbook
of metacognition and learning technologies, 427–449.
Amsterdam, the Netherlands: Springer.
Buckingham Shum, S. and Deakin Crick, R. (2012).
Learning Dispositions and Transferable Competencies:
Pedagogy, Modelling and Learning Analytics.
In S. Buckingham Shum, D. Gasevic and R. Ferguson
(Eds.). Proceedings of the 2nd International
Conference on Learning Analytics and Knowledge, 92-
101. ACM, New York, NY, USA. DOI:
10.1145/2330601.2330629.
Gašević, D., Jovanović, J., Pardo, A. and Dawson, S.
(2017a). Detecting learning strategies with analytics:
Links with self-reported measures and academic
performance. Journal of Learning Analytics, 4(1), 113–
128. DOI: 10.18608/jla.2017.42.10.
Gašević, D., Mirriahi, N., Dawson, S. and Joksimović, S.
(2017b). Effects of instructional conditions and
experience on the adoption of a learning tool
Computers in Human Behavior, 67, 207–220. DOI:
10.1016/j.chb.2016.10.026.
McLaren, B. M., van Gog, T., Ganoe, C., Karabinos, M. and
Yaron, D. (2016). The efficiency of worked examples
compared to erroneous examples, tutored problem
solving, and problem-solving in classroom
experiments. Computers in Human Behavior, 55, 87-
99. DOI: 10.1016/j.chb.2015.08.038.
McLaren, B.M., van Gog, T., Ganoe, C., Yaron, D. and
Karabinos, M. (2014) Exploring the assistance
dilemma: Comparing instructional support in examples
and problems. In S. Trausan-Matu et al. (Eds.)
Proceedings of the Twelfth International Conference on
Intelligent Tutoring Systems (ITS-2014). LNCS 8474.
(pp. 354-361). Springer International Publishing
Switzerland.
Nguyen, Q., Tempelaar, D.T., Rienties, B. and Giesbers, B.
(2016). What learning analytics based prediction
models tell us about feedback preferences of students.
In Amirault, R. and Visser, Y., (Eds.), e-Learners and
Their Data, Part 1: Conceptual, Research, and
Exploratory Perspectives.
Quarterly Review of
Distance Education. 17(3).
Papamitsiou, Z. and Economides, A. (2014). Learning
Analytics and Educational Data Mining in Practice: A
Systematic Literature Review of Empirical Evidence.
Educational Technology & Society, 17 (4), 49–64.
Pekrun, R., Vogl, E., Muis, K.R. and Sinatra, G.M. (2017).
Measuring emotions during epistemic activities: the
Epistemically-Related Emotion Scales, Cognition and
Emotion, 31(6), 1268-1276, DOI: 10.1080/
02699931.2016.1204989.
Renkl, A. (2014). The Worked Examples Principle in
CSEDU 2019 - 11th International Conference on Computer Supported Education
46
Multimedia Learning. In R. E. Mayer (Ed.), The
Cambridge Handbook of Multimedia Learning, 391-
412. Cambridge, UK: Cambridge University Press.
Rienties, B., Cross, S. and Zdrahal, Z (2017). Implementing
a Learning Analytics Intervention and Evaluation
Framework: What Works? In B. Kei Daniel (Ed.), Big
data and learning analytics: Current theory and
practice in higher education, 147–166. Cham: Springer
International Publishing. DOI: 10.1007/978-3-319-
06520-5_10.
Shih, B., Koedinger, K. R., & Scheines, R. (2008). A
response time model for bottom-out hints as worked
examples. In R. de Baker, T. Barnes and J. Beck (Eds),
Proceedings of the 1st International Conference on
Educational Data Mining, pp. 117–126. Retrieved from
www.cs.cmu.edu/~bshih/publications/Shih_Koedinger
_Scheines_2008_EDM.pdf
Tempelaar, D. T., Cuypers, H., Van de Vrie, E., Heck, A.
and Van der Kooij, H. (2013). Formative Assessment
and Learning Analytics. In D. Suthers and K. Verbert
(Eds.), Proceedings of the 3rd International
Conference on Learning Analytics and Knowledge,
205-209. New York: ACM. DOI:
10.1145/2460296.2460337.
Tempelaar, D. (2017). How dispositional Learning
Analytics helps understanding the worked-example
principle. In D. G. Sampson, J. M. Spector, D.
Ifenthaler and P. Isaias (Eds.), Proceedings 14th
International Conference on Cognition and
Exploratory Learning in Digital Age (CELDA 2017),
pp. 117-124. International Association for
Development of the Information Society, IADIS Press.
Tempelaar, D. T., Gijselaers, W.H., Schim van der Loeff,
S. and Nijhuis, J.F.H. (2007). A structural equation
model analyzing the relationship of student
achievement motivations and personality factors in a
range of academic subject-matter areas, Contemporary
Educational Psychology, 32(1), 105-131. DOI:
10.1016/j.cedpsych.2006.10.004.
Tempelaar, D. T., Rienties, B., Mittelmeier, J. and Nguyen,
Q. (2018). Student profiling in a dispositional learning
analytics application using formative assessment,
Computers in Human Behavior, 78, 408-420. DOI:
10.1016/j.chb.2017.08.010.
Tempelaar, D. T., Rienties, B. and Giesbers, B. (2015). In
search for the most informative data for feedback
generation: Learning Analytics in a data-rich context.
Computers in Human Behavior, 47, 157-167. DOI:
10.1016/j.chb.2014.05.038.
Tempelaar, D. T., Rienties, B. and Nguyen, Q. (2017a).
Towards actionable learning analytics using
dispositions. IEEE Transactions on Education, 10(1),
6-16. DOI:10.1109/TLT.2017.2662679.
Tempelaar, D. T., Rienties, B. and Nguyen, Q. (2017b).
Adding dispositions to create pedagogy-based Learning
Analytics. Zeitschrift für Hochschulentwicklung,
ZFHE, 12(1), 15-35.
Vermunt, J. D. (1996). Metacognitive, cognitive and
affective aspects of learning styles and strategies: A
phenomenographic analysis, Higher Education, 31, 25–
50. DOI: 10.1007/BF00129106.
Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context
47