Analysing the Use of Worked Examples and Tutored and Untutored

Problem-Solving in a Dispositional Learning Analytics Context

Dirk T. Tempelaar

, Bart Rienties

and Quan Nguyen

Maastricht University, School of Business and Economics, Tongersestraat 53, 6211 MD Maastricht, The Netherlands

Open University UK, Institute of Educational Technology, Walton Hal, Milton Keynes, MK7 6AA, U.K.

Keywords: Blended Learning, Dispositional Learning Analytics, Learning Strategies, Multi-modal Data, Prediction

Models, Tutored Problem-Solving, Untutored Problem-Solving, Worked Examples.

Abstract: The identification of students’ learning strategies by using multi-modal data that combine trace data with self-

report data is the prime aim of this study. Our context is an application of dispositional learning analytics in

a large introductory course mathematics and statistics, based on blended learning. Building on previous

studies in which we found marked differences in how students use worked examples as a learning strategy,

we compare different profiles of learning strategies on learning dispositions and learning outcome. Our results

cast a new light on the issue of efficiency of learning by worked examples, tutored and untutored problem-

solving: in contexts where students can apply their own preferred learning strategy, we find that learning

strategies depend on learning dispositions. As a result, learning dispositions will have a confounding effect

when studying the efficiency of worked examples as a learning strategy in an ecologically valid context.

1 INTRODUCTION

For many decades, research into student learning

tactics and strategies has primarily relied on self-

reports or think-aloud protocols, open to the bias

often present in self-reported perceptions, or

excluding naturalistic contexts from the analysis

(Azevedo et al., 2013; Gašević et al., 2017a; Gašević

et al., 2017b). The increasing use of blended learning

and other forms of technology-enhanced education

gave way to measure revealed learning strategies by

collecting traces of students’ learning behaviours in

the digital learning platforms. This new opportunity

of combining trace data with self-report data has

boosted empirical research in learning tactics and

strategies. Examples of such are Azevedo et al.

(2013), and research by Gašević and co-authors

(Gašević et al., 2017a; 2017b).

This type of research aims to investigate

relationships between learning strategies measured

by trace data, learning approaches measured by self-

reports, and academic performance as learning

outcomes. For instance, Gašević et al. (2017a) finds

https://orcid.org/0000-0001-8156-4614

https://orcid.org/0000-0003-3749-9629

https://orcid.org/0000-0001-8937-5121

that learning strategies are related to deep learning

approaches, but not to surface learning approaches. In

the experimental study Gašević et al. (2017b), the role

of instructional conditions and prior experience with

technology-enhanced education is investigated.

However, most of these studies do not take individual

differences into account, as expressed in Gašević et

al. (2017b, p. 216): ‘Future studies should also

account for the effects of individual differences -e.g.,

motivation to use technology, self-efﬁcacy about the

subject matter and/or technology, achievement goal

orientation, approaches to learning, and

metacognitive awareness’.

Our paper aims to contribute to this lack of

empirical work incorporating individual differences,

by addressing students’ learning strategies within a

dispositional learning analytics context. The

Dispositional Learning Analytics (DLA)

infrastructure, introduced by Buckingham Shum and

Crick (2012), combines learning data (generated in

learning activities through technology-enhanced

systems) with learner data (student dispositions,

values, and attitudes measured through self-report

Tempelaar, D., Rienties, B. and Nguyen, Q.

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context.

DOI: 10.5220/0007674900380047

In Proceedings of the 11th International Conference on Computer Supported Education (CSEDU 2019), pages 38-47

ISBN: 978-989-758-367-4

surveys). Learning dispositions represent individual

difference characteristics that impact all learning

processes and include affective, behavioural and

cognitive facets (Rienties et al., 2017). Student’s

preferred learning approaches are examples of such

dispositions of both cognitive and behavioural type.

The current study builds on our previous DLA-

based research (Nguyen et al., 2016; Tempelaar et al.,

2013; 2015; 2017a; 2017b; 2017c; 2018). One of our

empirical findings in these studies was that traces of

student learning in digital platforms show marked

differences in the use of worked examples (Nguyen et

al., 2016; Tempelaar et al., 2017a; 2017b; 2018). The

merits of the worked examples principle Renkl (2014)

in the initial acquisition of cognitive skills are well

documented. The use of worked solutions in multi-

media based learning environments stimulates

gaining deep understanding (Renkl, 2014). When

compared to the use of erroneous examples, tutored

problem-solving, and problem-solving in computer-

based environments, the use of worked examples may

be more efficient as it reaches similar learning

outcomes in less time and with less learning efforts.

The mechanism responsible for this outcome is

disclosed in Renkl (2014, p. 400): ‘examples relieve

learners of problem-solving that – in initial cognitive

skill acquisition when learners still lack

understanding – is typically slow, error-prone, and

driven by superficial strategies. When beginning

learners solve problems, the corresponding demands

may burden working memory capacities or even

overload them, which strengthens learners’ surface

orientation. … When learning from examples,

learners have enough working memory capacity for

self-explaining and comparing examples by which

abstract principles can be considered, and those

principles are then related to concrete exemplars. In

this way, learners gain an understanding of how to

apply principles in problem-solving and how to relate

problem cases to underlying principles’.

However, empirical research based on measured

learning behaviour suggests that students may abuse

help facilities available in digital learning

environments through bypassing more abstract hints

and going straightforwardly to concrete solutions

(Shih et al., 2008). Analysing log behaviour of

students, distinguishing proper use and abuse of help

facilities, would allow creating profiles of adaptive

and maladaptive learning behaviours (Shih et al.,

2008; see also Papamitsiou and Economides, 2014).

Following research by McLaren and co-authors

(McLaren et al., 2014; 2016), we extend the range of

preferred learning strategies taken into account to

include, beyond worked-examples, the tutored and

untutored problem-solving strategies. In the tutored

problem-solving strategy, students receive feedback

in the form of hints and an evaluation of provided

answers, both during and at the end of the problem-

solving steps. In untutored problem-solving,

feedback is restricted to the evaluation of provided

answers at the end of the problem-solving steps

(McLaren et al., 2014; 2016).

Evidence for the worked examples principle is

typically based on laboratory-based experimental

studies, in which the effectiveness of different

instructional designs is compared (Renkl, 2014).

McLaren and co-authors take the research into the

effectiveness of several learning strategies a step into

the direction of ecological validity, by choosing for

an experimental design in a classroom context,

assigning the alternative learning approaches

worked-examples, tutored and untutored problem-

solving, and erroneous examples as the conditions of

the experiment (McLaren et al., 2014; 2016). In our

research, we increase ecological validity one more

step by offering a digital learning environment that

encompasses all learning strategies of worked-

examples, tutored and untutored problem-solving,

and observing the revealed preference of the students

in terms of learning strategy they apply. In this

naturalistic context, the potential contribution of LA-

based investigations is that we can observe students’

revealed preferences for a specific learning strategy,

how these preferences depend on the learning task at

hand, and how these preferences link to other

observations, such an individual difference

characteristics. By doing so, we aim to derive a

characterization of students who actively apply

worked examples or tutored problem-solving, and

those not doing so. In line with contemporary

research into learning strategies applying trace data

(Gašević et al., 2017a; 2017b), we adopt two research

questions: 1) how does the choice for learning

strategy relate to learning dispositions? and 2) how

does the learning strategy of using worked examples

or tutored problem-solving relate to learning

outcomes?

2 METHODS

2.1 Context of the Empirical Study

This study takes place in a large-scale introductory

mathematics and statistics course for first-year

undergraduate students in a business and economics

programme in the Netherlands. The educational

system is best described as ‘blended’ or ‘hybrid’. The

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context

main component is face-to-face: Problem-Based

Learning (PBL), in small groups (14 students),

coached by content expert tutors (in 78 parallel

tutorial groups). Participation in tutorial groups is

required. Optional is the online component of the

blend: the use of the two e-tutorials SOWISO

(https://sowiso.nl/) and MyStatLab (MSL) (Nguyen

et al., 2016; Tempelaar et al., 2013; 2015; 2017a;

2017c). This design is based on the philosophy of

student-centred education, placing the responsibility

for making educational choices primarily on the

student. Since most of the learning takes place during

self-study outside class through the e-tutorials or

other learning materials, class time is used to discuss

solving advanced problems. Thus, the instructional

format shares most characteristics of the flipped-

classroom design. Using and achieving good scores

in the e-tutorial practice modes is incentivized by

providing bonus points for good performance in

quizzes that are taken every two weeks and consist of

items that are drawn from the same item pools applied

in the practising mode. This approach was chosen to

encourage students with limited prior knowledge to

make intensive use of the e-tutorials.

The subject of this study is the full 2016/2017

cohort of students (1093 students). A large diversity

of the student population was present: only 19% were

educated in the Dutch high school system, against

81% being international students, with 50

nationalities present. A large share of students was of

European nationality, with only 3.9% of students

from outside Europe. High school systems in Europe

differ strongly, most particularly in the teaching of

mathematics and statistics. Therefore, it is crucial that

this introductory module is flexible and allows for

individual learning paths. Students spend on average

24 hours in SOWISO and 32 hours in MSL, which is

30% to 40% of the available time of 80 hours for

learning both topics.

2.2 Instruments and Procedure

Both e-tutorial systems SOWISO and MSL follow a

test-directed learning and practising approach. Each

step in the learning process is initiated by a question,

and students are encouraged to (attempt to) answer

each question. If a student does not master a question

(completely), she/he can either ask for hints to solve

the problem step-by-step, or ask for a fully worked

example. After receiving feedback, a new version of

the problem loads (parameter based) to allow the

student to demonstrate his/her newly acquired

mastery. Students’ revealed preferences for learning

strategies are related to their learning dispositions, as

we demonstrated in previous research (Nguyen et al.,

2016; Tempelaar et al., 2017a; 2017c) for the use of

worked-examples in SOWISO), and the use of

worked-examples in MSL (Tempelaar, 2017b). This

study extends Nguyen et al. (2016) and Tempelaar et

al. (2017a; 2017c) by investigating three learning

strategies in the SOWISO tool: worked examples, and

supported and tutored problem-solving.

Figure 1 demonstrates the implementation of the

alternative learning strategies students can opt for a

sample exercise:

• Check: the untutored problem-solving

approach, offering only correctness

feedback after problem-solving;

• Hint: the tutored problem-solving approach,

offering feedback and hints to assist the

student in the several problem-solving steps;

• Solution: the worked-examples approach;

• Theory: asking for a short explanation of the

mathematical principle.

Our study combines trace data of the SOWISO e-

tutorial with self-report survey data measuring

learning dispositions. Clicks in the two e-tutorial

systems represent an important part of that trace data,

and in that respect, our research design is aligned with

the research by Amo-Filvà and co-authors (Amo et

al., 2018; Amo-Filvà et al., 2019) who use a tool

called Clickstream to describe click behaviour in the

digital learning environment. But trace data can

include more than click data only. Azevedo (Azevedo

et al., 2013) distinguishes between trace data of

product and process type, where click data is part of

the category of process data. In this study, we will

combine both process data, as, e.g. the clicks to

initiate the above-mentioned learning supports of

Check, Hint, Solution and Theory, but also product

data, as, e.g. mastery in the tool, as discussed below.

SOWISO reporting options of trace data are very

broad, requiring making selections from the data.

First, all dynamic trace data were aggregated over

time, to arrive at static, full course period accounts of

trace data. Second, from the large array of trace

variables, a selection was made by focusing on

process variables most strongly connected to

alternative learning strategies.

In total, five trace variables were selected:

• Mastery in the tool, the proportion of

exercises successfully solved as product

indicator;

• #Attempts: the total number of attempts of

individual exercises;

CSEDU 2019 - 11th International Conference on Computer Supported Education

Figure 1: sample of SOWISO exercise with the options Check, Theory, Solution, and Hint.

• #Hints: the total number of Hints called for.

• #Examples: the total number of worked-out

examples called.

To disentangle the effects of learning intensity from

learning strategy, we restricted the sample to those

students who have been very active in the e-tutorial

and achieved at least a 70% mastery level (that is,

successfully solved at least 162 of the 231 exercises):

860 of the 1080 students. The next step in the analysis

is to create profiles of learning behaviours by

distinguishing different patterns of student learning in

the e-tutorials, as in Amo et al. (2018) or Amo-Filvà

et al. (2019). Rather than applying advanced

statistical techniques to create different profiles of

using worked examples, as in Gašević and co-authors

(Gašević et al., 2017a; 2017b) or our previous

research (Nguyen et al., 2016; Tempelaar, et al.,

2017a; 2017c), we applied quartile splits: the selected

students were split into four equal-sized groups

according to intensity of using worked examples, as

well the intensity of using hints. Table 1 provides

descriptive statistics of these four times four sub-

samples.

The operationalization of revealed preferences for

learning strategies follow these quartile splits. The

revealed preference for the worked-examples strategy

is operationalized as students calling for many

examples, ending up in the higher quarters of the

quartile-split for examples. The revealed preference

for the tutored problem-solving strategy is

operationalized as calling for a large number of hints,

thus ending in the higher quarters of the quartile-split

for hints. As is clear from Table 1, revealed

preferences for learning strategies are not disjunct. A

student can combine the strategies of worked-

examples and tutored problem-solving, calling for an

above-average number of hints as well as above

average number of examples.

The strategy of untutored problem-solving is a

necessary component of any of the revealed

preferences, since students can only build mastery

through untutored problem-solving, and the students

included in this analysis all obtained high mastery

levels.

Mastery level is indeed invariant over groups, and

never below 96%: there is a large majority of students

in all sub-samples that reached full mastery. There is

considerable variation in the use of hints, and the use

of examples. The use of hints and examples seems

only weakly associated, except for the quarter of

students using most hints: HintsQ4. In that quarter,

the use of hints and examples is positively correlated

(in HintsQ4, the correlation of hints and examples

equals 0.23, against 0.07 in all four quarters).

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context

Table 1: Descriptive statistics of four times four sub-samples of students achieving at least 70% mastery level.

Group N Mastery #Attempts #Hints #Examples

HintsQ1&ExamplesQ1 72 96.2% 340 4.3 38.7

HintsQ1&ExamplesQ2 41 97.8% 418 4.8 88.7

HintsQ1&ExamplesQ3 59 98.2% 534 4.2 139.9

HintsQ1&ExamplesQ4 59 97.7% 729 4.5 157.7

HintsQ2&ExamplesQ1 64 97.5% 367 20.8 41.9

HintsQ2&ExamplesQ2 55 99.2% 432 21.8 85.3

HintsQ2&ExamplesQ3 39 99.7% 520 21.0 134.5

HintsQ2&ExamplesQ4 45 98.8% 758 20.3 269.9

HintsQ3&ExamplesQ1 53 97.0% 370 57.4 48.5

HintsQ3&ExamplesQ2 53 98.8% 433 53.8 88.6

HintsQ3&ExamplesQ3 56 99.3% 528 56.7 136.1

HintsQ3&ExamplesQ4 52 99.5% 786 59.5 275.4

HintsQ4&ExamplesQ1 33 99.2% 486 135.8 49.1

HintsQ4&ExamplesQ2 60 98.4% 476 137.9 89.7

HintsQ4&ExamplesQ3 61 99.3% 587 157.2 140.8

HintsQ4&ExamplesQ4 58 99.4% 764 174.9 249.1

Total 860 98.4% 530 58.1 135.5

In this study, we will focus on a selection of the

self-report surveys measuring student learning

dispositions. More than a dozen were administered,

ranging from affective learning emotions to cognitive

learning processing strategies:

• Epistemological self-theories of

intelligence;

• Epistemological views on role effort in

learning;

• Epistemic learning emotions;

• Cognitive learning processing strategies;

• Metacognitive learning regulation

strategies;

• Subject-specific (math & stats) learning

attitudes;

• Academic motivations;

• Achievement goals;

• Achievement orientations;

• Learning activity emotions;

• Motivation & Engagement wheel;

• Cultural intelligence;

• National cultural values; and

• Help-seeking behaviour

Main self-report instruments measuring learning

dispositions used in this study are shortly described in

the following subsections. For more extensive

coverage, please see previous studies by the authors

(Nguyen et al., 2016; Tempelaar et al., 2015; 2017a,

2017c; 2018). The description of the research

outcomes will focus on specific aspects of learning

dispositions: learning approaches, anxiety and

uncertainty as aspects of students’ attitudes and

learning emotions.

Course performance data is based on the final

written exam, as well as the three intermediate

quizzes. Quiz scores are averaged, and both exam and

quiz are decomposed into two topic scores, resulting

in MathExam, StatsExam, MathQuiz and StatsQuiz.

2.2.1 Learning Approaches

Students’ learning approaches are based on

Vermunt’s Inventory of Learning Styles (ILS)

instrument (Vermunt, 1996). Our study focused on

two of four domains of the ILS: cognitive processing

strategies and metacognitive regulation strategies.

The instrument distinguishes three different

processing strategies: deep approaches to learning,

stepwise or surface approaches to learning and

concrete or strategic approaches to learning, as well

as three regulations strategies: self-regulation,

external regulation and lack of regulation.

2.2.2 Dispositional Attitudes Data

Attitudes towards learning of mathematics and

statistics were assessed with the SATS instrument

(Tempelaar et al., 2007). The instrument contains six

quantitative methods-related learning attitudes:

• Affect: students’ feelings concerning

mathematics and statistics,

• CognComp: students’ self-perceptions of

their intellectual knowledge and skills when

applied to mathematics and statistics,

CSEDU 2019 - 11th International Conference on Computer Supported Education

• Value: students’ attitudes about the

usefulness, relevance, and worth of

mathematics and statistics in their personal

and professional life,

• NoDifficulty: students’ perceptions that

mathematics and statistics as subjects are not

difficult to learn,

• Interest: students’ level of individual interest

in learning mathematics and statistics,

• Effort: the amount of work students are

willing to undertake to learn the subjects.

2.2.3 Dispositional Epistemic Emotions Data

Epistemic emotions are related to the cognitive

aspects of a learning task. Prototypical epistemic

emotions are curiosity and confusion. In this study,

epistemic emotions were measured with the

Epistemic Emotion Scales (EES; Pekrun et al., 2017).

That instrument includes the scales:

• Surprise: neutral epistemic emotion,

• Curiosity: positive, activating epistemic

emotion,

• Confusion: negative, deactivating epistemic

emotion,

• Anxiety: negative, activating epistemic

emotion,

• Frustration: negative, deactivating epistemic

emotion,

• Enjoyment: positive, activating epistemic

emotion,

• Boredom: negative, deactivating epistemic

emotion.

3 RESULTS

3.1 Previous Research

In previous research (Nguyen, 2016; Tempelaar et al.,

2015; 2017a; 2017c; 2018), we investigated the role

of worked examples in LA applications and found

that a range of dispositions predict the use of worked

examples as a learning strategy. Demographic

variables, student-learning approaches, learning

attitudes and learning emotions influenced the use of

worked examples, with effect sizes up to 7% for

individual dispositions. In our profiling study

(Tempelaar et al., 2017a) we found that the use of

worked examples and the total number of attempts to

be the two variables shaping most of the characteristic

differences between different profiles in the use of the

e-tutorial. The use of hints did not strongly contribute

to the creation of the student use profiles. As a

consequence, we expect dispositions to play a less

strong role in the explanation of the use of hints as a

learning strategy than it has in the explanation of the

use of worked examples. This expectation does

indeed come true, and in the reporting of the

empirical outcomes in the next sections, we will focus

on the cases where dispositions matter in the

explanation of both learning strategies, leaving out

the cases where the impact is primarily on the use of

worked examples, that are described in previous

research (Nguyen et al., 2016; Tempelaar et al., 2015;

2017a; 2017c; 2018).

3.2 Demographics

Demographic variables have no practical significance

in the explanation of the use of hints: gender and

international status have statistically non-significant

relationships with the intensity of use of hints. Math

prior education has a marginally significant effect

with limited size (p-value=.04, eta squared=1.2%).

Differences in national cultural values follow this

pattern, with the single exception that students from

cultures that assign greater value to long-term

orientation tend to apply the learning strategy of using

hints more often than students from other cultures (p-

value=.004, eta squared=1.2%).

3.3 Learning Approaches

Although the use of learning strategies is the explicit

focus of learning approaches frameworks, the

learning strategy of supported problem solving by

using hints was not adequately captured in our

learning approaches instrument. Hence, we found no

differences in learning approaches for the use of hints.

In the use of worked examples, there were significant

differences for both the deep and stepwise processing

strategies and self-regulation of learning.

3.4 Prior Knowledge

Differences induced by different levels of prior math

schooling are enlarged in the first measurement of

cognitive type: the math entry test, administered at the

very start of the course. The score of diagnostic test

was strongly associated with the intensity of using

hints, and the use of worked examples. Significance

levels are .006, <.001 and .018 for the hints effect, the

examples effect, and the interaction effect,

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context

respectively, with a total effect size of eta

squared=11.9%. Figure 2 provides a graphical

illustration of the effects in the several quarters,

where we applied reversed scaling to the several

quarters to facilitate readability. Students with the

highest prior knowledge levels tend to use fewer hints

and fewer examples than the other students. However,

there was more consistency in the pattern for the use

of examples than that for the use of hints: in the

quarter of students with the highest intensity of using

examples, both the Q1 and Q4 quarters of hint use

demonstrate low levels of prior knowledge.

Figure 2: Quarter differences for prior math knowledge, as

measured by a diagnostic test (reversed scaling).

3.5 Learning Attitudes

Since learning attitudes as Affect and Cognitive

Competence are associated with levels of prior

knowledge, it is to be expected that the intensity of use

of both learning strategies is associated with learning

attitudes. That was indeed the case: Affect (p-value

hints<.001, p-value examples<.001, total eta

squared=8.9%), Cognitive competence (p-value

hints<.001, p-value examples<.001, total eta

squared=8.8%) demonstrated clear linear effects in the

absence of interaction effects. Value and Interest had

no role in explaining difference in strategy use,

whereas the NoDifficulty variable was only weakly

associated with both strategies (p-value hints=.034, p-

value examples=.008, total eta squared=2.9%), and the

Effort variable is associated with only the examples

strategy (p-value hints=.461, p-value examples=.002,

total eta squared=3.5%). Figure 3 provides a graphical

presentation for the case of Affect. As in the previous

figure, we see that the highest levels of Affect are to be

found in the group of students who use both hints and

examples least frequently and that intensive use of both

strategies is associated with low levels of Affect.

Figure 3: Quarter differences for learning attitude Affect

(reversed scaling).

3.6 Epistemic Emotions

Epistemic emotions demonstrated group differences

for the negative emotions Confusion (p-value

hints=.004, p-value examples<.001, total eta

squared=7.0%) and Frustration (p-value hints<.001,

p-value examples<.001, total eta squared=6.4%).

Frustration was one of the few disposition variables

that was associated with the use of hints (partial eta-

squared=2.7%) more than with the use of examples

(partial eta-squared=2.4%). Epistemic Enjoyment

makes an even stronger case: here the only significant

relationship is with the use of hints (p-value

hints=.001, p-value examples=.083, total eta

squared=4.8%). Figure 4 demonstrates the

association for Epistemic Frustration, Figure 5 that

for Epistemic Enjoyment.

Figure 4: Quarter differences for Epistemic Frustration

(non-reversed scaling).

CSEDU 2019 - 11th International Conference on Computer Supported Education

Figure 5: Quarter differences for Epistemic Enjoyment

(reversed scaling).

3.7 Learning Outcomes

In the main outcome variable of the learning process,

Math Exam or the achievement in the math section of

the final written exam, only associations with the

learning strategy of using examples can be found, be

it that the interaction term is significant (p-value

hints=.106, p-value examples<.001, p-value

interaction=.013, total eta squared=10.9%). Figure 6

provides a graphical description: math exam score is

generally increasing for less intensive use of

examples, but the pattern is not identical for all

quarters of hint use intensity. Specifically, in students

in the third quarter of hint use intensity, the use of

examples and performance seem fairly unrelated.

Figure 6: Quarter differences for Math Exam score

(reversed scaling).

4 DISCUSSION AND

CONCLUSIONS

Existing studies into the efficiency of alternative

learning strategies, both in lab settings (Renkl, 2014)

and in classroom settings (McLaren et al., 2014;

2016), point in the direction of worked-examples

being superior to tutored and untutored problem-

solving. These are generic conclusions, which do not

differentiate between types of academic tasks or types

of students. The main contribution of this study was

the emphasis on the individual differences: learning

dispositions make a difference, academic tasks make

a difference. Allowing for individual differences and

task differences also changes the first order

conclusions.

Regarding the first research question, we found

that students who had less prior knowledge sought

more support from both worked-examples and hints.

Similarly: students who experienced more negative

epistemic emotions such as confusion and frustration,

examples of mal-adaptive dispositions, sought more

support from both worked-examples and hints.

Students who scored higher in the prior knowledge

test usually took on the task by themselves without

seeking help from hints or examples. At the same

time, students who used fewer hints and worked

examples scored higher on the math exam (second

research question). This implies that worked-

examples are only superior to tutored and untutored

problem-solving when the latter two learning

strategies are not sufficient to achieve proficiency.

The initial acquisition of complex knowledge is an

example of such a context. In cases the cognitive

challenges of the learning tasks are less, this

superiority may break down, and worked-examples

may be less efficient learning strategies than problem-

solving approaches.

Transferring the findings of the Renkl (2014) and

the McLaren et al. (2014; 2016) studies to our

context, suggests that the superiority of the worked-

examples strategy may be the result of the tasks

offered to the participants of these studies to be of

such type that students in their studies had little or no

prior knowledge. Our context has been different:

given the wide variety of the tasks and the large

diversity in prior knowledge of students, there exists

a wide range of relevant prior knowledge levels for

any task at hand. In such a context, where students are

expected to demonstrate mastery, a mastery that can

only be acquired in the untutored problem-solving

mode, the use of examples and hints is inevitably a

roundabout route, adding inefficiency to the most

direct way to mastery. That route of using tutored

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context

problem-solving and worked-examples is taken by

students who assessed the direct way of untutored

problem-solving to be -still- impassable, explaining

the relationship with prior knowledge.

Our study is based on creating a taxonomy of

learning behaviours by measuring trace data

generated by student activity in e-tutorials. That

taxonomy corroborates the concept of ‘help abuse’

developed by Shih et al. (2008). Rather than trying to

solve problems by asking for hints, some students

bypass these hints and directly call for complete

solutions. Table 1 makes clear that there exist huge

differences in the ratio of hints called for and

solutions called for between the several categories

generated on the quartile splits. That finding is in line

with the hypothesis of help abuse. However, we

cannot easily characterize the extreme categories of

few hints and many solutions versus many hints and

few solutions in terms of the learning dispositions

included in this study. That is: although we find

categories that might represent help abuse, they are

not easily connected with the notions of good and bad

student use as introduced in Shih et al. (2008).

We also corroborate the findings of, e.g. Amo et

al. (2018) and Amo-Filvà et al. (2019) that traces of

learning processes represent useful sources of data for

profiling learning behaviour. At the same time: these

data do capture only part of the learning process. That

is: the main limitation of this research approach is that

all learning that takes place outside the traced e-

tutorials remains unobserved.

The current study has focussed on individual

differences between students in their preference for

learning strategies, and the relationship with learning

dispositions. In future research, we intend to

additionally include the task dimension, by

investigating student preference for learning

strategies as a function of both individual differences

in learning dispositions and task characteristics.

REFERENCES

Amo, D., Alier, M., García-Peñalvo, F. J., Fonseca, D. and

Casañ, M. J. (2018). Learning Analytics to Assess

Students’ Behavior With Scratch Through Clickstream.

In M. Á.Conde, C. Fernández-Llamas, Á. M. Guerrero-

Higueras, F. J. Rodríguez-Sedano, Á. Hernández-

García and F. J. García-Peñalvo (Eds.), Proceedings of

the Learning Analytics Summer Institute Spain 2018 –

LASI-SPAIN 2018, pp. 74-82. Aachen, Germany:

CEUR-WS.org.

Amo-Filvà, D. A., Alier Forment, M., García-Peñalvo, F.

J., Fonseca-Escudero, D. and Casañ, M. J. (2019).

Clickstream for learning analytics to assess students’

behaviour with Scratch. Future Generation Computer

Systems, 93, 673-686. DOI: 10.1016/j.future.2018.

10.057. 2019.

Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-

Behnagh, R., Bouchet, F., et al. (2013). Using trace data

to examine the complex roles of cognitive,

metacognitive, and emotional self-regulatory processes

during learning with multi-agents systems. In R.

Azevedo and V. Aleven (Eds.), International handbook

of metacognition and learning technologies, 427–449.

Amsterdam, the Netherlands: Springer.

Buckingham Shum, S. and Deakin Crick, R. (2012).

Learning Dispositions and Transferable Competencies:

Pedagogy, Modelling and Learning Analytics.

In S. Buckingham Shum, D. Gasevic and R. Ferguson

(Eds.). Proceedings of the 2nd International

Conference on Learning Analytics and Knowledge, 92-

101. ACM, New York, NY, USA. DOI:

10.1145/2330601.2330629.

Gašević, D., Jovanović, J., Pardo, A. and Dawson, S.

(2017a). Detecting learning strategies with analytics:

Links with self-reported measures and academic

performance. Journal of Learning Analytics, 4(1), 113–

128. DOI: 10.18608/jla.2017.42.10.

Gašević, D., Mirriahi, N., Dawson, S. and Joksimović, S.

(2017b). Effects of instructional conditions and

experience on the adoption of a learning tool

Computers in Human Behavior, 67, 207–220. DOI:

10.1016/j.chb.2016.10.026.

McLaren, B. M., van Gog, T., Ganoe, C., Karabinos, M. and

Yaron, D. (2016). The efficiency of worked examples

compared to erroneous examples, tutored problem

solving, and problem-solving in classroom

experiments. Computers in Human Behavior, 55, 87-

99. DOI: 10.1016/j.chb.2015.08.038.

McLaren, B.M., van Gog, T., Ganoe, C., Yaron, D. and

Karabinos, M. (2014) Exploring the assistance

dilemma: Comparing instructional support in examples

and problems. In S. Trausan-Matu et al. (Eds.)

Proceedings of the Twelfth International Conference on

Intelligent Tutoring Systems (ITS-2014). LNCS 8474.

(pp. 354-361). Springer International Publishing

Switzerland.

Nguyen, Q., Tempelaar, D.T., Rienties, B. and Giesbers, B.

(2016). What learning analytics based prediction

models tell us about feedback preferences of students.

In Amirault, R. and Visser, Y., (Eds.), e-Learners and

Their Data, Part 1: Conceptual, Research, and

Exploratory Perspectives.

Quarterly Review of

Distance Education. 17(3).

Papamitsiou, Z. and Economides, A. (2014). Learning

Analytics and Educational Data Mining in Practice: A

Systematic Literature Review of Empirical Evidence.

Educational Technology & Society, 17 (4), 49–64.

Pekrun, R., Vogl, E., Muis, K.R. and Sinatra, G.M. (2017).

Measuring emotions during epistemic activities: the

Epistemically-Related Emotion Scales, Cognition and

Emotion, 31(6), 1268-1276, DOI: 10.1080/

02699931.2016.1204989.

Renkl, A. (2014). The Worked Examples Principle in

CSEDU 2019 - 11th International Conference on Computer Supported Education

Multimedia Learning. In R. E. Mayer (Ed.), The

Cambridge Handbook of Multimedia Learning, 391-

412. Cambridge, UK: Cambridge University Press.

Rienties, B., Cross, S. and Zdrahal, Z (2017). Implementing

a Learning Analytics Intervention and Evaluation

Framework: What Works? In B. Kei Daniel (Ed.), Big

data and learning analytics: Current theory and

practice in higher education, 147–166. Cham: Springer

International Publishing. DOI: 10.1007/978-3-319-

06520-5_10.

Shih, B., Koedinger, K. R., & Scheines, R. (2008). A

response time model for bottom-out hints as worked

examples. In R. de Baker, T. Barnes and J. Beck (Eds),

Proceedings of the 1st International Conference on

Educational Data Mining, pp. 117–126. Retrieved from

www.cs.cmu.edu/~bshih/publications/Shih_Koedinger

_Scheines_2008_EDM.pdf

Tempelaar, D. T., Cuypers, H., Van de Vrie, E., Heck, A.

and Van der Kooij, H. (2013). Formative Assessment

and Learning Analytics. In D. Suthers and K. Verbert

(Eds.), Proceedings of the 3rd International

Conference on Learning Analytics and Knowledge,

205-209. New York: ACM. DOI:

10.1145/2460296.2460337.

Tempelaar, D. (2017). How dispositional Learning

Analytics helps understanding the worked-example

principle. In D. G. Sampson, J. M. Spector, D.

Ifenthaler and P. Isaias (Eds.), Proceedings 14th

International Conference on Cognition and

Exploratory Learning in Digital Age (CELDA 2017),

pp. 117-124. International Association for

Development of the Information Society, IADIS Press.

Tempelaar, D. T., Gijselaers, W.H., Schim van der Loeff,

S. and Nijhuis, J.F.H. (2007). A structural equation

model analyzing the relationship of student

achievement motivations and personality factors in a

range of academic subject-matter areas, Contemporary

Educational Psychology, 32(1), 105-131. DOI:

10.1016/j.cedpsych.2006.10.004.

Tempelaar, D. T., Rienties, B., Mittelmeier, J. and Nguyen,

Q. (2018). Student profiling in a dispositional learning

analytics application using formative assessment,

Computers in Human Behavior, 78, 408-420. DOI:

10.1016/j.chb.2017.08.010.

Tempelaar, D. T., Rienties, B. and Giesbers, B. (2015). In

search for the most informative data for feedback

generation: Learning Analytics in a data-rich context.

Computers in Human Behavior, 47, 157-167. DOI:

10.1016/j.chb.2014.05.038.

Tempelaar, D. T., Rienties, B. and Nguyen, Q. (2017a).

Towards actionable learning analytics using

dispositions. IEEE Transactions on Education, 10(1),

6-16. DOI:10.1109/TLT.2017.2662679.

Tempelaar, D. T., Rienties, B. and Nguyen, Q. (2017b).

Adding dispositions to create pedagogy-based Learning

Analytics. Zeitschrift für Hochschulentwicklung,

ZFHE, 12(1), 15-35.

Vermunt, J. D. (1996). Metacognitive, cognitive and

affective aspects of learning styles and strategies: A

phenomenographic analysis, Higher Education, 31, 25–

50. DOI: 10.1007/BF00129106.

Analysing the Use of Worked Examples and Tutored and Untutored Problem-Solving in a Dispositional Learning Analytics Context