Bridging Skills and Scenarios: Initial Steps Towards Using Faded

Worked Examples as Personalized Exercises in Vocational Education

Torben Soennecken

, Anan Sch

utt

, Bj

orn Petrak

and Elisabeth Andr

Human-Centered Artiﬁcial Intelligence, University of Augsburg, Augsburg, Germany

{ﬁrstname.lastname}@informatik.uni-augsburg.de

Keywords:

Personalized Learning, Faded Worked Examples, Exercise Generation, Vocational Networking Education,

Scenario-Based Learning, Adaptive Educational Systems.

Abstract:

In this paper, we present a method for generating faded worked examples as personalized exercises aimed at

bridging the gap between knowledge of theoretical concepts and their application in the real world, which is

particularly important in vocational education. Previous works suggest that faded worked examples are effec-

tive learning material that can also adapt to learners of different levels. Yet, there is no formulated method for

automatically generating faded worked examples personalized to different learners in real-time. We develop

a method for generating faded worked examples from scenarios, changing the faded positions and degree of

fading based on the targeted skills and the learner’s proﬁciency level. We evaluate our method through a

user study involving 13 computer science students from a German university, who practice speciﬁc computer

networking skills. The results indicate signiﬁcant improvement in the targeted skill over the untargeted one,

highlighting the potential of our approach in vocational education settings. Our study is an early but promising

step towards the future of personalized learning, paving the way for further research in adaptive and personal-

ized vocational training.

1 INTRODUCTION

In vocational education, the primary goal is the suc-

cessful translation of theoretical concepts to real-

world application (Hippach-Schneider et al., 2007).

Practice exercises act as vital instruments for the

transfer of learning (Newell and Rosenbloom, 1993).

To achieve effective learning outcomes from prac-

tice exercises, it is crucial to limit the engagement of

working memory in non-learning related activities, in

line with the principles of Cognitive Load Theory as

proposed by Sweller (1988, 2010). Since the working

memory has a limited capacity, it is important to keep

it focused on learning.

To address the limited working memory, Atkin-

son et al. (2003) proposed using faded worked exam-

ples as learning material. Faded worked examples, as

exercises, provide learners with incomplete solutions,

fading out speciﬁc steps for them to identify and com-

plete. The ﬂexibility in faded worked examples lies in

https://orcid.org/0009-0006-7706-1807

https://orcid.org/0009-0006-2459-719X

https://orcid.org/0000-0002-0687-9795

https://orcid.org/0000-0002-2367-162X

adjusting the extent and speciﬁc positions of fading,

enabling them to meet the unique needs of different

learners.

When working on a faded worked example based

on a real-world scenario, a learner must be able to

understand and use the various skills relevant to the

current problem state. Beyond just skill application,

deeper comprehension of the context and the ability

to exercise critical thinking are essential (Anderson

et al., 1995). Given the wide range of proﬁciency in

both skills and context, the exercises should be tai-

lored to meet the needs of the learners at all levels, al-

lowing for individual progression, as noted by Butler

et al. (2015). This underscores the need for automated

faded worked example generation.

The development of automatically generated

faded worked examples faces technical challenges.

Central to these challenges is the algorithm’s capac-

ity to intelligently select steps for fading that align

with the desired learning outcomes and to modulate

the level of fading to avoid cognitive overload. Our

proposed method adapts existing exercise generation

methods with novel techniques for customizing faded

worked examples, offering a more adaptive approach

to exercise creation and individualized fading.

Soennecken, T., Schütt, A., Petrak, B. and André, E.

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education.

DOI: 10.5220/0012565000003693

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024) - Volume 1, pages 43-53

ISBN: 978-989-758-697-2; ISSN: 2184-5026

To evaluate the effectiveness of our approach, we

carry out a user study, using the proposed method to

generate exercises in the form of faded worked ex-

amples, and evaluating whether learners learn the tar-

geted skills. We use the ﬁeld of computer networking

within vocational education as our domain of inter-

est, due to its complexity and the limited exploration

it has received in the realm of personalized education.

The ill-structured (Renkl, 2023) nature of network-

ing, with its varied solutions and reliance on expert

heuristics, makes it an exemplary ﬁeld for our pro-

posed method. In networking, a single scenario can

cover various skills, ﬁtting our method that fades parts

of a scenario to focus on developing a speciﬁc skill.

In this paper, we assess the potential of automati-

cally generating faded worked examples and their ef-

fectiveness in focusing on certain skills in practice-

oriented scenarios. For this, we present the following

contributions:

• We develop an algorithmic approach for the au-

tomated generation of networking faded worked

examples as exercises, tailored to different target

skills.

• We carry out a structured user study to evaluate

the effectiveness of the automatically generated

faded worked examples in enhancing the acqui-

sition of targeted networking skills.

2 RELATED WORKS

Designing effective learning material is a focused

goal of educational literature. To achieve this, the

Cognitive Load Theory (CLT) provides the ground-

work for understanding the limits of cognition. CLT

studies the limited capacity of the working memory

in the brain, and thus suggests that learning materials

should minimize the use of the limited working mem-

ory for non-learning related activities (Cooper, 1990).

One practical implementation following the sug-

gestions of CLT is the worked example (Sweller,

1994). In this approach, the learner is presented with a

fully worked-out solution to a practice problem rather

than being tasked with solving the problem indepen-

dently (Atkinson et al., 2000). This leads to en-

hanced schema construction, equipping learners with

the ability to classify various problem structures and

efﬁciently select relevant problem-solving strategies,

aligning with established theories on schema induc-

tion and analogical transfer (Sweller et al., 1998; Gick

and Holyoak, 1983; Spencer and Weisberg, 1986).

When a beginner starts out on a task, worked ex-

amples have been demonstrated to be beneﬁcial, as

they contribute to managing cognitive load and fa-

cilitate learning (Paas and van Gog, 2006; Kirschner

et al., 2006). For experts, however, worked examples

might present redundant information, thereby hin-

dering rather than promoting learning (Renkl et al.,

2004). This difference in the effectiveness of in-

structional techniques between beginners and experts

is known as the expertise reversal effect (Kalyuga,

2009) and calls for the personalization of the learn-

ing material.

As learners gain expertise, one should transition

from fully worked-out examples to examples that

offer less guidance, eventually leading the learner

to solve problems independently. This approach,

known as faded worked examples, involves pro-

gressively reducing the number of provided solution

steps (Atkinson et al., 2003). Research has demon-

strated that such a fading technique enhances learning

outcomes (Renkl et al., 2004) by providing the guid-

ance that is required for a learner to allow effective

learning. Importantly, it is not the position of a step

in a solution that is signiﬁcant, but the fundamental

concepts that underpin that step, with learners learn-

ing most about the principles that were faded (Renkl

et al., 2004). The possible variety of fading a worked

example makes it suitable for personalization.

Effectively implementing faded worked examples

as personalized exercises requires an automated sys-

tem for constructing a diverse array of such exercises.

Early initiatives in automated exercise generation in-

clude the work of Sadigh et al. (2012), who proposed

a system for generating model-based problems using

mutation operators from generalized templates. This

technique of problem generation, which correlates the

complexity of the problem with the number of muta-

tion operators, has been a guiding principle for our

method.

Andersen et al. (2013) built upon this by using

procedural programs to generate solvable domain-

speciﬁc problems and characterizing “traces” as a

proxy for difﬁculty. Later, Butler et al. (2015) in-

troduced a system capable of generating exercises

for composite concepts formed by combining funda-

mental concepts based on existing solution features.

The techniques of Andersen et al. (2013) and Butler

et al. (2015) innovatively automate learner progres-

sion within procedural and non-procedural domains.

Unlike generating real-time, personalized exercises,

these approaches are designed for learners to progress

through a set of static problems.

Waldmann (2014) and

Abrah

am et al. (2023) in-

troduced methods for auto-generating solvable ex-

ercises in constraint programming, permitting auto-

matic assessment and grading. Our approach inte-

CSEDU 2024 - 16th International Conference on Computer Supported Education

grates the emphasis on guaranteeing solvability from

these methods. However, the methods are limited to

their domain and are therefore incompatible with the

demands of networking exercises.

In the domain of exercise generation, the reverse

solution generation methodology has been previously

highlighted. Taylor and Parberry (2011) employed

this approach in multi-solution puzzles, while Ahmed

et al. (2013) applied it within the ﬁeld of natural

deduction to ensure problem solvability. Informed

by these studies, our proposed exercise generation

method adopts the reverse generation technique, em-

phasizing its potential to guarantee solvability in the

generated exercises.

Further advancements in the ﬁeld include the work

of Srivastava and Goodman (2021), and Cui and

Sachan (2023), who contributed to the development

of algorithms for the automatic generation of per-

sonalized textual problems, primarily demonstrated in

language translation tasks. Nonetheless, these meth-

ods lack a robust domain model, leading to the poten-

tial generation of invalid problems, which is crucial

to avoid when exercises are automatically served to

learners.

In contrast, Polozov et al. (2015) introduced an ap-

proach that incorporates instructional (relating to tu-

tor requirements) and individual aspects (relating to

learner requirements), effectively preventing the cre-

ation of invalid problems by integrating a robust do-

main model during mathematical problem generation.

Smith et al. (2013) carried out one of the most com-

pelling studies on the generation of mathematical puz-

zles, a task that shares signiﬁcant similarities with

practical vocational tasks in terms of an open solu-

tion space. They not only ensured the solvability of

the generated problems but also maintained solution

features (like preventing undesired shortcuts) over the

entire solution space.

The methods we introduced serve as a basis for

exercise generation, yet they need to be modiﬁed to

work with real-world networking scenarios. Several

methodologies are tailored to speciﬁc domains, mak-

ing them unsuitable for computer networks, our area

of interest (e.g. Waldmann (2014), Taylor and Par-

berry (2011)). Notably, some methods (Srivastava

and Goodman, 2021; Cui and Sachan, 2023) even risk

producing invalid problems. Additionally, the com-

putational demands of other approaches (e.g. Ander-

sen et al. (2013), Butler et al. (2015)) hinder on-the-

ﬂy generation of personalized exercises. These chal-

lenges present an opportunity for continued research

and advancement in generating networking exercises

for vocational education, particularly through the use

of faded worked examples for individualized learning.

3 GENERATION

In this section, we introduce our method for gener-

ating faded worked examples as exercises, following

the notion of Renkl et al. (2004) that learners learn

most about faded principles. A key feature of our

method is its adaptability, allowing exercises to ad-

just according to the learner’s evolving proﬁciency.

This adaptability positions our method as a tool for

implementing personalized, progressive instructional

design (Tetzlaff et al., 2020). The generated exercises

are designed to be open-ended, supporting multiple

valid solutions.

To describe our method, we start with a general

overview and then provide a comprehensive descrip-

tion of its structure and steps. The process begins by

selecting an appropriate scenario, which is equivalent

to a worked example, and consists of components and

corresponding conﬁguration values. The valid prop-

erties of the scenario serve as candidates for the ex-

ercise’s goals, which are selected based on the spe-

ciﬁc skills the learner needs to practice. From this

scenario, we fade out some conﬁguration values, in-

validating the goals in the process, turning a worked

example into a faded worked example. The posi-

tions to fade are determined according to the skills the

learner needs to practice. Fading is done using muta-

tion operators on the scenario, a method inspired by

Sadigh et al. (2012). To adapt to the learner, the muta-

tion operators are selected depending on the learner’s

proﬁciency. Adopting the approach by Ahmed et al.

(2013), learners should solve the exercise by adapting

the mutated scenario to fulﬁll the goals, in a sense “re-

verting” back to the original scenario, although there

are multiple ways to solve the exercise.

In this paper, we apply the proposed generation

method to networking exercises, but the framework

can also be extended to exercises with variable solu-

tion characteristics having a bounded and veriﬁable

structure, such as step-by-step programming tasks

(Kumar, 2022), RAID (Redundant Array Of Indepen-

dent Disks) design scenarios, and mathematical puz-

zles (Smith et al., 2013).

3.1 Structure

We introduce the concepts and variables that formal-

ize an exercise. An exercise, denoted by E, comprises

a mutated scenario S

′

and a set of associated, veriﬁ-

able goals G :=



, . . . , G



that the learner aims

to achieve within that scenario. The mutated scenario

′

, is the modiﬁed version of a starting scenario S,

and establishes the overarching context of the exer-

cise. A goal is a property of the scenario that the

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education

learner aims to solve, and can be veriﬁed automati-

cally, as was done in Sadigh et al. (2012). Mathemat-

ically, a goal can be seen as a function, mapping a

scenario to a truth value, indicating whether the sce-

nario fulﬁlls the goal, as noted in Equation 1. As the

goals are derived from the properties of the initial sce-

nario S, they always hold true on S, but potentially not

for the mutated scenario S

′

. A scenario includes com-

ponents C :=



, . . . ,C



and their corresponding

sets of conﬁgurations CF

i,1

, . . . ,CF

i,n

, as

written in Equation 1. Every conﬁguration CF

i, j

con-

sists of a set of conﬁguration values V

(l)

i, j

. The compo-

nents C

serve as the foundational elements of a sce-

nario, with their conﬁgurations CF

directing the inter-

action within the scenario. The conﬁguration values

are what the learner has to provide to solve an exer-

cise.

To guide the learner in terms of what to explore

and edit, a conﬁguration can be ﬁxed by the system

(locked). Conversely, other conﬁgurations are left

modiﬁable (unlocked) for the learner to edit. This de-

sign helps communicate to the learner which compo-

nents and conﬁguration values in the exercise should

be the focus of attention.

To illustrate, consider a scenario that describes a

computer network. Here, a component might repre-

sent a switch, a router, or a computer. The address-

ing conﬁguration for a computer, for instance, might

specify the IPv4 address of the computer along with

its associated subnet mask. An exemplary goal could

be ensuring that two computers can communicate via

IPv4 with each other.

i, j



(1)

i, j

, . . . ,V

i, j

)

i, j



i,1

, . . . ,CF

i,n

S :=



⟨

,CF

⟩

, . . . ,



,CF



G :=



, . . . , G



, G

(S) ∈

{

T, F

}

E :=



′

, G



(1)

3.2 Generation Parameters

To generate an exercise tailored to the skills a learner

needs to practice, we supply the necessary generation

parameters. Given the need to practice a certain type

of scenario T and set of skills K :=

{

, . . . , k

}

. Our

method incorporates two types of proﬁciency: skill

proﬁciency P (k

) ∈ [0, 100] and scenario type proﬁ-

ciency P (T ) ∈ [0, 100]. Skill proﬁciency indicates

a learner’s foundational understanding of a particu-

lar skill, encompassing both declarative knowledge

and the ability to apply it in isolation. In the context

of networking, proﬁcient learners can respond accu-

rately to questions such as “What is an IPv4 address?”

or “What is a valid IPv4 address for a computer within

the subnet 192.168.0.0/24?”. In addition to being pro-

ﬁcient in a skill, the learner must also recognize where

the skill should be applied to solve a scenario-based

exercise (Renkl et al., 1994). This contextual under-

standing is reﬂected by the scenario type proﬁciency.

For instance, within networking, a learner’s scenario

type proﬁciency would enable them to address prob-

lems like “In the presented network, what changes are

required for two speciﬁc computers to be able to com-

municate with each other?”. Hence, by integrating

both scenario type and skill proﬁciency in the gen-

eration process, we aim to create exercises that not

only reinforce speciﬁc skills but also promote the in-

tegrated application of these skills within complex, re-

alistic scenarios.

To relate skills to the exercise and the structure we

use, we associate a goal with the set of skills that are

required for a learner to address that goal. We also

associate the individual conﬁgurations with the skills

that are required to understand that conﬁguration.

Networking often involves tasks like ensuring,

preventing, or modifying component reachability.

This property can be examined at different network-

ing layers (Zimmermann, 1980), each necessitating

unique skills and perspectives. With this in mind, the

goals can be changed to match the skills the learner

should practice. A network exercise could require

knowledge about virtual network trafﬁc separation us-

ing VLANs (IEEE, 2018), conﬁguration of addresses,

or static routes. Depending on the supplied skills K

and the learner’s proﬁciency P (k

), conﬁgurations of

different components will be faded, as shown in Fig-

ure 1.

Figure 1: Fading directs the exercise to certain skills. The

same starting scenario can be faded in different ways to ac-

commodate different requirements. The faded components

are the ones that are mutated from the original scenario in

terms of their conﬁgurations, which correspond to certain

skills, and highlighted in the ﬁgure.

CSEDU 2024 - 16th International Conference on Computer Supported Education

3.3 Generation Steps

3.3.1 Selecting Scenario

Before the generation process, we have a list of pre-

deﬁned scenarios available. We select a scenario to

base the exercise on by considering the set of skills

K the learner should practice. The scenario needs to

contain this set of skills, which is queried with Q

(K).

We deﬁne skills as being contained within a scenario

by determining if there are goals that address these

skills, as shown in Equation 2. What skills a goal ad-

dresses is pre-deﬁned based on its type and can be

queried with Q(G

). If there are many suitable sce-

narios, we choose an arbitrary one.

(S, K) :=

{

| (Q (G

) ∩ K) ̸=

}

(K) :=

{

S | Q

(S, K) ̸=

}

(2)

3.3.2 Selecting Goals

After selecting the scenario S of type T and its set of

supported goals, we set the number of goals included

within an exercise to match the learner’s scenario type

proﬁciency P (T ). Within our method, we approxi-

mate that the number of goals is correlated with the

complexity of an exercise. We ﬁnd this approxima-

tion is suitable in our type of exercises, because new

goals increase the criteria the learner needs to con-

sider before ﬁlling in conﬁguration values. We select

a suitable number of goals depending on the scenario

type proﬁciency, as shown in Equation 3. If there are

many suitable goal combinations, an arbitrary one is

chosen from Q

(S, K, P (T )).

(P(T )) =











1 P(T ) ∈ [0, 50]

2 P(T ) ∈ (50, 75]

3 P(T ) ∈ (75, 100]

(S, K, P (T )) :=

{

G ⊆ Q

(S, K) | |G| = n

(P(T ))

}

(3)

3.3.3 Determining Faded Positions

As a next step, to determine the faded positions, we

consider the scenario S and the skills to be practiced

K along with their respective proﬁciencies P (k

). We

determine potential pairs of conﬁgurations and com-

ponents F (S, K) for fading based on the set of skills

associated with a conﬁguration Q (CF

i, j

), as described

in Equation 4.

F(S, K) =



,CF

i, j





⟨

,CF

⟩

∈ S

∧CF

i, j

∈ CF

∧ Q (CF

i, j

) ⊆ K



(4)

Currently, we determine all potentially faded po-

sitions F (S, K) as actually faded positions.

3.3.4 Mutation

Finally, we obtain a mutated scenario S

′

by mutating

the positions determined for fading F (S, K) with the

mutation operator M, therefore constructing a chal-

lenging exercise E =

⟨

′

, G

⟩

. To mutate a conﬁgura-

tion, we choose M based on the learner’s proﬁciency

level within the skills that are required by that conﬁg-

uration.

In our study, we have three mutation operators,

, M

, and M

∞

, that remove either one, two, or all

values from the conﬁguration, respectively. The mu-

tation M to use at a position



,CF

i, j



∈ F(S, K) is

selected from these three, depending on the learner’s

lowest proﬁciency level among all the relevant skills

i, j

= min

k∈Q(CF

i, j

)

(P(k)), as described in Equation

M =











i, j

∈ [0, 50]

i, j

∈ (50, 75]

∞

i, j

∈ (75, 100]

(5)

Importantly, the mutations are designed such that

the learner can reverse them, namely by ﬁlling in the

correct conﬁguration values, which means that the ex-

ercise is guaranteed to be solvable after the mutation.

Following this, our method unlocks all the conﬁgura-

tions that were mutated, so that the learner can ﬁll in

these values. The conﬁguration values that have not

been changed remain locked so that they are visible

but cannot be changed by the learner.

3.4 Example Generation

Consider a learner needing to practice skills related to

the conﬁguration of addresses within a network setup.

For this purpose, the system chooses a network sce-

nario with a router connected to two computers, as

illustrated in Figure 2.

Figure 2: Diagram depicting the initial state of a network

scenario.

Upon scenario selection, speciﬁc exercise goals

are formulated, informed by the valid properties of

the chosen scenario. This formulation is visualized in

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education

Figure 3. Two goals are selected, calculated from the

learner’s current scenario type proﬁciency of 60.

Properties

PC 2 reaches

Router 1 on Layer 3

addressing,

static-routing

PC 1 reaches

Router 1 on Layer 3

addressing,

static-routing

Router 1 reaches

PC 2 on Layer 3

addressing,

static-routing

Router 1 reaches

PC 1 on Layer 3

addressing,

static-routing

PC 2 reaches

PC 1 on Layer 3

addressing,

static-routing

PC 1 reaches

PC 2 on Layer 3

addressing,

static-routing

Goals

Make PC 1 reach

PC 2 on Layer 3

addressing

Make PC 2 reach

PC 1 on Layer 3

addressing

Skills

addressing

Scenario Type Proﬁciency

65 out of 100

Figure 3: Diagram illustrating the formulation of exercise

goals.

After setting the goals, the system identiﬁes the

address conﬁgurations of the computer ports and

router ports for fading. Given the learner’s low com-

prehension of addresses, each conﬁguration is mu-

tated by a singular value, as showcased in Figure 4.

Lastly, the method determines the mutated conﬁgura-

tions to be unlocked.

Figure 4: Diagram illustrating a scenario where conﬁgura-

tions have been mutated, highlighted in red. As the method

is targeting the skill of address conﬁguration, the relevant

port conﬁgurations are selected for fading. To not over-

whelm the learner, only one value is mutated for each con-

ﬁguration, creating an exercise.

4 USER STUDY

The user study aims to test the efﬁcacy of our pro-

posed method of generating faded worked examples

as exercises, with a particular emphasis on its abil-

ity to facilitate the learning of a speciﬁc skill from a

shared pool of scenarios that support multiple skills.

We use 11 scenarios throughout our study, all of

which are considered the same type of scenario, as

they share a similar, basic topology consisting of one

to two computer subnetworks. The study focuses on

two fundamental networking skills: IPv4 static rout-

ing and VLAN conﬁguration. Each participant is as-

signed to practice one of these skills.

4.1 Research Question

The primary question guiding this user study is:

“Does our method for automatically generating faded

worked examples as exercises, tailored to speciﬁc

skills, enhance learning the trained skills compared

to non-trained skills?”.

4.2 Procedure

Introductory contents

Pre-test

Practice ipv4 in exercises

ipv4 goals

ipv4 vlan

fade

Practice vlan in exercises

vlan goals

ipv4 vlan

fade

Practice phase

Post-test

Figure 5: Diagram showing the stages of the user study.

Learners in different conditions practice with different prac-

tice exercises, where the goals are focused on the trained

skill and the conﬁguration values associated with the trained

skill are faded.

The study is conducted on-site at the university cam-

pus. The steps of the user study are depicted in Fig-

ure 5. Before participating, participants completed

a consent form. The study begins with introductory

content, which is a brief overview of computer net-

working skills, reinforcing the knowledge they have

previously acquired in their education. Subsequently,

they took a paper-based pre-test consisting of two ex-

ercises for each skill. These exercises permit partial

correctness, for instance, awarding one point for each

accurately identiﬁed value within a static routing en-

try. After the pre-test, participants engage in a 30-

minute learning session using the provided platform,

with access to written instructions about the introduc-

tory content. Following the learning phase, the partic-

CSEDU 2024 - 16th International Conference on Computer Supported Education

ipants take a post-test, which encompasses the same

topics and skills as the pre-test but with different ques-

tions. During the whole user study, participants are

not allowed to pose content-related questions to the

experimenters.

4.3 Learning Platform

The central component of the user study is the learn-

ing platform, which supplies networking exercises for

participants to practice. The platform is designed as a

distributed application, accessed via participants’ web

browsers. The network is depicted as a graph with

components as vertices. Participants can modify the

conﬁguration of each component by selecting it and

entering values. The connections, on the other hand,

remain ﬁxed. To visualize the learning platform, we

present two screenshots that show important user in-

teractions within the application.

The ﬁrst screenshot (Figure 6) showcases the ap-

plication presenting two goals that require IPv4 ad-

dress conﬁguration and IPv4 static routing for a valid

answer. Although VLAN is also part of the scenario,

it is locked, indicating that it is not required for the

current exercise. This is shown by a grey lock on the

vertex and dashed lines around the ports within the

network plan.

The second screenshot (Figure 7) demonstrates

how the application presents the assessment of sub-

missions. After a learner submits a solution attempt

for an exercise, the learning platform provides feed-

back on their performance by adjusting the color of

the goals to red or green to indicate incorrect or cor-

rect conﬁgurations. Because of the open-ended de-

sign of the exercises, we assess the correctness by de-

termining which goals are fulﬁlled, and which are not.

As outlined in Equation (1), our goals are properties

of the scenario and, therefore, can be used for an au-

tomated assessment. This method allows for partial

correctness: solving some goals but failing others.

4.4 Conditions

The study has two conditions. Each condition has a

trained skill, either IPv4 static routing or VLAN con-

ﬁguration, with the other skill referred to as the un-

trained skill. Exercises generated for both conditions

originate from a shared pool of scenarios. The trained

skill is incorporated during the generation process for

goal selection and conﬁguration mutation, as shown

in Figure 5. Throughout this process, all conﬁgura-

tions associated with a target skill are completely re-

set. The following parameters are used to generate

exercises for participants based on their condition:

{

ipv4-static-routing

}

{

vlan-conﬁguration

}

P(k

) = 100

P(network-plan) = 75

(6)

4.5 Measured Variables

The normalized learning gain (NLG) of students was

measured based on the relative scores obtained from

the pre- and post-tests, as described in Equation 7

(Marx and Cummings, 2007).

NLG =











Post − Pre

Full − Pre

if Post > Pre

Post − Pre

Pre

if Post ≤ Pre

(7)

where Pre and Post are the scores from pre- and post-

test, Full is the full score on the test, and NLG is

the normalized learning gain, respectively. The pre-

and post-test each contain questions related to the two

mentioned skills. We sum the scores from each skill

in the test, and calculate a separate NLG for each skill.

4.6 Participants

For the goals of the user study, we require the partici-

pants to have basic background in computer science

and networking. Accordingly, we recruit 13 com-

puter science students from a German University by

approaching them on campus, and giving them a short

description of the experiment. The exercises within

the user study are relevant to their academic curricu-

lum, offering an inherent advantage to their participa-

tion. Participants were not paid for their participation.

5 RESULTS

We show the measured normalized learning gains,

separated between the trained and untrained skill, in

Figure 8. For the statistical evaluation, we employed

the one-tailed paired t-test, with the alternative hy-

pothesis that the NLG of the trained skill is higher

than the untrained skill. The decision to use the one-

tailed paired t-test was supported by the outcomes of

a Shapiro-Wilk test, which indicated the normality

of the data distribution (p = .118 for trained skills,

p = .379 for untrained skills). There was a signif-

icant difference in the normalized learning gain be-

tween the trained (M = 0.50, SD = 0.37) and un-

trained skills (M = 0.14, SD = 0.39); t(12) = 2.31,

p = .020, with a large effect size as indicated by

Cohen’s d = 0.92.

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education

Figure 6: Screenshot showing the application with two goals on the top right corner, addressing the skills IPv4 address

conﬁguration and IPv4 static routes. Here, the learner is adding a default gateway to the routing table of the computer “PC 1”.

Figure 7: Screenshot displaying the assessment of a learner’s submission. The goals are color-coded (red or green) to indicate

incorrect or correct conﬁgurations, respectively, with corresponding icons.

5.1 Observations

In addition to the primary results, we explored the

potential of our methods for real-time exercise gen-

eration. While formal time measurements were not

conducted, the distributed system demonstrated efﬁ-

ciency by delivering a newly generated exercise to the

web browser of participants within a delay of mere

seconds. Additionally, participants reported instances

of participant frustration in the conversation following

the experiment, which stemmed from the high difﬁ-

culty level of some exercises or the absence of explicit

hints during the problem-solving phase.

6 DISCUSSION

In this paper, we proposed and evaluated an au-

tomated faded worked example generation system

aimed at enhancing speciﬁc vocational skills. The

empirical evidence from our study showed a signiﬁ-

cant difference in learning gains between trained and

untrained skills, which indicates the system’s ability

to generate exercises for different skills from a shared

pool of scenarios. This aligns with prior research by

Renkl et al. (2004) regarding the skill enhancement at

the faded positions of a worked example, and furthers

the evidence to the domain of networking.

The real-time generation capability of our sys-

tem, as evidenced during the study, makes it suitable

for serving personalized exercises matching the dy-

CSEDU 2024 - 16th International Conference on Computer Supported Education

Figure 8: Boxplot showing the Normalized Learning Gain

of the trained (faded in exercise) and untrained (unfaded in

exercise) skills. Each condition has one trained skill, and

one untrained skill. This plot combines the results from

both conditions.

namic learner’s needs. Notably, the system’s ability

to tailor exercises from a common pool of scenarios

for varied skills highlights its adaptability and cost-

effectiveness.

However, our observations also revealed chal-

lenges faced by participants, notably frustration and

difﬁculty. This suggests a need to integrate more ex-

plicit instructional guidance, in line with Clark et al.

(2012). Presenting efﬁciently structured and fully-

explained worked examples before fading, as ex-

tensively researched by Sweller and Cooper (1985);

Cooper and Sweller (1987); Ward and Sweller (1990),

would provide a more structured learning pathway,

especially for beginners. Another step towards an efﬁ-

ciently structured worked example could be including

the rationale behind the provided solution steps (van

Gog et al., 2004).

Additionally, providing corrective feedback

mechanisms, for example as hints, as proposed by

Shute (2008) and Stamper et al. (2013), would further

support learners, especially when they encounter

hurdles.

6.1 Limitations

There are two limitations to our work. First, the user

study only contained 13 participants, which means

that the effect size would be better estimated with

a larger population. Second, the study was carried

out on-site and with experimenters. This might have

caused the learners’ behavior to be different than in

self-study, which would be the main setting of such

learning platforms.

6.2 Future Work

Future research should focus on reﬁning the selec-

tion of fading positions in exercises by considering

the steps in the solution path, rather than consider-

ing single conﬁguration values alone. This is because

even though the conﬁguration values that are mutated

can be attributed to some skills, the steps in a solu-

tion path can provide additional information. To more

closely relate to a real use case, our exercise genera-

tion approach should be coupled with knowledge trac-

ing (Abdelrahman et al., 2023), so that the learner’s

proﬁciency levels can be determined and correspond-

ing skills can be targeted by practice exercises. Focus

on explicit instructional support, direct feedback, and

hinting mechanisms would further enrich the learn-

ing process. First steps towards the automatic genera-

tion of feedback are given by O’Rourke et al. (2019),

who utilized the problem-generation model encoded

in Answer Set Programming to guide the learner. Fi-

nally, exploring the system’s transferability to other

educational domains and quantifying the effort re-

quired for such adaptations will be crucial for its

broader application.

7 CONCLUSION

This paper outlines a new method for generating vo-

cational education exercises based on practical sce-

narios and the concept of faded worked examples.

The contributions put forth are twofold: an algorith-

mic approach for generating faded worked examples

as exercises tailored to speciﬁc learner skills and a

user study conducted within the computer network-

ing domain to evaluate the approach. The prelimi-

nary results from the user study indicate potential ef-

fectiveness in enhancing the trained skills, but further

enhancements are needed in guidance and feedback,

along with broader domain testing.

ACKNOWLEDGEMENTS

We thank Luis Schweigard and Felix Rinderer for

their support in implementing and carrying out the

user study. This work was supported by using

ChatGPT-4, a language model developed by OpenAI,

assisting in the formulation of some parts of the text to

enhance the readability of the paper (OpenAI, 2023).

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education

REFERENCES

Abdelrahman, G., Wang, Q., and Nunes, B. (2023). Knowl-

edge tracing: A survey. ACM Computing Surveys,

55(11):1–37.

Abrah

am, E., Nalbach, J., and Promies, V. (2023). Auto-

mated Exercise Generation for Satisﬁability Check-

ing. In Formal Methods Teaching, pages 1–16.

Springer Nature Switzerland.

Ahmed, U. Z., Gulwani, S., and Karkare, A. (2013). Auto-

matically Generating Problems and Solutions for Nat-

ural Deduction. In Twenty-Third International Joint

Conference on Artiﬁcial Intelligence.

Andersen, E., Gulwani, S., and Popovic, Z. (2013). A

Trace-based Framework for Analyzing and Synthesiz-

ing Educational Progressions. In Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems. ACM.

Anderson, J. R., Corbett, A. T., Koedinger, K. R., and Pel-

letier, R. (1995). Cognitive Tutors: Lessons Learned.

Journal of the Learning Sciences, 4(2):167–207.

Atkinson, R. K., Derry, S. J., Renkl, A., and Wortham, D.

(2000). Learning from Examples: Instructional Prin-

ciples from the Worked Examples Research. Review

of educational research, 70(2):181–214.

Atkinson, R. K., Renkl, A., and Merrill, M. M. (2003).

Transitioning From Studying Examples to Solving

Problems: Effects of Self-Explanation Prompts and

Fading Worked-Out Steps. Journal of Educational

Psychology, 95(4):774–783.

Butler, E., Andersen, E., Smith, A. M., Gulwani, S., and

Popovi

c, Z. (2015). Automatic Game Progression De-

sign through Analysis of Solution Features. In Pro-

ceedings of the 33rd Annual ACM Conference on Hu-

man Factors in Computing Systems. ACM.

Clark, R., Kirschner, P. A., and Sweller, J. (2012). Putting

Students on the Path to Learning: The Case for Fully

Guided Instruction. American Educator, 36(1):5–11.

Cooper, G. (1990). Cognitive load theory as an aid for

instructional design. Australasian Journal of Educa-

tional Technology, 6(2).

Cooper, G. and Sweller, J. (1987). Effects of schema acqui-

sition and rule automation on mathematical problem-

solving transfer. Journal of Educational Psychology,

79(4):347–362.

Cui, P. and Sachan, M. (2023). Adaptive and Personalized

Exercise Generation for Online Language Learning.

Gick, M. L. and Holyoak, K. J. (1983). Schema Induc-

tion and Analogical Transfer. Cognitive Psychology,

15(1):1–38.

Hippach-Schneider, U., Krause, M., and Woll, C. (2007).

Berufsbildung in Deutschland: Kurzbeschreibung.

Ofﬁce for Ofﬁcial Publications of the Europ. Commu-

nities.

IEEE (2018). IEEE Standard for Local and Metropoli-

tan Area Network–Bridges and Bridged Networks.

IEEE Std 802.1Q-2018 (Revision of IEEE Std 802.1Q-

2014), pages 1–1993.

Kalyuga, S. (2009). The Expertise Reversal Effect. In Man-

aging Cognitive Load in Adaptive Multimedia Learn-

ing, pages 58–80. IGI Global.

Kirschner, P. A., Sweller, J., and Clark, R. E. (2006).

Why Minimal Guidance During Instruction Does Not

Work: An Analysis of the Failure of Constructivist,

Discovery, Problem-Based, Experiential, and Inquiry-

Based Teaching. Educational Psychologist, 41(2):75–

86.

Kumar, A. N. (2022). Solvelets: Tutors to Practice the Pro-

cess of Programming. In Proceedings of the 27th ACM

Conference on on Innovation and Technology in Com-

puter Science Education Vol. 1, pages 151–157.

Marx, J. D. and Cummings, K. (2007). Normalized change.

American Journal of Physics, 75(1):87–91.

Newell, A. and Rosenbloom, P. (1993). Mechanisms of skill

acquisition and the law of practice. Cognitive Skills

and Their Acquisition, Vol. 1.

OpenAI (2023). ChatGPT-4. Large language model.

O’Rourke, E., Butler, E., Tolentino, A. D., and Popovi

c, Z.

(2019). Automatic Generation of Problems and Ex-

planations for an Intelligent Algebra Tutor. In Lecture

Notes in Computer Science, pages 383–395. Springer

International Publishing.

Paas, F. and van Gog, T. (2006). Optimising worked exam-

ple instruction: Different ways to increase germane

cognitive load. Learning and Instruction, 16(2):87–

91.

Polozov, O., O’Rourke, E., Smith, A. M., Zettlemoyer, L.,

Gulwani, S., and Popovi

c, Z. (2015). Personalized

Mathematical Word Problem Generation. In Twenty-

Fourth International Joint Conference on Artiﬁcial In-

telligence.

Renkl, A. (2023). Using worked examples for ill-structured

learning content. Their Own Words: What Scholars

and Teachers Want You To Know About Why and How

To Apply the Science of Learning in Your Academic

Setting, pages 207–224.

Renkl, A., Atkinson, R. K., and Große, C. S. (2004). How

Fading Worked Solution Steps Works – A Cognitive

Load Perspective. Instructional Science, 32(1/2):59–

82.

Renkl, A., Gruber, H., Mandl, H., and Hinkhofer, L. (1994).

Hilft Wissen bei der Identiﬁkation und Kontrolle eines

komplexen

okonomischen Systems?

Sadigh, D., Seshia, S. A., and Gupta, M. (2012). Automat-

ing Exercise Generation: A Step towards Meeting the

MOOC Challenge for Embedded Systems. In Pro-

ceedings of the Workshop on Embedded and Cyber-

Physical Systems Education, WESE ’12, New York,

NY, USA. Association for Computing Machinery.

Shute, V. J. (2008). Focus on Formative Feedback. Review

of educational research, 78(1):153–189.

Smith, A. M., Butler, E., and Popovic, Z. (2013). Quantify-

ing over Play: Constraining Undesirable Solutions in

Puzzle Design. In FDG, pages 221–228.

Spencer, R. M. and Weisberg, R. W. (1986). Context-

dependent effects on analogical transfer. Memory &

Cognition, 14(5):442–449.

CSEDU 2024 - 16th International Conference on Computer Supported Education

Srivastava, M. and Goodman, N. (2021). Question Gener-

ation for Adaptive Education. In Proceedings of the

59th Annual Meeting of the Association for Compu-

tational Linguistics and the 11th International Joint

Conference on Natural Language Processing (Volume

2: Short Papers). Association for Computational Lin-

guistics.

Stamper, J., Eagle, M., Barnes, T., and Croy, M. (2013). Ex-

perimental Evaluation of Automatic Hint Generation

for a Logic Tutor. International Journal of Artiﬁcial

Intelligence in Education, 22(1-2):3–17.

Sweller, J. (1988). Cognitive Load During Problem

Solving: Effects on Learning. Cognitive Science,

12(2):257–285.

Sweller, J. (1994). Cognitive load theory, learning difﬁculty,

and instructional design. Learning and Instruction,

4(4):295–312.

Sweller, J. (2010). Element interactivity and intrinsic, ex-

traneous, and germane cognitive load. Educational

Psychology Review, 22(2):123–138.

Sweller, J. and Cooper, G. A. (1985). The Use of

Worked Examples as a Substitute for Problem Solv-

ing in Learning Algebra. Cognition and Instruction,

2(1):59–89.

Sweller, J., van Merrienboer, J. J. G., and Paas, F. G. W. C.

(1998). Cognitive Architecture and Instructional De-

sign. Educational Psychology Review, 10(3):251–

296.

Taylor, J. and Parberry, I. (2011). Procedural Generation of

Sokoban Levels. In Proceedings of the International

North American Conference on Intelligent Games and

Simulation, pages 5–12.

Tetzlaff, L., Schmiedek, F., and Brod, G. (2020). Develop-

ing Personalized Education: A Dynamic Framework.

Educational Psychology Review, 33(3):863–882.

van Gog, T., Paas, F., and van Merri

enboer, J. J. (2004).

Process-oriented worked examples: Improving trans-

fer performance through enhanced understanding. In-

structional Science, 32(1/2):83–98.

Waldmann, J. (2014). Automated Exercises for Constraint

Programming. In WLP/WFLP, pages 66–80.

Ward, M. and Sweller, J. (1990). Structuring Effective

Worked Examples. Cognition and Instruction, 7(1):1–

39.

Zimmermann, H. (1980). OSI Reference Model–The ISO

Model of Architecture for Open Systems Intercon-

nection. IEEE Transactions on Communications,

28(4):425–432.

Bridging Skills and Scenarios: Initial Steps Towards Using Faded Worked Examples as Personalized Exercises in Vocational Education