Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses

in Multimodal Immersive Learning Environments

Bibeg Limbu

and Irene-Angelica Chounta

Department of Human-centered Computing and Cognitive Science, University of Duisburg-Essen, Germany

{bibeg.limbu, irene-angelica.chounta}@uni-due.de

Keywords:

Multimodal Learning, Handwriting, Haptic Senses, Embodied Learning.

Abstract:

This study explores the role of motor activity and tactile perception in Multimodal Immersive Learning En-

vironments (MILEs) within the context of handwriting. A 2x2 factorial experimental design was used to

investigate the impact of the intensity of motor activity and the sensitivity of tactile perception on learning

performance, measured as memory recall. Mental effort and perceived workload were monitored as the me-

diating variables. Participants (N=20) completed a handwriting task, that is, copying text displayed on a

prompter using a tablet and a stylus. During the task, the participants used additional pressure to increase

the intensity of the motor activity and/or wore gloves to reduce the sensitivity of tactile perception. Results

indicate no signiﬁcant effect of either manipulation on recall, mental effort, or perceived workload. This may

suggest that integrating supplementary haptic feedback technologies in MILEs does not impose additional

cognitive load or obstruct learning. The ﬁndings contribute to the design of MILEs by informing the effective

integration of wearable sensors to support authentic practice for skill acquisition. The study can inform future

research that explores the effects of haptic senses and their broader applications in other learning contexts,

contributing to a deeper understanding of embodied learning and dual-coding theory.

1 INTRODUCTION

As multimodal immersive technologies mature, au-

thentic computer-supported learning environments

in education have become increasingly accessible

(Di Mitri et al., 2024). These environments facilitate

authentic practice, which simulates realistic contexts

that reﬂect the way knowledge will be used in real

life (Di Mitri et al., 2022; Horz, 2012). Multimodal

immersive learning environments (MILEs), supported

by technologies such as sensors and mixed reality, en-

able authentic practice by engaging all the learners’

senses, akin to the realistic contexts in the real world

(Specht et al., 2019; Limbu et al., 2018). Schneider

et al. (Schneider et al., 2019) used virtual reality to

support the authentic practice of presentation skills

by simulating the audience. In addition, sensors and

actuators in MILEs enable the augmentation of expe-

rience, which can be used to provide in-situ learning

support (Meik et al., 2021) required for the authen-

tic practice of complex skills. For example, Limbu et

al. (Limbu et al., 2019) used sensors to augment the

https://orcid.org/0000-0002-1269-6864

https://orcid.org/0000-0001-9159-0664

pen strokes with different colors to provide supple-

mentary feedback on handwriting pressure. With the

growing use of MILEs-supported authentic practice

in acquiring complex skills (Taguma and Frid, 2024),

questions about how multimodality affects learning

emerge; for example, in computer-supported authen-

tic practice with virtual reality, where the learning ex-

perience can be manipulated, how does the ampliﬁca-

tion or diminishing of senses, such as haptic senses,

affect learning?

Related research (Giannakos and Cukurova, 2023;

Lee et al., 2023; Limbu et al., 2022) argues for the ef-

ﬁcacy of multimodal and immersive technologies to

support cognitive learning theories, such as Cognitive

Load Theory and embodied learning. The Cognitive

Load Theory (Sweller et al., 2011) assumes that learn-

ing imposes mental effort or cognitive load. Cognitive

load signiﬁcantly impacts the learning process, that is,

the assimilation and retention of information in long-

term memory (Paul A. Kirschner and Clark, 2006).

In a speciﬁc context, information from the environ-

ment is received simultaneously through the multiple

senses or modalities. Processing this incoming in-

formation imposes cognitive load. The dual-coding

theory (Clark and Paivio, 1991) postulates that in-

Limbu, B. and Chounta, I.-A.

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments.

DOI: 10.5220/0013285600003932

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 17th International Conference on Computer Supported Education (CSEDU 2025) - Volume 1, pages 72-82

ISBN: 978-989-758-746-7; ISSN: 2184-5026

coming verbal and imagery information through the

senses is processed through distinct cognitive chan-

nels. In other words, the cognitive load imposed by

the two types of contextually related incoming infor-

mation does not impose or impose minimal additional

mental effort during the learning process. In embod-

ied learning, as is often the case in authentic prac-

tice, multiple senses, and motor activities are involved

(Clark et al., 2019). In this case, how the physical

embodiment of the learner (in the case of embodied

cognition) or the inclusion of additional senses (in the

context of classical cognition theories), such as haptic

sense/modality, impact learning is unclear. If all non-

verbal information from senses, including the haptic

sense/modality, is processed through a common chan-

nel (i.e., non-verbal) sharing the working memory as

Dual-coding theory postulates, do embodied learning

environments lead to a loss in learning performance?

However, integration of multiple senses has been

found to signiﬁcantly improve task performance in

high load conditions (Marucci et al., 2021). While

the use of additional senses may enhance learning in

speciﬁc scenarios, Vermeulen et al. (Vermeulen et al.,

2008), in contrast, also found that sensory overload

often leads to an increase in mental effort for a spe-

ciﬁc modality. The design of effective MILEs neces-

sitates an understanding of such implications of mul-

tisensory learning design. Therefore, in this paper, we

investigate the association between haptic senses and

learning via mental effort and perceived workload in

the context of handwriting. We aim to address the fol-

lowing research questions:

RQ1. How does the sensitivity of tactile percep-

tion affect learning performance?

RQ2. How does the intensity of motor activity af-

fect learning performance?

Haptic senses constitute multiple factors, such as

tactile perception and motor activities. Tactile per-

ception is the ability to sense and interpret incoming

information through touch. First, we examine the im-

pact of the sensitivity of tactile perception on learning

(RQ1). Furthermore, tactile perceptions are tightly

intertwined with motor activities. In an embodied

learning design, all learning, regardless of whether

it concerns sports or mathematics, emerges through

action, which includes motor activities (Abrahamson

and Lindgren, 2014). Notably, using motor activities

has been shown to beneﬁt students’ performance, es-

pecially in maths (Shvarts and van Helden, 2023; Li

et al., 2023). Since motor activities are indispensable

to tactile perception and can impact learning, we sec-

ondly examine the impact of the intensity of motor

activities on learning (RQ2).

In the following, we provide the background in

the context of handwriting, followed by the research

methods. Then, we provide the results, after which

we present our discussions and conclusions.

2 BACKGROUND

In this work, we explored the association between

haptic senses and learning in the context of hand-

writing. Handwriting remains a prevalent motor ac-

tivity in educational contexts and has proven beneﬁ-

cial for learning. Handwriting has been found to be

a predictor of literacy skills in primary grade students

(Skar et al., 2022; McCarroll and Fletcher, 2017) and

in Kindergarten students (Ray et al., 2022). This

highlights the importance of internalizing competent

handwriting ﬂuency and is a prerequisite for all fur-

ther handwriting-related learning beneﬁts. For exam-

ple, several studies (Mueller and Oppenheimer, 2014;

Wrigley, 2019; Flanigan et al., 2024) found that stu-

dents who took notes on laptops performed worse on

conceptual questions than students who took hand-

written notes. The authors argued that laptop note-

takers tend to transcribe lectures verbatim, which is

detrimental to learning. In contrast, handwriting ne-

cessitates processing information and reframing it in

their own words, as handwriting is slower than typing,

and students need to condense the incoming informa-

tion. This requires more mental effort, which means

the student must cognitively engage with the mate-

rial. Corroborating this, Longcamp et al., (Longcamp

et al., 2005) and Smoker et al., (Smoker et al., 2009)

argued that the brain receives multiple signals dur-

ing writing (visual, motor, and kinaesthetic), which

typing does not do in the same manner. This re-

sults in increased activation in brain regions associ-

ated with language processing, working memory, and

executive functions during handwriting tasks, which

is beneﬁcial for learning (Van der Weel and Van der

Meer, 2024). By inference, based on the Dual-coding

Theory, we can conjecture that the haptic senses dur-

ing handwriting potentially allow students to expend

more mental effort, which results in better assimila-

tion and recall of information.

2.1 Handwriting and Memory

Related research demonstrated that taking notes by

hand – instead of typing – can lead to improved mem-

ory (Smoker et al., 2009; Bouriga and Olive, 2021;

Van der Weel and Van der Meer, 2024). This may

seem counterintuitive, as the higher-level cognitive

processes involved in handwriting seem to place extra

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments

mental effort on working memory, inhibiting the abil-

ity to store information in short-term memory (Pev-

erly, 2006). Working memory holds a limited amount

of information during the assimilation and recall of

information in the learning process (Baddeley, 1990),

and requiring a student to hold more information than

possible at any given time can lead to cognitive over-

load. While handwriting demands more mental ef-

fort during the assimilation process than typing, hand-

writing leads to better short-term and long-term re-

call than typing (Bouriga and Olive, 2021). Learn-

ing strategies referred to as “Desirable Difﬁculties”

(Bjork and Bjork, 2011) slow down the learner, such

as requiring them to write with their hands, forc-

ing learners to expend more mental effort leading to

deeper levels of processing, and consequently better

memory (Craik and Tulving, 1975). Weel & Meer

(Van der Weel and Van der Meer, 2024), and James &

Engelhardt (James and Engelhardt, 2012) found that

handwriting, compared to typewriting, enhances brain

connectivity patterns that support learning. Similarly,

Ose Askvik et al. (Ose Askvik et al., 2020) found that

handwriting with a digital pen induces theta-range

synchronized activity in brain areas linked to memory

and learning, which supports encoding new informa-

tion.

2.2 Multimodal Immersive Learning

Technologies in Handwriting

Multimodal Immersive Learning Environments

(MILEs), using sensors and actuators, are capable of

supporting authentic practices (Di Mitri et al., 2022;

Limbu et al., 2018). Moreover, they are capable

of augmenting authentic practices by amplifying

or inhibiting various stimuli in the environment.

Danna and Velay (Danna and Velay, 2015) delineate

boundaries between the various types of stimuli in

authentic handwriting practice. According to the

authors, primary stimuli are naturally present in

writing, namely visual, proprioceptive feedback from

the hand, etc. The supplementary stimuli, on the

other hand, are consequences of applied technologies

and their affordances, which can potentially beneﬁt

learning (Danna and Velay, 2015; Kiefer and Velay,

2016). Loup Escande et al., (Loup-Escande et al.,

2017) and Limbu et al., (Limbu et al., 2019) have

used color gradients in their applications to provide

supplementary visual information about pen pressure

while writing. Loup Escande et al., (Loup-Escande

et al., 2017) found that the supplementary informa-

tion led to an increase in mental effort, while Limbu

et al., (Limbu et al., 2019) found no signiﬁcant

difference. It should be noted that Limbu et al.,

(Limbu et al., 2019) have limited participants to make

deﬁnite conclusions.

In summary, handwriting beneﬁts memory by ac-

tivating multiple sensory pathways, while the addi-

tional mental effort required enhances cognitive en-

gagement, fostering deeper learning. This encourages

MILE designers to use haptic senses during learning,

but it is essential to exercise caution. First, Yoshida et

al., (Yoshida et al., 2015) postulate a smaller memory

capacity for haptic senses than for visual senses. A

complex implementation of haptic stimuli can more

easily create cognitive overload. Second, while the

advantages of handwriting on paper have been well

studied, there is a lack of understanding of how those

beneﬁts will transfer to digital devices, such as in the

case of MILEs (Kiefer and Velay, 2016). In this study,

we explore the timely need to better understand the

implications of haptic senses for learning to support

the design of MILEs.

3 METHODOLOGY

To investigate our research questions, we designed

a formative study. For this study, we deﬁned haptic

sense across two dimensions: a) the sensitivity of the

tactile perception and b) the intensity of motor activ-

ity. We experimentally compared the learning per-

formance of the four groups (see Section 3.1, Table

1) with a memory-based post-test. Additionally, we

compared the mental effort and perceived workload

of the groups as the mediator variables.

3.1 Experimental Design

Table 1: Four study groups (P=Pressure, G=Glove): three

experimental groups (Group 1, 2 and 3) and one control

group (Group 4).

Study groups

Group Condition N

1 P + G 5

2 P + ¬G 6

3 ¬P + G 5

4 ¬P + ¬G 4

This study investigated the effect of speciﬁc hap-

tic senses, namely, handwriting pressure and tac-

tile perception, on learning as memory recall (Paul

A. Kirschner and Clark, 2006). The experimental de-

sign used between-subjects 2x2 factorial design (see

Table 1). The two treatment conditions were: 1. in-

tensity of motor activity implemented by additional

writing Pressure (P), and 2. inhibiting the sensitivity

of tactile perception by use of a Glove (G). The partic-

CSEDU 2025 - 17th International Conference on Computer Supported Education

ipants of Group 1 used additional pressure while writ-

ing and used a glove (P+G). In Group 2, the partic-

ipants only used additional pressure (P+¬G), Group

3 did not use additional pressure but wore gloves

(¬P+G), and the participants in Control Group 4 did

neither (¬P+¬G). Participants who were required to

use additional pressure while writing, i.e., Group 1

and 2, were informed about the distinct audio signal

that would occur to notify when pressure was below

the required threshold. This audio signal was notice-

ably different from the auditory stimuli for the sec-

ondary task (see Section 3.5.2).

3.2 Participants

Twenty (20) participants took part in the study; twelve

(12) were female, and eight (8) were male. The mean

age of the sample was 22.09 (SD=2.09). The female

participants were, on average, younger (M = 21.6, SD

= 1.83) than the male participants (M = 23.1, SD =

2.23). The participants were university students in the

Computer Science faculty and were ﬂuent in German,

as the study was conducted in German. Only right-

handed students were invited to control the variation

that may arise from the dominant hand.

3.3 Apparatus

A WACOM One™graphic tablet with a display and

complementary stylus pen was used by the partici-

pants to copy the text displayed on the adjacent mon-

itor (see Figure 1). The graphic tablet was connected

to a PC running a custom application developed to log

the pressure. It also reminded the participants in the

“P” treatment group to exert more writing pressure

when it was low. A gardening glove with minimal

impact on the pen grip was used to reduce the sensi-

tivity of the tactile perception for the group with the

“G” treatment. A keyboard was also placed next to

the participants. The participants were instructed to

react to the auditory signal associated with the sec-

ondary task (see Section 3.5.2) as fast as possible by

pressing the spacebar button on the keyboard.

3.4 Procedure

Once the participants arrived, they were briefed about

the study’s objectives and the experimental task. They

were provided with a consent form and informed

that they could withdraw from the study at any point

and they could ask questions throughout its duration.

Once the consent form was signed, the participants

received a unique identiﬁer code. This code was used

to anonymize participants’ data. The study was ap-

Figure 1: Experimental setup in which the teleprompter

(monitor) displays text that the participants copied with a

stylus pen.

proved by the ethics board of the university where the

study was conducted.

First, the participants were required to familiarise

themselves with the apparatus. Then, they were ran-

domly assigned to one of the study groups. They were

once again briefed about the experimental task, that

is, to copy the text shown in the prompter, and the

secondary task, that is, to react to the auditory stimuli

by pressing a button on the keyboard with their left

hand. The text for the experimental task that the par-

ticipants were required to copy was displayed on the

prompter sentence by sentence. Based on the treat-

ment conditions, participants were assigned to either

wear a glove, write with extra pressure, do both, or

write normally (see Section 3.1).

After completing the experimental task, partici-

pants answered a multiple-choice test that checked

their recall of the text displayed in the prompter. They

also responded to the NASA-Task Load Index (Hart,

1986) to measure their perceived workload. The se-

quence in which these two tests were administered

was randomized to reduce potential ordering effects

(for example, the recall test might inﬂuence the per-

ceived workload test and vice versa). The whole pro-

cedure lasted approximately 30 minutes.

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments

3.5 Materials and Measures

3.5.1 Perceived Workload

Perceived workload, i.e., the amount of physical and

mental effort invested, is measured using the Nasa-

TLX instrument (Hart, 1986). The NASA-TLX

evaluates perceived workload across six dimensions:

mental demand, physical demand, temporal demand,

performance, effort, and frustration. Participants rate

each dimension on a scale from 0-100, reﬂecting the

perceived intensity of each factor. A pairwise compar-

ison between the 15 pairs from 6 dimensions is then

performed in which the participant selects the most

inﬂuential dimension from the two. Based on this, the

individual weights for each dimension are calculated

and then used to calculate the overall workload.

3.5.2 Mental Effort

Additionally, mental effort is also measured using the

dual-task method. The dual-task paradigm is a be-

havioural method to estimate mental effort over time

by providing a second task in addition to the primary

task (such as reacting to auditory stimuli by pressing

a switch, (Limbu et al., 2019)) to measure a decay in

performance in the secondary (or primary) task (Es-

maeili Bijarsari, 2021). As a secondary task in this

study, participants responded to the auditory stimuli

by pressing the spacebar key on a wireless keyboard

as fast as possible. Their reaction time was logged in

milli-seconds.

3.5.3 Immediate Recall

Recall is measured using a knowledge test. The

knowledge test is administered using a multiple-

choice questionnaire, which assesses the participant’s

ability to remember and correctly recall the informa-

tion presented during the experiment. Due to the im-

mediate administration of the test following the ex-

periment, we regard the recall knowledge test as im-

mediate recall. The knowledge test consists of 10

questions related to the content presented during the

experiment that the participant was required to copy

during the experiment. For each question, there were

four possible choices with only one correct answer.

Additionally, a control question was added to the test

to ensure that the participants were not randomly ﬁll-

ing in the questionnaire. Furthermore, participants

were also instructed to skip the question if they did

not know the answer rather than guessing.

4 RESULTS

The data collected as part of this study is publicly ac-

cessible (Limbu and Chounta, 2025).

Table 2: Mean and standard deviation for Immediate Recall,

Perceived Workload, and Mental Effort.

Results [

x, sd]

Group Immediate Recall Perceived Workload Mental Effort (sec)

Max=10 Max=100 Min-Max = 2.74-14.88

1 7.20, 1.64 57.2, 21.0 5.42, 1.18

2 7.17, 1.72 46.8, 16.2 4.55, 1.12

3 8.40, 1.52 40.2, 14.2 8.77, 4.07

4 8.25, 1.71 52.0, 13.6 4.77, 2.09

4.1 Perceived Workload

A two-way analysis of variance (ANOVA) was con-

ducted to examine the effects of handwriting pressure

(additional pressure vs. normal pressure) and glove

(wearing a glove vs. no glove) on perceived work-

load. Levene’s test for homogeneity of variances was

conducted to assess the assumption of equal variances

across the four groups. The result was not statistically

signiﬁcant, indicating that the assumption of homo-

geneity of variances was met [F(3, 16) = 0.001, p =

.999]. Therefore, the variances can be assumed to be

equal across groups. However, a relatively small sam-

ple size (N=20) across four groups means a higher

probability of Type II error.

The ANOVA test showed that there was no statis-

tically signiﬁcant difference between the handwriting

pressure conditions [F(1, 16) = 0.664, p = .427, η

0.04]. Similarly, the effect of wearing a glove on per-

ceived workload was not statistically signiﬁcant [F(1,

16) = .003, p = .957, η

= 0.22]. The interaction ef-

fect between handwriting pressure and a glove was

not statistically signiﬁcant [F(1, 16) = 2.167, p = .160,

= 0.12], suggesting that the two conditions did not

affect each other’s inﬂuence on the perceived work-

load. While the η

value of 0.12 suggests a moderate

effect of the handwriting pressure and wearing a glove

on perceived workload, the small sample size (N=20)

prohibits drawing further conclusions.

Descriptive statistics (see Table 2) showed that

Group 1 (M = 57.2, SD = 21.0) perceived the biggest

workload, followed by the control Group 4 (M=52,

SD = 13.6).

4.2 Mental Effort

A two-way analysis of variance (ANOVA) was con-

ducted to examine the effects of handwriting pres-

sure (additional pressure vs. normal pressure) and

glove (wearing a glove vs. no glove) on mental ef-

fort. Levene’s test for homogeneity of variances was

CSEDU 2025 - 17th International Conference on Computer Supported Education

conducted to assess the assumption of equal variances

across the four groups. The result was not statistically

signiﬁcant, indicating that the assumption of homo-

geneity of variances was met [F(3, 16) = 1.887, p =

.172]. Therefore, the variances can be assumed to be

equal across groups.

ANOVA showed that there was no statistically sig-

niﬁcant difference in the reaction time, thus the men-

tal effort, between the handwriting pressure condi-

tions [F(1, 16) = 3.63, p = .075, η

= 0.18]. How-

ever, the effect of wearing a glove on reaction time

while writing was marginally signiﬁcant [F(1, 16) =

4.50, p = .050, η

= 0.22]. The interaction effect be-

tween handwriting pressure and a glove was not sta-

tistically signiﬁcant [F(1, 16) = 2.11, p = .166, η

0.12], suggesting that the two conditions did not af-

fect each other’s inﬂuence on the mental effort. While

the η

values suggest a moderate-to-large effect of the

handwriting pressure and wearing a glove on reaction

time, the small sample size (N=20) prohibits drawing

further conclusions.

Descriptive statistics (see Table 2) showed that

Control Group 3 (M = 8.77, SD = 4.07) had the

longest reaction time. However, the large SD of 4.07

is caused by an outlier, with one of the participants

taking 14.88 ms, which is drastically different from

the other participants in the group. Group 2, which

applied additional handwriting pressure but did not

wear a glove, had the shortest mean reaction time

(M=4.55, SD = 1.12).

4.3 Mediation Analysis

Mediation analysis was conducted using bootstrapped

estimates with 5000 draws to examine whether men-

tal effort, and perceived workload (mediating vari-

ables), mediated the relationship between the group

(independent variable) and recall results (dependent

variable) in a sample of 20 participants. Group 4

(¬P+¬G) was treated as the reference group. The

analysis was conducted using structural equation

modeling (SEM) with the maximum likelihood esti-

mator in lavaan in Rstudio™.

Model Speciﬁcation. The following relationships

were tested

• a paths: The effects of Group on the mediator

(Mental Effort or Perceived Workload).

• b paths: The effect of the mediator on Recall.

• c paths: The direct effects of group assignment on

Recall, controlling for the mediator.

• Indirect effects (a*b): The mediated effects of

group assignment on Recall via the mediator.

• Total effects: The overall effects of Group on Re-

call (direct + indirect effects)

4.3.1 Mental Effort

Group 1 Group 2 Group 3

Mental

Effort

Recall

Group 4

REF

= 0.653

p = 0.560

= −0.217

p = 0.844

= 4.001

p = 0.052

b= −0.07

p = 0.761

= −1.096

p = 0.345

= −1.068

p = 0.334

= −0.134

p = 0.929

Figure 2: Path-diagram with Mental Effort as mediating

variable.

Table 3: Effects of treatment (group) on Recall via Mental

Effort.

Mediation analysis on Mental Effort

Group Estimate Signiﬁcance 95% Conﬁdence Interval

Direct Effects

1 -1.096 .345 [-3.427, 1.167]

2 -1.068 .334 [-3.233, 1.087]

3 -0.134 .929 [-3.198, 2.465]

Indirect Effects

1 0.046 .904 [-0.420, 1.137]

2 -0.015 .957 [-0.646, 0.549]

3 0.284 .787 [-1.086, 2.745]

Total Effects

1 -1.050 .344 [-3.167, 1.250]

2 -1.083 .321 [-3.225, 1.111]

3 0.150 .891 [-1.833, 2.533]

The direct effects of the three groups (1,2,3) on re-

call were not statistically signiﬁcant compared to the

control (group 4). Similarly, the indirect effects of the

three groups on recall via mental effort were not sig-

niﬁcant. The total effects of the groups on recall were

also not statistically signiﬁcant (see Table 3). The

mediation analysis indicates that the mental effort did

not signiﬁcantly mediate the relationship between the

three groups (various treatments) and recall (mem-

ory). None of the indirect or direct effects reached

statistical signiﬁcance, suggesting that the differences

in group treatments (relative to Group 4) did not sig-

niﬁcantly impact recall either directly or indirectly via

the mental effort.

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments

Group 1 Group 2 Group 3

Perceived

Workload

Recall

Group 4

REF

= 5.200

p = 0.639

=−5.167

p = 0.579

= −11.800

p = 0.195

b= −0.044

p = 0.150

= −0.821

p = 0.419

= −1.311

p = 0.175

= −0.369

p = 0.778

Figure 3: Path-diagram with Perceived Workload as a me-

diating variable.

Table 4: Effects of treatment (Group) on Recall via Per-

ceived Workload.

Mediation analysis on Perceived Workload

Group Estimate Signiﬁcance 95% Conﬁdence Interval

Direct Effects

1 -0.821 .419 [-2.765, 1.269]

2 -1.311 .175 [-2.768, 1.006]

3 -0.369 .778 [-2.836, 2.414]

Indirect Effects

1 -0.229 .716 [-1.669, 0.979]

2 00.227 .615 [-0.895, 0.994]

3 0.519 .406 [-0.585, 1.943]

Total Effects

1 -1.050 .344 [-3.167, 1.250]

2 -1.083 .321 [-3.225, 1.111]

3 0.150 .891 [-1.833, 2.533]

4.3.2 Perceived Workload

These results indicate that the three groups (1,2,3) did

not have a statistically signiﬁcant effect (relative to

Group 4), directly or indirectly, on recall (see Table

4). This suggests that perceived workload did not me-

diate the effect of group assignments on immediate

recall. None of the indirect or direct effects reached

statistical signiﬁcance, suggesting that the differences

in group treatments (relative to Group 4) did not sig-

niﬁcantly impact recall either directly or indirectly via

the perceived workload.

5 DISCUSSION

The sensitivity (or depravity of tactile perception

here) was enforced by requiring the participants to

wear gloves during the writing process (RQ1). The

intensity of motor activity was enforced by requir-

ing the participants to write with additional pressure

(RQ2). The learning performance was measured by

the participant’s performance on the immediate recall

test. The mental effort and the perceived workload

were recorded as the mediating variables. There was

no signiﬁcant effect of the intensity of motor activity

and the sensitivity of tactile perception on the medi-

ating variables. Furthermore, the two treatments also

did not interact with each other. The mediation anal-

ysis (see Section 4.3) showed no signiﬁcant direct or

indirect effect of the groups on the learning perfor-

mance, both via mental effort or perceived workload.

Additionally, the statistically non-signiﬁcant total ef-

fects of the groups showed that neither wearing gloves

to deprave the tactile perception nor writing with ad-

ditional pressure has any impact on the recall. How-

ever, the consistency of the total effect of perceived

workload and mental effort on recall across the two

models suggests that they are both constructs mea-

suring the same or very similar variable, and/or they

do not signiﬁcantly contribute to the relationship be-

tween group assignment (independent) and immedi-

ate recall (dependent) in comparison to the reference

group (Group 4).

The results indicate that the mental effort and/or

perceived workload did not signiﬁcantly differ across

the two treatment conditions, nor did the treatments

interact. As the learning in the context of this study

was purely cognitive,i.e., assimilation and recall from

memory, the ﬁnding is in line with Ray et al. (Ray

et al., 2022), who only found weak evidence for the

effect of psychomotor aspects on cognitive learning.

The documented beneﬁts of handwriting for memory

are often associated with literacy (Skar et al., 2022;

Ray et al., 2022). In the cases where beneﬁts were ob-

served in regards to memorization of concepts in uni-

versity students (Flanigan et al., 2024; Wrigley, 2019;

Mueller and Oppenheimer, 2014), the learners were

cognitively engaged due to the time constraint which

required them to actively process information to con-

dense them. Consequently, the experimental task used

in this study, that is, copying text shown in a prompter,

might not have cognitively engaged the learner.

Further, the treatment (haptic) conditions were in-

tended to stress the working memory through the non-

verbal or imagery system of the Dual-coding model.

However, the treatment is also contextually closely

coupled with the information received by the verbal

system (Text/words), as the experimental task uses the

verbal channel in the form of printed texts. This aligns

with suggestions from Danna and Velay (Danna and

Velay, 2015) and potentially hints towards the inte-

gration of haptic modality in the Modality principle

in Multimedia Learning (Mayer, 2005), which sug-

gests that the use of multiple modalities results in ef-

CSEDU 2025 - 17th International Conference on Computer Supported Education

ﬁcient learning when the information is contextually

coupled.

5.1 Theoretical and Practical

Implications

As the education landscape shifts towards

competency-based frameworks, multimodal immer-

sive learning environments (MILEs) for promoting

authentic practice of complex skills are becoming

increasingly common. MILEs require tracking the

learner’s actions in the environment. This is often

accomplished by using wearable sensors, which

paradoxically are added on top of the actual authentic

settings. For example, Mat Sanusi et al., (Mat Sanusi

et al., 2021) used smartphone sensors by attaching a

smartphone to the learner’s body, arguably affecting

the learner’s authentic performance. Thus, under-

standing the effect of such additions on learning is of

utmost importance for designing efﬁcacious MILEs.

The study results suggest that wearing a glove

and/or exerting additional effort in the form of pen

pressure does not affect recall, mental effort, or per-

ceived workload. Thus, such manipulations in MILEs

may not lead to cognitive overload and, therefore,

have minimal impact on learning. However, Ver-

mulen et al., (Vermeulen et al., 2008) found that sen-

sory overload resulted in cognitive overload when

additional stimuli were presented, but the increase

in the intensity of existing primary stimuli seemed

to have no impact. This potential to provide addi-

tional feedback and support during authentic prac-

tice in MILEs can improve learning and acquisition

of complex skills (Danna and Velay, 2015).

This may posit that wearable sensors can po-

tentially be safely used for learning handwriting.

Doug (Doug, 2019) found that the students’ hand-

writing performance is continuously degrading, af-

fecting their academic performance. While occupa-

tional therapy-based interventions have proven bene-

ﬁcial (Hoy et al., 2011), they cannot address the issue

at the required scale. MILEs can contribute towards

solving this problem by automating educational as-

pects surrounding handwriting. For example, auto-

mated systems utilizing consumer tablets have been

developed to diagnose handwriting difﬁculties such as

dysgraphia (Asselborn et al., 2020).

Similarly, Dikken et al., (Dikken et al., 2022)

also developed a sensor-based application for train-

ing handwriting that provided real-time feedback on

various handwriting attributes based on the teacher’s

expertise. Such attributes, like pen pressure and

perceptual-motor abilities, directly impact handwrit-

ing itself(Dennis and Swinth, 2001), which further

affects academic performance. The potential to use

increasingly more invasive sensors, such as the wear-

able glove from SenseGlove™without adversely af-

fecting learning, broadens the possibilities for more

inclusive multisensory and seamless learning design

(Specht et al., 2019). For example, such environ-

ments can cater to people with hearing loss by the use

of vibration motors to provide a sense of direction in

mixed-reality environments.

5.2 Limitations

The study is limited by the amount of participants

(N=20). Despite the use of bootstrapping with 5000

draws, the limited number of participants divided into

four groups is not sufﬁcient to overlook this limita-

tion.

The study was also limited by the choice to use a

text excerpt for recall. This is in contrast to other stud-

ies (Smoker et al., 2009; Bouriga and Olive, 2021;

Van der Weel and Van der Meer, 2024), which used

an array of unrelated words to test the memory. Struc-

tured text excerpts may not necessarily overload the

working memory, especially in the presence of prior

knowledge. No pre-test was performed to test the

presence of prior knowledge. However, the text ex-

cerpt was taken from a geology book under the pre-

tense that the students from the computer science fac-

ulty would have minimal knowledge of the topic, if

any at all. Using a text excerpt was a conscious choice

to test the effects of handwriting in the absence of rep-

etition.

The experimental treatment involved manipulat-

ing two aspects of the haptic sense, but each was only

altered in one direction, even though opposite manip-

ulations might also inﬂuence learning. For example,

writing on paper is superior in terms of brain activa-

tion in comparison to writing on digital tablets (Ume-

jima et al., 2021), conceivably due to the increased

friction provided by the paper’s rough texture. In

this study, the tactile perception was reduced, and the

writing pressure was increased. However, we did not

study the effects of their corresponding reverse ma-

nipulation.

Lastly, the NASA Task Load Index (NASA TLX)

questionnaire used in the study consists of only 14

categories compared to its 15 categories. One cate-

gory was removed as it was irrelevant to this study.

This discrepancy can impact the reliability of the in-

strument.

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments

6 CONCLUSION

In this study, we experimentally investigated the ef-

fect of manipulating two attributes associated with

haptic sense, namely tactile perception and motor ac-

tion, on learning. Learning is deﬁned as the assimila-

tion and recall of information from memory. As the

learning process is impacted by cognitive load, which

correlates to mental effort, mental effort was observed

as the mediating variable. Additionally, perceived

workload, which represents both the mental and the

physical effort, was also included due to the emphasis

on the haptic senses. The use of gloves manipulated

the tactile perception, while the motor action was ma-

nipulated using software to enforce higher pressure

while writing. The results of the study showed no sta-

tistically signiﬁcant effect of the treatments, individ-

ually or combined, on the learning performance via

mental effort or perceived workload compared to the

control. No signiﬁcant effect of the treatment was ob-

served on the mediating variables as well. The ﬁnd-

ings of this study contribute to our understanding of

the design of multimodal immersive learning environ-

ments and embodied learning to enhance memory and

the acquisition of complex skills. It also contributes

to improving our understanding of dual-coding theory

in the light of haptic senses.

ACKNOWLEDGEMENTS

We want to thank the students at the University of

Duisburg-Essen who helped execute the study in var-

ious roles. Namely, Anna Luisa F

arber, B

unyamin

Yilmaz, Hannah Holland, Joshua Jung, Katarzyna

Pogorzala, Van Hoang, and Zhang Xinyu.

REFERENCES

Abrahamson, D. and Lindgren, R. (2014). Embodiment and

Embodied Design, page 358–376. Cambridge Hand-

books in Psychology. Cambridge University Press.

Asselborn, T., Chapatte, M., and Dillenbourg, P. (2020).

Extending the Spectrum of Dysgraphia: A Data

Driven Strategy to Estimate Handwriting Quality. Sci-

entiﬁc Reports, 10(1):1–11. ISBN: 4159802060.

Baddeley, A. D. (1990). Human memory: Theory and prac-

tice. Human memory: Theory and practice. Allyn &

Bacon, Needham Heights, MA, US. Pages: xi, 515.

Bjork, E. L. and Bjork, R. A. (2011). Making things hard on

yourself, but in a good way: Creating desirable difﬁ-

culties to enhance learning. In Psychology and the real

world: Essays illustrating fundamental contributions

to society., pages 56–64. Worth Publishers, New York,

NY, US.

Bouriga, S. and Olive, T. (2021). Is typewriting more

resources-demanding than handwriting in undergrad-

uate students? Reading and Writing, 34(9):2227–

2255.

Clark, J., Crandall, P. G., Pellegrino, R., and Shabatura, J.

(2019). Assessing Smart Glasses-based Foodservice

Training: An Embodied Learning Theory Approach.

Canadian Journal of Learning and Technology, 45(2).

Number: 2.

Clark, J. M. and Paivio, A. (1991). Dual coding theory and

education. Educational Psychology Review, 3(3):149–

210.

Craik, F. I. M. and Tulving, E. (1975). Depth of pro-

cessing and the retention of words in episodic mem-

ory. Journal of Experimental Psychology: General,

104(3):268–294. Place: US Publisher: American Psy-

chological Association.

Danna, J. and Velay, J.-L. (2015). Basic and supplementary

sensory feedback in handwriting. Frontiers in Psy-

chology, 6:169.

Dennis, J. L. and Swinth, Y. (2001). Pencil Grasp and

Children’s Handwriting Legibility During Different-

Length Writing Tasks. The American Journal of Oc-

cupational Therapy, 55(2):175–183.

Di Mitri, D., Limbu, B., Schneider, J., Iren, D., Gian-

nakos, M., and Klemke, R. (2024). Multimodal and

immersive systems for skills development and edu-

cation. British Journal of Educational Technology,

55(4):1456–1464.

Di Mitri, D., Schneider, J., Limbu, B., Mat Sanusi, K. A.,

and Klemke, R. (2022). Multimodal Learning Ex-

perience for Deliberate Practice, pages 183–204.

Springer International Publishing, Cham.

Dikken, O., Limbu, B., and Specht, M. (2022). Deliber-

ate Practice of Handwriting: Supervision Under the

Ghost of an Expert. In Hilliger, I., Mu

noz-Merino,

P. J., De Laet, T., Ortega-Arranz, A., and Farrell, T.,

editors, Educating for a New Future: Making Sense

of Technology-Enhanced Learning Adoption, pages

434–440, Cham. Springer International Publishing.

Doug, R. (2019). Handwriting: Developing Pupils’ Identity

and Cognitive Skills. International Journal of Educa-

tion and Literacy Studies, 7(2):177–188.

Esmaeili Bijarsari, S. (2021). A Current View on Dual-Task

Paradigms and Their Limitations to Capture Cognitive

Load. Frontiers in Psychology, 12. Publisher: Fron-

tiers.

Flanigan, A. E., Wheeler, J., Colliot, T., Lu, J., and Kiewra,

K. A. (2024). Typed Versus Handwritten Lecture

Notes and College Student Achievement: A Meta-

Analysis. Educational Psychology Review, 36(3):78.

Giannakos, M. and Cukurova, M. (2023). The role

of learning theory in multimodal learning analyt-

ics. British Journal of Educational Technology,

54(5):1246–1267.

Hart, S. G. (1986). Nasa task load index (tlx).

Horz, H. (2012). Situated Prompts in Authentic Learning

Environments. In Seel, N. M., editor, Encyclopedia of

the Sciences of Learning, pages 3086–3088. Springer

US, Boston, MA.

CSEDU 2025 - 17th International Conference on Computer Supported Education

Hoy, M. M. P., Egan, M. Y., and Feder, K. P. (2011). A

Systematic Review of Interventions to Improve Hand-

writing. Canadian Journal of Occupational Therapy,

78(1):13–25. Publisher: SAGE Publications Inc.

James, K. H. and Engelhardt, L. (2012). The effects of

handwriting experience on functional brain develop-

ment in pre-literate children. Trends in Neuroscience

and Education, 1(1):32–42.

Kiefer, M. and Velay, J.-L. (2016). Writing in the digital

age. Trends in Neuroscience and Education, 5(3):77–

81.

Lee, Y., Limbu, B., Rusak, Z., and Specht, M. (2023).

Role of Multimodal Learning Systems in Technology-

Enhanced Learning (TEL): A Scoping Review. In

Viberg, O., Jivet, I., Mu

noz-Merino, P. J., Perifanou,

M., and Papathoma, T., editors, Responsive and Sus-

tainable Educational Futures, pages 164–182, Cham.

Springer Nature Switzerland.

Li, D., Wang, D., Zou, J., Li, C., Qian, H., Yan, J., and He,

Y. (2023). Effect of physical activity interventions on

children’s academic performance: a systematic review

and meta-analysis. European Journal of Pediatrics,

182(8):3587–3601.

Limbu, B. and Chounta, I.-A. (2025). Dataset for the paper

”can the mighty pen be mightier? investigating the

role of haptic senses in multimodal immersive learn-

ing environments”.

Limbu, B., van Helden, G., Barnes, J., and Specht, M.

(2022). We can teach more than we can tell: com-

bining Deliberate Practice, Embodied Cognition, and

Multimodal Learning, volume 3247 of CEUR Work-

shop proceedings, pages 15–21. CEUR-WS.

Limbu, B. H., Jarodzka, H., Klemke, R., and Specht, M.

(2018). Using sensors and augmented reality to train

apprentices using recorded expert performance: A

systematic literature review. Educational Research

Review, 25(June 2017):1–22. Publisher: Elsevier.

Limbu, B. H., Jarodzka, H., Klemke, R., and Specht, M.

(2019). Can You Ink While You Blink? Assessing

Mental Effort in a Sensor-Based Calligraphy Trainer.

Sensors, 19(14):3244. Number: 14 Publisher: Multi-

disciplinary Digital Publishing Institute.

Longcamp, M., Zerbato-Poudou, M.-T., and Velay, J.-L.

(2005). The inﬂuence of writing practice on letter

recognition in preschool children: A comparison be-

tween handwriting and typing. Acta Psychologica,

119(1):67–79.

Loup-Escande, E., Frenoy, R., Poplimont, G., Thouvenin,

I., Gapenne, O., and Megalakaki, O. (2017). Contri-

butions of mixed reality in a calligraphy learning task:

Effects of supplementary visual feedback and exper-

tise on cognitive load, user experience and gestural

performance. Computers in Human Behavior, 75:42–

49.

Marucci, M., Di Flumeri, G., Borghini, G., Sciaraffa, N.,

Scandola, M., Pavone, E. F., Babiloni, F., Betti, V.,

and Aric

o, P. (2021). The impact of multisensory in-

tegration and perceptual load in virtual reality settings

on performance, workload and presence. Scientiﬁc

Reports, 11(1):4831. Publisher: Nature Publishing

Group.

Mat Sanusi, K. A., Mitri, D. D., Limbu, B., and Klemke, R.

(2021). Table Tennis Tutor: Forehand Strokes Clas-

siﬁcation Based on Multimodal Data and Neural Net-

works. Sensors, 21(9):3121. Number: 9 Publisher:

Multidisciplinary Digital Publishing Institute.

Mayer, R. E. (2005). Cognitive Theory of Multimedia

Learning, page 31–48. Cambridge Handbooks in Psy-

chology. Cambridge University Press.

McCarroll, H. and Fletcher, T. (2017). Does handwriting in-

struction have a place in the instructional day? The re-

lationship between handwriting quality and academic

success. Cogent Education, 4(1):1386427.

Meik, A., Schneider, J., and Schiffner, D. (2021). Get your

back straight! learn pilates with the pilates correction

app. In DELFI 2021, pages 163–168. Gesellschaft f

Informatik e.V., Bonn.

Mueller, P. A. and Oppenheimer, D. M. (2014). The Pen

Is Mightier Than the Keyboard: Advantages of Long-

hand Over Laptop Note Taking. Psychological Sci-

ence, 25(6):1159–1168. Publisher: SAGE Publica-

tions Inc.

Ose Askvik, E., van der Weel, F. R. R., and van der Meer,

A. L. H. (2020). The Importance of Cursive Handwrit-

ing Over Typewriting for Learning in the Classroom:

A High-Density EEG Study of 12-Year-Old Children

and Young Adults. Frontiers in Psychology, 11. Pub-

lisher: Frontiers.

Paul A. Kirschner, J. S. and Clark, R. E. (2006). Why

minimal guidance during instruction does not work:

An analysis of the failure of constructivist, discovery,

problem-based, experiential, and inquiry-based teach-

ing. Educational Psychologist, 41(2):75–86.

Peverly, S. T. (2006). The Importance of Handwriting

Speed in Adult Writing. Developmental Neuropsy-

chology, 29(1):197–216.

Ray, K., Dally, K., Rowlandson, L., Tam, K. I., and Lane,

A. E. (2022). The relationship of handwriting abil-

ity and literacy in kindergarten: a systematic re-

view. Reading and Writing, 35:11191155. Publisher:

Springer Science and Business Media B.V.

Schneider, J., Romano, G., and Drachsler, H. (2019). Be-

yond Reality—Extending a Presentation Trainer with

an Immersive VR Module. Sensors, 19(16):3457.

Number: 16 Publisher: Multidisciplinary Digital Pub-

lishing Institute.

Shvarts, A. and van Helden, G. (2023). Embodied learn-

ing at a distance: from sensory-motor experience to

constructing and understanding a sine graph. Mathe-

matical Thinking and Learning, 25(4):409–437.

Skar, G. B., Lei, P.-W., Graham, S., Aasen, A. J., Johansen,

M. B., and Kvistad, A. H. (2022). Handwriting ﬂu-

ency and the quality of primary grade students’ writ-

ing. Reading and Writing, 35(2):509–538.

Smoker, T. J., Murphy, C. E., and Rockwell, A. K. (2009).

Comparing Memory for Handwriting versus Typing.

Proceedings of the Human Factors and Ergonomics

Society Annual Meeting, 53(22):1744–1747. Pub-

lisher: SAGE Publications Inc.

Specht, M., Hang, L. B., and Barnes, J. S. (2019). Sen-

Can the Mighty Pen Be Mightier? Investigating the Role of Haptic Senses in Multimodal Immersive Learning Environments

sors for Seamless Learning, pages 141–152. Springer

Singapore, Singapore.

Sweller, J., Ayres, P., and Kalyuga, S. (2011). Cognitive

Load Theory. Explorations in the Learning Sciences,

Instructional Systems and Performance Technologies.

Springer New York, NY, 1 edition.

Taguma, M. and Frid, A. (2024). Curriculum Frame-

works and Visualisations Beyond National Frame-

works: Alignment with the OECD Learning Compass

2030. Technical report, OECD, Paris.

Umejima, K., Ibaraki, T., Yamazaki, T., and Sakai, K. L.

(2021). Paper Notebooks vs. Mobile Devices: Brain

Activation Differences During Memory Retrieval.

Frontiers in Behavioral Neuroscience, 15. Publisher:

Frontiers.

Van der Weel, F. R. R. and Van der Meer, A. L. H. (2024).

Handwriting but not typewriting leads to widespread

brain connectivity: a high-density EEG study with im-

plications for the classroom. Frontiers in Psychology,

14. Publisher: Frontiers.

Vermeulen, N., Corneille, O., and Niedenthal, P. M. (2008).

Sensory load incurs conceptual processing costs. Cog-

nition, 109(2):287–294.

Wrigley, S. (2019). Avoiding ‘de-plagiarism’: Explor-

ing the affordances of handwriting in the essay-

writing process. Active Learning in Higher Education,

20(2):167–179. Publisher: SAGE Publications.

Yoshida, T., Yamaguchi, A., Tsutsui, H., and Wake, T.

(2015). Tactile search for change has less memory

than visual search for change. Attention, Perception,

& Psychophysics, 77(4):1200–1211.

CSEDU 2025 - 17th International Conference on Computer Supported Education