HELP DESIGN FOR THE METACOGNITIVE GUIDANCE
OF THE LEARNER
A Proposition of Computer-Based System
J-C. Sakdavong
1
, F. Adreit
2
and N. Huet
1
1
CLLE-LTC, UMR CNRS 5263
2
IRIT, Toulouse le Mirail University
5, Allée A. Machado, 31058 Toulouse cedex 1, France
Keywords: Learning, Distance education, Computer supported education, ICT, AIED, Modelling, Help, Guidance,
Metacognition, Metacognitive guidance, Multiagent system.
Abstract: This paper presents the framework and the software system we have built in order to provide metacognitive
guidance help in Computer Supported Education. The goal is to assist self-regulation of the learner thanks to
a dynamic help system which takes into account in real-time the learner's behavior and his profile. The
software system is a multiagent system which captures the learner's behavior, analyse it and define the help.
Used currently in the step of conception, the software system will become the learning system by successive
learning and enrichment. We present its principles and its operational aspects. The application field of this
work is the French certificate “C2i”.
1 INTRODUCTION:
SELF-REGULATION AND
METACOGNITIVE HELPS
We are in the context of learning throughout life
supported by Information and Communication
Technologies (ICT). Students, workers, retired
people are more and more often faced with learning
alone, at home or on their workplace. In this context,
there is usually no teacher present to monitor and
assist them. To meet this need, a lot of learning is
given through computer-based learning systems:
online tutorials, courses, or more integrated systems
(Learning Management Systems, Learning and
Content Management Systems). In this way, learners
have spatial (through remote systems) and temporal
(the learners can learn according to their availability)
autonomy. If they are properly designed, these
systems allow education adapted to the profile of the
learner: his knowledge, learning experience,
metacognitive profile.
Unfortunately, it is now clear that these systems
do not often achieve their goal (Osman and
Hannafin, 1992; Winne and Stockley, 1998). One of
the problems of the effective use of ICT for learning
is that these systems require that the learners
regulate their own learning (Avezedo, 2005). Few of
them have the required skills for taking in charge
ones cognitive functioning (e.g. Hannafin & Land,
1997). Acquiring new knowledge and ability appeals
not only to cognitive processes (activation of
knowledge, use of adapted learning strategies, and
memorization of new knowledge) but also to
metacognitive processes (planning, self-evaluation,
learning regulation). In face to face education, some
of these metacognitive activities are provided by the
teacher; in the context of computer supported
education, they fall to the learner. It is why we speak
about self-regulated learning for this kind of
learning.
To help learners to regulate themselves, the
designers of computer-based learning systems have
added learning helps to support planning, self-
evaluation and learning regulation. The table 1 gives
some examples of help available on computer-based
learning systems; we can see what may be provided
by the teacher (in face to face education) and what is
left to the learner in the context of computer
supported education.
210
Sakdavong J., Adreit F. and Huet N. (2009).
HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based System.
In Proceedings of the First International Conference on Computer Supported Education, pages 209-215
Copyright
c
SciTePress
Table 1: Examples of helps.
Metacognitive
activity
Help provided by the teacher (in face to face
education)
Help available in computer-based learning systems
Planning The teacher defines the purposes and the
duration of the lesson.
The teacher defines the distribution of learning
activity.
The system asks the learner to define a priori
his learning time.
Self-
evaluation
The teacher defines exercises or questions to
verify the control of a concept or a part of the
lesson.
The learner can ask questions to the teacher or
to other learners.
The system provides exercises, questions and
feedback.
The learner can ask questions to the teacher or to
others learners via the forum.
Regulation If the learner meets with difficulties, the teacher
can provide help (for example, an explanation
in a scheme-form), an advice ("read again a
chapter"), an indication ("look for the definition
of a concept”), an answer (a solution).
The learner can approach other learners.
Depending on the answers to questions, the system
may propose to the learner to revise a concept.
The learner can approach other learners via the
forum.
However, the efficiency of these learning helps
has not been really evaluated and some recent works
show difficulties in their use. We can notice that
some of them are underused or even unused
(Narciss, Proske, Körndle, 2007; Narciss, Körndle,
Dupeyrat, 2002). Other works highlight inadequate
use of help; it is what Roll et al. call metacognitive
bugs (Roll et al., 2005): for example, unorganized
over-use of help (Roll et al.,2005), exclusive use of
help that provides an answer rather than an
indication to look for the answer (Aleven et al.,
2003).
This rather disappointing acknowledgement
raises questions about the metacognitive abilities
which are necessary for adequate use of help: to be
aware of needing help, to choose an appropriate type
of help, to detect the usefulness of help, to realize
when help is necessary, and, after failing, to detect
which help to revise (Aleven and al., 2003 ;
Puustinen, 1998). Works in metacognition field
show clearly the difficulties to acquire and to apply
such metacognitive abilities, even in face to face
education. They show also that these skills change
according to factors like learner's knowledge and
age.
2 THE CEAGMATIC PROJECT
2.1 An Original Project
Our work is a part of the CEAGMATIC project of
the French National Research Agency (ANR).
Researchers involved in this project are members of
the CLLE-LTC (Laboratoire Travail et Cognition)
and IRIT (Institut de Recherche en Informatique de
Toulouse)laboratories.
The main goal of this project is to design and
build a help guidance system to improve learners’
metacognitive abilities. This system has to analyze
the learner’s profile and to react in real time to
learner’s behavior. The project team is composed of
researchers in the fields of cognitive psychology and
computer science.
To the best of our knowledge, only one research
center has made such a system (the Human
Computer Interaction Institute of Carnegie Mellon
University, Roll et al., 2005) but with only one kind
of metacognitive guidance. Our system will go
further by proposing and comparing many kinds of
guidance: one proposed and then accepted by the
learner and another one, imposed by the system (in
order to compensate for a learner’s metacognitive
lack or inappropriate behavior).
Another important part of this project is the
learners’ profiling. We take into account learners’
demographics, cognitive and metacognitive profiles:
the system will build a learner profile through
questionnaires and real time activity analysis. This
profile will be used in order to select how and when
to help and to guide the learner. It will evolve
according to the effectiveness of the helps and
guidance. This designing choice allows us to target
heterogeneous categories of learners (workers,
students, …) as we can adapt the system’s help and
action to the learner’s profile without overloading
him with useless interactions and documents.
HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based
System
211
Moreover, we can provide the help progressively,
when the learner (in fact, his profile) evolves.
2.2 The Project’s Steps
The project begun on 2007 and will end on 2011. In
order to design and build the help and guidance
system, we have defined the following steps:
Preliminary: Defining learner’s regular and
inappropriate behaviors by analyzing learners'
behaviors on provided interactive lessons and
exercises (the first version of the system does
not provide specific helps) ; this step is
completed
Providing Cognitive Helps: Adding helps
devoted to “inappropriate behaviors” identified
during the previous step. Then analyzing again
learner’s regular and inappropriate behaviors
integrating these new helps (we will target bad
use of these helps as the “metacognitive”
inappropriate or missing behaviors) ; this step is
in progress
Providing Metacognitive Helps: Adding
metacognitive guidance actions devoted to
metacognitive needs identified in the previous
step. These actions will be either guidance
actions or metacognitive profiling actions.
In order to support these 3 steps, we have built a
multi-agent system which can capture and analyze
learners’ behavior while they study. This system has
to provide the lessons and exercises to the learners
(Figure 1). The main goal of this paper is to describe
and explain how and why we are building this
system. The other results of our experiments will be
presented later after step 3 will be finished.
2.2.1 Step 1: Preliminary
A psychologist has studied and analyzed
learners’ behaviors during face to face lessons
Experienced teachers have specified the
learning activity and the optimal behavior in
terms of tasks and knowledge (Paquette and al.
2002)
e-Learning engineers have built an online
course according to the previous specifications
The online course has been tested over 100
learners and the multi-agent system has
recorded all the learners’ behaviors into activity
graphs (Figure 2)
Psychologists are analyzing the activity graphs
of each learner in order to identify the
characteristic behaviors e.g. the learners’
regular and inappropriate (mistakes) behaviors
while doing exercises.
Figure 1: User interface of the multiagent system.
Figure 2: Activity graph recorded by the system.
2.2.2 Step 2: Providing Cognitive Helps
This step consists in conceiving actions associated
with each characteristic behavior identified during
the step 1. These actions will help the learner when
he makes “cognitive mistakes”.
The cognitive helps will be designed by
psychologists
These helps will be included in the multi-agent
system as helper agents
CSEDU 2009 - International Conference on Computer Supported Education
212
The online course including helps will be tested
over many groups of learners and the multi-
agent system will again record all the learners’
behaviors into graphs
Psychologists will analyze these activity graphs
in order to identify the characteristic behaviors
while using the new helps e.g. the learners
regular and inappropriate regulation actions
while doing exercises (for example, a learner
who never accepts to read again the lesson
when the helper agents propose to do it).
2.2.3 Step 3: Providing Metacognitive Helps
This step consists in conceiving metacognitive
guidance actions associated with each characteristic
behavior identified during the step 2 e.g. bad use of
helps. These actions can be assistances but also
refinements of learners’ metacognitive profiles.
Thus, the system will progressively build precise
profiles.
We speak here of “metacognitive guidance”
because our hypothesis is that if learners do not use
correctly the helps, it is because they have a lack of
metacognitive abilities: they do not regulate
correctly their learning behavior.
Two types of metacognitive guidance will be
proposed:
A suggested guidance that the learner can
accept or refuse (Noury and al., 2006), (for
example, “you should look at the glossary”;
“You should do the exercises before doing the
test”)
An imposed guidance if the system identifies a
recurrent metacognitive mistake or lack. (for
example, a definition from the glossary is
presented to the learner)
This is the principal specificity of our approach
from the point of view of psychology.
Then:
The two types of metacognitive guidance will
be included in the multi-agent system as new
helping agents
The online course including guidance will be
tested over many groups of learners and the
multi-agent system will again record all the
learners’ behavior into graphs
Psychologists will analyze the graphs of
learners’ behaviors in order to check if the
metacognitive helps are useful.
2.4 Experiment: The Chosen Target
Learners
To experiment, we have chosen the French
certificate “C2i (level 1)” (Computer Science
and Internet Certificate). The learners have
different backgrounds, levels of study and ages.
Moreover, we have a large population of
students for testing and teachers experienced in
this training (in the universities of the two
involved research laboratories).
After having analyzed the results over the tests
of C2i, we have chosen to target the
“Formatting documents with style sheets”
lesson. Indeed, this lesson presents cognitive
and metacognitive difficulties which can be
supported by the help guidance system.
3 DESIGN PRINCIPLES OF THE
COMPUTER-BASED SYSTEM
3.1 Dynamics, Flexibility and
Scalability of the System
Our system is based on the observation and analysis
of the learner's behavior. So, it is based on a
dynamic component (the activity) from which the
help is constructed dynamically (by observation and
analysis). Therefore, the system has to be able to
observe, analyze the learner's behavior and to
construct real-time help.
Moreover, we propose a general help principle
which can be used in any learning situation.
Therefore, the system has to be easily adapted,
keeping the generic functions of the system, just
modifying the learning situation. For example, in the
experimentation, it is used for the learning situation :
"Formatting documents with style sheets". It should
be used for other learning situations like "Using of
table of contents or index" or, beyond word
processing, like "Using functions in spreadsheet".
Finally, while the system is currently used as a
workshop which allows psychologists to observe the
learners' behaviors, it will be able (in the third step)
to integrate into a dynamic whole the observation-
analysis-help process.
3.2 Bootstrap, Grading and Regulation
of the System
To analyze the learner's behavior, we use the
description of the regular activity and of the possible
HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based
System
213
deviations which are linked to the knowledge and
the ability of the current exercise. For example,
"Formatting a paragraph" is linked to paragraph and
alignment concepts (knowledge), and to the
designation of a paragraph and of an alignment
(ability). The regular activity and the possible
deviations compose the system’s bootstrap.
They are completed and refined during the
analysis of the activity by the psychologists. This
analysis allows also designing the action system: an
action (cognitive profile modification, help
suggestion) is associated with each characteristic
behavior. It is what we call the grading of the
system.
Finally, the regulation of the system consists in
adapting the help as the system runs. Ideally, it
should be a self-regulation of the system. But for the
project, we will allow only a dynamic regulation
within the action system defined by the
psychologists in the second step.
3.3 Functional Aspects
Our help system is based on the observation and
analysis of the learner's behavior. Therefore, it is
necessary that it places the learner in a position to do
and can observe his actions. It is why the system has
to integrate learning interactive tools. For example,
we have integrated a text editor for the
experimentation.
To analyze the activity, we have integrated a tool
which allows the observation of the learner's
behavior. We wondered about the granularity of the
observed actions. Technically, it was possible to
observe elementary actions (click, mouse moved,
…). But, after a first test, we realized that the
important actions were:
Actions on interactive objects ; for example, a
selection in a menu or the validation of a dialog
box
Semantic actions linked to the learning context;
for example, putting a word in italics
Therefore, the system observes and records these
actions which are analyzed by the psychologists,
observed and then processed by the system to
generate the help in the next step.
The psychologists analyze the learner's behavior
in relation to the regular activity. To make the
analysis of the activity easier, we have integrated
tools to describe the regular activity and to represent
the observed one with the same graphic formalism;
these tools can also represent the differences
between regular and observed activities.
Finally, the system contains a tool of automatic
analysis of the activity which allows detecting
characteristic behaviors, and an action system able
to activate help and to modify the profile.
All these functional aspects have been integrated
to the computer system.
4 SOFTWARE ARCHITECTURE
OF THE MULTI-AGENT
SYSTEM
4.1 Why a Multi-Agent System?
We have implemented the device to deliver online
course and to provide helps as a multi-agent system
(Wooldridge, 2002). This choice of implementation
allows us:
To have software elements (agents) able
intrinsically to observe the activity and to
produce a behavior, also to communicate
between them;
To obtain a dynamic behavior of the device,
creating agents during the learner's behavior
(for example, creating a new helping agent
when an exercise starts) or modifying in real
time the behavior of agents (for example, an
helping agent can change of behavior
according to an evolution of the learner’s
profile);
An incremental construction of the device;
A flexible and dynamic construction of the
device; for example, we can replace the agent
“text editor” by an agent “spreadsheet”
according to the situation of training, without
modifying the remainder of the device;
To consider a distributed runtime of the various
elements of the device on various computers
(the learners’ computers, the LMS’s computers
and the learners profiler computer).
We have used the framework JADE
(Bellefemine, F. and al., 2004) and programmed the
agents in the Java language.
4.2 Agents of the System
The system is composed by several types of agents:
”Principal”, “Exercise” and “Applicative”
agents which constitute the LMS (Learning
Management System);
An “Historical” agent;
CSEDU 2009 - International Conference on Computer Supported Education
214
”Helper”, “Scrutinizing” and “Profile” agents
which constitute the system of analysis and the
system of help.
The LMS includes a “Principal” agent which
implements the teaching scenario. The figure 1
shows the human-computer interface of the LMS:
the summary (“Résumé”), the course and the
exercises (“Cours et exercices”), the self-assessment
(“Autoévaluations”), the external references
(“Références”), the glossary and the index
“(“Glossaire, index)”. When the learner chooses an
exercise, the “Principal” agent creates an “Exercise
agent implementing the scenario of the
corresponding exercise; we can have thus
simultaneously several “Exercise” agents An
“Exercise” agent is always associated to an
“Applicative” agent which implements the
interactive system necessary to the realization of the
exercise; in the current project, this agent
implements a word processor.
The “Historical” agent records the learner’s
behavior as a sequence of actions (the activity
graph). It thus communicates with the previous
agents: it records the activity with respect to the full
teaching scenario (for example, it records if the
learner consults the exercises, then reaches the
course), to the scenario for a particular exercise (for
example, when the learner answers the first question,
then the second one, then returns to the first one), to
the “Applicative” agent (for example, the learner
selects a paragraph then clicks on the shortcut button
“centering the paragraph”).
“Scrutinizing” agents allow observing and
analyzing the activity of learning. These agents are
charged to identify characteristic behaviors,
according to the profile. They are created
dynamically by the “Exercise” agents. They have a
mechanism of subscription which enables them to
receive from the “Historical” agent the sequences of
actions they are charged to analyze. According to
their analysis, they will create “Helper” agents or
will communicate with the existing “Helper” agents.
They will also communicate with the “Profile” agent
charged to dynamically adapt the profile of the
learner.
The “Helper” agents provide the assistance by
giving feedback, displaying solution, procedure,
chapter corresponding to the difficulty, asking
questions to the learners. In the last step of the
project, they will give the metacognitive guidance to
the learners. They also will communicate with the
“Profile” agent.
5 CONCLUSIONS
We have presented the process and the software
device that we have developed, associated to the
design of a new kind of help. The multiagent
architecture used to implement the software system
is an original way to deal with the complex problem
of a dynamic and contextual learning help. It allows
to meet the dynamic, flexibility and scalability
requirements of the device.
We are testing it with the learning of the C2i
certificate. At present, we have realized the first
step of the process (we have defined the regular
behavior and the possible deviations) and constituted
the bootstrap of the software device. Then we have
recorded the behavior of a troop of learners with the
software device. Currently, a psychologist is
analysing these recordings (step 2 of the process).
Afterwards, the results of this analysis will be
integrated into the system and will be evaluated.
At the same time, we are working on the
specification of « Helper » agents to add syntactic
analysis abilities to them: each « Helper » agent will
be defined by an abstract grammar which will be
specific to a learning behavior. Then, the
psychologists would just have to define abstract
grammars and associated semantic actions.
REFERENCES
Osman, M. E., Hannafin, M.J. 1992. Metacognition
research and theory: Analysis and implications for
instructional design. Educational Technology
Research & Development, 40(2), 83-89.
Winne, P. H., Stockley, D. B. 1998. Computing
technologies as sites for developing self-regulated
learning. D. H. Schunk and B. J. Zimmerman (Eds.),
Developing self-regulated learning: From teaching to
self-reflective practice 106-136. New York: Guilford.
Azevedo, R. 2005. Using hypermedia as a metacognitive
tool for enhancing student learning? The role of self-
regulated learning. Educational Psychologist, 40(4),
199–209.
Hannafin, M. J., Land, S. M. 1997. The foundations and
assumptions of technology-enhanced, student-centered
learning environments. Instructional Science, 25, 167-
202.
Narciss, S., Proske, A., Körndle, H. 2007. Promoting self-
regulated learning in web-based learning
environments. Computers in Human Behavior, 23,
1126-1144.
Narciss, S., Körndle, H., & Dupeyrat, C. 2002.
Promouvoir l’apprentissage auto-régulé avec
l’Internet. Colloque Compréhension et Hypermédia,
Albi, 10-11 octobre 2002.
HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based
System
215
Roll, I., Baker, R.S., Aleven, V., McLaren, B., &
Koedinger, K. 2005. Modeling students’ metacognitive
errors in two intelligent tutoring systems. In L.
Ardissono, P. Brna and A. Mitrovic (Eds.),
Proceedings of the 10th International Conference on
User Modeling, UM'2005, 379-388. Berlin: Springer-
Verlag.
Aleven, V., Fischer, F., Schworm, S., Stahl, E., Wallace,
R. 2003. Help Seeking and Help Design in Interactive
Learning Environments. Review of educational
Research, 73, 3, 277-320.
Puustinen, M. 1998. Help-seeking behavior in a problem-
solving situation: Development of self-regulation.
European Journal of Psychology of Education, 13,
271-282.
Paquette G., De la Teja, I, Lundgren-Cayrol, K, Léonard,
M., Ruelland, D. 2002. La modélisation cognitive, un
outil de conception des processus et des méthodes
d’un campus virtuel. Revue de l’Éducation à distance,
Edmonton, Numéro spécial RCE-Téléapprentissage,
vol. 17, No3, 4-25.
Noury, F., Hansel, J., Huet, N. 2006. Metacognition and
motivation on Help-seeking in learning statistics on a
web site., EARLI-SIG-Metacognition 2006,
Cambridge, UK, 19-21/07/2006.
Wooldridge M. 2002. An introduction to MultiAgent
Systems, John Wiley & sons, Ltd, 2004.
Bellefemine, F., Caire, G., Trucco, T., Rimassa, G. 2004.
Jade programmer’s guide (jade 3.2). Technical
Report, TILab S.p.A., 2004.
CSEDU 2009 - International Conference on Computer Supported Education
216