HELP DESIGN FOR THE METACOGNITIVE GUIDANCE

OF THE LEARNER

A Proposition of Computer-Based System

J-C. Sakdavong

, F. Adreit

and N. Huet

CLLE-LTC, UMR CNRS 5263

IRIT, Toulouse le Mirail University

5, Allée A. Machado, 31058 Toulouse cedex 1, France

Keywords: Learning, Distance education, Computer supported education, ICT, AIED, Modelling, Help, Guidance,

Metacognition, Metacognitive guidance, Multiagent system.

Abstract: This paper presents the framework and the software system we have built in order to provide metacognitive

guidance help in Computer Supported Education. The goal is to assist self-regulation of the learner thanks to

a dynamic help system which takes into account in real-time the learner's behavior and his profile. The

software system is a multiagent system which captures the learner's behavior, analyse it and define the help.

Used currently in the step of conception, the software system will become the learning system by successive

learning and enrichment. We present its principles and its operational aspects. The application field of this

work is the French certificate “C2i”.

1 INTRODUCTION:

SELF-REGULATION AND

METACOGNITIVE HELPS

We are in the context of learning throughout life

supported by Information and Communication

Technologies (ICT). Students, workers, retired

people are more and more often faced with learning

alone, at home or on their workplace. In this context,

there is usually no teacher present to monitor and

assist them. To meet this need, a lot of learning is

given through computer-based learning systems:

online tutorials, courses, or more integrated systems

(Learning Management Systems, Learning and

Content Management Systems). In this way, learners

have spatial (through remote systems) and temporal

(the learners can learn according to their availability)

autonomy. If they are properly designed, these

systems allow education adapted to the profile of the

learner: his knowledge, learning experience,

metacognitive profile.

Unfortunately, it is now clear that these systems

do not often achieve their goal (Osman and

Hannafin, 1992; Winne and Stockley, 1998). One of

the problems of the effective use of ICT for learning

is that these systems require that the learners

regulate their own learning (Avezedo, 2005). Few of

them have the required skills for taking in charge

ones cognitive functioning (e.g. Hannafin & Land,

1997). Acquiring new knowledge and ability appeals

not only to cognitive processes (activation of

knowledge, use of adapted learning strategies, and

memorization of new knowledge) but also to

metacognitive processes (planning, self-evaluation,

learning regulation). In face to face education, some

of these metacognitive activities are provided by the

teacher; in the context of computer supported

education, they fall to the learner. It is why we speak

about self-regulated learning for this kind of

learning.

To help learners to regulate themselves, the

designers of computer-based learning systems have

added learning helps to support planning, self-

evaluation and learning regulation. The table 1 gives

some examples of help available on computer-based

learning systems; we can see what may be provided

by the teacher (in face to face education) and what is

left to the learner in the context of computer

supported education.

210

Sakdavong J., Adreit F. and Huet N. (2009).

HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based System.

In Proceedings of the First International Conference on Computer Supported Education, pages 209-215

 SciTePress

Table 1: Examples of helps.

Metacognitive

activity

Help provided by the teacher (in face to face

education)

Help available in computer-based learning systems

Planning The teacher defines the purposes and the

duration of the lesson.

The teacher defines the distribution of learning

activity.

The system asks the learner to define a priori

his learning time.

Self-

evaluation

The teacher defines exercises or questions to

verify the control of a concept or a part of the

lesson.

The learner can ask questions to the teacher or

to other learners.

The system provides exercises, questions and

feedback.

The learner can ask questions to the teacher or to

others learners via the forum.

Regulation If the learner meets with difficulties, the teacher

can provide help (for example, an explanation

in a scheme-form), an advice ("read again a

chapter"), an indication ("look for the definition

of a concept”), an answer (a solution).

The learner can approach other learners.

Depending on the answers to questions, the system

may propose to the learner to revise a concept.

The learner can approach other learners via the

forum.

However, the efficiency of these learning helps

has not been really evaluated and some recent works

show difficulties in their use. We can notice that

some of them are underused or even unused

(Narciss, Proske, Körndle, 2007; Narciss, Körndle,

Dupeyrat, 2002). Other works highlight inadequate

use of help; it is what Roll et al. call metacognitive

bugs (Roll et al., 2005): for example, unorganized

over-use of help (Roll et al.,2005), exclusive use of

help that provides an answer rather than an

indication to look for the answer (Aleven et al.,

2003).

This rather disappointing acknowledgement

raises questions about the metacognitive abilities

which are necessary for adequate use of help: to be

aware of needing help, to choose an appropriate type

of help, to detect the usefulness of help, to realize

when help is necessary, and, after failing, to detect

which help to revise (Aleven and al., 2003 ;

Puustinen, 1998). Works in metacognition field

show clearly the difficulties to acquire and to apply

such metacognitive abilities, even in face to face

education. They show also that these skills change

according to factors like learner's knowledge and

age.

2 THE CEAGMATIC PROJECT

2.1 An Original Project

Our work is a part of the CEAGMATIC project of

the French National Research Agency (ANR).

Researchers involved in this project are members of

the CLLE-LTC (Laboratoire Travail et Cognition)

and IRIT (Institut de Recherche en Informatique de

Toulouse)laboratories.

The main goal of this project is to design and

build a help guidance system to improve learners’

metacognitive abilities. This system has to analyze

the learner’s profile and to react in real time to

learner’s behavior. The project team is composed of

researchers in the fields of cognitive psychology and

computer science.

To the best of our knowledge, only one research

center has made such a system (the Human

Computer Interaction Institute of Carnegie Mellon

University, Roll et al., 2005) but with only one kind

of metacognitive guidance. Our system will go

further by proposing and comparing many kinds of

guidance: one proposed and then accepted by the

learner and another one, imposed by the system (in

order to compensate for a learner’s metacognitive

lack or inappropriate behavior).

Another important part of this project is the

learners’ profiling. We take into account learners’

demographics, cognitive and metacognitive profiles:

the system will build a learner profile through

questionnaires and real time activity analysis. This

profile will be used in order to select how and when

to help and to guide the learner. It will evolve

according to the effectiveness of the helps and

guidance. This designing choice allows us to target

heterogeneous categories of learners (workers,

students, …) as we can adapt the system’s help and

action to the learner’s profile without overloading

him with useless interactions and documents.

HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based

System

211

Moreover, we can provide the help progressively,

when the learner (in fact, his profile) evolves.

2.2 The Project’s Steps

The project begun on 2007 and will end on 2011. In

order to design and build the help and guidance

system, we have defined the following steps:

• Preliminary: Defining learner’s regular and

inappropriate behaviors by analyzing learners'

behaviors on provided interactive lessons and

exercises (the first version of the system does

not provide specific helps) ; this step is

completed

• Providing Cognitive Helps: Adding helps

devoted to “inappropriate behaviors” identified

during the previous step. Then analyzing again

learner’s regular and inappropriate behaviors

integrating these new helps (we will target bad

use of these helps as the “metacognitive”

inappropriate or missing behaviors) ; this step is

in progress

• Providing Metacognitive Helps: Adding

metacognitive guidance actions devoted to

metacognitive needs identified in the previous

step. These actions will be either guidance

actions or metacognitive profiling actions.

In order to support these 3 steps, we have built a

multi-agent system which can capture and analyze

learners’ behavior while they study. This system has

to provide the lessons and exercises to the learners

(Figure 1). The main goal of this paper is to describe

and explain how and why we are building this

system. The other results of our experiments will be

presented later after step 3 will be finished.

2.2.1 Step 1: Preliminary

• A psychologist has studied and analyzed

learners’ behaviors during face to face lessons

• Experienced teachers have specified the

learning activity and the optimal behavior in

terms of tasks and knowledge (Paquette and al.

2002)

• e-Learning engineers have built an online

course according to the previous specifications

• The online course has been tested over 100

learners and the multi-agent system has

recorded all the learners’ behaviors into activity

graphs (Figure 2)

• Psychologists are analyzing the activity graphs

of each learner in order to identify the

characteristic behaviors e.g. the learners’

regular and inappropriate (mistakes) behaviors

while doing exercises.

Figure 1: User interface of the multiagent system.

Figure 2: Activity graph recorded by the system.

2.2.2 Step 2: Providing Cognitive Helps

This step consists in conceiving actions associated

with each characteristic behavior identified during

the step 1. These actions will help the learner when

he makes “cognitive mistakes”.

• The cognitive helps will be designed by

psychologists

• These helps will be included in the multi-agent

system as helper agents

CSEDU 2009 - International Conference on Computer Supported Education

212

• The online course including helps will be tested

over many groups of learners and the multi-

agent system will again record all the learners’

behaviors into graphs

• Psychologists will analyze these activity graphs

in order to identify the characteristic behaviors

while using the new helps e.g. the learners’

regular and inappropriate regulation actions

while doing exercises (for example, a learner

who never accepts to read again the lesson

when the helper agents propose to do it).

2.2.3 Step 3: Providing Metacognitive Helps

This step consists in conceiving metacognitive

guidance actions associated with each characteristic

behavior identified during the step 2 e.g. bad use of

helps. These actions can be assistances but also

refinements of learners’ metacognitive profiles.

Thus, the system will progressively build precise

profiles.

We speak here of “metacognitive guidance”

because our hypothesis is that if learners do not use

correctly the helps, it is because they have a lack of

metacognitive abilities: they do not regulate

correctly their learning behavior.

Two types of metacognitive guidance will be

proposed:

• A suggested guidance that the learner can

accept or refuse (Noury and al., 2006), (for

example, “you should look at the glossary”;

“You should do the exercises before doing the

test”)

• An imposed guidance if the system identifies a

recurrent metacognitive mistake or lack. (for

example, a definition from the glossary is

presented to the learner)

This is the principal specificity of our approach

from the point of view of psychology.

Then:

• The two types of metacognitive guidance will

be included in the multi-agent system as new

helping agents

• The online course including guidance will be

tested over many groups of learners and the

multi-agent system will again record all the

learners’ behavior into graphs

• Psychologists will analyze the graphs of

learners’ behaviors in order to check if the

metacognitive helps are useful.

2.4 Experiment: The Chosen Target

Learners

• To experiment, we have chosen the French

certificate “C2i (level 1)” (Computer Science

and Internet Certificate). The learners have

different backgrounds, levels of study and ages.

• Moreover, we have a large population of

students for testing and teachers experienced in

this training (in the universities of the two

involved research laboratories).

• After having analyzed the results over the tests

of C2i, we have chosen to target the

“Formatting documents with style sheets”

lesson. Indeed, this lesson presents cognitive

and metacognitive difficulties which can be

supported by the help guidance system.

3 DESIGN PRINCIPLES OF THE

COMPUTER-BASED SYSTEM

3.1 Dynamics, Flexibility and

Scalability of the System

Our system is based on the observation and analysis

of the learner's behavior. So, it is based on a

dynamic component (the activity) from which the

help is constructed dynamically (by observation and

analysis). Therefore, the system has to be able to

observe, analyze the learner's behavior and to

construct real-time help.

Moreover, we propose a general help principle

which can be used in any learning situation.

Therefore, the system has to be easily adapted,

keeping the generic functions of the system, just

modifying the learning situation. For example, in the

experimentation, it is used for the learning situation :

"Formatting documents with style sheets". It should

be used for other learning situations like "Using of

table of contents or index" or, beyond word

processing, like "Using functions in spreadsheet".

Finally, while the system is currently used as a

workshop which allows psychologists to observe the

learners' behaviors, it will be able (in the third step)

to integrate into a dynamic whole the observation-

analysis-help process.

3.2 Bootstrap, Grading and Regulation

of the System

To analyze the learner's behavior, we use the

description of the regular activity and of the possible

HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based

System

213

deviations which are linked to the knowledge and

the ability of the current exercise. For example,

"Formatting a paragraph" is linked to paragraph and

alignment concepts (knowledge), and to the

designation of a paragraph and of an alignment

(ability). The regular activity and the possible

deviations compose the system’s bootstrap.

They are completed and refined during the

analysis of the activity by the psychologists. This

analysis allows also designing the action system: an

action (cognitive profile modification, help

suggestion) is associated with each characteristic

behavior. It is what we call the grading of the

system.

Finally, the regulation of the system consists in

adapting the help as the system runs. Ideally, it

should be a self-regulation of the system. But for the

project, we will allow only a dynamic regulation

within the action system defined by the

psychologists in the second step.

3.3 Functional Aspects

Our help system is based on the observation and

analysis of the learner's behavior. Therefore, it is

necessary that it places the learner in a position to do

and can observe his actions. It is why the system has

to integrate learning interactive tools. For example,

we have integrated a text editor for the

experimentation.

To analyze the activity, we have integrated a tool

which allows the observation of the learner's

behavior. We wondered about the granularity of the

observed actions. Technically, it was possible to

observe elementary actions (click, mouse moved,

…). But, after a first test, we realized that the

important actions were:

• Actions on interactive objects ; for example, a

selection in a menu or the validation of a dialog

box

• Semantic actions linked to the learning context;

for example, putting a word in italics

Therefore, the system observes and records these

actions which are analyzed by the psychologists,

observed and then processed by the system to

generate the help in the next step.

The psychologists analyze the learner's behavior

in relation to the regular activity. To make the

analysis of the activity easier, we have integrated

tools to describe the regular activity and to represent

the observed one with the same graphic formalism;

these tools can also represent the differences

between regular and observed activities.

Finally, the system contains a tool of automatic

analysis of the activity which allows detecting

characteristic behaviors, and an action system able

to activate help and to modify the profile.

All these functional aspects have been integrated

to the computer system.

4 SOFTWARE ARCHITECTURE

OF THE MULTI-AGENT

SYSTEM

4.1 Why a Multi-Agent System?

We have implemented the device to deliver online

course and to provide helps as a multi-agent system

(Wooldridge, 2002). This choice of implementation

allows us:

• To have software elements (agents) able

intrinsically to observe the activity and to

produce a behavior, also to communicate

between them;

• To obtain a dynamic behavior of the device,

creating agents during the learner's behavior

(for example, creating a new helping agent

when an exercise starts) or modifying in real

time the behavior of agents (for example, an

helping agent can change of behavior

according to an evolution of the learner’s

profile);

• An incremental construction of the device;

• A flexible and dynamic construction of the

device; for example, we can replace the agent

“text editor” by an agent “spreadsheet”

according to the situation of training, without

modifying the remainder of the device;

• To consider a distributed runtime of the various

elements of the device on various computers

(the learners’ computers, the LMS’s computers

and the learners profiler computer).

We have used the framework JADE

(Bellefemine, F. and al., 2004) and programmed the

agents in the Java language.

4.2 Agents of the System

The system is composed by several types of agents:

• ”Principal”, “Exercise” and “Applicative”

agents which constitute the LMS (Learning

Management System);

• An “Historical” agent;

CSEDU 2009 - International Conference on Computer Supported Education

214

• ”Helper”, “Scrutinizing” and “Profile” agents

which constitute the system of analysis and the

system of help.

The LMS includes a “Principal” agent which

implements the teaching scenario. The figure 1

shows the human-computer interface of the LMS:

the summary (“Résumé”), the course and the

exercises (“Cours et exercices”), the self-assessment

(“Autoévaluations”), the external references

(“Références”), the glossary and the index

“(“Glossaire, index)”. When the learner chooses an

exercise, the “Principal” agent creates an “Exercise”

agent implementing the scenario of the

corresponding exercise; we can have thus

simultaneously several “Exercise” agents An

“Exercise” agent is always associated to an

“Applicative” agent which implements the

interactive system necessary to the realization of the

exercise; in the current project, this agent

implements a word processor.

The “Historical” agent records the learner’s

behavior as a sequence of actions (the activity

graph). It thus communicates with the previous

agents: it records the activity with respect to the full

teaching scenario (for example, it records if the

learner consults the exercises, then reaches the

course), to the scenario for a particular exercise (for

example, when the learner answers the first question,

then the second one, then returns to the first one), to

the “Applicative” agent (for example, the learner

selects a paragraph then clicks on the shortcut button

“centering the paragraph”).

“Scrutinizing” agents allow observing and

analyzing the activity of learning. These agents are

charged to identify characteristic behaviors,

according to the profile. They are created

dynamically by the “Exercise” agents. They have a

mechanism of subscription which enables them to

receive from the “Historical” agent the sequences of

actions they are charged to analyze. According to

their analysis, they will create “Helper” agents or

will communicate with the existing “Helper” agents.

They will also communicate with the “Profile” agent

charged to dynamically adapt the profile of the

learner.

The “Helper” agents provide the assistance by

giving feedback, displaying solution, procedure,

chapter corresponding to the difficulty, asking

questions to the learners. In the last step of the

project, they will give the metacognitive guidance to

the learners. They also will communicate with the

“Profile” agent.

5 CONCLUSIONS

We have presented the process and the software

device that we have developed, associated to the

design of a new kind of help. The multiagent

architecture used to implement the software system

is an original way to deal with the complex problem

of a dynamic and contextual learning help. It allows

to meet the dynamic, flexibility and scalability

requirements of the device.

We are testing it with the learning of the C2i

certificate. At present, we have realized the first

step of the process (we have defined the regular

behavior and the possible deviations) and constituted

the bootstrap of the software device. Then we have

recorded the behavior of a troop of learners with the

software device. Currently, a psychologist is

analysing these recordings (step 2 of the process).

Afterwards, the results of this analysis will be

integrated into the system and will be evaluated.

At the same time, we are working on the

specification of « Helper » agents to add syntactic

analysis abilities to them: each « Helper » agent will

be defined by an abstract grammar which will be

specific to a learning behavior. Then, the

psychologists would just have to define abstract

grammars and associated semantic actions.

REFERENCES

Osman, M. E., Hannafin, M.J. 1992. Metacognition

research and theory: Analysis and implications for

instructional design. Educational Technology

Research & Development, 40(2), 83-89.

Winne, P. H., Stockley, D. B. 1998. Computing

technologies as sites for developing self-regulated

learning. D. H. Schunk and B. J. Zimmerman (Eds.),

Developing self-regulated learning: From teaching to

self-reflective practice 106-136. New York: Guilford.

Azevedo, R. 2005. Using hypermedia as a metacognitive

tool for enhancing student learning? The role of self-

regulated learning. Educational Psychologist, 40(4),

199–209.

Hannafin, M. J., Land, S. M. 1997. The foundations and

assumptions of technology-enhanced, student-centered

learning environments. Instructional Science, 25, 167-

202.

Narciss, S., Proske, A., Körndle, H. 2007. Promoting self-

regulated learning in web-based learning

environments. Computers in Human Behavior, 23,

1126-1144.

Narciss, S., Körndle, H., & Dupeyrat, C. 2002.

Promouvoir l’apprentissage auto-régulé avec

l’Internet. Colloque Compréhension et Hypermédia,

Albi, 10-11 octobre 2002.

HELP DESIGN FOR THE METACOGNITIVE GUIDANCE OF THE LEARNER - A Proposition of Computer-Based

System

215

Roll, I., Baker, R.S., Aleven, V., McLaren, B., &

Koedinger, K. 2005. Modeling students’ metacognitive

errors in two intelligent tutoring systems. In L.

Ardissono, P. Brna and A. Mitrovic (Eds.),

Proceedings of the 10th International Conference on

User Modeling, UM'2005, 379-388. Berlin: Springer-

Verlag.

Aleven, V., Fischer, F., Schworm, S., Stahl, E., Wallace,

R. 2003. Help Seeking and Help Design in Interactive

Learning Environments. Review of educational

Research, 73, 3, 277-320.

Puustinen, M. 1998. Help-seeking behavior in a problem-

solving situation: Development of self-regulation.

European Journal of Psychology of Education, 13,

271-282.

Paquette G., De la Teja, I, Lundgren-Cayrol, K, Léonard,

M., Ruelland, D. 2002. La modélisation cognitive, un

outil de conception des processus et des méthodes

d’un campus virtuel. Revue de l’Éducation à distance,

Edmonton, Numéro spécial RCE-Téléapprentissage,

vol. 17, No3, 4-25.

Noury, F., Hansel, J., Huet, N. 2006. Metacognition and

motivation on Help-seeking in learning statistics on a

web site., EARLI-SIG-Metacognition 2006,

Cambridge, UK, 19-21/07/2006.

Wooldridge M. 2002. An introduction to MultiAgent

Systems, John Wiley & sons, Ltd, 2004.

Bellefemine, F., Caire, G., Trucco, T., Rimassa, G. 2004.

Jade programmer’s guide (jade 3.2). Technical

Report, TILab S.p.A., 2004.

CSEDU 2009 - International Conference on Computer Supported Education

216