INTELLIGENT TUTORING SYSTEM: AN ASSESSMENT

STRATEGY FOR TUTORING ON-LINE

Francesco Colace, Massimo De Santo, Mario Vento

DIIIE-Università degli Studi di Salerno

Via Ponte Don Melillo, 1 84084 Fisciano (Sa)

Keywords: E-Learning, Assessment, Bayesian Networks, Metadata.

Abstract: In this paper we introduce a tutoring approach for E-Learning formative process. This approach is strictly

related to the assessment phase. Assessment in the context of education is the process of characterizing what

a student knows. The reasons to perform evaluation are quite varied, ranging from a need to informally

understand student learning progress in a course to a need to characterize student expertise in a subject.

Otherwise finding an appropriate assessment tool is a central challenge in designing a tutoring approach. In

this paper we propose an assessment method based on the use of ontologies and their representation through

a Bayesian Networks. The aim of our approach is the generation of adapted questionnaires in order to test

the student’s knowledge in every phase of learning process. Analyzing the results of the evaluation an

intelligent tutoring system can help students offering an effective support to learning process and adapting

their learning paths.

1 INTRODUCTION

Our “information-oriented” society shows an

increasing demand of life-long learning. In such

framework, on-line learning is becoming a real

solution that allows flexibility and quality in the

learning process. In this field the assessment phase is

acquiring a strategic interest. In fact, assessment is

an important and difficult task in the whole teaching

and learning process (Dahlgren, 1984). In fact it can

furnish timely feedback through just-in-time

problem solving aid that turns the learning process

into a highly interactive learning experience. On the

other hand the implementation of an effectiveness

on-line evaluation strategy can be very difficult. In

fact experts consider the evaluation phase as the

weak point in an E-learning process. Many of

currently existing assessment systems focus on

simple assessment strategies, e.g. only on single or

multiple-choice questions with several answers, and

radio-buttons to select the correct answer.

Furthermore most of them are unable to support the

different needs of individual users and focus mainly

on the assessment of the “average user”. In this way

teachers can give only a mere quantitative evaluation

of students’ knowledge and can not fill in the gaps in

their learning approach. Assessment provides an

effective method to gather information about

student’s learning and it is a good starting point in

order to arrange feedback’s strategy. Finding an

appropriate assessment tool is a central challenge in

designing an assessment approach (Royer, 1993).

The difficulty arises because of the diversity of

learning objectives, the diversity in what counts as

evidence of learning, the diversity of tools available,

the varying resources available, and the varying

assessment contexts. One way to address these

various assessment goals and challenges is through

the use of concept maps (Turns, 2000). So starting

from this general framework in this paper we

describe our system for assessment and tutoring

based on Ontology formalism (the generalization of

concept maps) and metadata standards. In this paper

we represent ontology through Bayesian Networks

formalism because in this way we can depict and

estimate the preparatory links between the various

subjects belonging to knowledge domain. In this

way it is easier to understand the real knowledge

shortage of students (Colace, 2003). In this paper we

design and implement a tool that arranges the best

assessment strategy according to the information

inferred by the analysis of questionnaires. Our tool

can test the knowledge of students on every subject

of ontology adopting various approaches. The

Bayesian approach allows quantizing the probability

of correct answer of students in a particular subject.

In this way tool can propose to the student the

question with the lower (or higher) probabilities of

correct answer. At the end of the assessment

157

Colace F., De Santo M. and Vento M. (2006).

INTELLIGENT TUTORING SYSTEM: AN ASSESSMENT STRATEGY FOR TUTORING ON-LINE.

In Proceedings of the Eighth International Conference on Enterprise Information Systems - HCI, pages 157-160

DOI: 10.5220/0002457001570160

 SciTePress

student’s profile is updated. The paper is organized

as follows: in Section 2, we provide the motivations

and the details of the proposed assessment tool. In

particular we give some details on ontologies and

their mapping through Bayesian Networks. In

section 3 we describe the proposed approach. In

section 4 the experimental results are reported.

Finally, in the last section we draw conclusions and

indicate future directions of our research.

2 THE TRACKING AND

ASSESSMENT PROBLEM

Some of the tasks that an E-Learning platform

should carry out aim to allow people to find,

evaluate and acquire adapted learning objects. These

activities are common and easy to carry out in

traditional learning processes: we can not say the

same in distance learning field. In the designing and

organizing phase of a course, a teacher has to choose

the most appropriate contents: this selection shows

notable difficulties, also due to the huge amount of

information available, of which only a minimum

part really meets teachers’ needs. The possibility of

accessing to contents that could be useless or not

related to the subjects of interest is considerable. In

this framework a very important role is played by

the assessment phase. Assessment gives to the

learning environment the most direct information

about the student’s knowledge. The best assessment

approach provides questionnaires that are built

dynamically on the basis of the student model.

Questions have to cover the topics most recently

completed, as well as those that should be reviewed.

Each question has a level of difficulty, which is also

used in the updating phase of student model.

Correctly answering a harder question demonstrates

greater ability than correctly answering an easier one

(Shang, 2001). The assessment framework combines

researches from two major research disciplines:

adaptive educational hypermedia and semantic web

technologies. In this paper we focus our attention on

the assessment phase and in particular in the

designing of an adapted assessment generator and

assessment module.

2.1 Ontologies

The concept of ontology was taken from philosophy

where it means a systematic explanation of being. In

recent years, however, this concept has been

introduced and used in different contexts, thereby

playing a predominant role in knowledge

engineering and in artificial intelligence. Ontologies

could be represented as taxonomic trees of

conceptualizations: they are general and domain-

independent at a superior level, but become more

and more specific when one goes down the

hierarchy. In other words, when we move from the

highest taxonomic levels to the lowest ones,

characteristics and aspects typical of the domain

under examination are showed. In order to point out

this difference in literature we call them

heavyweight (deeper ontology) and lightweight

(advances taxonomy) ontology respectively. In this

paper we will adopt the last one approach keeping in

mind this definition of ontology: “ontology may take

a variety of forms, but it will necessarily include a

vocabulary of terms and some specification of their

meaning.

2.2 Ontologies and Bayesian

Networks

Bayesian networks have been successfully used to

model knowledge under conditions of uncertainty

within expert systems, and methods have been

developed from data combination and expert system

knowledge in order to learn them. A Bayesian

network is a graph-based model encoding the joint

probability distribution of a set of random variables

X ={X

, …, X

). It consists of a directed acyclic

graph S where each node is associated with one

random variable Xi and each arc represents the

conditional dependence among the nodes that it

joints and a set P of local probability distributions,

each of which is associated with a random variable

Xi and conditioned by the variables corresponding to

the source nodes of the arcs entering the node with

which Xi is associated. The lack of an arc between

two nodes involves conditional independence. On

the other hand, the presence of an arc from the node

Xi to the node Xj represents that Xi is considered a

direct cause of Xj. In this paper we use an algorithm,

based on the formalism of the Bayesian networks,

able to infer propaedeutic relationships among

different subjects belonging to the knowledge

domain of a course (Colace, 2003). The first step of

this algorithm is the introduction of a mapping

between Ontology and Bayesian Network. In our

ontology model nodes represent the subjects

belonging to the course knowledge domain and the

arcs mean a propaedeutic relationship among the

nodes. We can map this ontology graph in a

Bayesian network in the following way: the

Bayesian networks nodes can model the subjects

belonging to the course Knowledge Domain and the

knowledge of subject by students while arcs in the

same way can mean the propaedeutic relationships

ICEIS 2006 - HUMAN-COMPUTER INTERACTION

158

among the nodes. Given the previous mapping

strategy our aim is to use and update the ontology

used by teacher in his/her course. Obviously we

must define data type and data set for this approach.

As previously said the student’s answers to the end

course evaluation tests represent a source of implicit

evidence. In fact, teachers through the end-of-course

evaluation tests not only assess student’s knowledge

for every subject, but describe the course ontology

and outline the propaedeutic aspects that relate

subjects each other. In this way we can use an

updated ontology and we can measure the effective

propaedeutic links between the various topics of a

course.

3 THE PROPOSED SYSTEM

In this section we will describe in detail the

architecture of our tool (named Virtual Teacher) and

the proposed assessment tutoring strategy. As

previously said we aimed to design a tool for

assessment able to assist students and teachers in the

formative process. We designed our tool keeping in

mind the main needs of students and teachers. From

a technological point of view we designed the tool

according these constraints: Web based approach,

aesthetic and minimalist design, flexibility and

efficiency of use, help users recognize, diagnose,

and recover from errors. In the first phase of the

designing we pointed out the actors of the system

and the use cases. We identified three typologies of

actors in the system: Administrators, Teachers and

Students. Each of these figures has a well defined

role and tasks. In particular Administrators can

introduce new courses, describe new ontologies and

manage the accesses to the tool. Teachers can design

the reference ontology, describe the learning objects

and the questions linked to the nodes of ontology.

Teachers can also manage the reports of every

student in order to better supervise the learning

process. Students can use tool in three different

ways: Exam, Normal test, Bayesian test. In the

Exam way our tool arranges a classical final test

exam according to the teacher’s strategy. At the end

of the exam the system produces a report analyzing

the performance of student in every subject. The

normal test approach can be used during some

module of the course. The more interesting service

offered by our tool is the Bayesian test. This service

makes the most of the matching between ontology

and Bayesian network. In fact the first step is the

introduction of a mapping strategy between

Ontology and Bayesian Network. In our ontology

model nodes represent the subjects belonging to the

knowledge domain of the course and the arcs mean a

preparatory relationship among the nodes. In this

way we can map the ontology graph in a Bayesian

network in the following way: the nodes of Bayesian

Network model the subjects belonging to the course.

The states (two: yes and not) of nodes represent the

knowledge of student in the subject. The arcs mean

the propaedeutic relationships among the nodes. In

other words a node of Bayesian network-ontology

represents the Knowledge domain of a course and

quantizes student’s knowledge of this node. First of

all the system select a set of questions associated to

every network node. At the end of this first phase

system, through a Bayesian approach infers what

subjects the students knows better than others. In

fact through the Bayesian analysis the system can

measure the percentage of correct answer in a

subject. In particular it can predict the percentage of

correct answer to a subject after a correct (or not)

answer to questions related to propaedeutic subjects.

At this point it can apply various strategies: for

example it can select and propose to the student the

question with the smaller percentage of correct

answer. At the end of Bayesian test a detailed report

on the knowledge of student in the various subjects

is sent to teacher and to student himself. In particular

after the Bayesian test the system proposes to the

student some learning object for deepening some

subjects. At the end of Bayesian Test the system

updates the user profile of students.

4 EXPERIMENTAL RESULTS

In order to test the effectiveness of our tool we used

it during the course of Introduction to Computer

Science at Foreign Literature and Language Faculty

of University of Salerno. This course is composed

by seven modules: Introduction to PC Architecture,

Introduction to Operative System, Microsoft Word,

Microsoft Excel, Microsoft Access, Microsoft Power

Point, and Internet. On the basis of the

considerations of previous section, teacher designed

the reference ontology. Each node of the networks

has two states and shows the probability that a

generic learner knows the subject associated with the

same node. We have supposed that each node can

assume only the following two states: state ‘Yes’

complete knowledge of the subject and state ‘Not’

total ignorance on the subject. The student level of

knowledge could be evaluated on the basis of the

answers given to the questions (a set of questions is

proposed for each subject). At the end of the course

students have to get through a final examination’s

test composed by forty questions. The questions

INTELLIGENT TUTORING SYSTEM: AN ASSESSMENT STRATEGY FOR TUTORING ON-LINE

159

belong to every subject of knowledge domain. The

number of student’s course was about 50 and at the

starting of the course we arranged them in two

groups (named blue and red). The first group had a

classical support to course activities and used only

the normal test approach while the latter <group

used also all functionalities of the tool as didactic

support. At the beginning of the course teachers

designed every module’s ontology in order to

organize the contents and an assessment test. The

results are in the table 1. The aim of this test is to

allow a first description of student through a

metadata structure. In this way teachers can obtain

information about the initial knowledge level of

students. This information is very useful in order to

describe for the first time the student profile. At this

point the system organized for the student of red

group a support material for every module of course.

In particular it proposes the most suitable contents

through a matching between the metadata of

learning objects and the description of the student.

As previously said during the course the students of

the two groups attended to the lessons and used the

virtual teacher tool. In particular students of red

group at the end of every module sustained a

Bayesian Test. At the end of course students had

their final course exam. In the table we depicted the

results:

Table 1: Results of Assessment Tool. The meaning of

range are: [0-10]: inadequate, [11-15]: poor progress, [16-

20]: adequate, [21-25]: good, [26-30]: very good.

Blue Group Red Group

Assessment

Test

Students Assessment

Test

Students

0-10 10 0-10 12

11-15 11 11-15 10

16-20 9 16-20 8

21-25 3 21-25 3

26-30 3 26-30 2

Total 36 Total 35

Table 2: Results of Final Test.

Blue Group Red Group

Final Test Students FinalTest Students

0-10 4 0-10 3

11-15 9 11-15 5

16-20 8 16-20 6

21-25 10 21-25 12

26-30 5 26-30 9

Total 36 Total 35

If we analyze the difference between the assessment

and the final exam (table 1 and 2) we can note that

the percentage of students that get through the

assessment test is 37% in the red group and 42% in

the blue group while in the case of the final

examination the percentage is 77% in the red group

and 64% in the blue group. We can note as more

students of red group get through the final exam and

improve their performance respect the assessment

test (about 40%). In particular we can note that the

students of the blue group have a minor

improvement (about 22%) than the students of the

red group. At the same time the percentage of red

group’s students that have a mark in the range 26-30

is higher than in the case of blue group: 26% to 8%.

In order to collect more information about the

effectiveness of our tool at the end of course we

submitted a questionnaire to every student. In the

questionnaire we asked the effectiveness of

Bayesian test and of learning objects furnished by

system at the end of the test. The 87 % of students

said that the support of Virtual Teacher tool was

very important in the learning process. In particular

the 73% of students declared that the supporting

learning object helped them in a better knowledge of

the various subjects.

5 CONCLUSION

In this paper we proposed a tool for the assessment

and tutoring of students during a learning process.

This is based on the use of ontology and Bayesian

Network. In particular through the matching

between ontology and Bayesian Network our tool

allow an effective tutoring and a better adaptation of

learning path to demands of students. The

assessment based on Bayesian approach allows a

deeper analysis of student’s knowledge. The

experimental results seem to confirm our approach.

As a future step of our research we intend to

evaluate the performance of the proposed system

when new features for tracking strategies are used.

REFERENCES

L. Dahlgren, Outcomes of learning, in The Experience of

Learning, F. Marton, D. J. Hounsell, and N. J.

Entwistle, Eds. Edinburgh, U.K.1984,

J. M. Royer, C. A. Cisero, and M. S. Carlo, Techniques

and procedures for assessing cognitive skills, Rev.

Educ. Res., vol. 63, no. 2, pp. 201–243, 1993.

J. Turns, C. J. Atman, and R. Adams, Concept Maps for

Engineering Education: A Cognitively Motivated Tool

Supporting Varied Assessment Functions, IEEE

Transactions On Education, Vol. 43, No. 2, 2000

F. Colace, M. De Santo, P. Foggia, M. Vento, Ontology

Learning Through Bayesian Networks, Proceedings of

ICEIS 2003 Conference, Angers, 2003

Yi Shang, Hongchi Shi, and Su-Shing Chen, An Intelligent

Distributed Environment for Active Learning, ACM

Journal of Educational Resources in Computing, Vol.

1, No. 2, 2001

ICEIS 2006 - HUMAN-COMPUTER INTERACTION

160