EXPERIMENTAL COMPARISON OF ADAPTIVE LINKS

ANNOTATION TECHNIQUE WITH ADAPTIVE DIRECT

GUIDANCE TECHNIQUE

Jozef Kapusta, Michal Munk and Milan Turčáni

Department of Informatics, Constantine the Philosopher University in Nitra, Tr. A Hlinku 1, Nitra, Slovakia

Keywords: Adaptive hypermedia systems, Links annotation, Direct guidance, Study time analysis.

Abstract: The problematics of educational environment adaptation when using the adaptive hypermedia systems

(AHS) not only includes the need to implement these systems, to develop the applicable adaptive problem

solving structures but also the evaluation of e-learning, pedagogical-psychological aspects of creating

materials supporting the education, scheming the subject matter, efficiency of the problematics presentation

etc. Based on their knowledge of the given field, the authors of this article have executed an experiment

aimed at the quantitative evaluation of results when searching for the options of AHS application in the

informatics courses at the Department of Informatics, Faculty of Natural Sciences, Constantine the

Philosopher University in Nitra. The gained experimental results have verified the didactical efficiency of e-

learning courses built by using adaptive hypermedia systems, the time-effectiveness of these courses, as

well as the choice of the best adaptation form. In the experiment, the adaptive annotation technique was

compared with the direct guidance technique. An important discovery coming from the results of the

executed experiment was that the direct guidance technique when compared with other techniques was the

least time-effective, but its didactical efficiency was the highest.

1 INTRODUCTION

The hypermedia materials lack personalization,

customizing and adapting of shown information to

individual needs of the user very often. Nowadays,

this disadvantage could be eliminated by application

of adaptive hypermedia systems (AHS) into

hypertext documents. This specific type of

applications combines hypermedia, user modeling

techniques and a certain type of artificial intelligence

that adapts the structure and contents of hypermedia

documents according to each user. The majority of

today’s AHS projects are aimed mostly at teaching

and presentation of information in education.

2 ADAPTIVE NAVIGATION

SUPPORT

Adaptive navigation support consists of influencing

user’s path in an information space. When using this

technique, the system’s adaptive core evaluates the

applicability of each shown link for the given user

and offers a result upon which it influences the

user’s path in the document system. This influence

can be directive in such a way that the system

disables the paths that aren’t applicable for the given

user and context or which are non-directive. In this

case, the system presents recommended (or not-

recommended) path in the information system to the

user by using various instruments. When using the

non-directive way, the system just sorts the links

according to their relevance or distinguishes the

important link differently (Brusilovsky, 2001).

To achieve the listed navigation methods in

information content, if using the directive or the

non-directive approach, the following techniques are

used mostly: direct guidance (the AHS guides the

user in an information space, which means it selects

the most applicable concepts and fragments assigned

to them), sorting links (links leading to other pages

are sorted hierarchically according to their relevance

(Kaplan, 1998)), links annotation (the adaptive

system marks links that are advisable for the user

(De Bra & Calvi, 1998)), hiding links (the links that

250

Kapusta J., Munk M. and TurÄ Ã ˛ani M.

EXPERIMENTAL COMPARISON OF ADAPTIVE LINKS ANNOTATION TECHNIQUE WITH ADAPTIVE DIRECT GUIDANCE TECHNIQUE.

DOI: 10.5220/0001824602500255

In Proceedings of the Fifth International Conference on Web Information Systems and Technologies (WEBIST 2009), page

ISBN: 978-989-8111-81-4

guide to the non-recommended information are

hidden (Paterno & Mancini, 1999)) etc.

3 THE RESEARCH

METHODOLOGY

There is a question needed to ask. Does a university

student need content and explanation-of-the-subject-

taught personalization when studying and is it even

appropriate to individualize the subject matter at a

university? In our opinion, we see as questionable if

it is effective to present the subject matter to

university students by using AHS or the ‘classical’

e-learning method. Even the using AHS, it is needed

to find the most effective technique of adaptation.

In our experiment, we have aimed at the

thematic field of ‘Programming Internet

Applications’ in a combined form of bachelor

studies of Applied Informatics. We have asked the

following research questions:

 Are e-learning courses built on the basis of

adaptive hypermedia systems didactically

effective?

 Are e-learning courses built on the basis of

adaptive hypermedia systems effective in

terms of the time needed to explain the given

subject matter?

 Which adaptive technique is the most

appropriate for students?

To prove the given questions, we have created

the following solution steps:

1. Creating of control and experimental groups.

2. Creating of reliable and valid measuring

procedures.

3. Realization of the experimental plan.

4. Understanding the data.

5. Checking validity of the used statistical

methods.

6. Data analysis and interpreting the results.

4 TECHNOLOGY USED

LMS Moodle was chosen to be the experimental

environment. This learning management system was

not only chosen because of its implementation at our

university as the university system dedicated for e-

learning and electronical study support, but also

because of its wide usage in the academic field when

managing education.

Except of several available activities in the

system that we have used to create the e-course, we

would like to point out the Lecture module that we

have used to create a lesson for direct student

guidance and the iLMS module that we have

implemented into LMS Moodle for the needs of our

experiment. This module has been used as an

adaptive system for link annotation.

4.1 The iLMS Module – A Link

Annotation Module

The iLMS module has been used for link annotation

which enables to recommend links to a user

according to metadata and defining of dependencies.

The system recommends the links by using four

tags: recommended link tag, ‘neutral’ link tag, a tag

when the system could not decide according to the

metadata and the not recommended link tag. The

iLMS module is an addition to the Moodle system.

From the technical point of view, the module

contains a new adaptive course format (the format

complements the traditional course formats, the

thematical and weekly ones) and some blocks for

creating adaptive content in LMS Moodle.

The module has been developed by Gert

Sauerstein as a part of his diploma thesis „KI-

Ansätze zur Lerner-Adaption in Lern-Management-

Systemen’ at the Technische Universität Ilmenau

(Ilmenau, Germany).

4.2 The Lesson Activity – A Module for

Direct Student Guidance

A lesson is an activity type that opens up a study

material in an interesting and flexible way. It

consists of many text pages so-called tabs that can

be extended by using pictures or hypertext links.

Each tab is enclosed by asking a question and the

student can choose from multiple possible answers.

If the student has answered this question right, he

can advance to the next page. If he has answered

false, he is being redirected to the previous page to

study the problematics again. The way of navigation

in the Lesson activity depends on setting the

parameters.

The Lesson is a universal module that can be

customized by the creator of the course according to

his ideas. He can decide to use a linear passing

through each pages of the lesson and he can close

each one by asking a check question. A more

effective, but also a more demanding way is when

the course creator decides to divide the pages of a

EXPERIMENTAL COMPARISON OF ADAPTIVE LINKS ANNOTATION TECHNIQUE WITH ADAPTIVE DIRECT

GUIDANCE TECHNIQUE

251

Lesson into more parallel paths and forces the

student to study the described problematics by

detail. (Švejda, 2006).

5 RESULTS

The experiment has taken place in the summer term

of 2007/2008 at the Department of Informatics. All

75 students of the second year of the bachelor’s

studies in Applied Informatics have participated in

this experiment.

5.1 Dividing into Groups

During the term, we have been monitoring four

groups of students in the ‘Programming Internet

Applications’ course that were created by regular

dividing into groups. The students were divided into

the following groups:

1. Without the LMS Moodle support

(Unsupported) – a group where classical F2F

teaching method was applied,

2. With ‘standard’ e-course support (Non-

Adaptation) - a group that was supervised by

using blended learning in LMS Moodle,

3. With direct guidance module support (Direct

Guidance) – a group that was studying by

using the direct guidance support,

4. With adaptive system for links annotation

support (Links Annotation) – a group where

the adaptive iLMS system was used.

Table 1: Experimental plan.

We have decided to use this classification

because the chosen groups had their virtual classes

already created in the system. The students without

the LMS support and studying with classical e-

course support build monitoring groups and students

with direct guidance module support and adaptive

system for links annotation support build the

experimental groups.

Courses in all three groups have had unified

structure. The learning matter of the thematic field

was divided into small parts, so that the part had had

the maximum height of 1.5 times area shown when

using typical display resolution. A test question was

asked after each part. Despite these adaptive systems

offer options for a much more interesting course

structure, this simple course structure has been used

intentionally to avoid distortion of results by

enriching adaptive courses or to avoid penalizing the

group with e-course support without adaptation.

5.2 Analysis of Pre-Test

The groups seem to be equivalent at first sight –

all groups have attended the same courses together

with the same teachers. Group allocation from the

sex and age point of view is also equivalent. We

have verified this fact statistically by using pre-test.

The pre-test in a way of entry test checking the basic

knowledge to pass the given problematics has been

applied. In the entry test that consisted of 15

questions, we have checked score and the time in

that the students were able to finish the test.

The following graph visualizes the MANOVA

results. It shows the point and confidence interval

estimates of pre-test score (Pre-Test) and the time

needed for its completion (Duration).

Figure 1: Means with Error Plot for Pre-Test.

Based on the MANOVA results, we do not reject

the null hypothesis stating that the difference among

the groups in pre-test score and the time needed to

process it is statistically insignificant, that means

that the vector of dependant variables (Pre-Test,

Duration) is independent from the Group factor.

This has confirmed the group equality presumption

and randomization wasn’t needed.

5.3 Analysis of Post-Test

After finishing their study we have evaluated the

knowledge of students by using an end-of-course

WEBIST 2009 - 5th International Conference on Web Information Systems and Technologies

252

test. The end-of-course test consisted of seven tasks.

These were ai med on proving the mastering of each

thematical area of the course.

Table 2: MANOVA, Multivariate Tests of Significance for

Post-Test.

Based on the MANOVA (Table 2) results, we

reject the null hypothesis with a 99% degree of

confidence, which means the vector of dependant

variables (Post-Test, Duration) is dependent on the

Group factor.

Table 3: ANOVA, Univariate Results for Post-Test.

From the univariate results of the analysis of

variance (Table 3) we reject the null hypothesis with

a 99% degree of confidence in case of a post-test

score, which means the Post-Test dependant variable

depends on the Group factor. Vice versa, the

differences in case of time needed to process the

post-test haven’t been proven. The following graph

visualizes the results of ANOVA/MANOVA.

Figure 2: Means with Error Plot for Post-Test.

After rejecting the null hypothesis in case of a

post-test, we ask what pairs are significantly

different.

Table 4: Tukey HSD test (Unequal N) for Post-Test.

Statistically significant differences have been

proven between Direct Guidance and Unsupported

(p<0,01) and between Direct Guidance and Non-

Adaptation (p<0,05) in favour of Direct Guidance.

Vice versa, an interesting discovery is that there

haven’t been detected any statistically significant

differences between Links Annotation and the other

Group factor levels.

Except of statistically significant differences,

based on the descriptive statistics (Figure 2), it can

be seen that the final test with better results have

passed those groups where AHS, this means Direct

Guidance a Links Annotation groups had been used.

From the adaptation methods it was naturally the

direct guidance group, of which the significant

differences we have proven. This fact is also

exponentiated by the statement that this particular

group had had the worst results in the entry test

(Figure 1).

Figure 3: Means with Error Plot for Post-Test.

We present the graph (Figure 3), in which the

end-of-course test results for each thematical field

are shown (Post-Test1, Post-Test2, Post-Test3),

because of data completeness, Also from the partial

point of view, the better post-test results score can

be seen in favour of the direct guidance method.

EXPERIMENTAL COMPARISON OF ADAPTIVE LINKS ANNOTATION TECHNIQUE WITH ADAPTIVE DIRECT

GUIDANCE TECHNIQUE

253

5.4 Study Time Analysis

The next field of experiment was to discover the

time needed to study the given problematics.

The Moodle system where the adaptive systems

were implemented contains a log file-creating

mechanism. The following graph shows an overview

of how much time the students in each group needed

to study the given problematics. From the available

time information, only the ‘pure’ study time values

were chosen, this means the intervals of signing onto

the system, ‘random’ course viewing or clicking on

activities like the forum, dictionary etc. have not

been included. There are only the values included

when the student has been working with each

chapter aimed at the particular thematic area in the

time values. The group without support is naturally

missing from the graphs; there has not been any

mechanism to monitor the relevant time data.

Figure 4: Means with Error Plot for Study Time.

Based on the ANOVA results, we do not reject

the null hypothesis stating that the study time score

difference among the groups is not statistically

significant, which means the Study Time dependant

variable does not depend from the Group factor.

Despite the fact that the statistically significant

differences have not been proven, the results when

taking study time into account are surprising.

Originally, we wanted to prove that using AHS is

not only didactically efficient, but also more time

effective than the classical e-course. But

surprisingly, in our conditions, we can state that the

students have spent more time studying when using

AHS than they would have spent with a classical e-

course. Even the didactically most efficient

adaptation method – direct guidance (the Direct

Guidance group) has been the least effective in

terms of study time. We explain this by the fact that

a course that leads a student by using direct guidance

can impress and motivate him in his next studies.

To complete the results, we show the graph (Figure

5), where the time needed to study a thematical field

(Study Time1, Study Time2, Study Time3) in terms of

the watched groups is shown. From these partial

values of study time, we can see even more

differences in using each guidance method.

Figure 5: Means with Error Plot for Study Time.

Based on results, we have decided to include

another factor into the model – Study time. By the

following analysis, we test if the adjusted group

means are different. The means are adjusted as if

there was the same (average) Study time factor value

in all groups.

Table 5: ANCOVA, Univariate Tests of Significance for

Post-Test.

We can see in the table (Table 5) that the

dependence between Post-Test and Study Time is

statistically insignificant (p>0,05), this means the

study time length needed has not affected the post-

test results. We reject the null hypothesis with a 95%

degree of confidence. The null hypothesis states that

the post-test score difference among groups are not

statistically significant, which means the Post-Test

dependant value depends from the Group factor.

The following graph visualizes the ANCOVA

results.

WEBIST 2009 - 5th International Conference on Web Information Systems and Technologies

254

Figure 6: Means with Error Plot for Post-Test.

After rejecting the null hypothesis, our question

is what pairs are significally different.

Table 6: Tukey HSD test (Unequal N) for posttest.

Statistically significant differences have been

proven between Direct Guidance and Non-

Adaptation (p<0,05) in favour of Direct Guidance.

6 DISCUSSION

Not to deemphasize the strength of statistical tests,

we have proven their validity. To gain data, we have

chosen to use reliable and valid measuring

procedures. To solve our research problem, we have

used two methods, the analysis of variance and the

analysis of covariance. Where the analysis of

variance is easier and does not require equation

regression premise in each group. On the other hand,

the interpretation is less valid if there exist

differences among groups in the inspected variable.

Similarly as in other situations, it is recommended to

execute both ways of analysis and to compare their

results. In our case, the results are identical and we

have a reason to consider them as sturdy.

By executing the previous analysis listed in

chapter 5, we have validated the didactical

efficiency of using adaptive hypermedia systems in

university education. We have proven statistically

that implementing a direct guidance system had had

positive impact on the end-of-course test results

among students studying with this support. Also

when using the adaptive links annotation, better end-

of-course test results can be seen according to

descriptive statistics (Figure 2), despite the fact that

a statistically significant difference hasn’t been

proven. Among the adaptive techniques, the most

advisable one seems to be the direct guidance

technique.

Interesting were the results when examining the

time-effectiveness where we haven’t proven that any

technique would be more time-demanding, but from

the results of descriptive statistics, we have

discovered time differences in favor of the non-

adaptive methods. The most time-demanding is the

direct guidance technique. In our opinion, this was

caused not only by the attractive form of presenting

the current problematics, but also by the technique

itself that avoids studying the problematics

improperly, which means it asks the student

mandatory questions that need to be answered

correctly in order to study further.

To approve the experimental results, we would

like to execute the described experiment again in the

winter term of 2008/2009 based on a bigger sample

of students. In this experiment, we plan to use

adaptive techniques only: the direct guidance

technique, links annotation and a new AHS with the

links sorting technique.

REFERENCES

Brusilovky, P., 2001. Adaptive Hypermedia. User

Modeling and User-Adapted Interaction, Vol. 11, pp.

87-110.

Brusilovsky, P., 2003. Developing adaptive educational

hypermedia systems: From design models to authoring

tools. In T. Murray (Eds.), Authoring Tools for

Advanced Technology Learning Environments.,

Kluwer Publishers.

De Bra, P. – Calvi, L., 1998. AHA: A generic adaptive

hypermedia system. In: Proc. of the 2nd Workshop on

Adaptive Hypertext and Hypermedia, (pp. 5–12),

Pittsburg, USA.

Kaplan, C. et. al., 1998: Adaptive hypertext navigation

based on user goals and context. In P. Brusilovsky

(ed.) Adaptive Hypertext and Hypermedia, Dordrecht,

Kluwer Publishers

Paterno, F., Mancini, C. 1999. Designing web interfaces

adaptable to different types of use. In: Proc. of the

Workshop Museums and the Web.

Švejda, G. et. al. 2006. Vybrané kapitoly z tvorby e-

learningových kurzov. Nitra : UKF

Švec, P. 2007. Využitie modulov tretích strán v LMS

Moodle. In: Divai 2007. (pp. 209-213) Nitra: UKF.

EXPERIMENTAL COMPARISON OF ADAPTIVE LINKS ANNOTATION TECHNIQUE WITH ADAPTIVE DIRECT

GUIDANCE TECHNIQUE

255