ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES

AND STUDENT KNOWLEDGE LEVEL

Boyan Bontchev and Dessislava Vassileva

Department of Software Engineering, Sofia University, 5, J. Baurchier blv., Sofia 1164, Bulgaria

Keywords: Adaptive assessment, Adaptive hypermedia systems, Learning styles, Learning object, Metadata,

Storyboard.

Abstract: Traditionally, adaptive assessment methods and tools have been addressed only by application of

computerized adaptive testing and item response theory as a key instrument for practical construction of

adaptive test assessments. The present paper tries to give a broader view of adaptive assessment, where tests

are not the only instrument for evaluation of learners outcomes gained during a course. It shows how

adaptive task-based assessment may be combined with traditional adaptive test assessment for achieving

better results and higher student satisfaction. Various types of tasks have been found suitable for adaptive

assessment based on learning styles and student knowledge level. By means of constructing course

storyboard with several branches for different learning styles, it has been proven that games, essay,

observation, comparative analysis tasks, projects and auto-generated tests may be used successfully for a

complex adaptive assessment. There have been explored approaches such as self-, peer- and teacher- (i.e.,

host) assessment by using appropriate types of estimable learning objects.

1 INTRODUCTION

Adaptive Web based learning plays a key role in

modern technology-enhanced learning research and

practical system implementation. First Brusilovsky

has pointed out (Brusilovsky, 1996) that learners

have different knowledge levels, needs and

preferences which should be considered for realising

personalized and adaptive hypermedia systems.

Moreover, learning styles have been used as a base

for creation of efficient adaptive hypermedia

systems (AHS) in many approaches (Milošević et al,

2007; Velsen, 2008). Learning style adaptivity may

be realised either by automatic generation of work

paths for each student character or, more precisely,

by construction of learning storyboards according

chosen instructional strategies (Vassileva et al,

2009a). In general, it has been proven in practice

that adaptation focussed on different aspects of

learning character is able to provide a higher

appealing and level of usefulness for the learner and,

thus, to lead to a better learning learning process

(Paramythis and Reisinger, 2003; Grimón et al,

2009).

Assessment is of key importance for both the

teacher and the learner because it tracks the learning

process, identifies strengths and weaknesses of the

learner and, thus, helps teacher in planning learning

steps. Introducing adaptivity to traditional

assessment may provide benefits not less then these

typical for AHS. However, until present, adaptive

assessment did not necessarily make a part of

adaptive hypermedia systems. Most of the present

works on adaptive assessment are focussed on

intelligent selection of questions based on prior

knowledge of the learner shown during a test run.

Thus, adaptive assessment tools use knowledge level

and proficiency of learners to select next question

within the test (Gouli et al, 2001). Testing is adapted

interactively to order to match the ability level of

any individual learner (Mansoor, 2006) by means of

using item response theory (IRT). En efficient

computer adaptive testing (CAT) system uses a

calibrated bank of questions developed according an

IRT model (Kovatcheva and Nikolov, 2009), initial

and intermediate questions' selection methods, a

scoring method, and terminating rules to stop (Gouli,

2001). For an intelligent selection of item (i.e.,

question) within an adaptive assessment run, proper

metadata of LOs of type questions has to be used. In

the approach of (Ibraheem, 2003), item metadata is

divided into two parts - descriptive and

psychometric.

449

Bontchev B. and Vassileva D..

ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL.

DOI: 10.5220/0003484104490454

In Proceedings of the 3rd International Conference on Computer Supported Education (ATTeL-2011), pages 449-454

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

The present paper describes a practical approach

of realization of assessment adaptive to both

learning styles and learner’s knowledge level and

achieved within the scope of a platform for adaptive

content delivery based on learner both character and

knowledge. An adaptation control engine chooses

within a course storyboard appropriate work paths

best suited to given learner style. In the course of the

work path, the engine shows to the learner only

learning objects (LO) having a difficulty appropriate

to the individual learner, and tracks these objects. At

a control page, the engine selects questions related to

the learning objects shown to the learner in an

adaptive mode. Besides adaptive tests, the approach

includes a more general adaptive assessment based

on learning styles - it includes assessment of

estimable learning objects such as essays, projects,

single- and multi-user games, comparative analysis

and others – all appropriate to a given learning style.

Experimental results are obtained for applying

adaptivity for Honey and Mumford (Honey and

Mumford, 2000) styles comprising of theorist,

pragmatist, reflector and activist. Authors argue the

appropriateness of such tasks to given learning style

and assessment method, such as self-, peer- and

teacher assessment.

2 ADOPTA – MODEL

AND PLATFORM

The assessment adaptive described in this paper is

implemented with the ADOPTA (ADaptive

technOlogy-enhanced Platform for eduTAinment)

platform for building edutainment (education plus

entertainment) content for both universities and

industry implementation. The triangular model of

ADOPTA is based principally on AHAM reference

model (De Bra et al, 1999) and its main idea is

explicitly separating narrative storyboard from the

content and adaptation engine. It adds some features

of contemporary adaptive e-learning systems as

support of different learning styles, learning content

and pedagogical strategy metadata and support of

several standards and more specifically LOM (Krull,

2004), SCORM (Rey-López et al, 2006) and OMV

(Vassileva et al, 2009).

The triangular model has hierarchical structure

with two levels (fig. 1). At each one level, it consists

of three sub-models. Thus its first level is divided

into the following three main models:

 Learner model – it structures data for learners

and it is divided into three sub-models - Goals and

preferences, Learning styles and Knowledge and

Performance. They provide data for learners, which

are used from adaptive engine for adaptive content

delivery.

 Adaptation model - it consists of following three

sub-models: Narrative metadata, Narrative

storyboard, and Storyboard rules. The sub-model

Narrative storyboard described course storyboard

graphs through directed graph. Each node of this

graph is narrative page or control page (CP).

Respectively narrative pages contain listed learning

object defined in different ontology graphs of the

Domain model, but CPs consist of randomly chosen

test questions which are based on visited learning

object of students. Each path from one CP to another

CP is called working path (WP). Each WP is

associated with a weight for each one learning

styles. These WP weights present how much they

are suitable for a particular learning style. The

metadata of narrative storyboard graphs are

described in the sub-model Narrative metadata. It

includes for example thresholds for each WP. These

thresholds gives the minimal results of test in CP

where the learner may continue to the next CP.

Rules for passing through narrative storyboard

graphs are stored in the sub-model Storyboard rules.

These rules define formulas which adaptive engine

used to calculate the most suitable WP for a learner

and conditions under which a learning object is

visible for a particular student.

 Domain model – it is responsible for creating of

learning objects (through the sub-model Learning

objects), for structuring it in ontology graphs

(through the sub-model Ontology graph) and

description of ontologies and learning objects

metadata (through the sub-models Content

Metadata). The sub-model Learning objects

determines several types of learning objects such as

narrative content, projects, tasks, essays, games,

assessment questions. Each of them has level of

difficulty (easy, medium, difficult, more difficult,

and most difficult). Using this, the instructor of a

course defines for what test results in CP are visible

various in difficulty learning objects. The other sub-

model Ontology graph organized learning object in

ontology graphs. In these graphs learning objects are

connected between themselves by two types of links:

o is-a – it is used for connection between

learning objects of type narrative content

o has-a – it defined reference relations and

learning object of type projects, tasks, essays,

games, assessment questions can be associated with

one and more learning objects of type narrative

content.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

450

Figure 1: Principal model and system workflow.

The architecture of ADOPTA system consists of

three main modules (fig. 1) – Authoring Tool,

Instructor Tool and Adaptation Control Engine.

The Authoring Tool is responsible for

implementation of functionalities of the Domain

model. It is used by content author for creating of

learning, for designing of ontology graphs and

describing of theirs metadata. For the last, this

module applies an inheritance mechanism.

After the learning content is created by the

content author, the instructor uses the Instructor

Tool for construction and for tuning of different

parameters of narrative storyboard and pages.

Finally, the Adaptation Control Engine uses all

available data for learning objects and storyboard

graphs for adaptive content delivery and adaptive

assessment. It can be used by instructor for

monitoring of the learning process, for controlling it

– adaptation behavior can be start or stop, for

correction of WP weights, for analysis of learner

performance, etc.

Each module includes three layers- persistence,

business and web (or client) layer. Each of the layers

is responsible for different specific problems. The

persistence layer stores and edits objects and it is

implemented with the Java Persistence API. The

business layer is build by EJB technology. The

business logic of application is presented by stateless

EJBs.

Finally the last web layer communicates with the

beans where the business logic resides with web

services and it is build with FLEX technology.

3 ADAPTIVE ASSESSMENTTO

LEARNING STYLES AND

KLOWLEDGE LEVEL

Realization of adaptive assessment within the

ADOPTA platform is a joint effort of content

authors, instructors and course supervisors. The

workflow phases are authoring of content for

adaptive courses, construction and tuning of

storyboard adaptive to learning styles and student

knowledge level and, finally, adaptive content

Figure 2: Partial view of ontology of the ADOPTA ontology authoring tool with LOs of a XML technology course.

ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL

451

delivery and assessment by means of estimable LOs

such as essays, projects, test, etc.

3.1 Content Authoring

Content authoring is possible by means of special

authoring tool providing LO metadata support based

on inheritance (Vassileva et al, 2009). Here, the

author should provide many LOs of various types

such as narrative LO (lesson), exercise, project,

essay, problem solving and others, in order to be

used next by instructors when creating course

storyboard graphs by means of the instructor tool.

The author organizes LOs within a domain

ontology having two main types of relationships:

IS_A and HAS_A. Fig. 2 shows a partial view of

such an ontology for a course of XML technologies

given to bachelor students. Questions are special

types of learning objects in the ontology and may be

related to the narrative LO they concern. While

IS_A relations are used for typefication of most LOs

of the course, HAS_A relations link question LOs to

the objects they refer to. Like the approach of (Wen,

2007), we use for types of relations “Enable”,

“Disable”, “Plus” and “Contradictory”. As well,

there may be used another types, as follows:

 Complementary – relation of such a type links

one question to other questions which are

complementary to the first one; means that one of

complementary questions may be used after the first

one;

 Opposite – links one question to another being

opposite of the first; means that these two questions

are adversative.

As well, for each of the LOs of type question the

author is supposed to provide item metadata divided

into two parts - descriptive and psychometric as

specified in (Ibraheem, 2003). For an easy

manipulation of LO metadata, non-monotonous

inheritance support is provided meaning metadata is

inherited within the ontology while some of the

fields may be overridden or added for given LO.

3.2 Course Storyboarding

While designing storyboard graphs by means of the

ADOPTA instructor tool, the instructor is supposed

to be concerned about two issues:

 development and tuning of a storyboard graph

containing paths with pages containing LOs

appropriate for different learning styles

 selection and distribution among the graph pages

of LOs with different level of difficulty in order to

be delivered next to learners with appropriate

knowledge level.

Figure 3: Partial view of XML course ontology.

The first task is not so trivial, as far as there is no

existing stable mapping between LOs types and

learning styles. Fig. 3 presents a possible mapping of

types of LOs to the four basic learning styles of

Honey and Mumford resulted from practical

experience of the paper authors in teaching students

for many years. It has been found that games,

essays, projects, problem-solving, comparative

analysis and observation tasks may be used for

assessment not less efficiently than traditional tests.

Thanks to their appropriateness to different learning

styles, LOs of these types are selected as candidates

for assessment adaptive to learning styles and have

names shown in black letters while narrative LOs

have names in white labels. For example, games and

problem-solving tasks are suitable for assessment of

both pragmatists and activists, while comparative

analysis tasks are mostly suitable to theorists and

reflectors. Note, that these types are distinguished

according their suitability for one or more of the

assessment approaches mentioned in section 2:

 for self assessment – there are nominated single

user games and test (auto-generated based on the

ontology relationships);

 for peer assessment – there are used multi-user

games (the opponent of a learner appears as peer

assessor), essay, observation, and comparative

analysis tasks;

 for teacher assessment - essay, observation, and

comparative analysis tasks plus projects and tests.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

452

At the very beginning of the storyboard design

process, the instructor is guided only by possible

learning styles of students. For this purpose, the

storyboard graph has to have a few initial nodes or

to start from a node giving basic information and,

next, to fork the working (i.e., learning) paths (WPs)

within the graph designed by the instructor tool. For

construction of the adaptive course storyboard

graph, the instructor may follow a strongly

connected storyboard approach or graph design

using parallel branches. Educational content with the

branch for predominant activists contains more

examples, while materials for theorists are given

with more detailed explanations and formalization

within another branch. LOs intended for pragmatists

are presented by practical tasks and exercises, while

these for reflectors are reporting, analysis and

comparisons.

Finally, the instructor has to tune the weights for

the existing working paths. Each weight consists of

values for activist, theorist, reflector and pragmatist,

stating the level of suitability of the path for each

style. As well, the instructor has to select LOs of

various level of difficulty in order to meet different

student knowledge levels.

3.3 Adaptive Content Delivery

and Assessment

Before starting a course, individual learners pass

through a pre-test determining their learning

character. Usually, a learning character contains a

combination of the four learning styles expressed in

different degree by different weights. Next, the

adaptation control engine will select a path best

suited to individual learner character of given

learner. According distribution of LOs types to

learning styles, different students will receive

different narrative and estimable LOs. Estimable

LOs will be used for a total adaptive assessment

where students with different styles will receive

different tasks (essays, projects, problems to be

solved, etc.) as shown in fig. 3. There different tasks

will be used for a complex final grade of the course.

Of course, adaptive assessment tests do participate

into the final grade – in control pages the control

engine generates automatically adaptive tests by

selecting questions related to the LOs delivered to a

learner via the work path. The selection is based on

the relation types explained in section 4.1.

After solving the test at a control point, the

learner is passed back to the previous control point if

his/her result is less than the threshold set for

passing this control point. In this case, the adaptation

control engine will select a different path for this

learner. Otherwise, to the learner there is proposed a

new path leading to the next control point and most

appropriate to his/her learning style. The result of

the assessment (equal or greater the threshold set for

passing that control point) will be used by the engine

for selection of LOs with appropriate level of

difficulty. Thus, individual learners undergo

assessment process adaptive to their learning styles

and, as well, knowledge level.

4 PRACTICAL RESULTS

The ontology, LOs, and storyboard shown above in

this paper have been used for preparing a part of an

adaptive course in the domain of XML technologies.

This part of the course has been given to more than

70 bachelor students (fourth year) by means of

ADOPTA as both adaptive courseware delivery and

adaptive assessment, while the rest of the course has

been presented as a traditional non-adaptive course.

Students have passed separate assessments for both

the adaptive and non-adaptive part of the course.

Next, they have filled an inquiry with many

questions aiming at revealing their attitude to the

adaptive and non-adaptive e-learning approaches.

Having summarized the inquiries and assessment

results, we are able to conclude the following:

 Students have shown better results when dealing

with adaptive content delivery and testing than with

non-adaptive ones;

 Students do prefer adaptive content delivery and

assessment based on learning styles and knowledge

level than traditional methods;

 All the learners like much more complex

assessment based on various types of tasks given

adaptively according learning styles, than traditional

test-based assessment;

 Students do prefer having self- and peer

assessment before official assessments;

 Students appreciate very much inclusion of

games - both single-user such as hangman, quizzes,

and word puzzles and multi-user games within the

ADOPTA platform

 Students require mobile games, tasks and

assessments to be included into the adaptive e-

learning process.

ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL

453

5 CONCLUSIONS

Until present, researchers and practitioners regard

adaptive assessment mainly as application of

computerized adaptive testing. Such systems make

use of item response theory as a key instrument for

practical construction of adaptive assessments based

on adaptive tests. The present paper presented a

broader view of adaptive assessment, where tests are

not the only (and, eventually, the best) instrument

for evaluation of learners outcomes gained during a

course. It has shown that adaptive task-based

assessment may be combined with traditional

adaptive test assessment for achieving better results

and, not last, student satisfaction and approval.

Authors have explored suitability of various

types of tasks for assessment adaptive to learning

styles and student knowledge level. By means of

constructing course storyboard with several branches

for different learning styles, it has been proven that

games, essay, observation, and comparative analysis

tasks plus projects and auto-generated tests may be

used successfully for a complex adaptive

assessment. There have been explored the

approaches of self-, peer- and teacher- (i.e., host)

assessment by using appropriate types of estimable

learning objects.

The future works on adaptive assessment by

using the ADOPTA platform will explore the

abilities of more types of relationships among

questions for realization of adaptive test assessment.

As well, authors plan to develop and experiment

more types of games for both self- and peer

assessment.

ACKNOWLEDGEMENTS

This work is partially supported by both the SISTER

project funded by the EC in FP7-SP4 Capacities via

agreement no. 205030 and the ADOPTA funded by

the Bulgarian NSF under agreement no. D002/155.

REFERENCES

Brusilovsky, P., 1996. Methods and Techniques of

Adaptive Hypermedia. in User Modeling and User-

Adapted Interaction, Vol. 6, No. 2-3, pp. 87-129.

De Bra P. et al., 1999. AHAM: A Dexter-based Reference

Model for adaptive Hypermedia. Proc. of the ACM

Conference on Hypertext and Hypermedia, ISBN:1-

58113-064-3, pp. 147-156.

Gouli, E., Kornilakis, H., Papanikolaou, K., Grigoriadou,

M., 2001. Adaptive Assessment Improving Interaction

in an Educational Hypermedia System, Proc. of the

PanHellenic Conf. in Human-Computer Interaction,

pp.217-222.

Grimón, F., Monguet, J. M., Ojeda, J., 2009. Knowledge

Based Information Retrieval with an Adaptive

Hypermedia System, in LNCS, ISBN 978-3-642-

02263-0, pp. 457-463.

Honey, P., Mumford, A., 2000. The Learning Styles

Helper's Guide. Maidenhead, Peter Honey

Publications.

Ibraheem, A., M., Shaalan, K, Riad, M. B., Darwish, M.

G., 2003, A Model and Supporting Mechanism for

Item Evaluation in Distance Learning-Based

Environment, in Egyptian Information Journal, Vol.4,

No. 2, December 2003, pp 169-186.

Kovatcheva, E., Nikolov, R., 2009. An adaptive feedback

approach for e-learning systems, in IEEE

Multidisciplinary Eng. Education Magazine, Vol. 4,

No. 1/2, JUNE 2009, pp.55-57.

Krull, G (2004). An investigation of the development and

adoption of educational metadata standards for the

widespread use of learning objects, Master Thesis,

Rhodes University, November 2004.

Mansoor A., 2006. IRT-item response theory assessment

for an adaptive teaching assessment system, Proc. of

MATH'06, Stevens Point, Wisconsin, USA, ISBN:999-

6666-22-1, pp.518-522.

Milošević, D. et al., 2007. Adaptive Learning by Using

SCOs Metadata, in Interdisciplinary Journal of

Knowledge and Learning Objects, Vol. 3, pp. 163-

174.

Paramythis, A. Reisinger, L., 2003. S Adaptive Learning

Environments and e-Learning Standards, Proc. of

Conference on e-Learning (ECEL2003), Glasgow,

Scotland, pp. 369-379.

Petrov M., Bontchev, B., 2004. Question and test

interoperability – organization, reuse and management

of assessments’ contents, Proc. of IADAT Int. Conf. on

Education – IADATe-2004, ISBN: 84-933971-0-5,

Bilbao, Spain, pp.435-439.

Rey-López, M., Fernández-Vilas A., Díaz-Redondo R.,

Pazos-Arias J., 2006. Providing SCORM with

adaptivity. Proc. of the 15

Int. Conf. on World Wide

Web, ISBN:1-59593-323-9, pp.981-982.

Vassileva D., B. Bontchev, Sl. Grigorov, 2009. Mastering

Adaptive Hypermedia Courseware, in Acta

Electrotechnica et Informatica, Vol. 9, No. 1, pp. 57–

62.

Velsen., L., 2008. User-centered evaluation of adaptive

and adaptable systems: a literature review, in The

Knowledge Engineering Review Journal, Cambridge

University Press, Vol. 23:3, pp. 261–281.

Wen, D., Graf, S., Lan, C. H., Anderson, T., Kinshuk, K.

D., 2007. Adaptive Assessment in Web-based

learning, Proc. of the IEEE Int. Conf. on Multimedia

and Expo (ICME'2007), Beijing, China, July 2007, pp.

1846-1849.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

454