GENERIC USER MODELING FOR ADAPTIVE ASSESSMENT

SYSTEMS

Alexander Heimbuch

, Christian Saul

and Heinz-Dietrich Wuttke

Data Representation & Interfaces Group, Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany

Integrated Communication Systems Group, Ilmenau University of Technology, Ilmenau, Germany

Keywords:

User Modeling, Adaptive Assessment Systems, Adaptive Systems, Overlay Approach, Bayesian Networks.

Abstract:

Personalization is becoming a crucial factor in many areas of life including education. Currently, established

e-learning systems mostly neglect the potential of user-speciﬁc adaptations such as optimized user orientation,

sequencing or presentation by not considering an (appropriate) user model. But, a user model is the crucial

factor how good and accurate the adaptations work. For that reason, this paper presents a generic approach for

user modeling in the context of Adaptive Assessment Systems (AASs). The approach (1) enables recording

skills and user characteristics, and the derivation of adaptable parameters; (2) allows incorporating additional

parameters to determine speciﬁc properties and (3) ensures the interoperability through the use of established

standards and speciﬁcations. In order to be generic and ﬂexible in conﬁguration, the overlay approach was

used in combination with Bayesian networks. In addition, the interoperability of the approach is ensured

through the use of the IMS LIP speciﬁcation. Finally, the implementation of the approach is demonstrated in

the AAS askme!. The work presented in this paper contributes to accurate characterizations of users, which in

turn allows adequate levels of adaptability to reﬂect the real intelligence of an e-learning system.

1 INTRODUCTION

Today, centralized information systems offer the op-

portunity of data organization and representation.

Based on intensive interactions between users and the

system, in combination with raising user information

recording, user-speciﬁc adaptations of the informa-

tion systems are possible. Established information

systems, especially e-learning systems, mostly ne-

glect the potential of user-speciﬁc adaptations such

as optimized user orientation, sequencing or presen-

tation. The user model represents the basis for user

orientation and therefore is an important and depen-

dent factor how good and accurate the adaptations in

the entire information system works.

For that reason, this paper presents a generic ap-

proach for user modeling in the context of Adaptive

Assessment Systems (AASs). The approach (1) en-

ables recording skills and user characteristics, and the

derivation of adaptable parameters; (2) allows incor-

porating additional parameters to determine speciﬁc

properties and (3) ensures the interoperability through

the use of established standards and speciﬁcation.

The work presented in this paper is part of an

overall project aiming at implementing a new inter-

active and personalized assessment system (Saul and

Wuttke, 2011b; Saul et al., 2011; Saul and Wuttke,

2011a).

The remainder of the paper is organized as fol-

lows: The second chapter gives an insight in related

work in terms of user modeling. The third chapter

states the shortcomings of existing approaches and

provides a distinct categorization of user preferences.

The fourth chapter proposes the generic approach for

user modeling in the context of AASs and chapter

ﬁve presents their implementation in the AAS askme!.

Chapter six discusses these ﬁndings and concluding

remarks and references complete the paper.

2 RELATED WORK

2.1 User Modeling in Adaptive

Hypermedia Systems

The ﬁrst steps towards considering user parameters

for adaptation purposes were made by De Bra et al.

in 1999 (De Bra et al., 1999) in the Adaptive Hyper-

media Application Model (AHAM). Like the Dexter

187

Heimbuch A., Saul C. and Wuttke H..

GENERIC USER MODELING FOR ADAPTIVE ASSESSMENT SYSTEMS.

DOI: 10.5220/0003920501870192

In Proceedings of the 4th International Conference on Computer Supported Education (CSEDU-2012), pages 187-192

ISBN: 978-989-8565-06-8

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

model (Halasz and Schwartz, 1994), AHAM focuses

on the storage layer, the anchoring and the presen-

tation speciﬁcation, but further subdivides the stor-

age layer into a domain model, a user model and a

teaching model. The user model keeps track of how

much the user knows about each of the concepts of the

application domain (conceptualized by the domain

model). Furthermore, it represents information about

a speciﬁc user and therefore sets up the basic require-

ment for effects of adaptation. Brusilovsky and Mil-

lan (Brusilovsky and Millan, 2007) distinguishes be-

tween three different information types in user models

namely:

• Application independent information (e.g., demo-

graphical data);

• Contextual information (e.g., user background);

• Information with a direct relation (e.g., through

the domain model).

The ﬁrst examples of systems with user models

reach back to the early nineties with implementations

of digital tutoring systems such as the Anatom-Tutor

developed at the Fraunhofer Institute of Biomedical

Engineering (IBMT) (Beaumont, 1995). Further de-

velopments were made by Weber and Brusilovsky

with the ELM-ART system (Weber and Brusilovsky,

2001), an adaptive versatile system for web-based in-

struction. Finally, De Bra et al. created the AHA!

system (De Bra et al., 2003) based on his assumptions

about AHAM.

In general, two user modeling mechanisms can be

distinguished (Brusilovsky and Millan, 2007):

• Stereotype-based user modeling.

• Feature-based user modeling.

In stereotype-based user modeling, the user is

classiﬁed according to his/her knowledge level of the

subject into different classes (e.g., novice, beginner,

intermediate or advanced) and the system adapts the

content or serves different content to users with dif-

ferent levels of knowledge. Examples of systems,

which use this modeling mechanisms are Anatom-

Tutor, AVANTI (Fink et al., 1996) and MetaDoc

(Boyle and Encarnacion, 1994). In feature-based user

modeling, the adaptation effect depends on the spe-

ciﬁc characteristic of each user. One of the most pop-

ular concepts in this domain is the overlay approach.

The overlay approach arranges the speciﬁc character-

istics of each user based on a predeﬁned schema. Ac-

cording to (Brusilovsky and Millan, 2007), overlay

concepts are especially useful for constructing com-

plex data structures. Examples of systems, which

make use of this modeling mechanisms are AHA!,

ISIS-Tutor (Brusilovsky and Pesin, 1994) and HY-

PERFLEX (Brusilovsky, 1996).

2.2 Standards and Speciﬁcation for

User-modeling

There are mainly two standards and speciﬁcations,

which are used to represent user information namely

IMS LIP and IEEE LTSC PAPI.

The IMS Learner Information Package (LIP)

speciﬁcation

addresses interoperability between dif-

ferent systems. It holds, maintains and manages

learner information in XML documents. Informa-

tion is represented in 11 core data structures such

as identiﬁcation, goal, qualiﬁcation, activity, compe-

tency, meaning skills, knowledge and abilities in the

cognitive, affective, and psycho-motor domains, etc.

However, IMS LIP makes no statement about the con-

tent structure in each category. Additionally, the spec-

iﬁcation records metadata such as timestamp, source

and privacy information. Examples of applications

include data recording and management of learning-

related history or user goals and skills (Deved

2006).

The IEEE LTSC Public and Private Informa-

tion (PAPI) speciﬁcation

is a standard to exchange

learner data between different systems. It represents

the learners’ knowledge by specifying the learner-

model. The learner information is divided into six

groups namely learner contact information, learner

relation information, learner security information,

learner preferences information, learner performance

information and learner portfolio information.

As there are disjoint attributes in both speciﬁca-

tions (e.g., privacy and security issues), there are often

combinations of elements of both speciﬁcations used

(Lazarinis and Retalis, 2006).

3 PROBLEM STATEMENT

Summarized it can be stated that all systems described

above use a user model for adaption, but the user in-

formation are mainly static and obtained explicitly.

Moreover, existing approaches mainly focus on the

entire learning process by providing educational hy-

permedia and often neglect the speciﬁc requirements

the assessment part actually requires. This includes,

for example, deriving and interpreting of user infor-

mation from assessment results.

In addition, knowledge, background, interests and

the personal characteristics of the user are important

aspects that have to be taken into account in adaptiv-

ity decisions (Brusilovsky and Millan, 2007). A strict

http://www.imsglobal.org/proﬁles/

http://www.cen-ltso.net/Main.aspx?put=230

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

188

distinction between these aspects is not possible due

to the mutual inﬂuence of the user properties. There-

fore, the user preferences in this paper are divided into

two categories namely user knowledge and individual

characteristics.

3.1 User Knowledge

The principles of teaching consider an interactive op-

erating learner, whose perception, interpretation and

processing of information is based on his experience

and progression. Therefore, the state of development

of an individual is built on the entity of his perception,

interpretation and processing of information abilities

(Tulodziecki, 2000). The development of these cog-

nitive abilities is seen as a process that can be mea-

sured by suitable means (e.g., tests, observation or

self-assessments). Therefore, the user model is re-

sponsible for the recording of this learning process.

The user development occurs within an e-learning

system by actively interacting with the system. User

information, which are obtained during the learning

process are in the focus of the user model. It can

be distinguished between explicit and implicit infor-

mation. Explicit information are gathered, for exam-

ple, by assessments, whereas implicit information are

gathered while the user interacts with the system. It is

important to note that the majority of the information

used for adapting the learning process is implicit. The

interpretation of this information is one of the major

problems a user model has to deal with.

3.2 Individual Characteristics

The second source for user information are the char-

acteristics of the individual user. These characteristics

can be divided into two different parts:

• General characteristics;

• Topic-speciﬁc characteristics.

General characteristics are, for example, the in-

trinsic or extrinsic motivation, cognition style or pref-

erences of the user. They have a general impact to the

learning process. In contrast, topic-speciﬁc character-

istics, for example, the prior knowledge of the user

are related to speciﬁc topics. It is important to note

that both characteristics are mainly explicit. For that

reason, recording, organizing and structuring of these

information is another main problem a user model has

to deal with.

4 PROPOSED SOLUTION

4.1 User Knowledge

To solve the problem of constructing (recording)

the user-speciﬁc process of learning, the overlay ap-

proach proposed by (Brusilovsky and Millan, 2007)

was chosen. An overlay model represents an individ-

ual user’s knowledge as a subset of the domain model,

which reﬂects the expert-level knowledge of the sub-

ject. Therefore, the model implies a close connection

between the domain and the user model. The tax-

onomies of the domain model build the basic struc-

ture for the user model. To illustrate the approach,

the domain model can be seen as an overlay of the

user model as shown in Figure 1. As shown, the user

model only contains a subset of all taxonomy ele-

ments available.

User-specific Overlay in User Model

Blueprint

Contains available User Information

arithmetic operations

addition subtraction

simple

complex

Taxonomy Definition Domain Model

arithmetic operations

addition subtraction

simple

complex

Figure 1: Overlay approach.

In addition to the recording of implicit user in-

formation, the interpretation of this information is

an important point as well. To deal with this chal-

lenge, the Bayesian network approach (Jensen, 1996)

was used. In general, Bayesian networks are acyclic

GENERICUSERMODELINGFORADAPTIVEASSESSMENTSYSTEMS

189

graphs that consist of a set of variables. Between

these variables a condition-dependent relation exists

except for variables in the same level. These vari-

ables are condition-dependent (Millan and Perez-De-

La-Cruz, 2002). Based on these relations, a probabil-

ity calculation of each network node can be made. In

this paper, the Bayesian network approach is used to

determine user-speciﬁc probability of achievements

for hierarchical taxonomies.

4.2 Individual Characteristics

To face the challenge of structuring and organizing in-

dividual user characteristics, an established standard

or speciﬁcation should be used. They play an impor-

tant role as they enable interoperability across differ-

ent systems. As mentioned in Section 2.2, there are

mainly two standards and speciﬁcations for represent-

ing user information. In this paper, IMS LIP was cho-

sen as data model for individual information, because

problems were identiﬁed in IEEE LTSC PAPI in the

lack of selectivity between the categories as well as

in the limited expandability. IMS LIP provides a ba-

sic vocabulary for declaring user information and pro-

vides the opportunity of ﬂexible and recursive struc-

tures. Arbitrary user information can be deﬁned using

11 categories.

5 IMPLEMENTATION

The user model approach proposed in Chapter 4 was

implemented in an AAS called askme! developed

by the Fraunhofer Institute for Digital Media Tech-

nology (IDMT). askme! follows the AHAM refer-

ence model meaning that it consists of a user model,

a domain model and an adaptation model, which

closely work together. In addition, it consists of an

adaptation engine, which performs the actual adap-

tation. For realizing adaptation during assessment,

askme! uses the adaptive question technique (Pitkow

and Recker, 1995). That means, it deﬁnes a dynamic

sequence of questions, which allow selecting ques-

tions dynamically. Based on rules and the last re-

sponse of the learner, appropriate questions are dy-

namically selected at runtime. askme! not only selects

and presents questions individually, but also takes so-

phisticated feedback techniques and methods into ac-

count resulting in providing feedback that is appropri-

ate for the learner’s context, knowledge level, individ-

ual characteristics and preferences (Saul et al., 2010).

Moreover, the sustainability of the questions and tests

created in askme! is guaranteed by the conformance

to the IMS QTI v2.1 speciﬁcation

. The user model

was implemented as a component in askme!. The re-

lations to the other components are presented in Fig-

ure 2.

askme!

Adaption-Engine

User Model Component

Domain Model

Component

Internal System Information

Interpreted User-Information

User-Information

Mapping

IMS-LIP

XML

External-Information

Figure 2: User model component relations.

As the system follows the Model-View-Controller

(MVC) architectural pattern, the user model compo-

nent is structured into several views and controllers

for administration and input purposes. In order to ac-

complish the objectives of Chapter 3, the implemen-

tation combines different approaches.

Firstly, the overlay approach to represent the user

knowledge was used. It was implemented in strong

connection with the domain model. Due to that syn-

ergy, it was possible to store the user information in

a single database table, which contains results from

the assessment tests, the identiﬁer for the user and the

addressed domain. To satisfy the demand of the adap-

tation engine, the user model (in combination with

domain information) is able to calculate the cumu-

lated results of each domain element (including the

children of each domain) for assessed questions. Fur-

thermore, it is possible for the adaptation engine to

request probabilities (p) that a user knows a speciﬁc

concept of a domain. For that, the relations between

the domain elements are used to calculate, in connec-

tion with the totally reached scores, the concept and

user-speciﬁc probabilities (see Figure 3). The overlay

approach was realized as a recursive algorithm, which

calculates the scores and probabilities from the leafs

of the domain (tree) to a given node. For that reason,

the run-time of the algorithm strongly depends on the

amount of child nodes given by the domain. Summa-

rized it can be stated that the user model component

provides different information to the adaption engine,

which can be used in adaptation rules as static (e.g.,

http://www.imsglobal.org/question/

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

190

scores) and dynamic (e.g., probabilities based on the

domain deﬁnitions) conditions.

arithmetic operations

addition

5 + 5

substraction

23 + 34

Figure 3: Bayesian network implementation.

Secondly, for recording of individual user char-

acteristics, the IMS LIP speciﬁcation was used as a

framework to categorize the user characteristics. In

order to face the recursive tree structure of IMS LIP,

all categories were stored in a traversed database ta-

ble. Additional data are stored in a dedicated database

table and linked through a unique identiﬁer of each

individual characteristic. This enables deﬁning own

characteristics, which nevertheless are grouped in the

11 categories of IMS LIP. These characteristics are

connected to speciﬁc users by a third database table

(see Figure 4). In order to gather these information,

different ways are possible, for example, explicit in-

formation can be given by the user himself or by an

evaluation given by the tutor of the assessment sys-

tem. The way of information gathering can be deﬁned

by the administration of the assessment system and

depends especially on the concrete characteristic. Ad-

ditionally, for interoperability purposes an IMS LIP

import and export was implemented.

IMS LIP

Categories

askme!

Specification

Identification

• John Doe

•052

• Main street

Accessibility

Goal

…

askme! Individual

Information

Identification

Accessibility

Goal

Qualifications

Activity

Transcript

...

Identification

• Name

• User-Id

• Address

Accessibility

Goal

…

LIP XML Definition

Additional Information

Database Table

Traversed

Database Table

Figure 4: IMS LIP implementation.

6 DISCUSSION

The proposed approaches for user modeling in AASs

also contain some critical aspects. First of all, the pre-

sented concept for recording user knowledge is ques-

tionable regarding the validity of the predictions. This

issue concerns the user-speciﬁc probability calcula-

tion of domain elements, because of the dependence

with the deﬁnition within the domain model. Each

prediction of the user model is based on the probabil-

ity parameters within the domain model. If the deﬁni-

tion within these relations fails, the prediction about

the users knowledge, made by the user model, will be

invalid. To face this problem, it is important to evalu-

ate the coherence of the relations within the domains.

Such an evaluation will result in more predictable and

also more precise user models.

The second aspect concerns the IMS LIP speci-

ﬁcation. The loose deﬁnition within the data model

does not provide only advantages. It can also cause

incompatibilities and problems with the differenti-

ation of the individual characteristics. Due to the

rough deﬁnition within the 11 categories, overlap-

ping dimensions are possible. Another issue concerns

the missing standardization of IMS LIP. As long as

there is no consensus within the e-learning commu-

nity, there will be no ﬁxed deﬁnition of important in-

dividual characteristics. But, IMS LIP has the poten-

tial to become a de facto standard and nevertheless

has strong synergies with the well known IMS QTI

speciﬁcation.

Finally, there is a general problem of user model-

ing in information systems. As described by Bloom

et al. (Bloom et al., 1956) and Anderson and Krath-

wohl (Anderson and Krathwohl, 2001), the process of

learning (and of course concerning the users knowl-

edge) contains more cognitive process dimensions

than remembering and understanding. The challenge

within systems in e-learning context (especially for

the user model) is to record advances thinking skills.

Referring to Wuttke et al. (Wuttke et al., 2008), it is

not really possible to measure these skills with tradi-

tional assessment techniques such as multiple-choice

questions. Therefore, more complex techniques like

interactive and immersive tools have to be integrated

and used in the assessment process. Until now, these

applications are developed stand-alone with no rela-

tion to learning or assessment systems. However, the

measurements of aggregated data could be beneﬁcial

to record advanced thinking skills. Standards like

HR-XML

already demonstrate how these interactive

applications could transfer their data to build more de-

tailed and precise user models.

http://www.hr-xml.org/

GENERICUSERMODELINGFORADAPTIVEASSESSMENTSYSTEMS

191

7 CONCLUSIONS AND FUTURE

WORK

User models provide necessary data to adapt digi-

tal information systems to the individual user. The

approach proposed in this paper especially concerns

user models for AASs. In this context, basic assump-

tions were made to create user-speciﬁc models that

support user-speciﬁc adaptations. As a result, the

highly efﬁcient overlay approach in combination with

Bayesian networks and the inclusion of the IMS LIP

speciﬁcation were presented. The presented approach

is highly generic and ﬂexible in conﬁguration. Fur-

thermore, critical aspects such as the validation and

interpretation problem of user information were dis-

cussed.

Future work of the institution of the main authors

will address the validation of the model by perform-

ing a comprehensive evaluation. It will point out the

educational beneﬁts (e.g., how precise are the gener-

ated user information) of the approach and their im-

plementation.

REFERENCES

Anderson, L. and Krathwohl, D. (2001). A Taxonomy for

Learning, Teaching, and Assessing: A Revision of

Blooms Taxonomy of Educational Objectives, Com-

plete Edition. Allyn & Bacon.

Beaumont, I. (1995). User modelling and system adaptation

in the interactive anatomy tutoring system anatom-

tutor. Advances in Human Factors/Ergonomics,

20:901–906.

Bloom, B., Engelhart, M., Furst, E., Hill, W., and Krath-

wohl, D. (1956). Taxonomy of Educational Objec-

tives, Handbook 1: Cognitive Domain. Longman.

Boyle, C. and Encarnacion, A. (1994). Metadoc: an adap-

tive hypertext reading system. User Modeling and

User-Adapted Interaction, 4(1):1–19.

Brusilovsky, P. (1996). Methods and techniques of adap-

tive hypermedia. User Modeling and User-Adapted

Interaction, 6(2-3):87–129.

Brusilovsky, P. and Millan, E. (2007). User Models for

Adaptive Hypermedia and Adaptive Educational Sys-

tems, chapter 1, pages 3–53. Berlin Heidelberg New

York: Springer-Verlag.

Brusilovsky, P. and Pesin, L. (1994). Isis-tutor: An adap-

tive hypertext learning environment. In JCKBSE, vol-

ume 94.

De Bra, P., Aerts, A., Berden, B., de Lange, B., Rousseau,

B., Santic, T., Smits, D., and Stash, N. (2003). AHA!

The Adaptive Hypermedia Architecture. In Proceed-

ings of the 14th ACM Conference on Hypertext and

hypermedia (HYPERTEXT 2003), pages 81–84.

De Bra, P., Houben, G., and Wu, H. (1999). AHAM: A

Dexter-based Reference Model for Adaptive Hyper-

media. In Proceedings of the 10th ACM Conference

on Hypertext and Hypermedia, pages 147–156.

Deved

c, V. (2006). Semantic Web and education. Springer

Verlag.

Fink, J., Kobsa, A., and Nill, A. (1996). User-oriented adap-

tivity and adaptability in the avanti project. In Design-

ing for the Web: empirical studies.

Halasz, F. and Schwartz, M. (1994). The Dexter hyper-

text reference model. Communications of the ACM,

37(2):30–39.

Jensen, F. (1996). An Introduction to Bayesian Networks.

Springer, London.

Lazarinis, F. and Retalis, S. (2006). Engineering an inter-

operable adaptive hypermedia testing tool supporting

user adaptable strategies. In Proceedings of the Artiﬁ-

cial Intelligence Applications and Innovations (AIAI)

Conference, volume 204, pages 393–400.

Millan, E. and Perez-De-La-Cruz, J. (2002). A bayesian di-

agnostic algorithm for student modeling and its evalu-

ation. User Modeling and User-Adapted Interaction,

12(2):281–330.

Pitkow, J. and Recker, M. (1995). Using the Web as a

Survey Tool: Results from the Second WWW User

Survey. Computer Networks and ISDN Systems,

27(6):809–822.

Saul, C., D

urrwald, F., and Wuttke, H.-D. (2011). Towards a

Generic Integration of Adaptive Assessment Systems

with Learning Environments. In Proceedings of the

3rd International Conference on Computer Supported

Education (CSEDU 2011), pages 117–122.

Saul, C., Runardotter, M., and Wuttke, H.-D. (2010). To-

wards Feedback Personalisation in Adaptive Assess-

ment. In Book of Abstracts & Workshop Programme

Sixth EDEN Research Workshop, pages 142–143, Bu-

dapest. European Distance and E-Learning Network.

Saul, C. and Wuttke, H.-D. (2011a). Feedback Personaliza-

tion as Prerequisite for Assessing Higher-order Think-

ing Skills. European Journal of Open, Distance and

E-learning (EURODL) - Special Issue: Best of EDEN,

pages 84–103.

Saul, C. and Wuttke, H.-D. (2011b). Personalized Assess-

ment of Higher-order Thinking Skills. In Proceedings

of the CSEDU Special Session on Assessment Tools

and Techniques for e-Learning (ATTeL 2011), pages

425–430.

Tulodziecki, G. (2000). Medienp

adagogik in der schule

der zukunft. Spannungsfeld Medien und Erziehung–

Medienp

adagogische Perspektiven, pages 13–32.

Weber, G. and Brusilovsky, P. (2001). ELM-ART: An

Adaptive Versatile System for Web-based Instruction.

International Journal of Artiﬁcial Intelligence in Edu-

cation, 12(4):351–384.

Wuttke, H.-D., Ubar, R., Henke, K., and Jutman, A. (2008).

The Synthesis Level in Bloom’s Taxonomy - a Night-

mare for an LMS. In Proceedings of the 19th EAEEIE

Annual Conference, pages 199–204.

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

192