GENERIC USER MODELING FOR ADAPTIVE ASSESSMENT
SYSTEMS
Alexander Heimbuch
1
, Christian Saul
1
and Heinz-Dietrich Wuttke
2
1
Data Representation & Interfaces Group, Fraunhofer Institute for Digital Media Technology, Ilmenau, Germany
2
Integrated Communication Systems Group, Ilmenau University of Technology, Ilmenau, Germany
Keywords:
User Modeling, Adaptive Assessment Systems, Adaptive Systems, Overlay Approach, Bayesian Networks.
Abstract:
Personalization is becoming a crucial factor in many areas of life including education. Currently, established
e-learning systems mostly neglect the potential of user-specific adaptations such as optimized user orientation,
sequencing or presentation by not considering an (appropriate) user model. But, a user model is the crucial
factor how good and accurate the adaptations work. For that reason, this paper presents a generic approach for
user modeling in the context of Adaptive Assessment Systems (AASs). The approach (1) enables recording
skills and user characteristics, and the derivation of adaptable parameters; (2) allows incorporating additional
parameters to determine specific properties and (3) ensures the interoperability through the use of established
standards and specifications. In order to be generic and flexible in configuration, the overlay approach was
used in combination with Bayesian networks. In addition, the interoperability of the approach is ensured
through the use of the IMS LIP specification. Finally, the implementation of the approach is demonstrated in
the AAS askme!. The work presented in this paper contributes to accurate characterizations of users, which in
turn allows adequate levels of adaptability to reflect the real intelligence of an e-learning system.
1 INTRODUCTION
Today, centralized information systems offer the op-
portunity of data organization and representation.
Based on intensive interactions between users and the
system, in combination with raising user information
recording, user-specific adaptations of the informa-
tion systems are possible. Established information
systems, especially e-learning systems, mostly ne-
glect the potential of user-specific adaptations such
as optimized user orientation, sequencing or presen-
tation. The user model represents the basis for user
orientation and therefore is an important and depen-
dent factor how good and accurate the adaptations in
the entire information system works.
For that reason, this paper presents a generic ap-
proach for user modeling in the context of Adaptive
Assessment Systems (AASs). The approach (1) en-
ables recording skills and user characteristics, and the
derivation of adaptable parameters; (2) allows incor-
porating additional parameters to determine specific
properties and (3) ensures the interoperability through
the use of established standards and specification.
The work presented in this paper is part of an
overall project aiming at implementing a new inter-
active and personalized assessment system (Saul and
Wuttke, 2011b; Saul et al., 2011; Saul and Wuttke,
2011a).
The remainder of the paper is organized as fol-
lows: The second chapter gives an insight in related
work in terms of user modeling. The third chapter
states the shortcomings of existing approaches and
provides a distinct categorization of user preferences.
The fourth chapter proposes the generic approach for
user modeling in the context of AASs and chapter
five presents their implementation in the AAS askme!.
Chapter six discusses these findings and concluding
remarks and references complete the paper.
2 RELATED WORK
2.1 User Modeling in Adaptive
Hypermedia Systems
The first steps towards considering user parameters
for adaptation purposes were made by De Bra et al.
in 1999 (De Bra et al., 1999) in the Adaptive Hyper-
media Application Model (AHAM). Like the Dexter
187
Heimbuch A., Saul C. and Wuttke H..
GENERIC USER MODELING FOR ADAPTIVE ASSESSMENT SYSTEMS.
DOI: 10.5220/0003920501870192
In Proceedings of the 4th International Conference on Computer Supported Education (CSEDU-2012), pages 187-192
ISBN: 978-989-8565-06-8
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
model (Halasz and Schwartz, 1994), AHAM focuses
on the storage layer, the anchoring and the presen-
tation specification, but further subdivides the stor-
age layer into a domain model, a user model and a
teaching model. The user model keeps track of how
much the user knows about each of the concepts of the
application domain (conceptualized by the domain
model). Furthermore, it represents information about
a specific user and therefore sets up the basic require-
ment for effects of adaptation. Brusilovsky and Mil-
lan (Brusilovsky and Millan, 2007) distinguishes be-
tween three different information types in user models
namely:
Application independent information (e.g., demo-
graphical data);
Contextual information (e.g., user background);
Information with a direct relation (e.g., through
the domain model).
The first examples of systems with user models
reach back to the early nineties with implementations
of digital tutoring systems such as the Anatom-Tutor
developed at the Fraunhofer Institute of Biomedical
Engineering (IBMT) (Beaumont, 1995). Further de-
velopments were made by Weber and Brusilovsky
with the ELM-ART system (Weber and Brusilovsky,
2001), an adaptive versatile system for web-based in-
struction. Finally, De Bra et al. created the AHA!
system (De Bra et al., 2003) based on his assumptions
about AHAM.
In general, two user modeling mechanisms can be
distinguished (Brusilovsky and Millan, 2007):
Stereotype-based user modeling.
Feature-based user modeling.
In stereotype-based user modeling, the user is
classified according to his/her knowledge level of the
subject into different classes (e.g., novice, beginner,
intermediate or advanced) and the system adapts the
content or serves different content to users with dif-
ferent levels of knowledge. Examples of systems,
which use this modeling mechanisms are Anatom-
Tutor, AVANTI (Fink et al., 1996) and MetaDoc
(Boyle and Encarnacion, 1994). In feature-based user
modeling, the adaptation effect depends on the spe-
cific characteristic of each user. One of the most pop-
ular concepts in this domain is the overlay approach.
The overlay approach arranges the specific character-
istics of each user based on a predefined schema. Ac-
cording to (Brusilovsky and Millan, 2007), overlay
concepts are especially useful for constructing com-
plex data structures. Examples of systems, which
make use of this modeling mechanisms are AHA!,
ISIS-Tutor (Brusilovsky and Pesin, 1994) and HY-
PERFLEX (Brusilovsky, 1996).
2.2 Standards and Specification for
User-modeling
There are mainly two standards and specifications,
which are used to represent user information namely
IMS LIP and IEEE LTSC PAPI.
The IMS Learner Information Package (LIP)
specification
1
addresses interoperability between dif-
ferent systems. It holds, maintains and manages
learner information in XML documents. Informa-
tion is represented in 11 core data structures such
as identification, goal, qualification, activity, compe-
tency, meaning skills, knowledge and abilities in the
cognitive, affective, and psycho-motor domains, etc.
However, IMS LIP makes no statement about the con-
tent structure in each category. Additionally, the spec-
ification records metadata such as timestamp, source
and privacy information. Examples of applications
include data recording and management of learning-
related history or user goals and skills (Deved
ˇ
zi
´
c,
2006).
The IEEE LTSC Public and Private Informa-
tion (PAPI) specification
2
is a standard to exchange
learner data between different systems. It represents
the learners’ knowledge by specifying the learner-
model. The learner information is divided into six
groups namely learner contact information, learner
relation information, learner security information,
learner preferences information, learner performance
information and learner portfolio information.
As there are disjoint attributes in both specifica-
tions (e.g., privacy and security issues), there are often
combinations of elements of both specifications used
(Lazarinis and Retalis, 2006).
3 PROBLEM STATEMENT
Summarized it can be stated that all systems described
above use a user model for adaption, but the user in-
formation are mainly static and obtained explicitly.
Moreover, existing approaches mainly focus on the
entire learning process by providing educational hy-
permedia and often neglect the specific requirements
the assessment part actually requires. This includes,
for example, deriving and interpreting of user infor-
mation from assessment results.
In addition, knowledge, background, interests and
the personal characteristics of the user are important
aspects that have to be taken into account in adaptiv-
ity decisions (Brusilovsky and Millan, 2007). A strict
1
http://www.imsglobal.org/profiles/
2
http://www.cen-ltso.net/Main.aspx?put=230
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
188
distinction between these aspects is not possible due
to the mutual influence of the user properties. There-
fore, the user preferences in this paper are divided into
two categories namely user knowledge and individual
characteristics.
3.1 User Knowledge
The principles of teaching consider an interactive op-
erating learner, whose perception, interpretation and
processing of information is based on his experience
and progression. Therefore, the state of development
of an individual is built on the entity of his perception,
interpretation and processing of information abilities
(Tulodziecki, 2000). The development of these cog-
nitive abilities is seen as a process that can be mea-
sured by suitable means (e.g., tests, observation or
self-assessments). Therefore, the user model is re-
sponsible for the recording of this learning process.
The user development occurs within an e-learning
system by actively interacting with the system. User
information, which are obtained during the learning
process are in the focus of the user model. It can
be distinguished between explicit and implicit infor-
mation. Explicit information are gathered, for exam-
ple, by assessments, whereas implicit information are
gathered while the user interacts with the system. It is
important to note that the majority of the information
used for adapting the learning process is implicit. The
interpretation of this information is one of the major
problems a user model has to deal with.
3.2 Individual Characteristics
The second source for user information are the char-
acteristics of the individual user. These characteristics
can be divided into two different parts:
General characteristics;
Topic-specific characteristics.
General characteristics are, for example, the in-
trinsic or extrinsic motivation, cognition style or pref-
erences of the user. They have a general impact to the
learning process. In contrast, topic-specific character-
istics, for example, the prior knowledge of the user
are related to specific topics. It is important to note
that both characteristics are mainly explicit. For that
reason, recording, organizing and structuring of these
information is another main problem a user model has
to deal with.
4 PROPOSED SOLUTION
4.1 User Knowledge
To solve the problem of constructing (recording)
the user-specific process of learning, the overlay ap-
proach proposed by (Brusilovsky and Millan, 2007)
was chosen. An overlay model represents an individ-
ual user’s knowledge as a subset of the domain model,
which reflects the expert-level knowledge of the sub-
ject. Therefore, the model implies a close connection
between the domain and the user model. The tax-
onomies of the domain model build the basic struc-
ture for the user model. To illustrate the approach,
the domain model can be seen as an overlay of the
user model as shown in Figure 1. As shown, the user
model only contains a subset of all taxonomy ele-
ments available.
User-specific Overlay in User Model
Blueprint
Contains available User Information
arithmetic operations
addition subtraction
simple
complex
Taxonomy Definition Domain Model
arithmetic operations
addition subtraction
simple
complex
Figure 1: Overlay approach.
In addition to the recording of implicit user in-
formation, the interpretation of this information is
an important point as well. To deal with this chal-
lenge, the Bayesian network approach (Jensen, 1996)
was used. In general, Bayesian networks are acyclic
GENERICUSERMODELINGFORADAPTIVEASSESSMENTSYSTEMS
189
graphs that consist of a set of variables. Between
these variables a condition-dependent relation exists
except for variables in the same level. These vari-
ables are condition-dependent (Millan and Perez-De-
La-Cruz, 2002). Based on these relations, a probabil-
ity calculation of each network node can be made. In
this paper, the Bayesian network approach is used to
determine user-specific probability of achievements
for hierarchical taxonomies.
4.2 Individual Characteristics
To face the challenge of structuring and organizing in-
dividual user characteristics, an established standard
or specification should be used. They play an impor-
tant role as they enable interoperability across differ-
ent systems. As mentioned in Section 2.2, there are
mainly two standards and specifications for represent-
ing user information. In this paper, IMS LIP was cho-
sen as data model for individual information, because
problems were identified in IEEE LTSC PAPI in the
lack of selectivity between the categories as well as
in the limited expandability. IMS LIP provides a ba-
sic vocabulary for declaring user information and pro-
vides the opportunity of flexible and recursive struc-
tures. Arbitrary user information can be defined using
11 categories.
5 IMPLEMENTATION
The user model approach proposed in Chapter 4 was
implemented in an AAS called askme! developed
by the Fraunhofer Institute for Digital Media Tech-
nology (IDMT). askme! follows the AHAM refer-
ence model meaning that it consists of a user model,
a domain model and an adaptation model, which
closely work together. In addition, it consists of an
adaptation engine, which performs the actual adap-
tation. For realizing adaptation during assessment,
askme! uses the adaptive question technique (Pitkow
and Recker, 1995). That means, it defines a dynamic
sequence of questions, which allow selecting ques-
tions dynamically. Based on rules and the last re-
sponse of the learner, appropriate questions are dy-
namically selected at runtime. askme! not only selects
and presents questions individually, but also takes so-
phisticated feedback techniques and methods into ac-
count resulting in providing feedback that is appropri-
ate for the learner’s context, knowledge level, individ-
ual characteristics and preferences (Saul et al., 2010).
Moreover, the sustainability of the questions and tests
created in askme! is guaranteed by the conformance
to the IMS QTI v2.1 specification
3
. The user model
was implemented as a component in askme!. The re-
lations to the other components are presented in Fig-
ure 2.
askme!
Adaption-Engine
User Model Component
Domain Model
Component
Internal System Information
Interpreted User-Information
User-Information
Mapping
IMS-LIP
XML
External-Information
Figure 2: User model component relations.
As the system follows the Model-View-Controller
(MVC) architectural pattern, the user model compo-
nent is structured into several views and controllers
for administration and input purposes. In order to ac-
complish the objectives of Chapter 3, the implemen-
tation combines different approaches.
Firstly, the overlay approach to represent the user
knowledge was used. It was implemented in strong
connection with the domain model. Due to that syn-
ergy, it was possible to store the user information in
a single database table, which contains results from
the assessment tests, the identifier for the user and the
addressed domain. To satisfy the demand of the adap-
tation engine, the user model (in combination with
domain information) is able to calculate the cumu-
lated results of each domain element (including the
children of each domain) for assessed questions. Fur-
thermore, it is possible for the adaptation engine to
request probabilities (p) that a user knows a specific
concept of a domain. For that, the relations between
the domain elements are used to calculate, in connec-
tion with the totally reached scores, the concept and
user-specific probabilities (see Figure 3). The overlay
approach was realized as a recursive algorithm, which
calculates the scores and probabilities from the leafs
of the domain (tree) to a given node. For that reason,
the run-time of the algorithm strongly depends on the
amount of child nodes given by the domain. Summa-
rized it can be stated that the user model component
provides different information to the adaption engine,
which can be used in adaptation rules as static (e.g.,
3
http://www.imsglobal.org/question/
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
190
scores) and dynamic (e.g., probabilities based on the
domain definitions) conditions.
arithmetic operations
addition
5 + 5
substraction
23 + 34
p
p
p
p
Figure 3: Bayesian network implementation.
Secondly, for recording of individual user char-
acteristics, the IMS LIP specification was used as a
framework to categorize the user characteristics. In
order to face the recursive tree structure of IMS LIP,
all categories were stored in a traversed database ta-
ble. Additional data are stored in a dedicated database
table and linked through a unique identifier of each
individual characteristic. This enables defining own
characteristics, which nevertheless are grouped in the
11 categories of IMS LIP. These characteristics are
connected to specific users by a third database table
(see Figure 4). In order to gather these information,
different ways are possible, for example, explicit in-
formation can be given by the user himself or by an
evaluation given by the tutor of the assessment sys-
tem. The way of information gathering can be defined
by the administration of the assessment system and
depends especially on the concrete characteristic. Ad-
ditionally, for interoperability purposes an IMS LIP
import and export was implemented.
IMS LIP
Categories
askme!
Specification
Identification
John Doe
•052
Main street
Accessibility
Goal
askme! Individual
Information
Identification
Accessibility
Goal
Qualifications
Activity
Transcript
...
Identification
Name
User-Id
Address
Accessibility
Goal
LIP XML Definition
Additional Information
Database Table
Traversed
Database Table
Figure 4: IMS LIP implementation.
6 DISCUSSION
The proposed approaches for user modeling in AASs
also contain some critical aspects. First of all, the pre-
sented concept for recording user knowledge is ques-
tionable regarding the validity of the predictions. This
issue concerns the user-specific probability calcula-
tion of domain elements, because of the dependence
with the definition within the domain model. Each
prediction of the user model is based on the probabil-
ity parameters within the domain model. If the defini-
tion within these relations fails, the prediction about
the users knowledge, made by the user model, will be
invalid. To face this problem, it is important to evalu-
ate the coherence of the relations within the domains.
Such an evaluation will result in more predictable and
also more precise user models.
The second aspect concerns the IMS LIP speci-
fication. The loose definition within the data model
does not provide only advantages. It can also cause
incompatibilities and problems with the differenti-
ation of the individual characteristics. Due to the
rough definition within the 11 categories, overlap-
ping dimensions are possible. Another issue concerns
the missing standardization of IMS LIP. As long as
there is no consensus within the e-learning commu-
nity, there will be no fixed definition of important in-
dividual characteristics. But, IMS LIP has the poten-
tial to become a de facto standard and nevertheless
has strong synergies with the well known IMS QTI
specification.
Finally, there is a general problem of user model-
ing in information systems. As described by Bloom
et al. (Bloom et al., 1956) and Anderson and Krath-
wohl (Anderson and Krathwohl, 2001), the process of
learning (and of course concerning the users knowl-
edge) contains more cognitive process dimensions
than remembering and understanding. The challenge
within systems in e-learning context (especially for
the user model) is to record advances thinking skills.
Referring to Wuttke et al. (Wuttke et al., 2008), it is
not really possible to measure these skills with tradi-
tional assessment techniques such as multiple-choice
questions. Therefore, more complex techniques like
interactive and immersive tools have to be integrated
and used in the assessment process. Until now, these
applications are developed stand-alone with no rela-
tion to learning or assessment systems. However, the
measurements of aggregated data could be beneficial
to record advanced thinking skills. Standards like
HR-XML
4
already demonstrate how these interactive
applications could transfer their data to build more de-
tailed and precise user models.
4
http://www.hr-xml.org/
GENERICUSERMODELINGFORADAPTIVEASSESSMENTSYSTEMS
191
7 CONCLUSIONS AND FUTURE
WORK
User models provide necessary data to adapt digi-
tal information systems to the individual user. The
approach proposed in this paper especially concerns
user models for AASs. In this context, basic assump-
tions were made to create user-specific models that
support user-specific adaptations. As a result, the
highly efficient overlay approach in combination with
Bayesian networks and the inclusion of the IMS LIP
specification were presented. The presented approach
is highly generic and flexible in configuration. Fur-
thermore, critical aspects such as the validation and
interpretation problem of user information were dis-
cussed.
Future work of the institution of the main authors
will address the validation of the model by perform-
ing a comprehensive evaluation. It will point out the
educational benefits (e.g., how precise are the gener-
ated user information) of the approach and their im-
plementation.
REFERENCES
Anderson, L. and Krathwohl, D. (2001). A Taxonomy for
Learning, Teaching, and Assessing: A Revision of
Blooms Taxonomy of Educational Objectives, Com-
plete Edition. Allyn & Bacon.
Beaumont, I. (1995). User modelling and system adaptation
in the interactive anatomy tutoring system anatom-
tutor. Advances in Human Factors/Ergonomics,
20:901–906.
Bloom, B., Engelhart, M., Furst, E., Hill, W., and Krath-
wohl, D. (1956). Taxonomy of Educational Objec-
tives, Handbook 1: Cognitive Domain. Longman.
Boyle, C. and Encarnacion, A. (1994). Metadoc: an adap-
tive hypertext reading system. User Modeling and
User-Adapted Interaction, 4(1):1–19.
Brusilovsky, P. (1996). Methods and techniques of adap-
tive hypermedia. User Modeling and User-Adapted
Interaction, 6(2-3):87–129.
Brusilovsky, P. and Millan, E. (2007). User Models for
Adaptive Hypermedia and Adaptive Educational Sys-
tems, chapter 1, pages 3–53. Berlin Heidelberg New
York: Springer-Verlag.
Brusilovsky, P. and Pesin, L. (1994). Isis-tutor: An adap-
tive hypertext learning environment. In JCKBSE, vol-
ume 94.
De Bra, P., Aerts, A., Berden, B., de Lange, B., Rousseau,
B., Santic, T., Smits, D., and Stash, N. (2003). AHA!
The Adaptive Hypermedia Architecture. In Proceed-
ings of the 14th ACM Conference on Hypertext and
hypermedia (HYPERTEXT 2003), pages 81–84.
De Bra, P., Houben, G., and Wu, H. (1999). AHAM: A
Dexter-based Reference Model for Adaptive Hyper-
media. In Proceedings of the 10th ACM Conference
on Hypertext and Hypermedia, pages 147–156.
Deved
ˇ
zi
´
c, V. (2006). Semantic Web and education. Springer
Verlag.
Fink, J., Kobsa, A., and Nill, A. (1996). User-oriented adap-
tivity and adaptability in the avanti project. In Design-
ing for the Web: empirical studies.
Halasz, F. and Schwartz, M. (1994). The Dexter hyper-
text reference model. Communications of the ACM,
37(2):30–39.
Jensen, F. (1996). An Introduction to Bayesian Networks.
Springer, London.
Lazarinis, F. and Retalis, S. (2006). Engineering an inter-
operable adaptive hypermedia testing tool supporting
user adaptable strategies. In Proceedings of the Artifi-
cial Intelligence Applications and Innovations (AIAI)
Conference, volume 204, pages 393–400.
Millan, E. and Perez-De-La-Cruz, J. (2002). A bayesian di-
agnostic algorithm for student modeling and its evalu-
ation. User Modeling and User-Adapted Interaction,
12(2):281–330.
Pitkow, J. and Recker, M. (1995). Using the Web as a
Survey Tool: Results from the Second WWW User
Survey. Computer Networks and ISDN Systems,
27(6):809–822.
Saul, C., D
¨
urrwald, F., and Wuttke, H.-D. (2011). Towards a
Generic Integration of Adaptive Assessment Systems
with Learning Environments. In Proceedings of the
3rd International Conference on Computer Supported
Education (CSEDU 2011), pages 117–122.
Saul, C., Runardotter, M., and Wuttke, H.-D. (2010). To-
wards Feedback Personalisation in Adaptive Assess-
ment. In Book of Abstracts & Workshop Programme
Sixth EDEN Research Workshop, pages 142–143, Bu-
dapest. European Distance and E-Learning Network.
Saul, C. and Wuttke, H.-D. (2011a). Feedback Personaliza-
tion as Prerequisite for Assessing Higher-order Think-
ing Skills. European Journal of Open, Distance and
E-learning (EURODL) - Special Issue: Best of EDEN,
pages 84–103.
Saul, C. and Wuttke, H.-D. (2011b). Personalized Assess-
ment of Higher-order Thinking Skills. In Proceedings
of the CSEDU Special Session on Assessment Tools
and Techniques for e-Learning (ATTeL 2011), pages
425–430.
Tulodziecki, G. (2000). Medienp
¨
adagogik in der schule
der zukunft. Spannungsfeld Medien und Erziehung–
Medienp
¨
adagogische Perspektiven, pages 13–32.
Weber, G. and Brusilovsky, P. (2001). ELM-ART: An
Adaptive Versatile System for Web-based Instruction.
International Journal of Artificial Intelligence in Edu-
cation, 12(4):351–384.
Wuttke, H.-D., Ubar, R., Henke, K., and Jutman, A. (2008).
The Synthesis Level in Bloom’s Taxonomy - a Night-
mare for an LMS. In Proceedings of the 19th EAEEIE
Annual Conference, pages 199–204.
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
192