TOWARDS A GENERIC INTEGRATION OF ADAPTIVE

ASSESSMENT SYSTEMS WITH LEARNING ENVIRONMENTS

Christian Saul, Felix Dürrwald

Fraunhofer Institute Digital Media Technology, Business Area Data Representation & Interfaces, Ilmenau, Germany

Heinz-Dietrich Wuttke

Faculty Informatics and Automation, Ilmenau University of Technology, Ilmenau, Germany

Keywords: Adaptive Assessment Systems, Learning Environments, Integration, Interoperability.

Abstract: Personalization is becoming a crucial factor in many areas of life including education. Currently, a learning

environment (LE) such as moodle, ILIAS, OLAT and dotLRN is far from being able to adapt the

assessment to the students’ individual context, prior knowledge and preferences, because personalization is

still insufficiently implemented or even not addressed in this system. In contrast, an adaptive assessment

system (AAS) takes the students’ characteristics into account in order to personalize the assessment, which

may result in more objective assessment findings. This paper analyzes how current open standards and

specifications can be used to achieve integration between LEs and AASs seamlessly, so that they can profit

from each other. For that reason, both LEs and AASs requirements are analyzed, because they have great

influence on the further considerations. LEs require control information as well as assessment information

from the AASs, which in turn require student as well as assessment information from the LEs. As a result of

the paper, an interworking of several standards and specifications (OPAQUE, IEEE LTSC PAPI, IMS LIP,

IMS QTI, etc.) is proposed, which can be used to achieve integration between LEs and AASs.

1 INTRODUCTION

Assessment in the educational settings is defined as

classifying, usually in measurable terms, knowledge,

skills, attitudes and beliefs of a student.

The functionalities to be provided by an

assessment system are: (1) identify students and

define which students have to take which tests at

what times and which questions make up each test,

(2) record the scores and other information that

result from students attempting the test and provide

reporting facilities for this information, (3) display

questions to students and process the responses to

generate scores and feedback and (4) allow authors

to create and configure the questions.

Currently, a learning environment (LE), also

known as learning management system (LMS), such

as moodle (http://www.moodle.org), ILIAS

(http://www.ilias.de), OLAT (http://www.olat.org)

and dotLRN (http://www.dotlrn.org) realizes

number 1 and 2 almost satisfying (Wuttke et al.,

2008). LEs usually have facilities to administrate

and support students during assessment, but they are

relatively weak in addressing functionalities 3 and 4.

Moreover, they are limited in posing question types

and using algorithms to analyze and process

students’ responses. In addition, if authors want to

create advanced questions and tests or content with

high levels of interaction, they are limited to the

tools provided by individual environments, which, in

turn, restrict the reusability and interoperability of

the created content. This often leads to an “one-size-

fits-all” approach, where all students are presented

the same questions. From a pedagogical point of

view, however, personalization support is crucial to

keep up the motivation and interests of the students,

which are critical success factors in the assessment

process.

An adaptive assessment system (AAS) poses a

better alternative. It takes the students’ individual

context, prior knowledge and preferences into

account in order to personalize the assessment,

which may result in more objective assessment

findings. The benefit of integrating LEs and AASs

117

Saul C., Dürrwald F. and Wuttke H..

TOWARDS A GENERIC INTEGRATION OF ADAPTIVE ASSESSMENT SYSTEMS WITH LEARNING ENVIRONMENTS.

DOI: 10.5220/0003344001170122

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 117-122

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

lies in combining the strengths of both systems. LEs

provide administration, learning and support

capabilities, while AASs provide personalized

assessment to students.

The focus of this paper is to analyze, how current

open standards and specifications can be used to

achieve integration of LEs and AASs. The work

presented in this paper is part of an overall project

aiming at implementing a new AAS (Saul et al.,

2010).

The remainder of the paper is organized as

follows: The second and third chapters describe LEs

and their requirements, which an AAS should meet

as well as AASs and their information requirements.

The fourth chapter deals with current standards and

specifications, which can be used to integrate LEs

and AASs. Chapter five proposes solutions, which

are discussed in chapter six. Concluding remarks

and references complete the paper.

2 LEARNING ENVIRONMENTS

LEs such as moodle, ILIAS, OLAT and dotLRN are

designed to support and enhance learning and

training in educational settings.

Generally, many LEs intersperse content with

assessments. The questions and tests, which make up

the assessments, are mostly created by tools

provided by the specific LE. The functionalities

provided by these tools encompass simple

monitoring mechanisms often used to unlock further

content. However, they do not provide sophisticated

control mechanisms, able to realize any kind of

individualization or personalization. In addition, the

lack of standardization on how these simple control

mechanisms are represented faces the authors of

questions and tests with the decision whether they

define control rules for a specific LE or not.

However, LEs generally offer extensive

management facilities to the authors such as student

and class management, course assembly and

publishing as well as student tracking across

courses. Therefore, LEs can offer student

management facilities to the AASs. In order to

enable a successfully integration of LEs with AASs,

the LEs require control information as well as

assessment information from the AASs.

2.1 Control Information

If a student executes a test, the LE needs to be

informed when the student has finished the test.

These control information can be used by the LE to

determine the state in the learning process, for

example, to lock further content or to start the test

environment again upon logging into the LE for the

next time. This will happen until the LE has received

the information from the AAS that the student has

taken the test completely. In addition, the AAS

should provide mechanisms to resume the

assessment in case the connection is interrupted or

lost.

2.2 Assessment Information

The LE needs to be informed about the results

achieved after the student has finished answering the

questions. This information encompass the question

that was asked, the final answer, the reached scores

for the answer and attempts the student took in

getting the final answer. The assessment information

can be used by the LE to report and compute overall

test scores.

3 ADAPTIVE ASSESSMENT

SYSTEMS

AASs such as SIETTE (Conejo et al., 2004), PASS

(Gouli et al., 2002), CosyQTI (Lalos et al., 2005)

and iAdaptTest (Lazarinis et al., 2009) take into

account the students’ individual context, prior

knowledge and preferences in order to personalize

the assessment. Although these systems adapt the

assessment process of each student resulting in

presenting different questions, they still enable a

better comparability between different individuals.

Moreover, they reveal the current areas of strength

and weakness of the students more precisely.

In order to seamlessly launch the AAS by any

LE, the AAS require student as well as assessment

information from the LE.

3.1 Student Information

Due to the fact that AASs personalize the

assessment, they need to be informed about learning

aspects of the student, such as prior knowledge,

learning preferences, etc. This, in turn, causes

several problems. The first is how to present the

information in a way that all LEs and AASs are able

to understand. Another problem arises, if the LEs are

CSEDU 2011 - 3rd International Conference on Computer Supported Education

118

not able to provide learning information about the

student to the AASs. In such cases, the AAS is

responsible to pre-test the student to obtain this

information. When the student launches the AAS

again, the AAS uses the previously determined

information and a new pre-test can be avoided.

Moreover, the information gathered during the

assessment process can also be used to determine

and refine the student information. A further

requirement of the AAS is to uniquely identify

students.

3.2 Assessment Information

When the LE asks the AAS to launch a particular

test, it needs to uniquely identify that test. The

required information encompasses the location of

the questions, the test identification and possibly the

version of the test.

4 STANDARDS AND

SPECIFICATIONS

The following standards and specifications can be

used to enable the integration of AASs with LEs.

4.1 Student -Modeling

The IMS Learner Information Package (LIP)

specification (2005) addresses interoperability

between internet-based LEs. It holds, maintains and

manages student information in XML documents.

Moreover the specifications records metadata such

as timestamp, source and privacy information.

The IEEE LTSC Public and Private Information

(PAPI) specification (2002) is a standard to

exchange student data between different systems. It

represents the students’ knowledge by specifying the

student-model.

As there are disjoint attributes in both

specifications, for example, privacy and security

issues, there are often combinations of elements of

both specifications used (Lalos et al., 2005;

Lazarinis et al., 2006).

4.2 Domain-Modeling

The XML Topic Maps (XTM) specification (2001)

defines a grammar for representing the structure of

information resources. XTM describe relations as

associations, mentioned in addressable information

resources (occurrences).

4.3 Questions and Tests

The IMS Question & Test Interoperability (QTI)

specification (2006) describes a data model for

representing question and test data and their

corresponding results supporting the exchange of

this material between authoring and delivery

systems. It structures material into assessments,

sections, and items.

4.4 Interworking

RQP (Remote Question Protocol) (Delius, 2005) is a

Web Service protocol based on SOAP (2007) and

has been developed by the Serving Mathematics

project (http://maths.york.ac.uk/serving_maths)

aiming at developing assessment tools in

mathematics education. Although RQP looked very

promising, the effort ran out of resources and the

protocol has never been finished. Conceptually

similar to RQP and in a working state is OPAQUE.

OPAQUE (Open Protocol for Accessing

QUestion Engines) (Hunt, 2008) is also based on

SOAP and allows LEs to delegate the presentation

of questions, the scoring of responses and the

generation of feedback to a remote question engine.

However, the LE takes full responsibility for

authenticating students and asks an appropriate

question engine to render each question. The

question engine will then process the request and

pass a response back to the calling LE. Although

OPAQUE has been implemented into the LE moodle

as well as into the question engines OpenMark

(https://openmark.dev.java.net) and STACK 2.0

(http://stack.bham.ac.uk), it is designed to allow

interoperability between arbitrary different types of

LEs and question engines.

5 PROPOSED SOLUTION

In this chapter a data structure and an interworking

of open standards and specifications is proposed to

achieve a seamless communication between LEs and

AASs. The proposed solution facilitates the

communication of control, student and assessment

information and thus enables personalized

assessment. The interworking includes the following

actions:

 Launching the test

 Communicating between AAS and LE

 Completing the test

TOWARDS A GENERIC INTEGRATION OF ADAPTIVE ASSESSMENT SYSTEMS WITH LEARNING

ENVIRONMENTS

119

5.1 Launching the Test

When the test is launched from the LE, it needs to

initialize its connection with the AAS. OPAQUE

defines the following Web Service message to

initiate a connection request between a LE and a

question engine. Please notice that AASs can simply

be referred to as question engines, although they do

much more than rendering questions.

<wsdl:message name="startRequest">

<wsdl:part name="questionID"

type="soapenc:string"/>

<wsdl:part name="questionVersion"

type="soapenc:string"/>

<wsdl:part name="questionBaseURL"

type="soapenc:string"/>

<wsdl:part name="initialParamNames

type="impl:ArrayOf_soapenc_string"/>

<wsdl:part name="initialParamValues"

type="impl:ArrayOf_soapenc_string"/>

...

</wsdl:message>

The message parts questionID and

questionVersion can be used to identify the adaptive

test, whereas questionBaseURL can be used to

define where the adaptive test resides. As mentioned

earlier, AASs not only require assessment

information, but also need student information. This

information can be submitted from the LE to the

AAS using IEEE LTSC PAPI/IMS LIP within the

message parts initialParamNames and

initialParamValues. After the AAS has received the

connection request, it will fetch the test, establish a

test session and return the corresponding session

identifier.

5.2 Communicating between AAS

and LE

When the LE has established a connection with the

AAS and has got its session identifier, the exchange

of data can be started. Independent of the internal

representation of the questions and tests and

processes taking place, each AAS has to render and

return questions in a format, which is generally

known by the LE. Due to the fact that almost all LEs

are web-based, the AAS should return data

conforming to established web standards. In

OPAQUE, the Web Service message, containing a

question, is defined as follows:

<wsdl:message name="processResponse">

<wsdl:part name="XHTML"

type="soapenc:string"/>

<wsdl:part name="CSS"

type="soapenc:string"/>

<wsdl:part name="resources"

type="impl:ArrayOfResource"/>

<wsdl:part name="progressInfo"

type="soapenc:string"/>

<wsdl:part name="questionEnd"

type="

xsd:boolean"/>

<wsdl:part name="results" type="

impl:Results"/>

</wsdl:message>

The message parts XHTML and CSS are

predestined to accommodate the HTML and the CSS

representation of the question, respectively. Further

needed resources like JavaScript libraries or the like

can be included using the resources message part.

Now, the LE can compile the question using the

several message parts and present it to the student.

After the student has answered the question, the LE

forwards the answer(s) to the AAS. In the following,

the Web Service message used to return the

answer(s) to the AAS for further processing are

described: The message part questionSession is

purposed to accommodate the session identifier of

the assessment. The proper answer(s) can be

included in the second and third message part.

Afterwards, the AAS has to compare the received

answer(s) with the correct answer(s) and decide how

to proceed. In case of a correct answer, the AAS has

to return the next question included in the test. In

case of an incorrect answer, however, the AAS has

to start its working. For example, personalized

feedback could be used to guide the student to the

correct solution or a slightly easier question

addressing the same topic.

5.3 Completing the Test

Finally, the student answers all questions of a test

and the test will be finished. Now, the AAS has to

inform the LE about the results achieved by the

student. This information is used by the LE for

reporting and computing overall test scores. In order

to inform the LE that the test is completely

answered, t

he AAS can use the questionEnd message part

included in every processResponse message (see

5.2). In this case the AAS simply has to set the value

to true and the LE knows that there are no questions

left and that the results of the test are included into

the results message part dedicated to hold this

information. Developed by the authors, a concluding

sequence diagram of the interworking of student, LE

and AAS is presented in Figure 1.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

120

6 DISCUSSION

In the previous chapter several open standards and

specifications and their interworking were presented

aiming at realizing integration of AASs in LEs. By

analyzing LEs and AASs important requirements

could be extracted, which had great influence on the

design of the interworking.

LEs require control information to determine the

state of the student in the learning process. Thus, the

interworking needs to be based on a session

management, which enables pausing and resuming

in case the student interrupts the assessment session

or lost the connection to the LE. OPAQUE uses a

session identifier to identify the requesting LE and

provides control messages to start, process, and stop

the assessment process. Among others, these are

reasons why we propose to use OPAQUE as general

interworking protocol. Moreover, although the LE

passes the assessment responsibilities to the AAS, it

needs to be informed how the student has performed

in the assessments completed. Based on this

information, the LE decides how to proceed in the

learning process. Although OPAQUE only presents

a structure dedicated to hold this information for one

question, it is flexible enough to accommodate the

student’s result of an entire test. IMS QTI was also

investigated to communicate of student results. It not

only defines a format for representing and

exchanging assessment content like questions and

tests, but also for assessment results (IMS QTI

Results Reporting package). The underlying data

structure is similar to the one provided by OPAQUE,

but the uncertainty as to whether an LE is able to

process this response could become a problem.

AASs require student information from the LEs.

This includes learning aspects, such as prior

knowledge, learning preferences, learning styles or

lifelong learning goals (i.e. career path). In chapter

4.1 candidate specifications including IEEE LTSC

PAPI and IMS LIP were presented, which aims at

presenting student information in a way that all LEs

and AASs can understand. Although there are some

standards for presenting this information, there is no

common vocabulary, which enables a common

understanding. Almost all AASs have their own

mechanisms and vocabularies for storing student

information. CosyQTI (Lazarinis et al., 2006), for

example, uses the IEEE/ACM vocabulary to

facilitate a common understanding. Such a

vocabulary combined with a unified domain-

modeling (see chapter 4.2) enables the AAS to

derive information about the current level of

knowledge of the student with respect to a specific

Figure 1: Sequence Diagram of Student, LE and AAS

Interaction.

topic. Based on this understanding, efficient

personalization by the AAS is possible. Moreover,

after the test is taken, the results can be returned to

the student-model by updating the competencies

included. As presented in chapter 5.1, OPAQUE

defines a message part to exchange student

information between LEs and AASs. The returning

session identifier can be regarded as key for further

communication and identifies the assessment

session. Another requirement is the uniform

identification of the student. OPAQUE does not

explicitly define a message part to identify the

specific student. In the IMS LIP specification it is

argued, that the source of the information record is

responsible for the uniqueness of the student

identifier. That would mean that the LE is

responsible to uniquely identify the student, in order

to track the learning process of the student. But, the

AAS has also an interest in identifying the

individuals, because it has to adapt the assessment

process accordingly. For that reason, the LE is

TOWARDS A GENERIC INTEGRATION OF ADAPTIVE ASSESSMENT SYSTEMS WITH LEARNING

ENVIRONMENTS

121

required to identify the student and to pass the

student information to the AAS (see chapter 5.1). In

addition, the AASs require assessment information

from the LEs in order to identify the specific test.

This includes the location of the questions, the test

identification and possibly the version of the test. As

presented in chapter 5.1, OPAQUE defines a Web

Service message to initiate a connection request

between LEs and AASs. Usually, questions are

defined according to the IMS QTI specification and

deposited in question banks.

7 CONCLUSIONS AND FUTURE

WORK

The objective of this paper was to analyze how

current open standards and specifications can be

used to achieve integration between LEs and AASs

seamlessly so that they can profit from each other.

The analysis was caused by an understanding of the

need of assessment adapted to the students’

individual context, prior knowledge and preferences

as well as the understanding that current LEs lack

possibilities to provide personalized assessment to

students. For that reason, both LEs and AASs were

subject of a requirement analysis. The results of the

analysis pointed out that LEs require control and

assessment information from the AASs, which in

turn require student and assessment information

from the LEs. After having analyzed the

requirements of both sides, related standards and

specifications were studied in detail and matched

against the requirements. As a result, an

interworking of several standards and specifications

were proposed, which could well be used to achieve

integration between LEs and AASs.

Future work of the institution of the main author

will implement a new AAS providing personalized

assessment. The system not only selects and presents

questions individually, but also takes sophisticated

feedback techniques and methods resulting in

providing feedback that is appropriate for the

students’ context, knowledge level, individual

characteristics and preferences into account. The

work provided in this paper helps integrating this

system in established LEs, which, in turn, can

contribute to a prompt and widespread adoption.

ACKNOWLEDGEMENTS

This work in this paper has partly been supported by

the European Project Nº 216746 PReservation

Organizations using Tools in Agent Environments

(PROTAGE) and by the Fraunhofer Institute for

Digital Media Technology, Ilmenau, Germany.

REFERENCES

Conejo , R., Guzman, E., Millan, E., Trella, M., Perez-de-

la-Cruz, J., Rios, A., 2004. SIETTE: A Web-Based

Tool for Adaptive Testing, In International Journal of

Artificial Intelligence in Education, 14, 29-61.

Delius, G., 2005. RQP – Remote Question Protocol, http://

maths.york.ac.uk/yorkmoodle/course/view.php?id=68

Gouli, E., Papanikolaou, K., Grigoriadou, M., 2002.

Personalizing assessment in adaptive educational

hypermedia systems. In Proceedings of the Second

International Conference on Adaptive Hypermedia

and Adaptive Web-Based Systems, 153-163.

Hunt, T., 2008. OPAQUE – Open Protocol for Accessing

Question Engines. http://docs.moodle.org/en/

Development:Opaque

IMS LIP, 2005. Learner Information Package,

http://www.imsglobal.org/profiles

IEEE PAPI, 2002. Public and Private Information,

http://www.cen-ltso.net/Main.aspx?put=230

IMS QTI, 2006. Question and Test Interoperability,

http://www.imsglobal.org/question

Lalos, P., Retalis, S., Psaromiligkos, Y., 2005. Creating

personalised quizzes both to the learner and to the

access device characteristics: the Case of CosyQTI. In

Proceeding of Workshop on Authoring of Adaptive

and Adaptable Educational Hypermedia, 1-7.

Lazarinis, F., Retalis, S., 2006. Engineering an

interoperable adaptive hypermedia testing tool

supporting user adaptable strategies. In Artificial

Intelligence, 204, 393-400.

Lazarinis, F., Green, S., Pearson, E., 2009. Focusing on

content reusability and interoperability in a

personalized hypermedia assessment tool. In

Multimedia Tools and Applications, 47, 257-278.

Saul, C., Runardotter, M., Wuttke, H.-D., 2010. Towards

Feedback Personalisation in Adaptive Assessment, In

Proceedings of the Sixth EDEN Research Workshop.

SOAP, 2007, Simple Object Access Protocol.

http://www.w3.org/TR/soap/

XTM, 2001. XML Topic Maps, http://topicmaps.org/xtm/

Wuttke, H.-D., Ubar, R., Henke, K., Jutman, A., 2008.

The synthesis level in Blooms Taxonomy a nightmare

for an LMS. In Proceedings of the 19th EAEEIE

Annual Conference, 199-204.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

122