Onjira Sitthisak, Lester Gilbert, Mohd T Zalfan and Hugh C. Davis
Learning Technologies Group, School of Electronics and Computing Science
University of Southampton, Highfield
Southampton, SO17 1BJ, United Kingdom
Keywords: Interactivity, Formative assessment, Learning design, IMS, QTI.
Abstract: We examine the integration of IMS Question and Test Interoperability (QTI) and IMS Learning Design
(LD) in implementations of E-learning from both pedagogical and technological points of view. We
propose the use of interactivity as a parameter to evaluate the quality of assessment and E-learning, and
assess various cases of individual and group study for their interactivity, ease of coding, flexibility, and
reusability. We conclude that presenting assessments using IMS QTI provides flexibility and reusability
within an IMS LD Unit Of Learning (UOL) for individual study. For group study, however, the use of QTI
items may involve coding difficulties if group members need to wait for their feedback until all students
have attempted a question, and QTI items may not be able to be used at all if the QTI services are
implemented within a service-oriented architecture.
E-learning can be viewed as the process of web-
based or online learning within an open, flexible,
and distributed learning environment (Westera et al.,
2005). Although several web-based educational
systems have been developed, these are ineffective
for facilitating the reuse and sharing of either
educational content or activities (Sampson et al.,
The IMS Learning Design (LD) specification
was introduced (IMS LD, 2003) to promote the
exchange and interoperability of E-learning
materials and to support pedagogical diversity. This
specification offered a standardized way to associate
educational content, activities and actors in the
design of any teaching-learning process. Educational
developers can use IMS LD to model who does
what, when, and with which content and services in
order to achieve the intended learning objectives.
The IMS QTI specification is used for
exchanging assessment information such as
questions, tests, and results. Similarly, it aims to
promote the exchange and interoperability of
assessment materials and services (IMS QTI, 2006).
We are looking into practices in the area of
integration between IMS LD and IMS QTI. IMS
QTI can be integrated with IMS LD in a number of
ways, and questions arise about integrating IMS LD
and IMS QTI from the point of view of pedagogical
effectiveness. In particular, implementations may
not sufficiently promote or control the desired
interactivity experienced by learners, or may present
an ineffective interactivity within the teaching-
learning process.
In this paper, we consider the presentation of
assessment and the resulting interactivity within an
Unit Of Learning (UOL) through the use of
IMS QTI. First, the role of assessments and
interactivity in the teaching-learning process is
explored. Second, the basic ideas of IMS QTI and
IMS LD are described. Third, various
implementation cases of assessment in IMS LD are
explained and their problems identified. Finally, the
joint use of IMS QTI and IMS LD are evaluated in
terms of improvement in flexibility, reusability, and
other parameters in the provision of best-case
interactivity expressible within a LD UOL.
Sitthisak O., Gilbert L., T Zalfan M. and C. Davis H. (2007).
In Proceedings of the Third International Conference on Web Information Systems and Technologies - Society, e-Business and e-Government /
e-Learning, pages 440-445
DOI: 10.5220/0001269304400445
The level of interactivity such as communication,
participation, activity, and feedback has a major
impact on the quality of technology enhanced
learning. Consequently, “interactivity does not
simply occur but must be intentionally designed”
(Berge, 1999, p.5) into an E-learning system.
In an E-learning systems context, a cycle of
interactivity occurs when the students are presented
with a number of choices that requires them to
actively process the course information and
materials, and are then given prompt, contingent,
and specific feedback about their particular choice.
This view of interactivity is based upon principles
from the psychology of learning. Figure 1 illustrates
these key characteristics of the interactivity cycle
(Gilbert and Gale, 2007).
Figure 1: Characteristics of the interactivity cycle.
Interactivity begins when the student is required
to process actively the materials and information. In
order to ensure active processing of the materials,
the student is posed a problem, question, or asked to
undertake an activity that offers a number of options
or choices. The student makes a choice, and
receives feedback about that choice. The
interactivity cycle completes, and the student
continues with the next learning activity.
Rolfe and McPherson (1995, p. 837) note that
“feedback or knowledge of results is the life-blood
of learning”. Appropriate feedback from assessment
can motivate students and redirect their learning
towards areas of deficiency, and can help teachers
improve their coursework and instructional methods.
Assessments may be categorized as diagnostic,
formative, or summative (McMillan, 2006).
Formative assessment should be followed by
feedback and remedial guidance so learners may
know their deficiency in understanding, knowledge,
or competence (Rolfe and McPherson, 1995).
Feedback is most effective and usable by the
student when it is immediate, specific, and
contingent (McKendree, 1990; King, 1999).
Generally, feedback should be given immediately, or
as soon as possible. Delayed feedback is usually
less useful the more it is delayed, and is of course
completely useless if it never arrives. The feedback
must be specific if it is to be optimally effective. It
is the specificity of the feedback that allows the
students to focus on exactly those aspects of their
learning that could do with improvement.
Most importantly, feedback must be contingent.
This is a technical term that means the feedback
must be functionally dependent upon, must follow
and be linked to, the student’s selective response.
Effective feedback should incorporate these three
factors in order to support a well-designed unit of
In this paper we implement and evaluate
interactivity using two specifications, IMS QTI and
IMS LD, by expressing an UOL which involves the
provision of feedback in formative assessment.
The IMS QTI specification (IMS QTI, 2006) is a
part of the same family of specifications as IMS LD.
It describes an information model for representing
questions, tests, and results. This specification
enables the exchange of item; test, and results data
between authoring tools, item banks, and test
construction tools, as well as learning systems and
assessment delivery systems. QTI version 2.0
processing is illustrated in Figure 2.
Figure 2: QTI version 2.0 processing.
When a learner accesses a Virtual Learning
Environment or Learning Management System
(VLE/LMS) to view and respond to a QTI question,
the system initially sends a QTI XML file to a QTI
processing service where a Question renderer
renders the question, the rendered question is sent
back to the VLR/LMS for display to the student.
The learner’s answer is sent to a QTI Response
renderer which marks the answer and provides
feedback. The rendered feedback is sent back to the
VLE/LMS for display to the learner.
IMS LD (IMS LD, 2003) is based on the following
principles: in a learning process each person has a
role (learner or teacher) and achieves learning
outcomes by carrying out learning activities within a
supportive environment. The major concept of the
IMS LD, the Method, is an element which allows the
coordination of activities of each role in the
designated environment to achieve learning
The learning process is modeled on a theatrical
play from a structural point of view. A Method
consists of one or more concurrent Play(s); a Play
consists of one or more sequential Act(s); an Act
consists of one or more concurrent Role-Part(s), and
each Role-Part associates exactly one Role with one
Activity or Activity-Structure.
In this study, we construct an IMS LD UOL to
provide questions, check the correct answer and give
feedback. We use Learning Design to orchestrate the
above processes according to the interactivity cycle
of Figure 1.
Activities in LD are associated with a Role in a
Role-Part, and they contain the actual instruction for
a person in that role. If the activity is directed at a
learner and aims to achieve a specific learning
outcome it is referred to as a learning activity.
A LD Method may contain conditions, i.e. If-Then-
Else rules that further refine the assignment of
activities and environment entities for persons and
roles. The ‘If’ part of the condition uses Boolean
expressions on the properties that are defined for
persons and roles in the LD UOL. Properties are
containers that can store information about persons’
roles and about the UOL itself, e.g. user profiles,
progression data (completion of activities), results of
tests (e.g. prior knowledge, competencies, learning
styles), or learning objects added during the
teaching-learning process (e.g. reports, essays or
new learning materials).
To explore assessments implemented using IMS LD,
a small UOL was developed incorporating question
and feedback activities. Students could see all
question activities, and could access each question in
turn. Interactivity was implemented as follows.
First, a question with multiple answer choices was
presented to each student. Next, the student
responded to the question by selecting one of the
choices. Then, the student’s response was
evaluated. Finally, the student received immediate,
specific feedback relating to his or her particular
answer. Then, the student moved forward to the
second question where the pattern was repeated.
This implementation may be considered ‘individual’
study. For a ‘group’ study implementation, a
student was given the feedback of a question only
after all students had finished answering it.
Using different mechanisms of Play, Act,
condition elements, and activity conditions within
IMS LD in order to control the interactivity, a
number of different UOL cases were developed for
the individual and group assessment scenarios.
Because IMS QTI has the capability to encapsulate
the question and feedback, each UOL case involving
a question activity immediately followed by a
feedback activity can be alternatively implemented
as an IMS QTI item instead of being implemented
within IMS LD.
Shows an implementation structure of the
assessment case for individual study using IMS LD
Figure 3: The structure of ‘individual’ study
implementation IMS LD (illustrated with two students).
WEBIST 2007 - International Conference on Web Information Systems and Technologies
alone. Figure 4 shows the implementation structure
for individual study using assessment implemented
as IMS QTI items.
Figure 5 shows the
implementation structure for group study using the
assessment implemented as IMS LD items.
Figure 4: The structure of ‘individual’ study
implementation IMS LD + QTI (illustrated with two
Figure 5: The structure of group study implementation
using IMS LD (illustrated with two students).
In this study, the criteria for evaluating the IMS LD
UOL of ‘individual’ and ‘group’ implementations
are as follows:
Interactivity quality
The four criteria of interactivity quality are: the
control of interactivity, and the specificity,
immediacy, and contingency of feedback.
Ease of coding
Ease of coding refers to the ease of providing the
functionality needed in the implementation of each
Flexibility and reusability
Flexibility and reusability refers to the ease with
which the properties of the UOL can be changed and
re-used in other contexts.
First, we consider the analysis of the simple
UOL ‘individual’ implementation with one Play, one
Act, and one Role-Part.
In , when the question activity is completed, the
feedback activity will be displayed immediately
based on the result of answering the question.
Hence, the IMS LD-only ‘individual’
implementation () provides full support for
specificity, immediacy, and contingency of
Using the capability of IMS QTI to encapsulate
the question activity and the feedback activity
Figure 4), the feedback message in the QTI activity
will be displayed immediately when learners answer
the question. As with the case of the LD-only
structure, the LD + QTI UOL ‘individual’
implementation provides effective interactivity since
feedback is immediate, specific, and contingent.
Because IMS LD and IMS QTI provide mechanisms
for controlling interactivity by using activity
conditions, sequence/selection properties, and QTI
mechanisms, the ‘individual’ implementations ( and
Figure 4) fully support ease of coding.
With regard to flexibility and reusability, the LD-
only structure () provides partial support because
changing, adding, or deleting the question and/or
feedback requires re-coding the UOL. This is due to
the dependency of the feedback activity on the result
of answering in the question activity. However, this
limitation may be addressed by implementing the
assessments as IMS QTI items (
Figure 4), increasing
the flexibility and reusability of the UOL. The IMS
QTI features provide for simpler coding within the
UOL and enhance its reusability.
Second, we analyze the UOL ‘group’
implementations as follows. Due to the nature of the
‘group’ study, all group members need to complete
the question activity before starting the feedback
activity. Therefore, the learners may not get their
feedback immediately after answering the question.
However, they can still get specific and contingent
feedback for their answers. As a result, a UOL
‘group’ implementation offers only partial support
for immediate feedback, but with full support for
specific and contingent feedback.
The LD-only UOL ‘group’ study implementation
Figure 5) provides for ease of coding through the
use of the ‘Act’ LD structure to control interactivity.
However, when implementing the assessment items
as QTI items, there may be difficulties with coding
within the UOL because IMS QTI may not provide
sufficient support for controlling group interactivity.
As in the LD-only UOL ‘individual’ study
implementation, the separation of question and
feedback activities in the IMS LD-only UOL ‘group’
study implementation (Figure 5) the may cause
difficulties with changing and re-using this UOL in
other contexts. The ‘group’ study implementation of
Figure 5 provides only partial support for flexibility
and reusability. The LD + QTI ‘group
implementation also provides only partial support
for flexibility and reusability of implementation, but
for a different reason. If group members need to wait
for their feedback until all students have attempted a
question, QTI items may not be appropriate.
According to QTI processing (Figure 2), feedback is
sent to the learner immediately after receiving the
answer. Hence, it may not be possible to implement
this version of ‘group’ study with QTI version 2.0
items by using rendering and response services
within a service-oriented architecture.
Table 1 shows the analysis of the ‘individual’
study and table 2 shows the analysis of ‘group’ study
assessment implementations using IMS LD alone
and with IMS QTI.
Table 1: Analysis of ‘individual’ study assessment
LD +
Figure Figure 4
Approach for
Activity condition,
Full support Full support
Full support Full support
Ease of coding Full support Full support
Flexibility and
Partial support Full support
Table 2: Analysis of ‘group’ study assessment
Figure Figure 5 Not illustrated
Approach for
Act mechanism May not be
depending upon
QTI service
Partial support Partial support
Full support Full support
Ease of coding Full support Partial support
Flexibility and
Partial support Partial support
Table 1 and Table 2 illustrate two important issues.
First, in measuring the pedagogical effectiveness
of any assessment, the model of interactivity shown
in Figure 1 provides key indicators. These include
the specificity, immediacy, and contingency of the
feedback given to the student upon completion of
the assessment. An instructional designer may
evaluate an implementation of IMS LD and IMS
QTI against these measures.
Second, the IMS QTI specification can be
considered as an integrative layer in implementing
IMS LD UOLs. However, there are some
shortcomings when integrating IMS QTI and IMS
LD implementations, as discussed in the evaluation
section. Instructional designers should consider this
issue when integrating IMS QTI items within an
The features of IMS QTI help the instructional
designer to implement an assessment within an IMS
LD UOL for individual study, solving the problems
that we found: ineffective interactivity, difficulty of
learning design coding, inflexibility, and poor
reusability. Teachers are increasingly expected to
create or adapt online activities without any
technical support from specialists, and the use of
IMS LD and QTI standards should help them meet
these expectations. Future developments in IMS LD
WEBIST 2007 - International Conference on Web Information Systems and Technologies
aim to improve the quality of e-learning, not only for
educators, but also for learners, and aim to increase
adaptation and reuse of UOLs (De Vries et al.,
Our study suggests that interactivity may be used
as a parameter for the pedagogical evaluation of
assessment and E-learning. As a result, instructional
designers are able to talk in terms of pedagogy rather
than technology, making explicit pedagogical
choices, subject to review, inspection, and critique.
Integration of IMS LD and IMS QTI would
increase the value of UOLs, but attention needs to be
paid to the usability of QTI items within ‘group’
study UOLs. The study and classification of group
activities and typical interactivities will provide
guidelines for developers to implement QTI and LD
authoring and run-time tools which allow
instructional designers to realize pedagogically
informed UOLs.
Berge, Z. L., 1999. Interaction in Post-Secondary Web-
Based Learning. Educational Technology, Vol.39,
No.1, pp 5-11.
De Vries, F., Tattersall, C. and Koper, R., 2006. Future
developments of IMS Learning Design tooling.
Educational Technology & Society, Vol.9, No.1, pp 9-
Gilbert, L. and Gale, V., 2007. Principles of E-learning
Systems Engineering, Chandos.
IMS LD, 2003. IMS Learning Design Best Practice and
Implementation Guide. Available from
IMS QTI, 2006. IMS Question and Test Interoperability
Overview. Available from
King, J., 1999. Giving feedback. British Medical Journal,
Vol.318, No.7200, pp 1-6.
McAndrew, P., Nadolski, R. and Little, A., 2005.
Developing an approach for Learning Design Players.
Journal of Interactive Media in Education 2005(14).
Available from
McKendree, J., 1990. Effective Feedback Content for
Tutoring Complex Skills. Human-Computer
Interaction, Vol.5, pp 381-413.
McMillan, J. H., 2006. Classroom Assessment: Principles
and Practice for Effective Instruction, Pearson
Technology Group.
Rolfe, I. and MaPherson, J., 1995. Formative assessment:
how am I doing? The Lacent, Vol.345, No.8953, pp
Sampson, D., Karampiperis, P. and Zervas, P., 2006.
Authoring web-based learning scenarios based on the
IMS Learning Design: Preliminary Evaluation of the
ASK Learning Design Toolkit. IEEE, pp 1003-1010.
Available from
Westera, W., Brouns, F., Pannekeet, K., Janssen, J. and
Manderveld, J., 2005. Achieving E-learning with IMS
Learning Design - Workflow Implications at the Open
University of the Netherlands. Educational
Technology & Society, Vol.8, No.3, pp 216-225.
Available from