ADDING TAGS TO COURSES TO IMPROVE EVALUATION

A Multiplatform LCMS Approach that Allows Multidimensional Analysis

Eduard Cespedes-Borras, Aitor Rodriguez, Jordi Carrabina and Javier Serrano

Centre de Prototips I Solucions HW/SW, Universitat Autonoma de Barcelona, ETSE, E-08193 Bellaterra, Spain

Keywords: Tag, Bloom’s Taxonomy, Educational Objectives, Metadata, Statistical Analysis, E-learning, SCORM, QTI,

Learning Design, Authoring Tool, LMS, Assessment Analysis Model.

Abstract: The main idea of this paper consists in adding tags to the contents available in any given course materials

structured according to any Learning Content Management System (LCMS). Tags, very popular in web 2.0

applications, give a free and flexible way of characterizing materials according to any criteria that a teacher

can imagine. Therefore, one can use them for any specific analysis and clustering of both teaching

methodology and students learning. Our approach claims to be platform independent in the sense that can be

applied on top of any current LCMS. To achieve that property, we define a XML specification that includes

specific, platform dependent, queries. This choice is much more efficient than building plug-ins or

hardcoded solutions for any existing learning platform (and its underlying data-base). At the end of the

paper, we show the powerfulness of this approach with a course example.

1 INTRODUCTION

A large number of specifications have been

generated in order to standardize some aspects of the

e-learning process, and also a large variety of

proposals which standardize the educative contents.

Successful examples are Sharable Content Object

Reference Model (SCORM) (ADL, 2008) or some

of IMS (IMS Content Package, IMS Simple

Sequencing, etc) (IMS, 2001).

There are also other specifications, like Learning

Design (LD) (IMS LD, 2003), that manage the

eLearning sequencing process. LAMS (LAMS,

2002) is the main tool for creating this type of

contents. However, its evaluation depends on the

LCMS like Moodle (Moodle, 2008) or Sakai (Sakai

Project, 2003), in which LAMS could be included.

Concerning SCORM, its evaluation capabilities

depend also on the LCMS that will manage it.

Besides, there are systems based on the Question

and Test Interoperability (QTI) (C. Smythe, 2002)

standard that focuses on the questions and their

evaluations. By this last, it is easy to find analytical

and statistical tools that allow us to increase the

evaluation performance.

Generally, there is a lack of standardized

reporting systems that could be used to achieve

conclusions on the efficiency of the teaching and

learning processes.

This paper proposes two new features: (1) a new

specification (with the corresponding tool) to tag e-

learning structures and (2) a methodology to

efficiently connect our tool to any eLearning

System. In this way, we pretend to improve the

management of the learning evaluation, and at the

same time, give flexibility to the evaluators to add

any criteria, that can later produce a high variety of

results using multidimensional analysis tools.

2 E-LEARNING STANDARDS

ANALYSIS

There exist different e-Learning standards that

emphasize different learning aspects. Following, we

will briefly review them.

2.1 SCORM Introduction

The Sharable Content Object Reference Model

(SCORM) is an aggregated specification for

asynchronous distance learning, organized by the

Advanced Distributed Learning Initiative (ADL).

Cespedes-Borras E., Rodriguez A., Carrabina J. and Serrano J. (2009).

ADDING TAGS TO COURSES TO IMPROVE EVALUATION - A Multiplatform LCMS Approach that Allows Multidimensional Analysis.

In Proceedings of the First International Conference on Computer Supported Education, pages 92-97

DOI: 10.5220/0001977900920097

 SciTePress

One important issue in SCORM is the technique

of packing course material sources, structure and

metadata into one exchangeable object.

The content packaging format is defined by the

IMS Content Packaging specification. Course

package uses zip format. In the zip file, an

imsmanifest.xml file exists, which is an eXtensible

Markup Language (XML) file used to express

organization and resources. Most standards follow

this format. SCORM organization defines the course

structure using a tree hierarchical model in which

each item could be either a simply html content or a

Sharable Content Object (SCO), an improved item

that defines questions and tests.

SCORM 1.2 version is the most used. It is rare

enough to find tools that implement version 1.3

completely (E. Cespedes-Borras, 2008). Evaluation

tools are also simple and only produce a global

mark for each course.

2.2 IMS Question & Test

Interoperability (QTI)

This specification describes a data model for

representation of question (item) and tests

(assessment) data and their corresponding results

reports. Therefore, the specification enables the

exchange of obtained data (item, test and results)

between authoring tools, item banks, test

constructional tools, Learning Management Systems

(LMS), and assessment delivery systems.

This specification is also divided in two parts:

Assessment, Section and Item (ASI), that defines the

structure of an exam with its questions; and Result

Reporting (RR), that defines the qualifications of

multiple students to any ASI element (from a single

question to a full exam).

Despite this, metadata in QTI is very complex.

There are eight question types, and for each

question, metadata have more than 20 attributes.

IMS QTI specification uses the industry standard

XML as the way to relate data model information

into physical representation.

QTI standard is widely accepted to design

examinations. So, it is easy to find it in all LCMS as

Moodle or Sakai. Analysis of exams can use the

recently published tool (X. Gumara, 2008), QTI

Result Reporting Stats Engine, one of the few

OpenSource analysis programs available for this

kind of applications.

2.3 IMS Learning Design (LD)

The IMS Learning Design specification supports the

use of a wide range of pedagogies in on-line

learning. Rather than attempting to capture the

specifics of many pedagogical methodologies, it

provides a generic and flexible language. This

language is designed to enable the description of

different pedagogical methodologies. A XML

document, with its typical tree structure, is used as

well. In this structure, LD defines activities and

grouping types, what is more flexible than SCORM

that only defines course items.

Learning Activity Management System (LAMS)

is the main LD platform. It provides a highly

intuitive visual authoring environment for creating

sequences of learning activities. Even though, their

tools are more focused on the course monitoring

than on the analysis of the evaluation.

Table 1: Brief learning standard comparison based on their

implementation state and tools.

Specification

Implemented

Tools

Evaluation

Method

SCORM 1.2 Moodle

One global final

result

QTI Complete

Moodle,

Sakai

QTI Result

Report

LD Level A LAMS

LAMS

monitoring and

final result

After studying these specifications, we can

conclude that: (1) most e-Learning models could be

defined as a tree structure; and (2) there is a lack of

analysis in most of tools (in the QTI case an

improvement of the statistical data provided can be

achieved); and (3) it does not exist a general

statistical tool that could be used in cooperation with

any standard or LCMS.

3 MULTIDIMENSIONAL

DESCRIPTIVE SYSTEM

In order to outperform the evaluation, we propose to

add new features, tags, to the descriptive data of any

eLearning specification, that could be represented

with an internal tree structure. Labeling can be

understood as adding descriptions at any different

depth levels of the tree.

Labels consist of one or several keywords, that

will allows us to create a non-structured description

to generate multiple classifications (see the example

in section 6). This descriptive system is

implemented as a set of descriptive tags. Tagging is

an easy method to implement and its use is very

ADDING TAGS TO COURSES TO IMPROVE EVALUATION - A Multiplatform LCMS Approach that Allows

Multidimensional Analysis

popular (i.e. gmail classification of emails, delicious

for bookmarks, etc), so that tagging is one of the key

points of the success of a large variety of context-

like web 2.0.

Our proposal of specification uses this concept

of tag to specify each node from the abstract tree.

Concerning the SCORM standard, the structure is a

complete course and each description is assigned to

each one of its items, and it does not matter if these

are ASSETS or Sharable Content Objects (SCO).

In the case of Learning Design, tags will increase

the data of each sequence item (activities or tool)

that describe a course.

For the third case considered, QTI, each node

represents an item or an assessment.

So, generally speaking, any system or educative

standard that structures their contents in a tree shape,

can be extended by the use of this methodology.

4 XML SPECIFICATION

PROPOSAL

The eXtensible Markup Language (XML) is well-

known language among the eLearning community

and will be used to describe the proposed Advanced

Statistical Evaluation of Education Contents

(ASEEC) specification.

The specification is divided in two sections: (1)

dataSource section, that defines the connection and

access parameters to the data repository, and (2)

dataOrganization, which defines the global shared

structure (shown in Figure 1).

The aim of the first section is directly connect

our analysis tool to data base through access

parameters to get the complete tree structure whose

tagging will be defined in the second section.

We consider that this option is better to generate

tags on top of QTI Result Reporting, that has been

studied as an interchange data format, due to the fact

that this one is not platform independent.

Figure 2 shows an example on how ASEEC

specification minimizes the complexity through this

direct connection. Section dataSource is split in

dbConnection and dbDependency. Login,

passphrase and url repository are defined in the

dbConnection subsection. Section dbDependency

contains the SQL sentence that will produce, as a

query result, the set of data to be further analyzed.

Each format and type of every column are defined in

this subsection.

Figure 1: XML specification proposed.

Figure 2: dataSource structure.

Second section, dataOrganization, contains the

abstract tree structure of the course. Each item of the

structure is formed by a title, a set of tags and the

corresponding recursive items.

5 PLATFORM INDEPENDENT

APPROACH

By implementing the specification presented in this

paper, it is possible to define a minimum connection

model between LCMS platform and an external

analysis tool (described in figure 3).

This type of connection will allow us to use the

specification of the course structure (as presented in

previous section) to extract the essential contents of

the course.

…

dbConnection

dependency

dataOrganization

Title

item

Title

1 Ta

…

item

dataSource

CSEDU 2009 - International Conference on Computer Supported Education

A critical technical point of our proposal is to

set-up, for each LCMS, the right parameters of the

query sentence to the corresponding data base. The

expected output of the query will be a table

containing one row per each user for a given course

and item. The format and value of each column will

be given by the specified parameters of the query.

Once the course structure is obtained, we can add

tags. The purpose of this tags is to add any attribute

that a teacher can consider useful to get a an

evaluation (for both students and teaching process).

SQL

uer

result

specification

Analys is

tool

eLearning

Course

LCMS

Data Base

format

wrapper

tag

adding

eLearning

Platform

Model

Proposed Model

(

Platform Inde

endent

)

Figure 3: Multiplatform design.

As a result, we will dispose of a tool able to

connect to any LCMS system, without the need to

build neither a plug-in nor modifying the code of the

LCMS. Though it was not our primary intention, this

would let to a unified analysis model able to be used

with different LCMS (nowadays it happens some

times that a professor has to use two different LCMS

even in the same institution).

5.1 Implementation Case Example

Figure 4 shows a potentially real implementation

application case of ASEEC specification. In a given

university campus professor can use either a Moodle

server, a proprietary SCORM LMS (i.e. for courses

rich in multimedia content), a proprietary LMS (i. e.

from the beginning of virtual university courses) or

any combination of them.

Finally, the proprietary LMS, that still contains a

large amount of course materials of that university,

has a proprietary format. Again, the hardest step is to

define the SQL query that would be injected.

Previous two cases would potentially be solved

quickly if the access policy to the corresponding

LMS managers is open enough.

Once the three ASEEC XML files are created,

our (or any) external analysis tool can connect to

each LMS platform transparently.

Figure 4: Real implementation case.

6 IMPROVEMENT OF ANALYSIS

AND EVALUATION

Adding tags to the course structure is oriented to

allow any multidimensional evaluation wanted by

the teacher, thanks to due the possibility of including

different tag sets to describe the contents, not only

questions and answers.

A set or group of tags can cluster for instance the

contents of the course, the taxonomy of educational

objectives (Bloom, B.S, 1956), the type of activities

made by the students, or any other knowledge the

teacher would like to include in.

These clusters will allow to group students

results according to the teacher needs at any moment

with the evaluation tool. This is helpful to improve

the evaluation of both course and individual

students.

The course evaluation improvement is possible

by detecting a general lack in some defined

educational objectives. On the other hand, the

teacher has the possibility to analyse the level of

knowledge acquisition of any single student (or

group of students) in any particular type of content

or activity.

6.1 Evaluation Use Case using Tags

In this section, we present an specific case devoted

Moodle

SCORM

Server

Propietary

LMS

Moodle

course

SCORM

course

Other

course

MySQL MySQL Oracle

XML

course

XML

course

XML

course

Evaluation Tool

Inde

endent Platform Model

Learnin

Platform Model

ADDING TAGS TO COURSES TO IMPROVE EVALUATION - A Multiplatform LCMS Approach that Allows

Multidimensional Analysis

to illustrate the potential use and improvement given

by our tool.

The Signals and Systems teacher of the computer

science degree wants to apply a taxonomy of

educational objectives in his subject and he wants to

know if all of them have been achieved by the

majority of students.

He also wants to detect possible failures in the

development of the different type of activities.

In order to get a better evaluation, the teacher

imports the course from the Moodle platform where

it is done and adds suitable tags (non exclusive) for

each course element (nodes in the course tree XML

specification).

He groups the course tags in three clusters (as

shown in Figure 5): content description, Bloom’s

Taxonomy and type of activity.

Figure 5: Tag adding to a defined course.

After the course is given, a set of results are

stored in the MySQL database by the Moodle

platform.

With the new evaluation tool, he will obtain the

database results for all students of the course and the

course schema enhanced with the tags he added.

Then, he will generate some figures (i.e. graphs)

about the results combining the multiple dimensions

of tags until he find something strange.

In the example given by figure 6, one can find

that qualifications for the analysis (educational

objective) are too low (4

set of bars). This is even

more important due to the fact that most of the

subject contents of that course (Signals and Systems)

are related to analysis.

In a deeper analysis, the teacher will try to find if

there is any specific module with low qualifications

than others or if this behavior is general for all

modules labeled as analysis.

He will generate a graph that shows the

qualification rate for all modules and elements with

that tag and discover that the analysis mark in

seminars has lower qualifications than in the other

activities (as shown in figure 7). So, at the end, he

will know that he must improve the way he is giving

the seminars.

Knowledge Understanding Application Analysis Synthesis Evaluation

Module 1 Module 2 Module 3

Figure 6: Clustering the course results in Bloom’s

Taxonomy educational objectives.

'Analysis' tag results

Lecture Seminar Homework Lab Group Work

Module 1 Module 2 Module 3

Figure 7: Clustering the analysis tag course results by the

type of activity done.

7 CONCLUSIONS

In this paper, we have proposed a new specification

to improve the management of the learning

assessment giving flexibility to the evaluators to

enhance the descriptions of any given eLearning

course contents with the tags (metadata) they decide

to add. The proposed model can be adapted to most

of the existing LCMS, as well as any other learning

models that structures its contents hierarchically.

This flexible addition of tag in any course

content structure allows a better assessment by

means of clustering the results according to any set

(multidimensional) of labels any teacher can

imagine.

We pretend in a close future to extend the

proposed XML specification to allow the acquisition

of results from other sources, not only from

databases. We are specially interested in the support

of the QTI Result Reporting format to make NOM

QTI compliance.

CSEDU 2009 - International Conference on Computer Supported Education

REFERENCES

Advanced Distributed Learning (ADL) – SCORM (2008).

Retrieved December 1, 2008 from

http://www.adlnet.gov/scorm/

IMS Global Learning Consortium (2001). Retrieved

December 1, 2008 from http://www.imsglobal.org

IMS Learning Design (2003). Retrieved December 1,

2008 from http://www.imsglobal.org/learningdesign/

Learning Activity Management System – LAMS (2002).

Retrieved December 1, 2008 from

http://www.lamsinternational.com

Moodle (2008). Retrieved December 1, 2008 from

http://www.moodle.org

Sakai Project (2003). Retrieved December 2, 2008 from

http://www.sakaiproject.org

C. Smythe, E. Shepherd, L. Brewer and S. Lay (2002,

February), IMS Question & Test Interoperability: An

Overview, Final Specification, Version 1.2, IMS

E. Cespedes-Borras, L. Vicent, M. Segarra (2008,

September), WIP- New Data Structure In SCORM

2004 Sequencing & Navigation, FIE.

X. Gumara, L. Vicent, M. Segarra (2008, July), QTI Result

Reporting Stats Engine for Question-Based Online

Tests, ICALT.

Bloom, B.S. (Ed.), Engelhart, M.D., Furst, E.J., Hill,

W.H., & Krathwohl, D.R. (1956). Taxonomy of

educational objectives: The classification of

educational goals. Handbook 1: Cognitive domain.

New York: David McKay.

ADDING TAGS TO COURSES TO IMPROVE EVALUATION - A Multiplatform LCMS Approach that Allows

Multidimensional Analysis