CASPER: FLEXIBLE SUPPORT FOR COLLABORATIVE
ASSESSMENTS AND E-PORTFOLIOS FOR WORK-BASED
LEARNING
Martin Stanton, Nick Whittaker
UbiqLab, Department of Computing and Mathematics, Manchester Metropolitan University, Manchester, U. K.
Christopher Tubb
Division of Computing, University of Wales Newport, Newport, U. K.
Keywords: Collaborative Assessment, Work-based learning, Semantic web, RFID.
Abstract: This work proposes a system for flexible support of collaborative assessments that combines both physical
and electronic artefacts. The project builds on current work using RFID for identifying, tracking and
cataloguing physical artefacts, and also leverages Semantic Web Technologies for maintaining the
relationships between physical and electronic artefacts, allowing multiple format submissions of assessment
material. These relationships can be extended to any other documentation associated with the assessment
(marks, feedback, moderation etc) thus permitting the use of innovative, multi-faceted assessments where
less time is spent managing the submissions and more time is spent assessing and feeding back to students.
The work will provide support for any collaborative assessment, but will be particularly useful in
constructing, maintaining and assessing e-portfolios for work-based learning.
1 INTRODUCTION
Innovation in assessment is becoming a requirement
in modern higher education and is widely reported
(Galloway 2007). As employability becomes
increasingly important in higher education,
assessment, teaching and assessment practices are
often required to reflect the multi-faceted problems
that students would face in the workplace. In
addition work based learning has become a strategic
goal of higher education and there is a requirement
to develop new forms of assessment based on prior
experience and portfolios.
Academics in HE institutions have often
proposed innovative approaches to Assessment,
however in many institutions the administration of
assessment has not reflected such innovation, often
stifling the implementation of innovative practices
because the processes and systems in place have
been developed with the traditional view of
coursework/exam, and one tutor per module/course.
Even modern VLEs tend have a relatively fixed
model of assessment which, whilst allowing for
multiple markers and electronic assessments still
maintain little information about the relationships
between these assessments, or their constituent parts.
Additionally, the push towards electronic
assessment/feedback has, thus far, taken little
account of those subjects where physical artefacts
are produced and the relationship between the
physical artefact and any electronic artefacts is
stored in the tutor’s memory or in their mark sheets.
This work proposes a system for flexible support
of collaborative assessments that combines both
physical and electronic artefacts. The project builds
on current work using RFID for identifying, tracking
and cataloguing physical artefacts, and also
leverages Semantic Web Technologies for
maintaining the relationships between physical and
electronic artefacts, allowing multiple format
submissions of assessment material. These
relationships can be extended to any other
documentation associated with the assessment
(marks, feedback, moderation etc) thus permitting
the use of innovative, multi-faceted assessments
where less time is spent managing the submissions
152
Stanton M., Whittaker N. and Tubb C. (2010).
CASPER: FLEXIBLE SUPPORT FOR COLLABORATIVE ASSESSMENTS AND E-PORTFOLIOS FOR WORK-BASED LEARNING.
In Proceedings of the Multi-Conference on Innovative Developments in ICT, pages 152-157
DOI: 10.5220/0003047001520157
Copyright
c
 SciTePress

and more time is spent assessing and feeding back to
students.
2 APPLICATION EXAMPLES
The following are based on real assessment
examples, but reflect the issues surrounding
assessment and feedback in many departments and
at many institutions.
2.1 Product Design Course
This course requires students to develop concept art,
CAD, and physical model artefacts for an
assessment. The concept art if often drawn by hand,
but may be submitted either physically or
electronically. The CAD drawings are submitted as
files on a CD, and then the physical model is housed
in the workshop where it was developed. The
assessor, and moderators (including external
examiners), view all or part of this submission, and
provide feedback, in the form or written reports.
These reports are collated and then returned to the
students, and are also made available to second
markers and external examiners, who will also add
some comment. The time taken in actually assessing
the material is almost equal to the time spent
distributing, collating and locating the different
artefacts associated with this assessment, which
therefore adds significant overhead to the
assessment process and therefore the timeliness of
feedback to the students. Some students will submit
additional supporting material for their work which
can often be overlooked by assessors as it is not part
of the ‘normal’ assessment process and there is often
insufficient time to give it proper consideration. This
tends to discourage students from going beyond the
assessment requirements, as they feel it is wasted
time and effort.
The class is currently small but the course
numbers are increasing. Therefore the assessment
team is under increasing pressure to streamline the
assessment process, and are considering removing
some of the requirement for concept art to reduce the
marking workload. This is not a satisfactory
solution as it reduces the authenticity of the
assignment.
2.2 Systems Development Course
Students studying a course on systems development
have a number of assessments throughout the
academic year. Some of these assessments are
online tests, others are taken as paper based tests and
exams, and others involve the development of a
system and the production of a report.
Online quizzes are employed to test student’s
knowledge of the subject and give some immediate
feedback to students on their progress. These are
submitted and marks distributed using a popular
VLE. Paper based tests and exams are used as it is
unfeasible to use the software for such large
numbers of students in one sitting. Marks for these
must all be entered into the Institution’s marks
database and additionally some of these marks are
entered into the VLE for feedback to students.
Feedback is also written on answer sheets but
students rarely see this as they are often never
collected from the tutor.
The development project involves modelling of a
given system, and implementation of a database and
a user interface. This involves a combination of
group and individual work, and is assessed by a
number of different tutors. Unit tutors want to
improve the quality of this assessment by combining
it with assessment for another unit on project
management, teamwork and communication skills.
Given the structure of the current VLE the students
may have to make two submissions of the same
material or make a dummy submission. Staff will
have to extract components for assessment for their
particular area from the submission and also ensure
that the two submissions are consistent. The
benefits to the student of assessment in this way are
that the project becomes more realistic and they can
be given credit for all the work they do on a single
project rather than having to carry out two separate
projects, which is the current situation.
The overhead in assessing the unit in this way is
significant, especially given the large numbers of
student taking the course. Tracking the individual
and group components of the submission is never
easy, and additionally extracting parts of the
submission for particular assessors and assessments
can be time consuming. This will give rise to the
overhead in managing the assessment and possibly
reduce the time available for assessing and feeding
back to students.
2.3 Work based Learning
Work based learning can take many forms but often
is reduced to compilation and submission of a
portfolio of work built up over time, using evidence
of prior experience and work that may have been
carried out sometime in the past. Locating elements
to be included is not always easy for the learner as
CASPER: FLEXIBLE SUPPORT FOR COLLABORATIVE ASSESSMENTS AND E-PORTFOLIOS FOR
WORK-BASED LEARNING
153

these may not have been intended to be anything
other than work with little consideration of future
usefulness or relevance at the time of creation. The
artefacts that are collected in the portfolio are often
in multiple formats, with physical artefacts and
paper based reports forming a significant component
alongside electronic artefacts. Additionally the
artefacts that satisfy a particular set of outcomes
may not all be located and submitted at the same
time and need to be submitted by the learner as and
when they are located or completed. Learners will
also need to be made aware of any gaps in their
portfolio which can either be filled when artefacts
are located or when new work has been completed.
Some of these artefacts may also be protected by
patents or disclosure agreements and the learner
needs to feel able to submit these without any fear
that an assessor might not be aware of such
constraints. Often in assessing work based learning
a single assessor may not have the necessary
knowledge to adequately assess all material
submitted and they will need to be able to easily
access additional assessors for parts of an
assessment. Some artefacts will satisfy many
outcomes, whilst some outcomes may be satisfied by
many artefacts, and an assessor needs to be able to
link individual artefacts with outcomes to ensure that
at some point all outcomes are covered and to be
able to advise the learner on the sort of material that
will fill the gaps in their learning (and thus their
portfolio). Management of such assessment is time
consuming and this leaves little time for the actual
assessment and feedback process.
3 THE PROBLEM
The common themes within the above examples are
as follows:
 The complex relationship between
assessments and outcomes.
 The requirement for extracting information
from assessments relating to multiple
outcomes.
 Adequately rewarding students for all the
work they submit.
 Tracking and locating multiple artefacts that
may be related to one or more assessments
 Use of multiple assessors for a single
assessment
 Reducing the management overhead of
assessments so that more quality time can be
spent on assessment and feedback
 Allowing academics the freedom to set
appropriate assessments without the
constraint of management overheads
 Handling physical and electronic artefacts in
a uniform manner
4 THE SOLUTION
There are many partial solutions to these problems;
some are technological, with others being more
managerial and process solutions. However, no one
system can satisfy all the assessment requirements
without being sufficiently flexible itself. Current
systems that support electronic submission of
assessments and marks are inflexible and force the
user (both student and assessor) to change their way
of working to suit the system rather than the
assessment requirements of their subject. There is,
as always, a real requirement for the timely reporting
of grades and marks, and the administrative systems
that require this information are necessarily
inflexible, and cannot be expected to support the
learning process.
Additionally there is the necessary and
invaluable timely feedback to students so that they
can learn from the assessments they are given and
gain more satisfaction from the hard work that they
have put in.
The technologies to support a solution to these
problems exist, although some are either new, or
have not yet been applied to their full potential. The
use of RFID to locate and track physical artefacts
can also be used to generate an electronic profile of
those artefacts. Ontology languages (such as RDF
or OWL) can be used to create and store complex
relationships between artefacts, and provides the
flexibility to add or generate new relationships.
When combined with RFID technology we can
include physical artefacts in those relationships. The
stored information can be queried in the same way
as a more traditional database, and using the power
of XML, results can be transformed into any
required format. Interaction design for computer
based systems has evolved from the traditional
WIMP interface and we now create web based
interactions allowing users to visualise their
information in a meaningful way.
There is a significant body of work on the
development of ontologies to model theories of
learning (Barros et al 2002) (Hayashi et al 2006)
(Mizoguchi et al. 2007). From a practical
perspective, the main advantage is the flexibility of
expression that is provided by, for example, RDF
INNOV 2010 - International Multi-Conference on Innovative Developments in ICT
154

triples. A more conventional systems design would
require a static structure, usually stored in a
relational database where the components of the
assessment are fixed at design time. These
components will be stored as fields in a table, and
often the only way to add flexibility is to increase
the number of fields to allow for a variety of options.
The storage of subject, predicate, object triples
enables the development of new fields without
affecting the design of the rest of the system. We are
already using a similar approach to allow user
defined fields to be added to products in a bespoke
e-commerce system. Here the products are so
diverse that it is difficult to create a standard product
table to include even the concept of a price.
5 THE SYSTEM: CAsPEr
CAsPEr is an ambitious project that has a number of
important facets:
 Storage of assessment artefacts (electronic
and physical)
 Storage of information about artefacts
 Storage of relationships between artefacts
 Reporting information to users and other
systems
These facets are drawn together to form a
number of major subsystems and these are details in
the following section.
5.1 System Architecture
An overview of the system architecture is given in
figure 1. This provides a very general view of the
system although each of the components will have
their own subsystems.
Figure 1: CAsPEr Architecture.
5.2 Components
The main components of CAsPEr will carry out the
following roles:
 RDF Data Store
This stores the information about Artefacts and
other ‘resources’ and the triples are stored in a
relational database and/or as RDFXML.
 File Storage
The RDF Data store points to any electronic
resources that are associated with an artefact.
These are stored in the file system and can be
retrieved and viewed locally. Additional files
can be added to an artefact, such as
assessments or feedback.
 GUI
The GUI allows users to add new information.
This includes the addition of new artefacts,
new resources, information about them and the
relationships between them. The GUI hides the
complexity of the underlying system and shows
the resources and artefacts as a connected set of
icons. New artefacts can be dragged from the
desktop (initiating the storage of files
associated with them) or can be created from
the GUI. Artefacts can be related to other
artefacts or to students, assessors, assessments
etc.
 RFID subsystem
The RFID subsystem acts as another means of
adding artefacts and information about them. It
manages the tagging, registration and location
of physical artefacts, and monitors any changes
to their status.
 Management Subsystem
Manages the creation and maintenance of
relationships between artefacts and resources.
Directs resources to their related files and
physical resources. Handles queries to the
RDF Data Store.
5.3 The CAsPEr Ontology
The CAsPEr Ontology is a lightweight ontology that
has been developed initially as a simple RDF graph.
It shows the potential for reasoning about new and
complex relationships. For example, Assessments
can already assess outcomes for a number of Units
and an Artefact can form part of a number of
different Assessments.
The Ontology is shown in Figure 2 as output
from Altova’s SemanticWorks software. As can be
seen, Artefacts play a central role in the ontology
with most other resources being related directly to
them in some way.
There is more work to be done refining this
Ontology but already it shows potential to add the
requisite flexibility to the system.
CASPER: FLEXIBLE SUPPORT FOR COLLABORATIVE ASSESSMENTS AND E-PORTFOLIOS FOR
WORK-BASED LEARNING
155

5.4 The CAsPEr GUI
The user interface for CAsPEr will be crucial for its
success and should derive from the nature of
assessments as being constructed from artefacts and
being related to students and units. These
relationships can be expressed graphically as shown
in Figure 3.
Figure 2: Part of the CAsPEr Ontology.
Figure 3: CAsPEr Graphical User Interface.
Artefacts can be built using the standard dialog
boxes, or can be dragged into the client window.
Relationships between artefacts and other resources
will reflect those of the underlying model.
6 RELATED WORK
Previous work has looked at the use of ontologies in
and for education but these tend to be developed for
the study of theoretical aspects of learning and
knowledge (Barros et al 2002) (Hayashi et al 2006)
(Mizoguchi et al. 2007), or to examine a specific
knowledge domain (Sosnovsky, and Gavrilova,
2006
). The CAsPEr ontology is designed
specifically to manage the operational features of
assessment, and whilst it does take into account
learning theories, such as constructive alignment, it
is based on the practical use of those theories to
form an information system rather than to theorise
about their nature.
Sánchez-Ruíz and Welling (2008) give details of
Ontology-Based assessment assistants, which whilst
covering some experimental work in assessment
automation, does provide a useful architecture for
assessment systems. This would complement
CAsPEr well as their ISA would form one part of
the assessment subsystem whereas CAsPEr would
handle the management of assessment artefacts.
Their artefacts may have domain specific properties,
and definitions, which could be merged with
CAsPEr’s.
Sicilia and Lytras (2005) provide a very useful
overview of recent work on the use of ontologies in
learning theory, and point towards the development
of systems that support learning design. This
provides some scope for the development of CAsPEr
to support generation of appropriate assessments
given the needs of learners and the design of
programmes of study.
Work is being carried out on integrating
semantics with existing LMSs, for example with
Moodle (Lukichev et al, 2007). CAsPEr is not
intended as a replacement for complex LMSs but it
will be able to integrate with them, managing the
assessments and providing an interface with them.
7 CONCLUSIONS
The use of innovative assessment methods to
support learning within higher education is
becoming more important, highlighted by strategies
such as the increase in work based learning.
CAsPEr provides a mechanism by which
assessment which may require a number of artefacts
across a number of learning outcomes and subjects
areas may be required. This is achieved through the
use of an RDF ontology and Semantic web
technologies.
Adoption of Semantic Web Technology is
finally becoming mainstream and, no doubt, as LMS
vendors scrabble to provide trendy social
functionality using emerging “open and social”
standards, we will see more systems built on RDF
Schemas. CAsPEr is not intended to replace an
LMS, as focus is on the management relationships
between artefacts rather than to control/monitor the
learning process. However it could be used to
INNOV 2010 - International Multi-Conference on Innovative Developments in ICT
156

complement an LMS by providing easy management
and collation of artefact and performance data.
Although other systems may employ similar
techniques CAsPEr has the advantage of providing a
flexible and domain independent method of
managing these assessments.
Importantly CAsPEr allows for the management
and control of the artefacts of an assessment without
imposing the rigid constraints on the forms and
structure of assessment seen in many other
assessment management technologies.
REFERENCES
Galloway, A., 2007. Diversity and Innovation in
Assessment Practices in Higher Education Institutions
and by Employers and Training Providers. Research
and Information Services Bulletin 25. Scottish
Qualifications Authority
Hayashi, Y., et al 2006. Ontological Modeling Approach
to Blending Theories for Instructional and Learning
Design, in Proc. of The 14th International Conference
on Computers in Education (ICCE2006)
Mizoguchi, R. et al. 2007. Inside Theory-Aware
Authoring System, in Proc. of The Fifth International
Workshop on Ontologies and Semantic Web for E-
Learning (SWEL'07)
Sánchez-Ruíz , A. and Welling, B., A Domain-Specific
Approach to the Development of Ontology-Based
Document Assessment Assistants, in Proc DSM 2008:
The 8th OOPSLA Workshop on Domain-Specific
Modeling. Nashville, TN, October 2008.
Barros, B., Verdejo, M.F., Read, T. and Mizoguchi, R.
(2002). Applications of a Collaborative Learning
Ontology. In Proceedings of MICAI’2002 Mexican
International Conference on Artificial Inteligence,
Springer-Verlag, pp. 301-310
Sicilia , M. Lytras M, On the representation of change
according to different ontologies of learning.
International Journal of Learning and Change . vol 1,
pp66—79, 2005
Sosnovsky, S. and Gavrilova, T. (2006). Development of
educational ontology for C-programming. Information
Theories & Applications, 13(4), 303–308.
Lukichev, S., Diaconescu, .I, and Giurca, A, Empowering
Moodle with Rules and Semantics. 3
rd
Workshop on
Scripting for the Semantic Web, Innsbruck, Austria
2007
CASPER: FLEXIBLE SUPPORT FOR COLLABORATIVE ASSESSMENTS AND E-PORTFOLIOS FOR
WORK-BASED LEARNING
157