MEASURING THE LEARNING PROGRESS IN A “LEARNING
BY AUTHORING” SEMANTIC WEB SERVICES
BASED ECOSYSTEM
Ivo Hristov, Gennady Agre and Danail Dochev
Institute of Information and Communication Technologies, Bulgarian Academy of Sciences
Acad. G. Bonchev St, block 2, Sofia, Bulgaria
Keywords: Learning by authoring, Active learning, Learning assessment, Learning task.
Abstract: The paper presents the approaches and methodology for defining learning tasks and measurement of the
learning outcomes in the learning by authoring environment. Assessment of the students work is made in
three directions: measurement of relevance of the embedded multimedia files; assessment of the quality of
the analysis made by the learner and visual appearance of entire multimedia document. The entire
architecture of the project is briefly presented in order to position the tool that realizes the approaches.
1 MOTIVATION
The evaluation and assessment of learner’s work has
never been an easy task in learning by doing
environment. Many practical solutions prioritize the
assessment methods suitable primary for automatic
processing. In such systems the learner answers are
compared to previously defined patterns. In order to
facilitate the evaluation of the learner’s performance,
many of the assessments restrict the potential steps
and the range of activities that the learners can take.
This in turn leads to restriction of the innovative
thinking and creative ideas that the learners might
apply. In our work we are targeting the methodology
that has two major goals: first, to allow the learner
wide area of possible solutions, and second, to
provide as much support as possible to the evaluator
when she is evaluating students work addressing
learners mistakes and weak points.
The paper presents a part of the ongoing work
on the SINUS project, which targets the
development of specialized e-learning facilities
allowing learning by-doing through learners’
authoring of specific learning materials by intensive
use of multimedia digital libraries (D. Dochev and
G. Agre, 2009). The SINUS environment is focused
on pro-active achievement of these goals by learn-
ers’ own actions, supported by the built-in domain
and pedagogical knowledge. The present paper
discusses the desired functionality of a SINUS
module called pedagogical knowledge editor (PKE)
used for defining learning objectives, tasks and
criteria for assessing the progress of the learners.
Such an assessment primary targets automatic or
semi - automatic support of the learners in their
work. Students can do self-assessment at every stage
of their work, receiving some hints and
recommendations from the system aiming at
improving the accuracy of the work and suggesting
concrete steps for such an improvement. On the
other hand, the instructor can see the learner
progress on demand; review the mistakes and helps
the learner in a proper way. The next section
introduces the general structure of the SINUS
project in order to position the PKE and its main
functionality in the entire architecture. The SINUS
underlying infrastructure – software services,
ontologies and semantic repositories used by the
PKE is briefly described. The third section presents
the approach taken for defining learning objectives
and learning tasks. The aim of the approach is to
enable the teacher to define the learning tasks
according to terms in the domain ontology and
criteria for successful task execution. The fourth
section deals with the assessment of learner’s work.
In our case it is a multimedia document created by
the student following the learning tasks. The section
describes in details the assessment of different stages
of the work – on learner’s request or as final grade
and explains how the system and the evaluator
413
Hristov I., Agre G. and Dochev D..
MEASURING THE LEARNING PROGRESS IN A “LEARNING BY AUTHORING” SEMANTIC WEB SERVICES BASED ECOSYSTEM.
DOI: 10.5220/0003472004130418
In Proceedings of the 3rd International Conference on Computer Supported Education (ATTeL-2011), pages 413-418
ISBN: 978-989-8425-50-8
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

measure the achievement of learning objectives. The
paper concludes with a short summary of the
presented approach and future work.
2 ARCHITECTURE OF THE
SINUS PROJECT
The SINUS project (sinus.iinf.bas.bg) is an
interdisciplinary 3 years research project aiming at
advancing the two of the fastest evolving
information technologies – Service Oriented
Computing and Technology Enhanced Learning by
applying the Semantic Web Service Methodology.
The project methodology is based on adaptation and
enhancement of some original methods and software
components developed by the project members
under FP-6 IST projects INFRAWEBS
(www.infrawebs.eu), LOGOS
(www.logosproject.com) and intensive study of
LT4eL project (http://www.lt4el.eu/).
2.1 Narrative Description of Learning
Scenario
According to the scenario the learning domain of the
project is the domain of Bulgarian Christian
Iconography. The main target group of learners
contains students following different classes of arts,
art history, theology etc. The learning scenario
(Pavlova-Draganova, L. and D. Paneva-Marnova,
2009) defines four groups of learners according to
their goals and interests: theological team; art critic’s
team; art technique’s team; artistic team. The
learners are given a general task to prepare a project
on the Iconography of Christ in the Historical
territories of Bulgaria. The set of tasks are different
for the different groups and teams. For example the
learners are expected to make: analysis of the
theological meaning of the Iconography of Christ
(theological team); art critic analysis of the
development of Christ’s image in the different
iconographical schools in Bulgaria (art critic’s
team); study of the main iconographic techniques
used in the historical territories of Bulgaria (art
technique’s team) and etc.
In order to do their work, the learners are
expected to prepare a multimedia document –
learners’ project by following the learning tasks. For
example learners are expected to prepare: (Pavlova-
Draganova, L. and D. Paneva-Marnova, 2009):
 analysis of the theological meaning of the
Iconography of Christ;
 art critic analysis of the development of Christ’s
image in the different iconographical schools in
Bulgaria;
 study of the main iconographic techniques used
in the historical territories of Bulgaria
In order to prepare the project, the students have to
develop scholarly essays/projects for pre-assigned
by the teacher analyses of given characteristics of
the objects under study. Each learner has to create
beforehand a limited sized task-focused collection of
multimedia objects to serve as a base for performing
the necessary analysis, as well as illustration of the
theses in the final analytical essay. The description
of a task aiming at developing a dedicated
collection, which is given to the users, may include,
for example, activities like:
 Search and preview of objects from given
centuries;
 Search and grouping the objects according to the
iconographic schools
 Filtering the results and grouping according to
iconographic technique.
 Presentation of the search results and grouping of
the objects according to given periodization.
According to the learning scenario, the PKE presents
learning tasks, links the learning tasks to the
predefined learning objective and enables definition
of assessment criteria for each task.
2.2 Technical Architecture
As presented on the Figure 1, the PKE is built on the
top of an extended digital library. By using PKE the
instructor creates the learning tasks and objectives
and stores them into the Pedagogical Knowledge
Repository that is then used by a learning
management system (LMS). The extended digital
library is a SOA-based architecture able to add and
exploit the semantics extending an existing digital
library – in our case the digital library of Bulgarian
Iconographic objects (digital library service – Figure
1). The digital library is used as an external source
and presents its functionalities as a web service. The
objects stored in the digital library can be
additionally annotated according to terns in selected
ontology (in our case Ontology of Bulgarian
Iconographic objects – OBIO) by using the Semantic
Object Annotator and Resource editor and the new
annotations are stored in the New Semantic
Annotation Repository. The Semantic Annotations
and Resource creation component uses concepts and
relations from different specialized domain
ontologies stored in the Ontologies Repository. Such
CSEDU 2011 - 3rd International Conference on Computer Supported Education
414

ontologies can be created and edited by means of the
Ontology Editor. The extended Search Engine is a
middleware software component used by all editors
for indexing, searching and retrieving of the
resources in all of the repositories.
Figure 1: SINUS Project architecture.
2.3 Functionality of the Pedagogical
Knowledge Editor
PKE is oriented to support the active learning
approach engaging the learner into the process of a
multimedia document creation. The active learning
approach gives people control over their own
learning (Bradford S. and W. J. Kozlowski, 2008). It
promotes an inductive learning process, in which
individuals must explore and experiment with a task
to infer the rules, principles, and strategies for
effective performance That is, the learner assumes
primary responsibility for important learning
decisions e.g., choosing learning activities,
monitoring and judging progress. The major
guideline in SINUS is to support the learners in their
activities. For this reason the learning tasks and the
assessment are subordinate to this guideline.
In general the learning task requires the learner
to take the appropriate set of actions, which satisfy
the predefined knowledge to be achieved in order to
satisfy the learning objective. Every learning task is
related to a learning activity. In our approach we
state that every single learning task has a range of
measurable learning outcome predefined on the task
creation. PKE is a design time tool, that implements
functionality of a simple specialized learning design
editor. In an extended learning scenario, where more
complicated learning objects are required, the tool
might require integration with an external learning
object creation tool as in (Hristov I., 2010).
3 LEARNING TASKS
DEFINITION
According to (Limberg, L., 2007), even though the
conceptions of a task can vary, the common trait is
that the task is seen as an activity to be performed in
order to accomplish a goal. Tasks are considered as
having a recognizable purpose, beginning, and end.
From our point of view, the definition of a learning
task is based on the following characteristics:
 Every learning task is related to a correspondent
learning objective.
 Learning tasks are defined in a way that does not
restrict the successful solution to the single possible
way for fulfilling the objective.
 Learning tasks contain descriptions of
measurable range of qualitative parameters for
successful completing the task that can be used by
the evaluator for assessing the work
In accordance with the learning scenario the learner
is required to analyze some aspects of the learning
domain – in our case the Bulgarian iconographic
objects (Staykova K. and Dochev D., 2009) and
prepare a multimedia document that contains the
analysis – i.e. the learning project. Actual learning
happens during proactive project creation. Learning
tasks are selected and ordered in a way to enable the
learner to achieve the learning objectives and
facilitate active learning. The learner deals with
three major generalized activities types:
 Collection creation – The learner searches the
repository for appropriate objects and selects some
of them according to the pre-defined learning task.
This is an activity that requires the learner to
differentiate the important aspects of iconographical
objects, their properties and the meanings. The result
of such a kind of activities is creation of collection
of iconographical objects that is based on a given set
of criteria.
 Preparation of analysis – The learner makes an
MEASURING THE LEARNING PROGRESS IN A "LEARNING BY AUTHORING" SEMANTIC WEB SERVICES
BASED ECOSYSTEM
415

analysis on some aspect of the collection. The result
of this type of activities is a textual description that
contains the analysis.
 Document structuring – The learner formats the
target document according to some visual
requirements and enter the textual description that
contains the analysis of the subject. The result of this
type of activities is a positioning and sizing of the
elements of the analysis
In order to support the evaluation of the learner
work, PKE requires every learning task to have
criteria for measuring of work success. The criteria
are primarily used for automatic support of the
learners when they are doing the task and for
preparing a list for assessment contours that the
instructor might use when tracking the learner
progress for final assessment or in order to support
them. As a generalization, the PKE has to model
three types of activities: Activities that require the
learner to search, select and insert multimedia
objects from the repository – actions related to
collection creation; activities that require performing
the analysis by comparing the objects from the
collection against the pre-assigned learning task; and
activities related to producing and formatting the
final multimedia document.
The instructor defines the required activities by
textual description of the learning task and the
criteria for the assessment. The text description is
presented to the learners. The way for defining
assessment criteria differs for different types of
tasks.
Criteria for assessment, related to the searching,
finding and inserting multimedia object tasks –
collection creation, are primarily defined by the
learning task description, containing instructor
recommendations about the possible steps to create
limited-sized task-focused collection which is rich
enough for the pre-assigned task (in our example
case – recommendations about minimal desirable
coverage of different authors, iconographic schools,
time periods, iconographic techniques etc.). By the
user interface of the system the instructor determines
the appropriate parameters for the objects, and the
system generates a query (formal representation of
the learning task using domain ontology terms). The
instructor can execute the query, review the result
set and refine the query if required. The key factor
here is that the system internally stores implicitly the
query or the union of queries (if needed) with
parameters that retrieve the successful multimedia
objects. The formal representation is used later for
checking the learner’s selection relevance. The
major advantage of the approach is that the new
objects, inserted in the repository are automatically
used even if they do not exist during the task
definition.
The preparation of analysis is another activity
that the learner can take. The quality of the analysis
prepared by the learner is a major subject of
assessment. In such a type of learning task the
instructor is required to enter explicitly through the
user interface a list of terms from the Ontology of
Bulgarian Iconographic objects (OBIO) (Staykova
K. and Dochev D., 2009), that have to be present in
the learner’s analysis.
Another option for the instructor is to provide an
exemplary textual analysis of the requited task. The
text entered by the instructor is annotated
automatically (or semi-automatically) with concepts
from the domain ontology (in our case - OBIO). The
system will infer the terms from the text and use
them for the evaluation of the text entered by the
learner. An advantage of the approach is that the
system can find the frequency of occurrence of the
terms used by the instructor for more precise
assessment. The usage of the ontology terms for the
assessment of the learner is disused in the next topic.
In the learning scenario the formatting activities
and visual arrangements are not directly related to
the learning goals. The instructor might prepare
some templates for storing the analysis, but the
visual appearance is intended to be used for
engaging the learners in a social collaborative
engagement since the assessment is planned to be
done by the other learners’ groups.
4 MEASURING THE LEARNER
PROGRESS
In order to support the learners in their work, we
propose basic mechanism for tracking their progress
by automatic identification and storing the
intermediate and final results of collection
development, presented by their semantic (meta)
descriptions (i.e. the corresponding instances of the
domain ontological model). This allows automatic
comparison with the formalised task description,
represented similarly.
In general this approach means that the system
stores incremental actions taken by the learner in
order to do the learning tasks. The approach enables
the system to recreate the student project following
the steps taken by the learner, since it stores the task
oriented activities, not only the current state of the
project. The system can track the user actions for
CSEDU 2011 - 3rd International Conference on Computer Supported Education
416

every task and generate automatically some hints
and warnings for errors. This can be used also for
identifying mistakes by the instructors.
Since the actions of the learners are stored task
by task, the criteria for task assessment are created
on the task definition it is possible to compare the
results of learners’ work to the results provided on
the task creation. The assessment can be done on the
entire work or on the set of tasks – partial
assessment. By registering the learner’s work task
by task and relating the learning tasks to learning
objectives we can assess the actual learning
progress.
4.1 Measure the Relevance of a
Task-focused Collection
The assessment of the relevance of the selected
multimedia objects (primary images) is important
part of evaluation of the learners work, because it
captures the ability of the learner to distinguish
different aspects and elements and symbolism of the
iconographical objects. The approach for assessment
is based on comparing the symbolic representation
of the learning task with the objects semantic
(metadata) descriptions. In this way the learning
environment may evaluate if the developed
dedicated task-focused collections of multimedia
objects contain sufficiently rich and various
illustrative material to back-up the analyses (e.g.
checking minimal number of analysed objects,
sufficient coverage of authors, iconographic schools,
time periods, diversity of desired characteristics
etc.).
4.2 Assessment of the Quality of the
Analysis
Analysis made by the learner is a process related to
comparing different characteristics of the
iconographic objects. The analysis requires the
learner to learn essential aspects, features, relations,
artefacts and directions in the learning domain. The
required knowledge is achieved by performing the
learning tasks. As a result of the analysis, the learner
prepares a textual description that contains the
learner contribution. The quality of the learner
analysis is the subject of the assessment. The
assessment of text contribution cannot be fully
automated. The essential part here is to support the
evaluators by presenting to them some measurable
counters. We intend to use the methodology
described as Knowledge rich approach in (Osenova
P. and Simov K., 2010). The knowledge rich
methods rely on the analysis of the text by using
knowledge sources, external to this text, such as
ontologies, lexicons and grammars. These sources
are used to achieve a semantically rich text analysis
which to explicate the conceptual content of the
learner’s answers.
The assessment of the learner analysis is based
on finding the terms for domain concepts (and their
possibly different linguistic forms) in the text
entered by the learner. The evaluator might use the
following parameters generated by the system: used
(obligatory or desirable) terms, missed terms,
frequency of terms occurrences, terms collocations
in a paragraph (a hint that they may be also
semantically related in the text) used by the learner
and instructor and the number of used terms by the
learner. It is obvious that, if the number of used
terms by the learner and number of terms used by
the instructor are approximately equal, the
probability of analysis to satisfy the learning
objectives is high. The system cannot automatically
grade the learner work, but it supports the
assessment by presenting to the evaluator
meaningful counters, for example – the concepts that
the learner missed or terms that the learner overused.
The presented counters can help the instructor, but
the final grade is done by the evaluator. We intend to
continuously use the approach in order to send some
hints to the learners when they missed some
concepts in their analysis.
4.3 Evaluation of the Visual
Appearance of the Entire
Document
Since the learning tasks are not highly prescriptive
by definition about formatting the document, the
evaluation of visual appearance of already prepared
multimedia document has very subjective elements.
In general the tasks do not prescribe anything about
visual appearance for the texts and the images into
the document. Nevertheless, the document is
structured in accordance with learning tasks, and
also the analysis seems to be suitable, according to
the terms uses, the entire document might not has
good appearance for the external users. Even the
visual appearance of the document is very subjective
it seems to be depended of the formatting features
like: the size and the position of the images and
texts; amount and balance of texts and images,
colours of the texts, background and many others.
Since there the criteria for assessment of visual
appearance are mainly qualitative, and the measure
of visual appearance is not the major goal, subject of
MEASURING THE LEARNING PROGRESS IN A "LEARNING BY AUTHORING" SEMANTIC WEB SERVICES
BASED ECOSYSTEM
417

learning, we assume that key factor for assessment
of visual appearance is the satisfaction of users, who
are using the created documents. We propose a
mechanism that allows the users to rate the
documents. After the document is checked by the
evaluator and it is marked as satisfying the required
learning tasks, it is opened for previewing and rating
by the other groups of learners. The learners might
rate the document, write a public or private comment
or use the document in their work. Since the work is
ongoing, we assume that this approach might
highlight the best student project. The students will
rate and comment the work of their colleagues,
according to their own knowledge. This feature will
give to the students some kind of reflectional view
of their own work, and the opportunity to change
some fragments according to the comments of the
others. In our view the social dimensions of the
learning are essential factor for the success, so the
students are encouraged to rate, comment and use
the work of their colleagues.
5 CONCLUSIONS
The paper describes an approach for helping the
evaluation of specific learning-by-authoring
activities, producing analytical essays/projects on
limited-sized dedicated collections of multimedia
objects, created by the learners according pre-
defined learning tasks.
The three types of learner activities are
continuously evaluated for giving the support to the
learner. The activities related to collection creation
are evaluated on the bases on the comparison of the
results of learner’s queries and formalised learning
task description prepared by the instructor. The
evaluation of quality of the analysis is made by the
evaluator. The evaluator is supported by the system
with measurable counters of concepts used or missed
by the learners and the percentage of conjunction in
concepts. The visual appearance, which, in our case,
is not subject of learning, is rated by the other
participations in the project.
The described approach is used as a base for
implementing the PKE – a part of the SINUS TEL-
oriented environment. In the moment the efforts are
concentrated mainly on a part related to
representation of the pedagogical knowledge
describing the process of creation and assessment of
the quality of a multimedia object collection that is
necessary for writing the analytical essay. The future
work will be focused on effective ways for formal
representation and use of pedagogical knowledge
needed for analysing the created collection.
ACKNOWLEDGEMENTS
The work on this paper was funded partially by the
Bulgarian NSF project D-002-189 SINUS “Semantic
Technologies for Web Services and Technology
Enhanced Learning”.
REFERENCES
Bradford S. Bell and Steve W. J. Kozlowski, 2008, Active
Learning: Effects of Core Training Design Elements
on Self-Regulatory Processes, Learning, and
Adaptability, Journal of Applied Psychology 2008,
Vol. 93, No. 2, 296–316
D. Dochev and G. Agre, 2009. Towards Semantic Web
Enhanced Learning. In: Proceedings of the 1st
International Conference on Knowledge Management
and Information Sharing (KMIS 2009), Funchal,
Madeira, Portugal, 6-8 October 2009, pp. 212-217
Hristov I., An Architecture of eLearning Enterprise
Authoring Studio, In.: Proceedings of Eighth
International Network Conference (INC2010),
Heidelberg, Germany, 6 - 8 July 2010, ISBN: 978-1-
84102-259-8, pp. 253 - 262
Limberg, L. 2007. "Learning assignment as task in
information seeking research" Information Research,
12(1) paper colis28. [Available at
http://InformationR.net/ir/12-1/colis/colis28.html]
Osenova P, Simov K., 2010, Semantic Annotation for
Semi-Automatic Positioning of the Learner, In:
Proceedings of the First Workshop on Supporting
eLearning with Language Resources and Semantic
Data, at LREC 2010, Malta, pp. 46-50.
Pavlova-Draganova L., D. Paneva-Marnova, 2009. A Use
Case Scenario for Technology-Enhanced Learning
through Semantic Web Services, International Journal
„Information Technologies & Knowledge”, vol. 3,
2009
Staykova K., Dochev D., 2009. Ontology “Bulgarian
Iconographical Objects” Creation and Experimental
Use, “Cybernetics and Information Technologies”,
Vol. 1, pp. 25-36
CSEDU 2011 - 3rd International Conference on Computer Supported Education
418