![](bg5.png)
4 EXPERIMENTAL VALIDATION
The course “Design and Production of Educational
Materials” is one of the e-learning courses in the
Education Degree in the University of Murcia. This
course took place in the second semester of
2008/2009, and had 25 students. All the work is
realized in our virtual campus SUMA
(http://suma.um.es/). The working processes of the
students are evaluated with an e-portfolio and other
different activities throughout the 9 themes of the
program and also the participation of the students in
several communication situations (videoconferences,
forum and collaborative works). The final evaluation
is carried out with two types of exams: multiple
choice, and open question test. For this second
phase, OeLE was used and it served for the
validation of the approach.
Now, we describe the process followed in this
validation experiment:
1) Development of the course ontology. The OWL
ontology has been done using Protégé, and it
has been imported into the OeLE platform. The
ontology has 111 classes, 71 object properties,
51 data type properties, including also disjoint
and cardinality constraints. Its consistency has
been checked using Pellet, and the ontology has
ALCHIN(D) DL expressivity.
2) Preparation of reinforcement contents: A series
of HTML learning objects were designed and
associated to the concepts of the course
ontology.
3) Design of the first exam: An exam containing 5
open questions was created using OeLE, and the
expected answers were annotated.
4) Execution of the exam: The students had to
answer this test using OeLE and with a time
limit. The students could review the contents of
the course in the virtual environment and could
search on internet to find answers during the
realization of the exam. This exam was taken by
21 students.
5) Assessment of the exams: The exams were
marked by a teacher and by OeLE.
6) Feedback: The students and the teacher received
the marks and the feedback generated by OeLE.
The students reviewed the reinforcement
learning objects associated to the knowledge
items suggested by OeLE.
7) Repetition of steps 3, 4 and 5 for the second
exam. This exam was taken by 20 students.
8) Evaluation of the feedback: This was done by
the students. Students were asked to answer a
questionnaire about the effectiveness and
usefulness of the learning objects and the
feedback received.
The whole experiment and the results can be found
at klt.inf.um.es/~oele/feedback. This includes: the
ontology, the questions of the exams, the annotations
of their expected answers, the reinforcement
contents, samples of the annotations extracted from
the students’ answers, samples of the semantic
feedback generated by OeLE for the teacher and for
a particular student, the marks of the students in both
exams, and the questionnaire filled by the students.
Next, some evaluation of the feedback results is
shown.
First, we compared the results obtained by the
students in both exams. If the feedback generated by
the system was effective, then the students should
have obtained a better mark in the second exam. The
maximum possible score in an exam is 10. The
average mark of the first exam was 6.18 (21
students)/6.12 (20 students) and the average mark
for the second one was 6.56 (20 students).12
students obtained a better mark, 4 obtained a worse
mark and 4 obtained a similar mark. For this
classification, we defined that a student obtained a
similar mark is the difference was not greater than
0.25. Consequently, it seems that the feedback
generated was useful for the students. However, this
is a single, and small experiment so strong
conclusions cannot be drawn from such results.
Therefore, we asked the students to answer a
questionnaire. This was designed from a pedagogical
perspective and it included questions related to
different issues such as usability, accessibility,
quality of the learning objects and usefulness of the
feedback.
Next, we discuss the results of the three
questions related to the feedback. A Likert scale was
used for answering to the questions. In this sense,
the students had to assign a value between
1(maximum disagreement) and 4(maximum
agreement). In order to summarize the results, two
groups were created: agreement (3-4) and
disagreement (1-2). The detailed results can be
checked at the aforementioned website. These are
the three questions:
• Question 1: Knowing the errors made in my
exam is a waste of time.
o Agreement: 16.7%
o Disagreement: 83.3%
• Question 2: Showing the feedback information
about the errors in my exam is positive.
GENERATION OF USEFUL SEMANTIC FEEDBACK FOR STUDENTS AND TEACHERS
225