EVALUATING AN E-LEARNING EXPERIENCE ORIENTED
TOWARDS ACCESSIBLE INSTRUCTION
Félix Buendía, Alberto González-Téllez, José Vte. Benlloch-Dualde, Germán Moltó,
Natividad Prieto, M. J. Castro-Bleda and Juan V. Oltra
Universitat Politècnica de València, València, Spain
Keywords: e-Learning Experience, Accessible Instruction, Quality Evaluation.
Abstract: e-Learning is becoming an essential tool in the field of accessible instruction. This work describes the
evaluation of an experience implemented at the Universitat Politècnica de València that addressed the
adaptation of its e-learning platform to make accessible its contents and resources. Such experience has
been developed in the context of the EU4ALL project which provided a general framework to cope with the
needs of accessible lifelong learning at a Higher Education level. The first part of the experience consisted
in developing specific components in Sakai, an open-source Learning Management System that is used at
UPV as the institutional e-learning platform under the name of PoliformaT. The second part dealt with the
evaluation of the developed Sakai version for a set of disabled students who registered in different
Engineering and Business courses at UPV and lecturers who contributed to adapt the required learning
resources. The results of the evaluation showed, on the one hand, the interest of students about the
availability of alternative accessible resources but, on the other hand, the point of view of lecturers who
were in charge of developing such resources and manifested the difficulty and effort to generate them.
1 INTRODUCTION
e-Learning is becoming an essential tool in the field
of accessible instruction. This kind of instruction is
growing in importance and there are several
initiatives to promote it. For example the Accessible
Instruction Pledge (Atomic Learning, 2011) states
the opportunity to “Understand the individual needs
and abilities of learners…” or “Create documents
and handouts that can be read and completed using
assistive technology” among other principles. The
current work agrees with this statement and it
addresses accessible instructional issues in a Higher
Education (HE) context.
Nowadays, there are multiple efforts in different
universities to make accesible their web sites and e-
learning platforms. However, most of these
initiatives have been focused on Web accessibility.
In this case, the focus is pointed at checking content
aspects and the way to match these instructional
contents to the user needs. Such circumnstance has
required the adaptation of the e-learning platform
used at the Universitat Politècnica de València
(UPV) with the purpose of improving their content
accessibility within the context of an European
project called EU4ALL. This project was funded by
the European Commission to construct a general
framework and extensible architecture of European-
wide services that enable all students, including
disabled students, or students with special needs, to
access HE studies, from enrolment to examination
and graduation (EU4ALL, 2011).
The remainder of the work is structured as
follows. The next section provides a general
overview about the context of the depeloped e-
learning experience. The third section introduces the
method used to evaluate such experience and then,
its implementation for enabling accessible
instruction in the given context. The fifth section
describes the evaluation results of the e-learning
experience. Finally, some conclusions and further
works are remarked.
2 CONTEXT
The following subsections describe the context of
the e-learning system developed at UPV within the
EU4ALL project. First, the higher education
scenario is introduced and then, the technological
381
Buendía F., González-Téllez A., Benlloch-Dualde J., Moltó G., Prieto N., Castro-Bleda M. and V. Oltra J..
EVALUATING AN E-LEARNING EXPERIENCE ORIENTED TOWARDS ACCESSIBLE INSTRUCTION.
DOI: 10.5220/0003910003810386
In Proceedings of the 4th International Conference on Computer Supported Education (CSEDU-2012), pages 381-386
ISBN: 978-989-8565-07-5
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
Figure 1: PoliformaT main screen.
support services provided by UPV in such
experience are reported.
2.1 Higher Education Scenario
The Universitat Politècnica de València is one of the
two main universities in the city of Valencia. The
interest of the UPV to assist people with special
needs in learning and research issues has been
growing recently. At the beginning of the year 2010,
the Information and Communication Technology
Office (ASIC) was invited to collaborate within the
EU4ALL project. The ASIC office has been in
charge of the institutional e-learning platform called
PoliformaT. The main goal for the participation of
the UPV in the EU4ALL project was to adapt the
PoliformaT platform to integrate it within the
EU4ALL framework. Once this adaptation was
performed, a set of evaluation experiments were
conducted by the paper authors.
2.2 Technological Support
The PoliformaT platform was developed from the
framework provided by the Sakai environment
(Mengod, 2006). Sakai is a consortium of
universities, colleges and commercial affiliates
working in open partnership with standards,
organizations and other open-source initiatives to
develop “community-source enterprise-scale
software applications to enhance collaboration,
research and teaching within higher education”
(White, 2005).
The UPV became a Sakai partner in 2005 and
adapted its components to produce the PoliformaT
platform. Some of the PoliformaT contributions
were the integration within the corporate systems
and applications, the customization of its appearance
and the internationalization, including the translation
to Spanish language.
Figure 1 shows a screenshot of
a PoliformaT site that displays some components of
the EU4ALL-UPV portal such as Tools and Content
areas. Several experiences have been developed in
the last years using the PoliformaT platform to
check its instructional potential in online courses
(Buendía & Hervás, 2008) and the current work has
enabled its adaptation to the EU4ALL requirements.
3 METHOD
The evaluation of the e-learning experience
developed within the EU4ALL project has been
based on a method addressed to check the user’s
point of view. There are multiple proposals in the
evaluation of e-learning experiences such as
quantitative vs. qualitative models, formative vs.
summative, internal vs. external, evaluation based on
experimental works or ethnographic studies
(Mandinach, 2005). In this case, the chosen method
was oriented towards gathering qualitative
information about the user experiences and
considering several phases in the evaluation process,
from the assessment of the user profiles (either
students or lecturers) to the final collection of their
perspectives and their analysis. In order to analyse
the obtained evaluation results, the UPV research
team selected a quality model (Pawlovsky, 2003)
proposed in the context of the European Quality
Observatory (EQO) and addressed to structure
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
382
quality approaches for evaluating e-learning
experiences. This model has also been applied to
evaluate experiences based on the use of the
Polifomat e-learning platform (Ejarque et al., 2007).
In summary, the method to evaluate the
EU4ALL e-learning experience at the UPV has been
divided in three main tasks.
First, to define specific learning scenarios in
which these experiences are developed, establishing
the elements to be addressed along the target
experiences.
Second, to state the main research questions to
be evaluated through the defined learning scenarios.
This research was mainly focused on checking the
Needs Assessment and the Authoring Support
services which are part of the EU4ALL project.
Additionally, several quality evaluation criteria were
assigned to the proposed research questions.
Third, to collect the evaluation results from users
who participated in the EU4ALL experiences. This
collection process was performed through
questionnaires submitted to users and their results
were analyzed by means of the referred quality
model.
3.1 Data Collection
Two different types of data collection techniques
were deployed. The first one was oriented to gather
the point of view of users through a personalized
interview, one addressed to students and a second
one for lecturers. The students’ interviews were
promoted by the technical responsible at CEDAT
(UPV disability office) who contacted with several
students with different disability issues. These
interviews revealed a huge variety of user profiles
and subjects and this fact moved the research team
to establish a set of basic teaching scenarios to be
evaluated:
Computing topics in several fields such as
Computer Fundamentals, Computer Technology, or
Data Structures.
Business topics such as Marketing or Legal
Economic aspects.
Another technique to collect data was the use of
questionnaires which were submitted to users after
the final tests at the end of the project. First a
questionnaire for students was prepared that
included three main sections: i) Demographic data,
ii) Filling out the profile form EU4ALL and iii)
Access to the course resources. A second
questionnaire was designed for lecturers with
question items classified also into three categories: i)
Demographic data, ii) Filling out the point of view
about the current PoliformaT version and iii) Filling
out the point of view about the new prototype of
PoliformaT.
3.2 Participants
The participants in the evaluation process were
classified in two main groups: i) lecturers who
provided course materials to be adapted in the
EU4ALL context and ii) disabled students who were
registered in different UPV courses, mainly in
Computer, Business and Engineering disciplines.
One of the main problems in the evaluation of the
UPV pilot site was the diversity and heterogeneity of
student profiles and courses. About ten lecturers
were asked for participating in the project but only
six of them were able to evaluate the UPV pilot site.
In the case of students, the CEDAT office contacted
with them and finally, ten students participated in
the experience (four with hearing impairment and
six with visual impairment).
3.3 Materials
The evaluation was focused on two main areas: i)
Computing courses and ii) Business courses. These
courses were face-to-face (classroom-based) and
they provided a set of instructional materials mainly
based on text documents (pdf in most cases), and
Microsoft PowerPoint® presentations. However,
they also included video or other multimedia
formats. For instance, in Computing courses, audio
versions were produced for pdf documents for
describing lab instructions or adding additional
information in graphical presentations (e.g. wiring
diagram), which were difficult to read for visually
impaired people. In the case of Business courses,
some deployed materials were screencasts, lecture
recordings and podcasts, which were captioned or
provided transcripts to students with hearing
troubles.
All the aforementioned materials were stored in a
Web site provided by the PoliformaT platform
which was assigned to the UPV-EU4ALL project
Moreover, this site has been used for lecturers and
students who provided samples of different kind of
teaching materials and resources, and also for other
lecturers and students as end-users who were
interested to check the produced materials before
testing them in the PoliformaT prototype adapted to
the EU4ALL requirements.
EVALUATINGANE-LEARNINGEXPERIENCEORIENTEDTOWARDSACCESSIBLEINSTRUCTION
383
4 IMPLEMENTATION
The UPV evaluation was carried out using the two
main data collection tools aforementioned. This
evaluation was performed on the PoliformaT
prototype developed within the EU4ALL project
with the collaboration of the ASIC office (Mengod,
2006). Next subsections describe the main steps to
implement such evaluation.
4.1 Interviewing Users
A first step consisted in the implementation of
interviews with both types of users. This step can be
considered crucial because it enabled the
configuration of the evaluation scenarios and the
teaching materials selected to check them. In
summary, it was unfeasible to implement formal full
courses due to the diversity of student profiles and
teaching disciplines. Nevertheless, the UPV research
team considered that the selected scenarios were
enough to obtain a meaningful sample of different
classroom-based contexts to be evaluated. This set
of learning scenarios allowed researchers to check
the main aspects of the EU4ALL potential to
provide an accessible content support.
4.2 User Experience
The second step in the evaluation procedure was
based on the EU4ALL research questions for testing
the user experience. A first version of the
PoliformaT prototype was developed including the
Preference form displayed on Figure 2 that allows
students to gather their needs and preferences about
the content features, for example to select text or
audio contents.
Once students select a specific course, they can
access the Resources provided by such course
adapted to their stated preferences. Figure 3 shows
part of the resources available for the Marketing
course students. In this case, there is an introductory
video to Marketing concepts based on a lecturer
interview and a transcription of a podcast about a
Marketing on-line topic.
In a similar way, lecturers who were involved in
the project checked the introduction and review of
resources. Figure 4 shows a screenshot that displays
the different options available for a specific resource
and the types of adaptation features which can be
selected. For example, in the case of multimedia
resources these can be based in audio-description or
captioned versions. These information items are then
stored in the Metadata Repository module provided
within the EU4ALL project. Such adaptation data
are used by the Content Personalization module to
offer users those resources that better fit their needs.
Figure 2: Student Preference form.
Figure 3: Resources in a Marketing course.
Figure 4: Edition of Resource attributes.
5 RESULTS
The final evaluation results have been divided in two
parts: the first one is addressed to show the point of
view of students regarding issues such as their
Preference inputs or if they take advantage of the
Resource access in the PoliformaT prototype and the
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
384
second one, to check the point of view of the
lecturers who produced the teaching materials and
stored them as Resources in the adapted PoliformaT.
In this first case, the starter questions were
addressed to gather demographic data about the
students who participated in the evaluation
experience. The average age was about 30 years
(only one person was older than 40 years). Only
30% of students stated that they deployed some kind
of assistive technology such as special headphones
for hearing impairment and text magnification tools
for visual aids. Figure 5 shows a chart that displays
percentages in the answers to questions about the
usage of the Preferences form by students. As
previously mentioned, a numeric scale was used to
assess the user answer from 1-strong disagree (dark
colour) to 5-strong agree (light colour).
Figure 5: Testing Preference form.
Four quality criteria were selected to measure the
required research questions in this case:
Easiness that asked if students had found easy
the process to enter the accessibility preferences.
The results were rather balanced and no specific
valuation was dominant.
Timeliness about the time needed to fulfil the
preferences form by students. Most of them (about
80%) were satisfied in this aspect.
Understandabilty represented the ability of the
students to process the preference information. In
this case, the percentage of indifferent-neutral
opinions and agree was quite similar (50% and 40%,
respectively).
Accuracy that referred if students consider that
the preference options display the required features
in a precise way. Also, the percentage of indifferent-
neutral opinions and agree was similar.
Concerning the lecturer point of view, a
questionnaire was submitted to 6 lecturers in
different disciplines. The main body of the
questionnaire was divided in two sections: the first
one, asking lecturers about their point of view about
the current version of PoliformaT in several issues
related to the process of introducing, authoring,
organizing and managing Resources in the
PoliformaT course. In general, comments of the
lecturers were sceptic about the efficacy of the
Authoring Support service and they realized that a
big effort developing accessible resources and
researching about the generation of accessible
contents was required.
5.1 Discussion
Once the results were collected, mainly through
questionnaires, and a preliminary analysis was
performed, based on the exposed quality criteria,
some issues could be commented. First of all,
although the number of interviewed students was
low and the gathered results could not be conclusive,
their point of view was globally positive and
research questions about User Experience such as
“Are the questions about user needs or preferences
clear?” or “Are these needs appropriately assessed?”
could be easily connected with the
Understandability and Accuracy quality criteria
which were checked in the Student view. In this
case, the average value of Understandability was 3.6
and the Accuracy criteria averaged 3.5 (scale from 1
to 5). The evaluation of research questions related to
the Framework adoption such as “What are the
benefits and drawbacks of the service?” could be
checked by means of the Usefulness and
Performance criteria which obtained average values
of 3.7 and 3.9, respectively. Therefore, a relatively
high benefit can be determined from these values.
The research question about the “integration with
planned/current UPV systems” was only evaluated
in an informal way by contacting with some
manager responsible who reported their interest in
incorporating this Needs Assessment service within
the Info-Accessibility initiative at the UPV (SGAU,
2010). Currently, this Info-Accessibility initiative is
only planned for UPV Library and Employment
services but mainly focused on physical
accessibility.
Another aspect that was informally evaluated
was the lack of official initiatives in the Sakai
context to make accessible this platform. A
preliminary work was presented (Buendia et al,
2011) but more research is required in this topic. As
an additional suggestion, evaluation methodologies
of this kind of accessible e-learning experiences
should be deeply investigated and reviewed.
EVALUATINGANE-LEARNINGEXPERIENCEORIENTEDTOWARDSACCESSIBLEINSTRUCTION
385
6 CONCLUSIONS
The current work has described the evaluation of an
e-learning experience developed in the context of the
EU4ALL project. EU4ALL implemented an open
and extensible architecture of services to provide
accessibility at HE scenarios. The project developed
a general infrastructure, composed by several
standards-based interoperable components such as
Need Assessment or Authoring Support services.
UPV participated in this project as a test partner to
demonstrate the applicability of the EU4ALL
architecture in a medium-size university. This
participation also included the adaptation of the e-
learning platform used at UPV (PoliformaT) to the
EU4ALL requirements.
Once the e-learning platform was adapted, a
group of UPV researchers in collaboration with the
Disability office (CEDAT) at this university
recruited users to check different issues within the
EU4ALL context. Members of this UPV research
team contacted with lecturers in Computing and
Business disciplines who were able to elaborate
different versions of the contents used in their
courses. The CEDAT office has assisted in the
selection of disabled students to test the EU4ALL
services developed in the PoliformaT platform.
The results of the evaluation showed, on the one
hand, the interest of students about the availability of
alternative accessible resources when accessing an
e-learning platform. On the other hand, lecturers
who were in charge of developing such resources
manifested the difficulty and effort to generate
accessible versions of them. Further works will be
oriented towards the research in tools to help
lecturers in the development of accessible contents
and the implementation of new e-learning
experiences.
ACKNOWLEDGEMENTS
This work is supported by the EU4ALL Project
(IST-FP6-034778) and the TEA project (PAID-
UPV/2791).
REFERENCES
Atomic Learning. 2011. Accessible instruction. Online
November 2011: http://www.atomiclearning.com/
allstudents.
Buendia, F., Hervás A., 2008. Evaluating an e-Learning
Experience Based on the Sakai Environment. Lecture
Notes in Business Information Processing, Volume 8,
Part 5, 338-346.
Buendía, F., Roldán, D. Mengod, R., Giménez, T., 2011.
Making accesible virtual learning environments and
their contents. Proceedings of the IADIS International
E-Society Conference, Avila, Spain.
Ejarque E., Buendía, F. Hervás A., 2007. Using a Quality
Framework to Evaluate e-Learning Based
Experiences. Proceedings of the 6th European
Conference on e-Learning, Copenhagen , Denmark.
EU4ALL-UPV, 2010. EU4ALL PoliformaT site. Online
November 2011: https://poliformat.upv.es/portal/site/
EU4ALL/.
EU4ALL, 2011. European unified approach for accessible
lifelong learning (IST-FP6-034778). Online November
2011: http://www.eu4all-project.eu/.
Mandinach, E.B., 2005. The development of effective
evaluation methods for e-learning: A concept paper
and action plan. Teacher College Record, vol. 107, no.
8, 1814-1835.
Mengod, R., 2006. PoliformaT, the Sakai-based on-line
campus for UPV - history of a success. Proceedings of
the 5th Sakai Conference, Vancouver, BC, Canada.
Pawlowski Jan M., 2003. The European Quality
Observatory (EQO): Structuring Quality Approaches
for e-Learning. Proceedings of the 3rd IEEE Int.
Conference on Advanced Learning Technologies.
SGAU, 2010. Sistema de Gestión de Accesibilidad
Universal. Online November 2011: http://www.upv.es/
entidades/SEPQ/.
White, A., 2005. Introducing the Sakai Community.
Proceedings of the 4th Sakai Conference, Austin, TX,
USA.
CSEDU2012-4thInternationalConferenceonComputerSupportedEducation
386
