Development and System Assessment of Learning Object
Recommendation based on Competency
Patrícia Alejandra Behar
, Ketia Kellen A. da Silva
, Daisy Schneider
, Sílvio César Cazella
Cristina A. W. Torrezzan
and Edimara Heis
Department of Education, Núcleo de Tecnologia Digital aplicada à Educação, UFRGS,
Av. Paulo Gama, 110 - Prédio 12105 - 4° andar sala 401, 90040-060, Porto Alegre, Brazil
Department of Exact Sciense and Social Applied, UFCSPA,
Rua Sarmento Leite, 245, CEP 90050-170, Porto Alegre, Brazil
Keywords: Recommendation Systems, Learning Objects, Competences.
Abstract: This article describes the development and evaluation of a learning objects (LOs) recommendation system
based on competences called RecOAComp. For such, the multidisciplinary team was composed by
educators, programmers and designers. Its purpose is to make recommendations based on the user profile.
Thus, it works by filtering and suggesting LOs that can support the subject in the construction of
competences according to her/his needs. RecOAComp was used in stricto sensu graduate courses on
Education and Informatics in Education between 2011 and 2015, involving more than 150 subjects. The
obtained results derived in system reprogramming, interface improving, student/teacher profile
characterization and usability parameters implementing; besides inserting new competences and LOs in the
databank. Currently, the project is aimed at implementation of collaborative filtering, with which students
evaluate the relevance of the indicated LOs, adding this information to the recommendation. In addition,
RecOAComp will be made available in plug-in format for Distance Education environments. Therefore,
this study aims to provide a LOs recommendation system in different educational modalities, supported on
the needs of each student in order to collaborate with her/his competence-building processes.
With Web 2.0, ordinary users can elaborate and
make available content. Thus, space was opened for
the production of materials for education, among
them learning objects (LOs). Defined as all modular
digital resource used to support classroom or
distance learning, they can still be reusable and
approach different media, supporting different
students profiles.
However, when users use search engines for
access to contents they are faced with a great
diversity and quantity of retrieved materials and
information. In the area of Education, this quest
requires a management to avoid excessive work of
the teacher during the selection of relevant materials;
or that the student spend a lot of time until finding a
suitable material to support her/his needs.
Thus, in order to store and organize LOs,
repositories have been created, aiming at the easing
of their selection from the knowledge areas, themes
and types (video, simulation, hypertext etc.). On the
other hand, it is observed that these repositories,
when consulted, still end up returning a lot of
content that is irrelevant to the subject's needs,
causing an overload of information.
This study deals with the construction and
evaluation process of a recommendation system
called RecOAComp (Learning Object
Recommender based on Competences,
Recomendador de Objetos de Aprendizagem
baseado em Competências in Portuguese), which
allows filtering of Learning Objects, according to
the competences that the user needs to build or
rebuild from her/his profile. The intention is to make
available to the academic community in general a
technology that can assist teachers and students in
the competence-building process.
The article is organized as follows: at first the
concept of competences in the context of education
Behar, P., Silva, K., Schneider, D., Cazella, S., Torrezzan, C. and Heis, E..
Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp.
In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2015) - Volume 3: KMIS, pages 274-280
ISBN: 978-989-758-158-8
2015 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
is discussed (section 2). In section 3 takes place a
reflection on the recommendation systems and their
relevance in assisting students and teachers access to
learning objects relevant to competences
construction. Then, in the course of sections 4 and 5,
the prototype in question is described, as well as the
process of elaboration and evaluation of the
recommendation system RecOAComp. In section 6
the analysis of data collected is presented with the
focus on the last validation and evaluation cycle
carried out in 2015. After that, the conclusions
identified up to the present stage of this study are
The term ‘competence’ was first used in the legal
field, employment given still today as ‘competence
to judge something’. Its use has been expanded to
the Administrative and Educational areas. In the
latter, its implementation started in professional
education, but was soon included in the educational
reforms in several countries with a perspective,
many times, behaviorist. In the late 1990s and early
2000s, the term was also earning a constructivist
bias, due especially to the works of Perrenoud
(1999; 2002).
The latter theoretical view about competences
has potential to contribute to a comprehensive
student training, as it goes beyond the simple content
memorization practice. It is understood this way,
since its elements are composed of Knowledge, but
also Abilities and Attitudes, abbreviated in the KAA
acronym. They can be related to the four pillars of
the 21
. century Education. (Delors, 1996): learning
to know (knowledge), learning to do (ability),
learning to live with others and learning to be
When building the KAA, along with its
mobilization in situations according to Le Boterf
(2004), one can put her/his competences into action.
Perrenoud (2004) adds that mobilization is,
especially, adaptation, generalization or
specification for orchestration and coordination of
the elements of the competences. When using a
competence, there is a number of procedures from
the identification of the problem, the means and
resources to solve it, to the evaluation, the making of
adjustments and documentation of actions
(Perrenoud, 1999).
Thus, in front of a scenario in which individuals
are faced constantly with scientific and
technological novelties, new socio-cultural and
economic pathways as well as great content
production, building competences becomes
necessary. In this sense, learning objects can
collaborate in the (re)construction of part of KAA or
of the competence in its entirety as they seek to
work with some of its elements in particular or in
whole. In this regard, in order to assist individuals to
achieve appropriate choices on reliable materials and
according to their needs, recommendation systems
were developed, which will be addressed in the next
Learning objects (LOs) are content modules or units
aiming at learning with support of digital
technologies (IEEE 2002, p.5 apud Coll and
Monereo, 2010, p.252). They have as characteristics
the possibility of being adapted and reused, and
besides being affordable, durable, they can be used
in different platforms (Fabre et al., 2003 apud
Tarouco et al., 2004). Haughey and Muirhead (2005)
note that “[ ... ] LOs have no value or utility out of
teaching contexts, its value lies in its application to
the classroom and online environments where
teachers may or may not be present”.
In order to aggregate these materials in a
common space, repositories of learning objects were
developed, which are databanks that store them, with
the aim to facilitate their access and organization.
Repositories allow indexing of these objects through
metadata filling, or, in other words, a set of
information that characterizes each LO registered,
with the objective to favor its search.
Recommendation Systems (RS) become, thus, a
strong ally to this educational context, as they are
applications that are intended to achieve the
appropriate combination between users expectations
(profile) and items to be recommended, i.e., to
define this interests relationship. According to
Cazella et al., (2010), RS use the repositories of LOs
coupled to users preference data to direct content
with potential interests.
In the educational context, they emerge as a tool
able to relate, more efficiently, educational resources
to students’ training needs. This is due to the types
of filtering providing, therefore, a personalized
recommendation with a reduced number of
Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp
irrelevant indications. Thus, it is understood that the
greater the diversity of filterings implemented in the
system, more refined will be its materials screening.
In general, there are seven types of filtering
recommendation systems, being: (1) Collaborative
Filtering; (2) Content-based Filtering; (3)
Demographic Filtering; (4) Knowledge-based
Filtering; (5) Utility-based Filtering; (6) Filtering
based in Other Contexts and (7) Hybrid Filtering. In
the case of RecOAComp, it has as filterings:
Collaborative, Content-based and Hybrid.
Below, RecOAComp prototype is detailed,
which uses the competences as one of the materials
filtering possibilities.
The Learning Objects Recommender based on
competences (RecOAComp) is available at, shows the
login screen.
The technologies used in prototyping process
were: Java Server Faces - JSF (in the view layer),
Prime Faces, Java Persistence API - JPA with
Hibernate (persistence layer) and MySQL. At this
stage of the research project to which the system is
linked, collaborative filtering is already inserted, but
knowledge-based filterings and filterings with
hybridism variations are being implemented. The
following describes the system operation logic in
educational context.
The teacher of a course first registers her/his
discipline in RecOAComp informing the
competences that may be built by students coursing
it and to what degree (1-5). Then, she/he links those
competences to learning objects (already inserted in
the system or registering new ones), as well as tells
which of these competences they can help to build
and to what degree, from the deepening of their
content and activities.
The student, in her/his turn, when registers
her/himself in the discipline, tells the system the
degree of construction (0-5) which she/he has, given
the competences linked to it. This will be the profile
of the student in the prototype.
After these procedures, a recommendation can be
requested, which is the indication to the student of
learning objects from the crossing between: the
degree of each competency addressed by the
discipline; how much the subject in question has
built each competence of the discipline based on
her/his perception; and the degree of contribution of
the learning object registered by the teacher for the
construction of competences related to it. The goal is
to assist them in the process of creation or
improvement of competences.
If the student understands that she/he built or
improved one or more competences with the use of
the indicated LOs, she/he can return to her/his
profile and change it, which will modify the next
recommendations. Similarly, after the use of
learning objects, students can provide feedback to
the system, evaluating them in a ranking (Likert
scale with 5 points represented visually by a set of
stars) according to their satisfaction with the
recommendation. As the Recommender is being
used and evaluated, recommendations will be
ordained again, using, also, in addition to
competences-based filtering, collaborative filtering
(arising from the evaluation carried out by users on
the relevance of each suggested recommendation).
The RecOAComp prototype was evaluated
through its use in some graduate disciplines. The
experiments are reported in section 5 that follows.
5 RecOAComp
This research follows an exploratory nature, with a
qualitative and quantitative approach. The
methodology involves the research of literature,
system prototyping and evaluation.
The project began in 2011, when the prototype
was built only with the competences-based filtering.
In this phase, the group performed its first
validation. From then, until the first half of 2015, the
system was used, validated and evaluated by more
than 150 participating subjects, who are students
from different areas of knowledge who attended
stricto sensu graduate disciplines on Education and
Informatics in Education. In the following sections,
this methodological pathway will be detailed, by
presenting the RecOAComp model and the system
validation process.
In order to evaluate the performance of the
RecOAComp recommendation system prototype,
applications were made in two graduate programs
disciplines from 2011 to 2015/1 (convenience
In the first stage of the research, which began in
2011, the recommender prototype was developed,
then the first system evaluation was promoted. It
took place in a discipline entitled ‘Competences for
Distance Education and the use of learning objects’,
KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing
involving 32 students. The registered learning
objects are part of the collection developed by the
research group, with the purpose to make
recommendations viable to users and to carry out the
first tests. Through feedback from students and
teachers who used RecOAComp, the need to create
and enhance features in order to refine and perfect
the prototype was verified.
In the second stage, which began in 2012,
prototype architecture was perfected with Java and
MySQL applications. The LOs evaluation by users
was refined through Likert scale. A tutorial video
was also incorporated into the tool in order to
facilitate its usability. The prototype had new
validation with the reissue of discipline, totaling 29
students participating. New LOs were added to those
already registered, this time with the inclusion of
those made available by repositories that are external
to the research group, allowing greater
diversification. It was found that the
recommendation system attended satisfactorily to
the needs of students, i.e., information filtering was
carried out correctly, showing a LOs base
sufficiently formed already. From the feedback
received, the need to enhance the existing features,
such as user profile and LOs evaluation, was found.
In 2013, characterizing the third step, data to be
filled in the profile, such as KAA detailing on the
form, were improved. Also, a new application was
created so that the user could indicate whether
she/he used or not the recommended object.
Administrator, to qualify prototype management,
and teacher profiles were also developed. The latter
can register disciplines/courses and indicate which
competences she/he wants her/his students to
construct. Furthermore, interface improvement
began, however, the need to improve some usability
issues was understood. A new validation was
performed, through the discipline ‘Pedagogical
Models and Competences in Distance Education’,
with the total of 35 students. In this application, the
students could also register learning objects from
different repositories in RecOAComp, contributing
even more to the diversification of its databank.
In the fourth stage, which took place in 2014,
the system has undergone a makeover in its PHP
databank, which aimed to rewrite the code and
perform refactoring, aiming to prepare it for the
incorporation of new tools. This modification
requested a restructuring of the whole interface
design, project that was initiated at this stage and
continued in 2015. Also, changes were made in the
profile filling form programming, reducing the scope
of KAA degrees from a limit of 10 to 5 points. The
prototype had the validation performed with 30
students of the discipline ‘Competences and
Learning Object Recommendation’, when
registration of learning objects activities and
competences-based recommendations were carried
out. In addition, through an online questionnaire,
with questions relating to operation, interface and
usability, it was possible to assess the contributions
of RecOAComp in the competences-based
recommendation and to identify necessary
improvements to the prototype. With this
information, significant changes and adjustments
were made, particularly regarding the drafting of the
new interface, at that time under development.
These validations, occurred between 2011 and
2014, used the following methodological
1) Student's Registration in the System by Filling
in a Form on the Definition of Profiles Related
with the Competences: The questionnaire
involved questions about ‘Teaching Experience
in E-learning’ (options ‘yes’ or ‘no’), assessment
competences based on built or not knowledge,
ability and/or attitudes. Information about
competences that the students believed they had
not developed yet guided the filtering procedure
because it was possible to link the profile of
these students to learning objects that could
assist them in building these still incipient
2) Registration of Learning Objects: Once their
registration is done, students had to insert objects
into a form based on metadata from a repository
developed at the research participant University.
This insertion characterized a teamwork. When
inserting the selected LO, the team informed the
system its General Category (ID information),
Life Category (creation description), Technical
Category (information to allow use), Educational
Category (educational description), and Rights
Category (use restrictions or not). After that,
students of the groups assessed whether
registered LOs supported (yes or no) the
construction of some specific competences
related to Distance Education. Thus, for each
LO, the group analyzed a series of 14
competences, which are also presented in the
profile. Thus, it was possible to have a good
basis for LOs recommendation.
3) Learning Objects Classification: After the
process of LOs registration and association with
the competences, the groups evaluated the
classification performed between them. This step
Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp
was fundamental to identify some small
distortions and to assess the filling process
provided by the prototype.
4) Recommendation Evaluation: The next step
was to start evaluating the use of RecOAComp
recommender. For each LO recommendation
made by the system, feedback was requested on
a Likert scale (a number scale of 5 points), being
the extremes: ‘horrible’ - when the suggested LO
was unrelated to the competences that the student
needed to build (indicated in her/his profile) and
‘excellent’ when the LO indicated was related to
the competences that the student needed to build.
In the fifth stage, which began in 2015 and, thus, in
development process, continuity was given to the
implementation of the new system interface, based
on usability issues and implementation of new
functionalities identified as necessary. A
collaborative system filtering was also implemented.
To date, the user evaluated the relevance of the LO
recommended, but the system did not use these data
in the content recommendation process; they served
only for a system evaluation. With the
implementation of collaborative filtering, the student
tells how relevant the recommended LO was to
her/his needs. The prototype collects this data and
incorporates them in the process of the forthcoming
recommendations, along with content filtering,
which was already used before.
The application of this new version of
RecOAComp was held on the discipline
‘Competences in Distance Education and
Recommendation of Learning Objects’ of the
Graduate Program in Education and Informatics in
Education, with 24 students. Again we used an
online questionnaire, in order to collect data about
the new interface and the operation, as well as the
relevance of the prototype as a whole. Regarding the
methodological pathway used in this RecOAComp
application, there were some changes due to
improvements made in the system. This time the
students, organized in groups, initially inserted
learning objects, informing the KAA of the
competences that each of them enabled to build and
to what degree (0-5). The same was done in relation
to the created discipline. In a second moment,
students individually accessed the system and
requested their enrollment in the disciplines created
by the other groups. Entering the disciplines for the
first time, each student reported how much (0-5)
she/he thought she/he already has developed the
KAA competences addressed by the discipline
through a form provided by the system. Then,
already inserted in the discipline, they requested a
recommendation of learning objects, analyzing if the
recommended contents were relevant to their real
needs, as shown in their profiles. This reevaluation
was conducted in 2015/1, by using the system at the
graduate course already mentioned and attented by
students in master's and doctorate, continuing the
data collection method that had been applied.
In section 6 data collected focusing in the latter
application-evaluation cycle are analyzed.
The prototype was used and evaluated by students in
a graduate discipline in 2015/1. To register the
evaluation, we used a questionnaire which was
answered by 11 students. The results of this last
evaluation are described and analyzed below.
The group considered the system as highly
relevant for education (63.6%), with high writing
and editing quality (72.7%), promoting
interdisciplinary use (63.6%). They also concluded
that the RecOAComp, through the recommended
LOs, enables the student to be challenged in
activities that give the opportunity for raising
hypotheses, the interaction, the reflection, the
exchange and the construction of knowledge, with
which they strongly (45.5%) or at least in part
(54.5%) agreed. In the same line, participants
answered that it favors the ability to elaborate and
create knowledge from the action-reflection-action
in 45.5% for both options. At the same time they
considered that it instigates the search for other
information on different sources of research, as
45.5% strongly and 54.5% partially agreed. Thus, it
was possible to observe that the participants
identified the RecOAComp recommendation system
as a possible ally of the teacher and of the students
in the teaching-learning process.
As for usability and design of recommender
interface, they considered it clear and concise, fully
(42.9%) or partially (50%), easy to use, with
information location presented in an intuitive way.
There was little disagreement on the aspects ‘clear
instructions’ (9.1%), ‘interactivity’ (9.1%), ‘easy
and consistent navigation’ (9.1%), ‘well organized
on-screen images’ (18.2 %) and ‘instructions
provided in a clear and objective way’ (18.2%). In
this sense, they point to the need for some
improvements, such as: inclusion of help icons,
updated tutorial provision, statements review and
navigation hierarchy insertion.
KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing
In the categories ‘colors adequacy’ and ‘fonts
size and pleasant style’, 9.1% had no opinion; the
remaining respondents agreed fully or in part,
adding 90.9% respectively for both options in these
statements. The same percentage of responses was
identified stating that the system is
‘engaging/motivating’. The topics ‘visually
attractive’, ‘flexible and reusable’, ‘high quality
graphic project (page design)’, ‘enough help
resources and usage tutorial’ had similar percentages
of disagreement and doubt in the positioning of
research participants, which were around 50%.
Regarding the learning objects registered in
RecOAComp, they were evaluated favorably. Thus,
the group of students considered fully or in part that
the recommended LOs presented concepts clearly.
There was disagreement of 9.1% in the categories
‘accurate and current information presentation’ and
‘inclusion of appropriate amount of material’. This
points to the requirement of registration with a
higher number of objects in the system.
Regarding the ‘good use of multimedia resources
(sound, pictures and video)’, some disagreement or
doubt (9.1%) arose. Also, for the ‘good use of
animations and simulations’ 18.2% did not know
how to answer and 9.1% disagreed. Such responses
demonstrate the need for inclusion of new LOs that
include greater interactivity and/or multimedia
resources diversity.
RecOAComp has been constantly improved from
the validations performed. These experiments
indicate its relevance for education supported in
digital resources in classroom or at distance
modalities. The intent, therefore, is that the
RecOAComp recommendation system can support
the building and improvement of competences, in
favor of a comprehensive and quality education.
This article presented the prototype development
process of the educational recommendation system
called RecOAComp, as well as its validations and
evaluations. It allows, through an Educational
Recommendation System (ERS), filtering learning
objects based on competences in order to help
students to build them according to what is
understood as necessary in a discipline.
As future works, we conjecture about the
insertion of a new functionality in the prototype.
This refers to the implementation of knowledge-
based filtering. This filtering technique proposes
specialized knowledge modeling (human specialist)
to assist in LOs recommendation. This modeling
will use domain ontology, refining filtering process
and, therefore, recommendation. For example, if
different teachers name the same competence as
‘Databank Model’ and ‘Databank Modeling’, the
agent will identify these similarities and rewrite the
competence title as "Databank Modeling". This is a
way to integrate content and meaning. Domain
Ontology application aims to work on this
integration problem.
Equally, we aim to elaborate a questionnaire as a
test to be presented to the student when she/he
enrolls in a discipline. This test is important in
helping the teacher to analyze if the KAA of the
students were undersized or oversized by the student
in relation to the degree of competences for the level
of discipline (for example: introductory,
intermediate, advanced).
Finally, it is expected that this work will assist
researches on the subject, given the relevance of the
contributions that recommendation systems can offer
to education.
Behar, Patricia, 2013. Competências em Educação a
Distância. Porto Alegre: Artmed, 2009.
Cazella, S. C., Silva, K. K. A., Behar, P., Schneider, D.,
Freitas, R., 2012. Recomendando objetos de
aprendizagem baseado em competências em EAD.
Revista Novas Tecnologias na Educação, v. 9, p. 1-10.
Cazella, S. C., Reategui, Eliseo, Machado, M., Barbosa, J.,
2010. Recomendação de Objetos de Aprendizagem
Empregando Filtragem Colaborativa e Competências.
In: Simpósio Brasileiro de Informática na Educação
(SBIE), Florianópolis.
Coll, C. and Monereo, C., 2010. Educação e aprendizagem
no séc XXI: novas ferramentas, novos cenários, novas
finalidades. In: Psicologia da Educação Virtual:
Aprender e Ensinar com as Tecnologias da
Informação e da Comunicação, Edited by Coll, C. e
Monereo, C, Artmed, Porto Alegre.
Delors, J. et al., 1996. Educação um tesouro a descobrir:
Relatório para a UNESCO da Comissão Internacional
sobre Educação para o século XXI, Cortez, 1ª edição.
Haughey, M. and Muirhead, B., 2005. Evaluating learning
objects for schools. E-Journal of Instructional Science
and Technology, 8(1). Retrieved April 13, 2011, from
Le Boterf, G., 2004. Construir as competências -
individuais e coletivas. Éditions d'Organization, 3ª
Nunes, M. A. and Cazella, S. C., 2011. O que sua
personalidade revela? Fidelizando clientes web
através de Sistemas de Recomendação e traços de
personalidade. In: Patrícia Vilain; Valter Roesler.
Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp
(Org.). Tópicos em banco de dados, multimídia e Web.
Tópicos em banco de dados, multimídia e Web. 1ed.
Florianópolis: Sociedade Brasileira de Computação, v.
1, p. 91-122.
Perrenoud, P., 1999. Construir as competências desde a
escola, Artmed, 1ª edição.
Perrenoud, P., 2004. De uma metáfora a outra: transferir
ou mobilizar conhecimentos? In: O enigma da
competência em educação, editado por DOLZ, J. e
OLLAGNIER, E. (Org.), Artmed, Porto Alegre.
Tarouco, L. M. R. et al., 2004. Objetos de Aprendizagem
para M-Learning", In: Anais do Congresso Nacional
de Tecnologia da Informação e Comunicação
(SUCESU), Florianópolis,
Wiley, D. A., 2000. Learning object design and
sequencing theory. (PhD Tesis). Provo, UT: Brigham
Young University.
KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing