Development and System Assessment of Learning Object

Recommendation based on Competency

RecOAComp

Patrícia Alejandra Behar

, Ketia Kellen A. da Silva

, Daisy Schneider

, Sílvio César Cazella

Cristina A. W. Torrezzan

and Edimara Heis

Department of Education, Núcleo de Tecnologia Digital aplicada à Educação, UFRGS,

Av. Paulo Gama, 110 - Prédio 12105 - 4° andar sala 401, 90040-060, Porto Alegre, Brazil

Department of Exact Sciense and Social Applied, UFCSPA,

Rua Sarmento Leite, 245, CEP 90050-170, Porto Alegre, Brazil

Keywords: Recommendation Systems, Learning Objects, Competences.

Abstract: This article describes the development and evaluation of a learning objects (LOs) recommendation system

based on competences called RecOAComp. For such, the multidisciplinary team was composed by

educators, programmers and designers. Its purpose is to make recommendations based on the user profile.

Thus, it works by filtering and suggesting LOs that can support the subject in the construction of

competences according to her/his needs. RecOAComp was used in stricto sensu graduate courses on

Education and Informatics in Education between 2011 and 2015, involving more than 150 subjects. The

obtained results derived in system reprogramming, interface improving, student/teacher profile

characterization and usability parameters implementing; besides inserting new competences and LOs in the

databank. Currently, the project is aimed at implementation of collaborative filtering, with which students

evaluate the relevance of the indicated LOs, adding this information to the recommendation. In addition,

RecOAComp will be made available in plug-in format for Distance Education environments. Therefore,

this study aims to provide a LOs recommendation system in different educational modalities, supported on

the needs of each student in order to collaborate with her/his competence-building processes.

1 INTRODUCTION

With Web 2.0, ordinary users can elaborate and

make available content. Thus, space was opened for

the production of materials for education, among

them learning objects (LOs). Defined as all modular

digital resource used to support classroom or

distance learning, they can still be reusable and

approach different media, supporting different

students profiles.

However, when users use search engines for

access to contents they are faced with a great

diversity and quantity of retrieved materials and

information. In the area of Education, this quest

requires a management to avoid excessive work of

the teacher during the selection of relevant materials;

or that the student spend a lot of time until finding a

suitable material to support her/his needs.

Thus, in order to store and organize LOs,

repositories have been created, aiming at the easing

of their selection from the knowledge areas, themes

and types (video, simulation, hypertext etc.). On the

other hand, it is observed that these repositories,

when consulted, still end up returning a lot of

content that is irrelevant to the subject's needs,

causing an overload of information.

This study deals with the construction and

evaluation process of a recommendation system

called RecOAComp (Learning Object

Recommender based on Competences,

Recomendador de Objetos de Aprendizagem

baseado em Competências in Portuguese), which

allows filtering of Learning Objects, according to

the competences that the user needs to build or

rebuild from her/his profile. The intention is to make

available to the academic community in general a

technology that can assist teachers and students in

the competence-building process.

The article is organized as follows: at first the

concept of competences in the context of education

274

Behar, P., Silva, K., Schneider, D., Cazella, S., Torrezzan, C. and Heis, E..

Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp.

In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2015) - Volume 3: KMIS, pages 274-280

ISBN: 978-989-758-158-8

is discussed (section 2). In section 3 takes place a

reflection on the recommendation systems and their

relevance in assisting students and teachers access to

learning objects relevant to competences

construction. Then, in the course of sections 4 and 5,

the prototype in question is described, as well as the

process of elaboration and evaluation of the

recommendation system RecOAComp. In section 6

the analysis of data collected is presented with the

focus on the last validation and evaluation cycle

carried out in 2015. After that, the conclusions

identified up to the present stage of this study are

displayed.

2 COMPETENCES: AN

EDUCATION VIEW

The term ‘competence’ was first used in the legal

field, employment given still today as ‘competence

to judge something’. Its use has been expanded to

the Administrative and Educational areas. In the

latter, its implementation started in professional

education, but was soon included in the educational

reforms in several countries with a perspective,

many times, behaviorist. In the late 1990s and early

2000s, the term was also earning a constructivist

bias, due especially to the works of Perrenoud

(1999; 2002).

The latter theoretical view about competences

has potential to contribute to a comprehensive

student training, as it goes beyond the simple content

memorization practice. It is understood this way,

since its elements are composed of Knowledge, but

also Abilities and Attitudes, abbreviated in the KAA

acronym. They can be related to the four pillars of

the 21

. century Education. (Delors, 1996): learning

to know (knowledge), learning to do (ability),

learning to live with others and learning to be

(attitudes).

When building the KAA, along with its

mobilization in situations according to Le Boterf

(2004), one can put her/his competences into action.

Perrenoud (2004) adds that mobilization is,

especially, adaptation, generalization or

specification for orchestration and coordination of

the elements of the competences. When using a

competence, there is a number of procedures from

the identification of the problem, the means and

resources to solve it, to the evaluation, the making of

adjustments and documentation of actions

(Perrenoud, 1999).

Thus, in front of a scenario in which individuals

are faced constantly with scientific and

technological novelties, new socio-cultural and

economic pathways as well as great content

production, building competences becomes

necessary. In this sense, learning objects can

collaborate in the (re)construction of part of KAA or

of the competence in its entirety as they seek to

work with some of its elements in particular or in

whole. In this regard, in order to assist individuals to

achieve appropriate choices on reliable materials and

according to their needs, recommendation systems

were developed, which will be addressed in the next

section.

3 OBJECTS OF LEARNING AND

RECOMMENDATION

SYSTEMS

Learning objects (LOs) are content modules or units

aiming at learning with support of digital

technologies (IEEE 2002, p.5 apud Coll and

Monereo, 2010, p.252). They have as characteristics

the possibility of being adapted and reused, and

besides being affordable, durable, they can be used

in different platforms (Fabre et al., 2003 apud

Tarouco et al., 2004). Haughey and Muirhead (2005)

note that “[ ... ] LOs have no value or utility out of

teaching contexts, its value lies in its application to

the classroom and online environments where

teachers may or may not be present”.

In order to aggregate these materials in a

common space, repositories of learning objects were

developed, which are databanks that store them, with

the aim to facilitate their access and organization.

Repositories allow indexing of these objects through

metadata filling, or, in other words, a set of

information that characterizes each LO registered,

with the objective to favor its search.

Recommendation Systems (RS) become, thus, a

strong ally to this educational context, as they are

applications that are intended to achieve the

appropriate combination between users expectations

(profile) and items to be recommended, i.e., to

define this interests relationship. According to

Cazella et al., (2010), RS use the repositories of LOs

coupled to users preference data to direct content

with potential interests.

In the educational context, they emerge as a tool

able to relate, more efficiently, educational resources

to students’ training needs. This is due to the types

of filtering providing, therefore, a personalized

recommendation with a reduced number of

Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp

275

irrelevant indications. Thus, it is understood that the

greater the diversity of filterings implemented in the

system, more refined will be its materials screening.

In general, there are seven types of filtering

recommendation systems, being: (1) Collaborative

Filtering; (2) Content-based Filtering; (3)

Demographic Filtering; (4) Knowledge-based

Filtering; (5) Utility-based Filtering; (6) Filtering

based in Other Contexts and (7) Hybrid Filtering. In

the case of RecOAComp, it has as filterings:

Collaborative, Content-based and Hybrid.

Below, RecOAComp prototype is detailed,

which uses the competences as one of the materials

filtering possibilities.

4 RecOAComp PROTOTYPE

The Learning Objects Recommender based on

competences (RecOAComp) is available at

http://www.recomendadorcomp.ufrgs.br/, shows the

The technologies used in prototyping process

were: Java Server Faces - JSF (in the view layer),

Prime Faces, Java Persistence API - JPA with

Hibernate (persistence layer) and MySQL. At this

stage of the research project to which the system is

linked, collaborative filtering is already inserted, but

knowledge-based filterings and filterings with

hybridism variations are being implemented. The

following describes the system operation logic in

educational context.

The teacher of a course first registers her/his

discipline in RecOAComp informing the

competences that may be built by students coursing

it and to what degree (1-5). Then, she/he links those

competences to learning objects (already inserted in

the system or registering new ones), as well as tells

which of these competences they can help to build

and to what degree, from the deepening of their

content and activities.

The student, in her/his turn, when registers

her/himself in the discipline, tells the system the

degree of construction (0-5) which she/he has, given

the competences linked to it. This will be the profile

of the student in the prototype.

After these procedures, a recommendation can be

requested, which is the indication to the student of

learning objects from the crossing between: the

degree of each competency addressed by the

discipline; how much the subject in question has

built each competence of the discipline based on

her/his perception; and the degree of contribution of

the learning object registered by the teacher for the

construction of competences related to it. The goal is

to assist them in the process of creation or

improvement of competences.

If the student understands that she/he built or

improved one or more competences with the use of

the indicated LOs, she/he can return to her/his

profile and change it, which will modify the next

recommendations. Similarly, after the use of

learning objects, students can provide feedback to

the system, evaluating them in a ranking (Likert

scale with 5 points represented visually by a set of

stars) according to their satisfaction with the

recommendation. As the Recommender is being

used and evaluated, recommendations will be

ordained again, using, also, in addition to

competences-based filtering, collaborative filtering

(arising from the evaluation carried out by users on

the relevance of each suggested recommendation).

The RecOAComp prototype was evaluated

through its use in some graduate disciplines. The

experiments are reported in section 5 that follows.

5 RecOAComp

IMPLEMENTATION AND

EVALUATION EXPERIENCES

This research follows an exploratory nature, with a

qualitative and quantitative approach. The

methodology involves the research of literature,

system prototyping and evaluation.

The project began in 2011, when the prototype

was built only with the competences-based filtering.

In this phase, the group performed its first

validation. From then, until the first half of 2015, the

system was used, validated and evaluated by more

than 150 participating subjects, who are students

from different areas of knowledge who attended

stricto sensu graduate disciplines on Education and

Informatics in Education. In the following sections,

this methodological pathway will be detailed, by

presenting the RecOAComp model and the system

validation process.

In order to evaluate the performance of the

RecOAComp recommendation system prototype,

applications were made in two graduate programs

disciplines from 2011 to 2015/1 (convenience

sample).

In the first stage of the research, which began in

2011, the recommender prototype was developed,

then the first system evaluation was promoted. It

took place in a discipline entitled ‘Competences for

Distance Education and the use of learning objects’,

KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing

276

involving 32 students. The registered learning

objects are part of the collection developed by the

research group, with the purpose to make

recommendations viable to users and to carry out the

first tests. Through feedback from students and

teachers who used RecOAComp, the need to create

and enhance features in order to refine and perfect

the prototype was verified.

In the second stage, which began in 2012,

prototype architecture was perfected with Java and

MySQL applications. The LOs evaluation by users

was refined through Likert scale. A tutorial video

was also incorporated into the tool in order to

facilitate its usability. The prototype had new

validation with the reissue of discipline, totaling 29

students participating. New LOs were added to those

already registered, this time with the inclusion of

those made available by repositories that are external

to the research group, allowing greater

diversification. It was found that the

recommendation system attended satisfactorily to

the needs of students, i.e., information filtering was

carried out correctly, showing a LOs base

sufficiently formed already. From the feedback

received, the need to enhance the existing features,

such as user profile and LOs evaluation, was found.

In 2013, characterizing the third step, data to be

filled in the profile, such as KAA detailing on the

form, were improved. Also, a new application was

created so that the user could indicate whether

she/he used or not the recommended object.

Administrator, to qualify prototype management,

and teacher profiles were also developed. The latter

can register disciplines/courses and indicate which

competences she/he wants her/his students to

construct. Furthermore, interface improvement

began, however, the need to improve some usability

issues was understood. A new validation was

performed, through the discipline ‘Pedagogical

Models and Competences in Distance Education’,

with the total of 35 students. In this application, the

students could also register learning objects from

different repositories in RecOAComp, contributing

even more to the diversification of its databank.

In the fourth stage, which took place in 2014,

the system has undergone a makeover in its PHP

databank, which aimed to rewrite the code and

perform refactoring, aiming to prepare it for the

incorporation of new tools. This modification

requested a restructuring of the whole interface

design, project that was initiated at this stage and

continued in 2015. Also, changes were made in the

profile filling form programming, reducing the scope

of KAA degrees from a limit of 10 to 5 points. The

prototype had the validation performed with 30

students of the discipline ‘Competences and

Learning Object Recommendation’, when

registration of learning objects activities and

competences-based recommendations were carried

out. In addition, through an online questionnaire,

with questions relating to operation, interface and

usability, it was possible to assess the contributions

of RecOAComp in the competences-based

recommendation and to identify necessary

improvements to the prototype. With this

information, significant changes and adjustments

were made, particularly regarding the drafting of the

new interface, at that time under development.

These validations, occurred between 2011 and

2014, used the following methodological

procedures:

1) Student's Registration in the System by Filling

in a Form on the Definition of Profiles Related

with the Competences: The questionnaire

involved questions about ‘Teaching Experience

in E-learning’ (options ‘yes’ or ‘no’), assessment

competences based on built or not knowledge,

ability and/or attitudes. Information about

competences that the students believed they had

not developed yet guided the filtering procedure

because it was possible to link the profile of

these students to learning objects that could

assist them in building these still incipient

competences.

2) Registration of Learning Objects: Once their

registration is done, students had to insert objects

into a form based on metadata from a repository

developed at the research participant University.

This insertion characterized a teamwork. When

inserting the selected LO, the team informed the

system its General Category (ID information),

Life Category (creation description), Technical

Category (information to allow use), Educational

Category (educational description), and Rights

Category (use restrictions or not). After that,

students of the groups assessed whether

registered LOs supported (yes or no) the

construction of some specific competences

related to Distance Education. Thus, for each

LO, the group analyzed a series of 14

competences, which are also presented in the

profile. Thus, it was possible to have a good

basis for LOs recommendation.

3) Learning Objects Classification: After the

process of LOs registration and association with

the competences, the groups evaluated the

classification performed between them. This step

Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp

277

was fundamental to identify some small

distortions and to assess the filling process

provided by the prototype.

4) Recommendation Evaluation: The next step

was to start evaluating the use of RecOAComp

recommender. For each LO recommendation

made by the system, feedback was requested on

a Likert scale (a number scale of 5 points), being

the extremes: ‘horrible’ - when the suggested LO

was unrelated to the competences that the student

needed to build (indicated in her/his profile) and

‘excellent’ when the LO indicated was related to

the competences that the student needed to build.

In the fifth stage, which began in 2015 and, thus, in

development process, continuity was given to the

implementation of the new system interface, based

on usability issues and implementation of new

functionalities identified as necessary. A

collaborative system filtering was also implemented.

To date, the user evaluated the relevance of the LO

recommended, but the system did not use these data

in the content recommendation process; they served

only for a system evaluation. With the

implementation of collaborative filtering, the student

tells how relevant the recommended LO was to

her/his needs. The prototype collects this data and

incorporates them in the process of the forthcoming

recommendations, along with content filtering,

which was already used before.

The application of this new version of

RecOAComp was held on the discipline

‘Competences in Distance Education and

Recommendation of Learning Objects’ of the

Graduate Program in Education and Informatics in

Education, with 24 students. Again we used an

online questionnaire, in order to collect data about

the new interface and the operation, as well as the

relevance of the prototype as a whole. Regarding the

methodological pathway used in this RecOAComp

application, there were some changes due to

improvements made in the system. This time the

students, organized in groups, initially inserted

learning objects, informing the KAA of the

competences that each of them enabled to build and

to what degree (0-5). The same was done in relation

to the created discipline. In a second moment,

students individually accessed the system and

requested their enrollment in the disciplines created

by the other groups. Entering the disciplines for the

first time, each student reported how much (0-5)

she/he thought she/he already has developed the

KAA competences addressed by the discipline

through a form provided by the system. Then,

already inserted in the discipline, they requested a

recommendation of learning objects, analyzing if the

recommended contents were relevant to their real

needs, as shown in their profiles. This reevaluation

was conducted in 2015/1, by using the system at the

graduate course already mentioned and attented by

students in master's and doctorate, continuing the

data collection method that had been applied.

In section 6 data collected focusing in the latter

application-evaluation cycle are analyzed.

6 DISCUSSION AND DATA

ANALYSIS

The prototype was used and evaluated by students in

a graduate discipline in 2015/1. To register the

evaluation, we used a questionnaire which was

answered by 11 students. The results of this last

evaluation are described and analyzed below.

The group considered the system as highly

relevant for education (63.6%), with high writing

and editing quality (72.7%), promoting

interdisciplinary use (63.6%). They also concluded

that the RecOAComp, through the recommended

LOs, enables the student to be challenged in

activities that give the opportunity for raising

hypotheses, the interaction, the reflection, the

exchange and the construction of knowledge, with

which they strongly (45.5%) or at least in part

(54.5%) agreed. In the same line, participants

answered that it favors the ability to elaborate and

create knowledge from the action-reflection-action

in 45.5% for both options. At the same time they

considered that it instigates the search for other

information on different sources of research, as

45.5% strongly and 54.5% partially agreed. Thus, it

was possible to observe that the participants

identified the RecOAComp recommendation system

as a possible ally of the teacher and of the students

in the teaching-learning process.

As for usability and design of recommender

interface, they considered it clear and concise, fully

(42.9%) or partially (50%), easy to use, with

information location presented in an intuitive way.

There was little disagreement on the aspects ‘clear

instructions’ (9.1%), ‘interactivity’ (9.1%), ‘easy

and consistent navigation’ (9.1%), ‘well organized

on-screen images’ (18.2 %) and ‘instructions

provided in a clear and objective way’ (18.2%). In

this sense, they point to the need for some

improvements, such as: inclusion of help icons,

updated tutorial provision, statements review and

navigation hierarchy insertion.

KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing

278

In the categories ‘colors adequacy’ and ‘fonts

size and pleasant style’, 9.1% had no opinion; the

remaining respondents agreed fully or in part,

adding 90.9% respectively for both options in these

statements. The same percentage of responses was

identified stating that the system is

‘engaging/motivating’. The topics ‘visually

attractive’, ‘flexible and reusable’, ‘high quality

graphic project (page design)’, ‘enough help

resources and usage tutorial’ had similar percentages

of disagreement and doubt in the positioning of

research participants, which were around 50%.

Regarding the learning objects registered in

RecOAComp, they were evaluated favorably. Thus,

the group of students considered fully or in part that

the recommended LOs presented concepts clearly.

There was disagreement of 9.1% in the categories

‘accurate and current information presentation’ and

‘inclusion of appropriate amount of material’. This

points to the requirement of registration with a

higher number of objects in the system.

Regarding the ‘good use of multimedia resources

(sound, pictures and video)’, some disagreement or

doubt (9.1%) arose. Also, for the ‘good use of

animations and simulations’ 18.2% did not know

how to answer and 9.1% disagreed. Such responses

demonstrate the need for inclusion of new LOs that

include greater interactivity and/or multimedia

resources diversity.

RecOAComp has been constantly improved from

the validations performed. These experiments

indicate its relevance for education supported in

digital resources in classroom or at distance

modalities. The intent, therefore, is that the

RecOAComp recommendation system can support

the building and improvement of competences, in

favor of a comprehensive and quality education.

7 CONCLUSIONS

This article presented the prototype development

process of the educational recommendation system

called RecOAComp, as well as its validations and

evaluations. It allows, through an Educational

Recommendation System (ERS), filtering learning

objects based on competences in order to help

students to build them according to what is

understood as necessary in a discipline.

As future works, we conjecture about the

insertion of a new functionality in the prototype.

This refers to the implementation of knowledge-

based filtering. This filtering technique proposes

specialized knowledge modeling (human specialist)

to assist in LOs recommendation. This modeling

will use domain ontology, refining filtering process

and, therefore, recommendation. For example, if

different teachers name the same competence as

‘Databank Model’ and ‘Databank Modeling’, the

agent will identify these similarities and rewrite the

competence title as "Databank Modeling". This is a

way to integrate content and meaning. Domain

Ontology application aims to work on this

integration problem.

Equally, we aim to elaborate a questionnaire as a

test to be presented to the student when she/he

enrolls in a discipline. This test is important in

helping the teacher to analyze if the KAA of the

students were undersized or oversized by the student

in relation to the degree of competences for the level

of discipline (for example: introductory,

intermediate, advanced).

Finally, it is expected that this work will assist

researches on the subject, given the relevance of the

contributions that recommendation systems can offer

to education.

REFERENCES

Behar, Patricia, 2013. Competências em Educação a

Distância. Porto Alegre: Artmed, 2009.

Cazella, S. C., Silva, K. K. A., Behar, P., Schneider, D.,

Freitas, R., 2012. Recomendando objetos de

aprendizagem baseado em competências em EAD.

Revista Novas Tecnologias na Educação, v. 9, p. 1-10.

Cazella, S. C., Reategui, Eliseo, Machado, M., Barbosa, J.,

2010. Recomendação de Objetos de Aprendizagem

Empregando Filtragem Colaborativa e Competências.

In: Simpósio Brasileiro de Informática na Educação

(SBIE), Florianópolis.

Coll, C. and Monereo, C., 2010. Educação e aprendizagem

no séc XXI: novas ferramentas, novos cenários, novas

finalidades. In: Psicologia da Educação Virtual:

Aprender e Ensinar com as Tecnologias da

Informação e da Comunicação, Edited by Coll, C. e

Monereo, C, Artmed, Porto Alegre.

Delors, J. et al., 1996. Educação um tesouro a descobrir:

Relatório para a UNESCO da Comissão Internacional

sobre Educação para o século XXI, Cortez, 1ª edição.

Haughey, M. and Muirhead, B., 2005. Evaluating learning

objects for schools. E-Journal of Instructional Science

and Technology, 8(1). Retrieved April 13, 2011, from

eval_learnobjects_school.htm

Le Boterf, G., 2004. Construir as competências -

individuais e coletivas. Éditions d'Organization, 3ª

edição.

Nunes, M. A. and Cazella, S. C., 2011. O que sua

personalidade revela? Fidelizando clientes web

através de Sistemas de Recomendação e traços de

personalidade. In: Patrícia Vilain; Valter Roesler.

Development and System Assessment of Learning Object Recommendation based on Competency - RecOAComp

279

(Org.). Tópicos em banco de dados, multimídia e Web.

Tópicos em banco de dados, multimídia e Web. 1ed.

Florianópolis: Sociedade Brasileira de Computação, v.

1, p. 91-122.

Perrenoud, P., 1999. Construir as competências desde a

escola, Artmed, 1ª edição.

Perrenoud, P., 2004. De uma metáfora a outra: transferir

ou mobilizar conhecimentos? In: O enigma da

competência em educação, editado por DOLZ, J. e

OLLAGNIER, E. (Org.), Artmed, Porto Alegre.

Tarouco, L. M. R. et al., 2004. Objetos de Aprendizagem

para M-Learning", In: Anais do Congresso Nacional

de Tecnologia da Informação e Comunicação

(SUCESU), Florianópolis, http://www.cinted.ufrgs.br/

CESTA/objetosdeaprendizagem_sucesu.pdf.

Wiley, D. A., 2000. Learning object design and

sequencing theory. (PhD Tesis). Provo, UT: Brigham

Young University.

KMIS 2015 - 7th International Conference on Knowledge Management and Information Sharing

280