Good and Similar Learners’ Recommendation in Adaptive Learning

Systems

Dade Nurjanah

School of Computing, Telkom University, Bandung, Indonesia

Keywords: Recommender Systems, Collaborative Filtering, Content-based Filtering, Good Learners, Similar Learners,

Adaptation Engine, Adaptive Learning Systems.

Abstract: Classic challenges in adaptive learning systems are about performing adaptive navigation that recommends a

topic or concept to be learned next and learning materials relevant to the topic. Both recommendations have

to meet active learners’ needs. As adaptive navigation problems have been solved using artificial intelligence

techniques, learning material recommendation problems can be solved using recommender techniques that

have been successfully applied to other problems. Until recently there have been a number of techniques that

come with certain advantages and disadvantages. This paper proposes a new technique for recommending

learning materials that combine content-based filtering and collaborative filtering based on the similarity

between learners and learners’ competence. It aims to diminish the drawback of classic collaborative filtering,

which is based on the similarities between learners and does not consider learners’ competence. It also

diminishes problems arising from collaborative filtering based on good learners’ competence, which

potentially produces recommended objects that do not meet the learners’ condition. The results of a recent

experiment show that the proposed technique performs well, as indicated by the MAE score of 0.96 for a

rating scale of 1 to 10.

1 INTRODUCTION

Adaptive learning technologies aim to individualise

learning by taking into account the learner’s needs,

learning styles, preferences, competence, and

learning goals in tailoring content and teaching

strategies. The adaptation is delivered in various

types of adaptive techniques, including adaptive

navigation that selects and recommends concepts to

be learned next by the active learner. The

recommended concepts are the most appropriate

concepts for the learner regarding their

characteristics. A challenge emerges when adaptive

learning systems use existing online materials. The

challenge deals with how to find appropriate learning

materials for the recommended concept. Challenges

occur in matching multi-dimensional learner models

and a large number of learning materials with various

formats (Knutov et al., 2009). Previous studies have

implied that an individual user model should find

appropriate learning objects (Sicilia et al., 2010;

Wang et al., 2007).

As learning tends to be a social process, other

learners’ experiences become important and act as

references. According to social learning theories, a

learner learns better when accompanied by

experienced learners (McLeod, 2007; Vygotsky,

1978). Furthermore, learners learn by observing the

behaviour of others and the outcomes, and they most

likely copy the behaviour if the outcome is positive.

This is supported by another study which reports that

learners can build their knowledge from the help and

support they receive from peers who perform well;

these are called peer helpers (Topping, 2005).

The principles of social learning have been

implemented in recommender systems with

collaborative filtering approaches. A

recommendation for a learner is performed based on

recommendations from peers who share certain

similarities with the learner (Indrayadi and Nurjanah,

2015; Lops et al., 2011; Wang et al., 2007). This

approach is considered to produce appropriate

recommendations since similar learners will like

similar objects. However, in the context of learning,

the recommended objects are probably not the objects

needed to boost the learner’s performance. Hence,

other approaches that consider recommendations

from good learners will be more useful. Learning

material recommendation systems based on good

learners’ recommendations have been studied before

434

Nurjanah, D.

Good and Similar Learners’ Recommendation in Adaptive Learning Systems.

In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016) - Volume 1, pages 434-440

ISBN: 978-989-758-179-3

(Ghauth and Abdullah, 2011). According to precision

and recall scores, it has been proved that

recommendation results are better than

recommendations based on similarities between

learners.

The inclusion of good learners in learning the

process is not a new concept. Topping’s theory on

peer learning spots the importance of peer helpers in

learning (Topping, 2005). Peer helpers are chosen out

of the best learners and they have superior mastery in

a small part of the curriculum. In terms of the

recommender systems of learning materials,

Topping’s theory is implemented by including good

learners’ experiences in the recommendation. It is

different from conventional collaborative filtering,

which considers the similarity between learners in

rating learning materials without considering how

good the peer learners are.

The inclusion of good learners in learning process

is not a new concept. Topping’s theory on peer

learning spots the importance of peer helpers in

learning (Topping, 2005). Peer helpers are chosen

amongst the best learners and they have superior

mastery in a small part of curriculum. In term of

recommender systems of learning materials, the

Topping’s theory is implemented by including good

learners’ experience in recommendation. It is

different from the conventional collaborative filtering

that considers the similarity among learners in rating

learning materials without considering how good the

peer learners.

In this study, we combine collaborative tagging

and filtering and introduce a new approach to

collaborative filtering by applying good learners’

recommendations combined with similar learners’

recommendations. By considering good learners’

recommendations, the recommender will produce

learning materials that meet the learners’ needs. On

the other hand, the conventional method produces

recommendations appropriate to the learners’

characteristics in that learners like the recommended

objects. Hence, we argue that a combination of the

two methods will improve the quality and the

suitability of the recommendations for learners.

The remaining part of this paper is organised as

follows. Section two, on related work, discusses

current studies on recommender systems. Section

three, on recommenders of learning materials,

discusses our proposed framework for recommending

learning materials based on recommendations from

good learners who have been rated as similar to the

active learner. Section four discusses the experiments

and the results. It is followed by section five, which

includes the conclusion and discusses future work.

2 RELATED WORK

The process of recommending learning objects can refer

to previous studies on recommendation systems for

various objects, such as movies, learning materials,

books, goods, et cetera. A recommender system is a tool

that identifies items that are similar to the active user’s

interests. Recommendation systems help users to choose

objects they might find in their interests or that are

useful. The main purpose of recommendation systems is

to choose certain objects that meet the users’

requirements. The quality of the recommendation

depends on the experience of the active user in rating the

objects and the rating patterns the objects have received.

There are two main approaches in recommender

systems: content-based and user-based approaches.

The first approach recommends objects that share

similarities with other objects liked by the active user

in the past (Lops et al., 2011). The key to this approach

is that the objects that might interest the user must be

similar to the objects he or she has liked previously.

Content-based approaches identify new, interesting

items based on the similarities between the features of

the items. Hence, new items share similarities with the

items that the user has previously viewed. It treats the

recommendation problem as a search for related

objects. When a user rates an item, the algorithm

constructs a search query to find other items with

similar keywords or subjects that have been given

similar ratings. Information about objects is stored and

considered in the recommendation process. In previous

studies, content-based approaches have been combined

with the user’s preferences. In learning material

recommender systems, for example, the preferences

could be media, language, or the topic being learned by

the user (Wang et al., 2007).

On the other hand, user-based approaches or

collaborative filtering make recommendations based on

similarities between the active user and other users

(Sicilia et al., 2010). The principle is that users with

similar profiles will like similar objects. The similarities

could be measured according to users’ competence

(Cazella et al., 2010), preferences and rating pattern

(Indrayadi and Nurjanah, 2015; Wang et al., 2007), or

other parameters (Verbert et al., 2012). In terms of the

use of rating in collaborative filtering, users are required

to express their preferences by rating items.

There have been many previous studies on

recommender types of learning systems (Chen et al.,

2005; Lu, 2004; Verbert et al., 2012) proposed a

framework for a recommender system that helped

learners to find learning materials that meet their

needs based on the learners’ abilities. Another study

implementing collaborative filtering for

Good and Similar Learners’ Recommendation in Adaptive Learning Systems

435

recommending learning materials was carried out by

(Soonthornphisaj et al., 2006). This research also

proposed a mash-up technique to aggregate

recommended materials from several websites.

A combination of content-based filtering and

collaborative filtering can be found in (Liang et al.,

2012) who applied knowledge discovery techniques

to perform personalised recommendations for a

courseware selection module. On the other hand, to

compute relevant links for active users (Khribi et al.,

2008) used web mining to process the recent

navigation histories of learners combined with the

similarities and dissimilarities between user

preferences and the learning resources.

A further study exploiting collaborative filtering

and learners’ preferences was proposed by (Wang et

al., 2007). They suggested a personalised

recommendation mechanism based on content and

user similarity to choose learning materials out of a

large number of materials available on the web. They

combined two algorithms, a preference-based

algorithm and a correlation-based algorithm, to rank

the recommended results to advise a learner about the

most suitable learning objects. This model uses a

specific ontology of a certain course to infer objects

required for a learner. The inference is based on his

or her past studying history, which is recorded as the

learner’s personal preference pattern. Another

consideration in selecting learning objects is to refer

to the experiences of similar learners. The similarities

between learners can be inferred from similar values

for certain parameters.

Until recently, improvements for recommender

systems for learning materials have been made by

taking into account learners’ competence. In a

previous study (Tai et al., 2008), learners’

competence was used to retrieve relevant learning

materials from the web. A combination of

collaborative filtering and learners’ competence was

proposed by (Cazella et al., 2010). A learner’s

competence is relative to other learners’ competence

as it is assessed by comparing it to the average of all

the learners’ competence. On the other hand, Ghaut

in (Ghauth and Abdullah, 2011) proposed a

collaborative filtering method based on good

learners’ recommendations, rather than similar

learners’ recommendations.

3 THE FRAMEWORK

The method we propose is for learning material

recommendations. The recommender is part of the

adaptation engine in our proposed adaptive learning

architecture, as described in Figure 1. The difference

between such architecture and the conventional

adaptive learning systems lies in the existence of an

adaptation engine that produces recommendations for

topics or concepts to be learned next and relevant

learning materials.

There are two modules in the adaptation engine:

1. An adaptive navigation engine that decides which

topic or concept the learner will learn next. This

module applies the Bayesian network, but this is

not discussed in this paper.

2. A recommender system for learning materials.

Once the adaptive navigation engine recommends

a concept, the recommender starts working to

choose learning materials relevant to the concept

and the learner. The proposed collaborative

filtering technique is applied in this module.

Figure 1: Architecture of an adaptive learning system.

A domain model is important for the

recommender system as it contains learning concepts,

learning content (materials), tags, and ratings. The

concepts and learning content are designed by

teachers, while the tags and ratings are given by

teachers and learners. In Figure 2, tags are described

as hubs that link learning materials to concepts.

Before the recommendation begins, the teacher

has developed learning concepts and uploaded

learning materials in the domain model. Afterwards,

learners tag and rate the learning materials. All the

tags and rates are recorded in the domain model and

will be used for the recommendation process.

Once the active learner receives the next concept

to be learned, the recommender will run content-

based filtering to find all the learning materials tagged

with the concept. Furthermore, the recommender will

calculate its ratings by good learners and the

similarity between good learners and the active

learner. When a good learner has not rated a learning

material, the recommender will predict the rating

based on the similarity between the material and other

materials that have been rated by the good learner.

CSEDU 2016 - 8th International Conference on Computer Supported Education

436

Figure 2: Domain model.

We define good learners as those who show good

track records in all topics they have learned in the

course and they have mastered a given concept, that

is the concept being learned by the active learning.

The processes to measure and maintain learners’

achievements are parts of student modelling in the

adaptive learning system in which the recommender

resides.

The framework of the proposed recommendation

strategy is described in Figure 3. To conclude, the

better a good learner in mastering a concept and the

more similar she/he is with the active learner in rating

learning materials, the more contribution she/he has

in the recommendation process.

Figure 3: Recommender framework for learning materials

based on similar-good learners’ recommendations.

We define good learners as those who show good

track records in all topics they have learned on the

course and have mastered a given concept, that is, the

concept being learned during active learning. The

processes to measure and maintain learners’

achievements are part of student modelling in the

adaptive learning system in which the recommender

resides.

The framework for the proposed recommendation

strategy is described in Figure 3. To conclude, the

better a good learner is at mastering a concept and the

more similar he or she is to the active learner in rating

learning materials, the more contribution he or she

makes in the recommendation process.

4 THE RECOMMENDATION

MODEL

The proposed technique consists of three steps:

content-based filtering, collaborative filtering, and

recommendation score calculation. The proposed

technique.

4.1 Content-based Filtering

Content-based filtering is aimed at selecting learning

materials that meet the concept being learned by the

active learner. One concept can relate to a number of

learning materials and vice versa, as shown in Figure

2. The hub is created through collaborative tagging by

learners. In the first step, the weight of each tag in the

learning materials is calculated using the following

formula:

i,lm

|C|Max

|C|

|C|Max

|C|

,i,i

(1)

where |C

i,lm

| is the frequency of tag C

on material lm,

which is similar to the number of learners tagging lm

with C, Max|C

| is the maximum frequency among the

tags given to material lm, while Max|C

| is the maximum

frequency of concept C

put as a tag on all the materials.

The first step produces weights for all the tags in all the

learning material and it would be normalised by

comparing w

i,lm

to the maximum of the weights.

In the second step, the relevance scores (RS) of

learning materials are calculated. The higher the RS is,

the more relevant the learning material to the concept

being learned. The relevance of learning material lm is

calculated according to the following formula:

|w||w|

w.w

lmu

(2)

Good and Similar Learners’ Recommendation in Adaptive Learning Systems

437

Variable w

is a weight vector of learner u’s

competence. It is a dynamic vector of the learner

model, which is dynamically updated in adaptive

learning systems. Since this paper focuses on the

recommender, we do not discuss when and how the

vector is updated. Another vector counted in the

relevance score is w

, a weight vector of tags given

to learning material lm. The second step produces L,

a set of materials that have the highest RS. At the end

of this process, a set of learning materials relevant to

the concept being learned by the active learner has

been defined. The kinds and number of tags are the

only parameters to determine the relevance or

irrelevance.

4.2 Collaborative Filtering based on

Recommendation from Similar

Good Learners

This part is the heart of the proposed collaborative

filtering technique. It aims to select good learners and

calculate the similarity scores between good learners

and the active learner based on ratings they have

given to learning materials. Good learners are those

who have mastered the concept being learned by the

active learner and consistently achieve well in all the

concepts they have learned so far. From the previous

parts, we have M, a set of learning materials with the

current learning context. We define G, a set of good

learners selected based on their overall competence

and their competence on the current topic being

learned by the active learner, al.

Once G has been defined, the similarity score

between the active learner and good learners will be

calculated. The similarity between learners is

identified from the ratings they have given to learning

materials. The similarity between the active learner,

al, and a good learner, gl, is calculated using the

following formula:

∑

∈

−−

|M|

1gm,1g

alm,al

|M|

1gm,1galm,al

)rr()rr(

)rr)(rr(

)1g,al(sim

(3)

where

al,mi

is the rating of material mi given by active

learner, al. Furthermore, r

gl,mi

is the rating of material

mi given by good learner, g1. The formula also

applies the average of the ratings given by active

learner, al,

, and the average of the ratings given

by active learner, g1,

. At the end of this module,

the similarity scores between the active learner and

each good learner will have been defined.

4.3 Recommendation Score Calculation

The final stage of the recommendation process is the

calculation of the recommendation scores. The

recommendation score of learning material lm is

calculated by considering the similarity between good

learners and active learners, and the ratings given by

good learners for lm. The formula is described as

follows:

∑

|G|

lm.gi

)g,al(sim

rating*)g,al(sim

(4)

where sim(al,g

) is the similarity score between the

active learner, al, and good learner, g

. On the other

hand, rating

is the rating for the learning material

lm given by the good learner, g

(equation 5). The

value of sim(al,g

) is set at 1 when the active learner

has not rated any learning material. In case a good

learner g has not rated learning material lm, then a

rating prediction will be calculated (equation 6).

|G|

rating

|G|

lm.g

∑

(5)

∑

|N|

|M|

mi,gi

lm,g

)m,lm(sim

rating*)m,lm(sim

(6)

where rating

is the rating of learning material mi

given by good learner g

and sim(lm,m

) is the

similarity score of the learning material lm and m

which is given by:

sim(lm,m

) =

|w||w|

w.w

milm

(7)

At the end of this stage, a set of learning materials

with their recommendation scores has been defined.

5 IMPLEMENTATION AND

TESTING

To test the proposed method, an experiment is

designed. For this experiment, we design 100

documents for 10 topics in a programming course,

including variables and data types, expressions and

assignments, case analysis, functions, recursion,

CSEDU 2016 - 8th International Conference on Computer Supported Education

438

loops, array, searching, sorting, and a matrix in the

forms of slides or short articles. Teachers have given

some tags to the learning materials and afterwards the

participants have been invited to tag and give ratings.

We invite 171 undergraduate students to

participate in the experiment. They are students who

are registering on the programming course or have

completed the course. The experiment consists of

various steps. Firstly, we need to recognise which are

the good learners among them. As we have discussed

previously, good or other types of learners are

recognised in adaptive learning systems by

processing learner models. However, as this research

is focused on the recommender system itself, we

conducted a pre-test covering the aforementioned 10

programming topics, two problems for each topic. At

the end of this activity, each participant has a

competence model that records his or her competence

with respect to the 10 topics examined. The status for

each topic consists of good mastering or 1 if they

could correctly solve both given problems, half

mastering or 0.5 if they correctly solve one of the

given problems, and need to learn, or 0 if none of the

given problems can be solved. Good learners are

those who pass a threshold, for example 0.75, for the

mean score of mastered and half-mastered topics.

Secondly, 51 of the participants are invited to

participate in testing the recommender. They are

requested to rate and tag learning materials for topics

they have mastered or half-mastered. Then, the

recommender testing is carried out for a topic they

have not yet mastered. As we have previously

discussed, the recommender of learning materials, as

part of adaptive learning systems, should receive

input from the adaptive navigation engine in the form

of a topic where the active learner can go next. As we

focus on the recommender, we simulate the input

process with a query asking for a topic input from the

active learner. Then the recommender suggests

learning materials discussing the topic and they have

not been rated by the learner, along with the

predictive scores. The participants are requested to

input ratings for the recommended materials on rating

scale up to 10.

We use the mean absolute error (MAE). MAE

represents the deviation between the predicted ratings

and the user-given ratings.

MAE =

∑

−

|rp|

(8)

The recent experiment produces an MAE score of

0.96 for the rating scale up to 10, which means that

the predicted and user-given ratings are not

significantly different. Compared to the standard for

the MAE score concluded from previous studies,

which is 0.73 for a rating scale up to 5, the proposed

algorithm shows better performance. The MAE score

will possibly change since more students will

participate in the experiment.

6 CONCLUSIONS

In this paper we have discussed our proposed

recommendation technique, which combines content-

based filtering and collaborative filtering, which

considers learners’ competence and similarities in

rating learning materials. The consideration of good

learners’ recommendations is inspired by the

existence of helpers in peer learning. The technique

aims to improve the suitability of the

recommendations with respect to learners’ needs. The

recent experiment to test the proposed technique

results in a low MAE score, 0.96 on rating scale up to

10. In comparison with the standard MAE score from

previous studies, which is 0.73 on a rating scale up to

5, the MAE score of the proposed method is relatively

low. The experiment is now still running so that more

participants can be invited to participate.

Following the results presented in this paper, there

is some work to carry out in the near future. Firstly,

since this recommender is part of adaptive learning

systems, this recommendation technique can be

improved by enhancing the method for identifying

good learners. This could be achieved by enhancing the

number of problems that learners have to solve in each

topic or using different types of problems, whereby

good learners can be more precisely identified.

Secondly, the collaborative filtering can be

extended by considering the similarities between

learners’ characteristics, for instance learners’

competence, in addition to similarities between rating

learning materials. By considering the learning

method, and various dimensions of learners’

characteristics, the recommendations are expected to

more precisely meet learners’ needs.

REFERENCES

Cazella, S. C., Reategui, E. B. & Behar, P., 2010.

Recommendation of Learning Objects Applying

Collaborative Filtering and Competencies. In the

proceeding of IFIP Advances in Information and

Communication Technology, pp. 35-43. Springer.

Chen, C. M., Lee, H. M. & Chen, Y. H., 2005. Personalized

E-Learning System Using Item Repository Theory.

Computers and Education, vol. 44, pp. 237-255.

Good and Similar Learners’ Recommendation in Adaptive Learning Systems

439

Ghauth, K. I. & Abdullah, N. A., 2011. The Effect of

Incorporating Good Learners’ Ratings in e-Learning

Recommender System. Educational Society and

Technology, vol. 14, pp. 248-257.

Indrayadi, F. & Nurjanah, D., 2015. Combining Learner's

Preference and Similar Peers' Experience in Adaptive

Learning. In the proceeding of The 7th International

Conference on Computer-Supported Education, pp.

486-493. ScitePress.

Khribi, M. K., Jemni, M. & Nasraoui, O., 2008. Automatic

Recommendations for E-Learning Personalization

Based on Web Usage Mining Techniques and

Information Retrieval. In the proceeding of Advanced

Learning Technologies, 2008. ICALT '08. Eighth IEEE

International Conference on, pp. 241-245.

Knutov, E., Bra, P. D. & Pechenizkiy, M., 2009. AH 12

Years Later: A Comprehensive Survey of Adaptive

Hypermedia Methods and Techniques. New Review of

Hypermedia and Multimedia vol. 15, pp. 5-38.

Liang, M., Guerra, J. & Brusilovsky, P., 2012. Building

multi-layer social knowledge maps with Google Maps

API. In the proceeding of Workshop on Semantic and

Adaptive Social Web (SASWeb 2012).

Lops, P., Gemmis, M. D. & Semerano, G., 2011. Content-

Based Recommender Systems: State of the Art and

Trends. In Recommender Systems Handbook, pp. 73-

105. Springer.

Lu, J., 2004. A Personalized E-Learning Material

Recommender System. In the proceeding of The 2nd

International Conference on Information Technology

for Application (ICITA 2004), pp. 374-379.

Mcleod, S., 2007. Vygotsky. From http://www.simplypsyc

hology.org/vygotsky.html. Accessed on 16 June, 2012.

Sicilia, M. A., Garcia-Barriocanal, E., Salvador Sanchez

Alonso & Cechinel, C., 2010. Exploring User-Based

Recommender Results in Large Learning Object

Repositories: The Case of MERLOT. In the proceeding

of 1st Workshop on Recommender Systems for

Technology Enhanced Learning (RecSysTEL 2010), pp.

2859–2864.

Soonthornphisaj, N., Rojsattarat, E. & Yim-Ngam, S.,

2006. Smart e-learning using recommender system. In

the proceeding of The 2006 international conference on

Intelligent computing: Part II, pp. 518-523. Springer-

Verlag.

Tai, D. W. S., Wu, H. J. & Li, P. H., 2008. Effective e

learning recommendation system based on self

organizing maps and association mining. The

Electronic Library, vol. 26, pp. 329-344.

Topping, K. J., 2005. Trends in Peer Learning. Educational

Psychology, vol. 25, pp. 631-645.

Verbert, K., Manouselis, N., Ochoa, X., Wolpers, M.,

Drachsler, H., Bosnic, I. & Duval, E., 2012. Context-

Aware Recommender Systems for Learning: A Survey

and Future Challenges. IEEE Transaction on Learning

Technology, vol. 5, pp. 318-335.

Vygotsky, L., 1978. Mind in Society. 1st edition. USA:

Harvard University Press. As cited by Mcleod, S.

(2007). Vygotsky. http://www.simplypsychology.org/v

ygotsky.html. Accessed on June 16, 2012.

Wang, T. I., Tsai, K. H., Lee, M. C. & Chiu, T. K., 2007.

Personalized Learning Objects Recommendation Based

on the Semantic-Aware Discovery and the Learner

Preference Pattern. Educational Technology and

Society, vol. 10, pp. 84-105.

CSEDU 2016 - 8th International Conference on Computer Supported Education

440