Labour Market Information Driven, Personalized, OER

Recommendation System for Lifelong Learners

Mohammadreza Tavakoli

1 a

, Stefan T. Mol

2 b

and G

abor Kismih

1 c

TIB Leibniz Information Centre for Science and Technology, Hannover, Germany

University of Amsterdam, Amsterdam, The Netherlands

Keywords:

Lifelong Learning, Open Education Resources, Recommender Systems, Labour Market Intelligence, Machine

Learning, Text Mining.

Abstract:

In this paper, we suggest a novel method to aid lifelong learners to access relevant OER based learning content

to master skills demanded on the labour market. Our software prototype 1) applies Text Classiﬁcation and Text

Mining methods on vacancy announcements to decompose jobs into meaningful skills components, which

lifelong learners should target; and 2) creates a hybrid OER Recommender System to suggest personalized

learning content for learners to progress towards their skill targets. For the ﬁrst evaluation of this prototype we

focused on two job areas: Data Scientist, and Mechanical Engineer. We applied our skill extractor approach

and provided OER recommendations for learners targeting these jobs. We conducted in-depth, semi-structured

interviews with 12 subject matter experts to learn how our prototype performs in terms of its objectives,

logic, and contribution to learning. More than 150 recommendations were generated, and 76.9% of these

recommendations were treated as useful by the interviewees. Interviews revealed that a personalized OER

recommender system, based on skills demanded by labour market, has the potential to improve the learning

experience of lifelong learners.

1 INTRODUCTION

The worlds of work and employment are changing

rapidly in our post-industrial societies. As a con-

sequence, matching processes between skill demand

and supply are getting more and more complicated as

skills dynamically evolve through an uncontrollable

process (Colombo et al., 2018; Castello et al., 2014).

These dramatic changes lead to a number of educa-

tional problems in relation to the gap between (dy-

namic) skills that job markets demand and the train-

ing that education programs offer (Smith and Ali,

2014; Wowczko, 2015; McGill, 2009). Furthermore,

being up to date about actual job market skills has

signiﬁcant importance for individuals to remain em-

ployed or climb workplace hierarchy during active

times of employment (Colombo et al., 2018; Kho-

breh et al., 2015). Notably, in order to mitigate mis-

matches between education and labour markets, we

need to 1) understand the dynamic nature of labour

https://orcid.org/0000-0002-7368-0794

https://orcid.org/0000-0002-9375-3516

https://orcid.org/0000-0003-3758-5455

markets, which requires the deconstruction of jobs

into required skills, and 2) match those skills to rel-

evant learning content.

In order to tackle the ﬁrst problem, governments

and international organizations have created a number

of occupational and skill taxonomies to provide struc-

ture for job-seekers and employers about skill compo-

nents of jobs (e.g. ESCO, ISCO, O*NET). However,

there are obstacles limiting the usefulness of these

taxonomies, such as keeping their information up-

dated (Djumalieva and Sleeman, 2018). At the same

time, researchers attempt to build ontologies to pro-

vide accurate representations of jobs and skills (e.g.

(Sibarani et al., 2017)), and machine learning models

to capture information from rich, text based, labour

market data sources, like job vacancy announcements

(Colace et al., 2019; Boselli et al., 2018b; Boselli

et al., 2018a; Kobayashi et al., 2018).

To address the second issue, educational services

should be tailored to the needs of individual lifelong

learners. In this respect, open education become a key

facilitator in many areas, including personal skills de-

velopment (Kanwar and Mishra, 2018). Open Edu-

cational Resources (OERs) are also gaining popular-

Tavakoli, M., Mol, S. and Kismihók, G.

Labour Market Information Driven, Personalized, OER Recommendation System for Lifelong Learners.

DOI: 10.5220/0009420300960104

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 2, pages 96-104

ISBN: 978-989-758-417-6

ity as content sources for open education (Ha et al.,

2011). Major OER repositories have large amount

of regularly updated learning content in wide range

of content areas. Therefore it is surprising that de-

spite their growing capacity, OER platforms still un-

der perform when offering personalised learning ser-

vices. As an example, OER users must consult and

search through several OER repositories (with dif-

ferent interfaces) manually in order to ﬁnd appropri-

ate learning content. Only few, initial efforts are re-

ported, which attempt to build OER recommendation

algorithms. These are done by collecting properties

of users and OERs using various approaches such as

building (or reusing existing) ontologies (Wan and

Niu, 2018; Sun et al., 2017), conducting user behav-

ior analysis in social networks (Lopez-Vargas et al.,

2014), or applying Text Mining techniques to identify

similar OER-Documents (Dufﬁn et al., 2007). Never-

theless, due to the lack of personalized services like

high quality search and recommendation, the popu-

larity of OERs has been limited in most user groups

(typically educators or lifelong learners) (Sun et al.,

2018; Ruiz-Iniesta et al., 2014; Chicaiza et al., 2015;

Ha et al., 2011; Chicaiza et al., 2017).

In this paper we address the above mentioned

challenges and report on the prototype building of a

personalized OER recommender system, which helps

lifelong learners 1) to be informed about necessary

skills required by their current or future jobs and 2)

recommend them OERs to facilitate their progress to-

wards mastering those skills. In this paper, after de-

picting the current state of the art, we reveal the meth-

ods and data we used to build up our skill classiﬁer

and OER recommender algorithms. Subsequently, we

showcase the validation of our ﬁrst prototype for two

jobs (Data Scientist and Mechanical Engineer), using

semi-structured interviews with domain experts. At

the end of the paper we conclude our experiences and

suggest further research directions.

2 STATE OF THE ART

2.1 Matching between Jobs and Skills

Having access to reliable labour market informa-

tion on skills and jobs is not easy. Currently, only

several governments or inter-governmental organiza-

tions (the most prominent actors are the US Govern-

ment, European Commission or Singapore) attempt to

build skill inventories and occupational taxonomies

(such as ESCO, ISCO or O*NET). Although these

taxonomy building efforts have created a stable ba-

sis for basic skill analytics (inter-skill relationships,

high level matching to competences and occupations),

most of these resources are created and maintained

by human experts in several time-consuming steps,

which makes them expensive and also susceptible to

out-dating (Djumalieva and Sleeman, 2018). It is

therefore not surprising that more and more commer-

cial and research attempts target new ways to obtain

real-time labour market information about skills, us-

ing and analysing alternative data sets like job va-

cancy announcement text, resume text, or social me-

dia data. These attempts can be clustered into the fol-

lowing three main categories:

2.1.1 Semantic-based Methods

This approach builds on ontologies to reveal and

organise components of jobs (e.g. skills, tasks)

(Sibarani et al., 2017; Castello et al., 2014; Khobreh

et al., 2015). These methods provide meaningful in-

formation for stakeholders (i.e. structure of existing

jobs, skills and their relationships), however, their dy-

namicity is limited, since building and maintaining

ontologies to cover a wide range of occupations and

skills, are currently done manually (by subject matter

experts), which is a very costly and time-consuming

exercise (Hepp, 2007).

2.1.2 Text Mining and Machine Learning

Methods

A number of studies analyze online vacancy an-

nouncements to classify job components (e.g. skills,

tasks) according to existing, static taxonomies (e.g.

ESCO). This is done to update taxonomies and pro-

vide fresh information about labour markets. Most of

these papers try to extract features from the vacancies

by applying embedding techniques (e.g. word2vec

and doc2vec)(Colombo et al., 2018), Topic Modeling

techniques (e.g. LDA) (Colace et al., 2019; Colombo

et al., 2018), TFIDF (Karakatsanis et al., 2017) and

use classiﬁcation techniques such as Logistic Regres-

sion, SVM, and Random Forest (Boselli et al., 2018b;

Boselli et al., 2018a) or calculate distance (Karakat-

sanis et al., 2017) to assign job vacancies to the their

closest job class. Furthermore, a number of papers are

focusing on using Text Mining and clustering tech-

niques to ﬁnd relationships between skills and jobs,

and to calculate similarity measures (Djumalieva and

Sleeman, 2018; Wowczko, 2015). These papers build

vectors for skills using embedding techniques, Bag of

Words (Djumalieva and Sleeman, 2018), and apply

clustering techniques such as K-Means (Djumalieva

and Sleeman, 2018) to ﬁnd the structure of related

skills and jobs. Contrary to the Ontology-based sys-

tems, given that a powerful model is constructed,

Labour Market Information Driven, Personalized, OER Recommendation System for Lifelong Learners

these methods can automatically extract the required

information form job vacancies. However, the identi-

ﬁcation of such general models remains challenging.

2.1.3 Content Analysis

Several papers focus on speciﬁc job areas, and collect

related job vacancies from various sources (e.g. job

boards, newspapers). Subsequently, they apply con-

tent analysis techniques such as counting the number

of skills occurrence and skills co-occurrence in order

to provide insights about skills in the investigated job

area (Verma et al., 2019; Gardiner et al., 2018; Maer-

Matei et al., 2019). Although, these methods are suc-

cessful when ﬁnding and identifying required skills

in a given job area, in most cases, they cannot scale.

The reason is that mostly these studies use static lists

of jobs and skills in their focus areas, which results in

a ”tunnel vision” and fail to detect new, emerging job

components.

2.2 OER Recommendation

The area of OER recommendation systems has enor-

mous development potential. The available literature

on OER based content recommendations to learners

is currently limited (Chicaiza et al., 2017) and there is

no signal that factors related to typical lifelong learn-

ing goals (skills, jobs) play any role here. To structure

recent developments, we clustered available studies

into the following four categories:

2.2.1 Heuristic Method

(Sun et al., 2018) examines the Cold Start problem

(Lam et al., 2008) in the case of new micro OERs.

The paper deﬁnes rules, based on recommended se-

quences of learning objects (e.g. some learning ob-

jects should be learnt before others) using an existing

ontology and calculates a Violation Degree accord-

ing to the rules. The more a learning path violates

the rules, the higher the Validation Degree is. Subse-

quently, the system recommends and adds new OERs

into users’ learning paths, based on minimizing the

violation degree.

2.2.2 Semantic and Ontology based Methods

(Wan and Niu, 2018) builds an ontology for learners,

learning objects, and their environments to establish

similarity measures between learning objects. This is

done in order to update learning objects’ properties

and provide diverse and adaptive recommendations.

Some studies make use of ontologies and open source

RDF data to leverage semantic content, and deﬁne

recommendation algorithms suitable for linked data

(Chicaiza et al., 2017; Ruiz-Iniesta et al., 2014; Sun

et al., 2017). Moreover, (Chicaiza et al., 2015) tries

to deﬁne an open linked vocabulary to describe user

proﬁles, in order to facilitate recommendations.

2.2.3 Social Network Analysis

(Lopez-Vargas et al., 2014) uses social networks to

build graphs of OERs and learners. Therefore, it

ﬁnds tweets which have valid urls, and builds a

graph, based on the co-occurrences of tweets’ hash-

tags. They also build a similar graph with users, based

on their mentions and retweet. Finally, they recognize

important and inﬂuential hashtags, and use density

and centrality measures from the graphs to provide

recommendations.

2.2.4 Machine Learning

(Sun et al., 2017) attempts to classify users (and

their demographic features) with the help of Deci-

sion Trees and Naive Bayes algorithms to recommend

them OERs. Furthermore, (Dufﬁn et al., 2007) uses

Document Clustering and LSA in order to ﬁnd simi-

lar OERs and use them for recommendations.

2.3 Research Question

Based on the state of the art, it is clear that 1) it is

worthwhile and timely to consider labour market in-

formation to deﬁne learning goals; 2) Efforts to de-

compose jobs into components suitable for educa-

tional purposes are still in their infancy, and 3) the

area of OER recommendation systems is an under-

researched area, with a number of challenges from a

technical (e.g. available algorithms, data integration,

scalability) perspective. For these reasons the main

research questions and objectives of this paper are:

• Empower lifelong learners to construct their own

learning trajectories on the basis of labour market

information and OER based learning content

• Create and evaluate a hybrid OER recommenda-

tion system prototype, relying on labour market

information, learner and OER properties

• Create an algorithm to decompose jobs into

unique skills and translate those skills into learn-

ing objectives

• Develop algorithms to match skills (learning ob-

jectives) with learning content available in OERs

on the basis of learner and OER properties

• Conduct an initial evaluation of our hybrid OER

recommendation system prototype against the

CSEDU 2020 - 12th International Conference on Computer Supported Education

general project objectives, the applied logic, and

its potential contribution to lifelong learning.

In general, with this work we expect to advance the

potential of OERs to handle the increasing need for

learning content and instruction (Ha et al., 2011),

through personalized services for learners, based on

labor market data.

3 METHODS

In this section, we detail the data and methods we

used to identify required skills and their importance

levels for jobs, followed by our OER recommendation

algorithms. Finally, we illustrate our prototype sys-

tem, which provides personalized OER recommenda-

tions to learners based on individual skill targets.

3.1 Data Collection

For the prototyping, we used a crawled sample data-

set from Monster.com containing 22,000 job vacan-

cies

. We used 80% of our dataset for training and

cross validation and 20% of them as our test set.

Moreover, for our OER recommendation, we have

used APIs, provided by the following OER providers:

SkillsCommons

and Wisc-Online

3.2 Labour Market Intelligence (LMI)

3.2.1 Extracting Skills from Job Vacancies

Since our aim was to avoid any dependency on exist-

ing taxonomies (which are updated slowly), we put

existing methods classifying jobs and skills into pre-

deﬁned classes aside, and created a dynamic job-skill

matching mechanism to detect skill changes in jobs

quickly. As the ﬁrst step, we constructed a model to

ﬁnd skill related sentences in job vacancies. After an

exploratory analysis, we concluded that large num-

ber of vacancies do not contain a ”Required Skills”

section. Therefore, in order to build our model, we

selected vacancies with an explicit ”Required Skills”

section and run the following preprocessing proce-

dure on each of those vacancies:

• Deletion of unimportant characters, punctuations

and bullet-points

• Removal of irrelevant stop words

The data-set is accessible from: https://www.kaggle.com/

PromptCloudHQ/us-jobs-on-monstercom

https://www.skillscommons.org/

https://www.wisc-online.com/

• Removal of conjunctions, articles, and preposi-

tions

• Sentence Tokenization

• Lowercase Conversion

• Lematization

Altogether we obtained more than 60,000 sentences

with this method. This corpus included both sen-

tences, which were mentioned in a ”Required Skills”

section (we set their label to 1), and also sentences

mentioned in other sections in vacancies (we set their

label to 0). As a result, we got around 15,000 sen-

tences related to ”Required Skills” and labelled as 1

and around 45,000 sentences not related to ”Required

Skills” and labelled as 0. Subsequently, we applied

embedding techniques on word-level n-grams, and

built sentence vectors with averaging word/n-gram

embeddings and using Multinomial Logistic Regres-

sion model to minimize the classiﬁcation error

. It

should be mentioned that word-level n-gram applies

the n-gram concept on character level and ﬁnd the

most common sequences of characters. Therefore,

vectors are created for each of the extracted sequence

of characters and it helps us build vectors for new

words (skills), based on our existing vector for the

new word’s sequences of characters (e.g. building

an initial vector for Mechatronic based on existing

vectors which are extracted from Elecronic and Me-

chanic). Applying our model on the test data-set

resulted in the detection of 88.7% balance-accuracy

(including precision and recall) of skill-related sen-

tences. Finally, we used TFIDF weighting to detect

skill terms in skill related sentences. It should be men-

tioned that we used Minimum Document Frequency of

3 as cut-off point in order to handle typing errors and

remove rare words.

3.2.2 Calculating Skills’ Importance for Jobs

To calculate the importance of particular skills associ-

ated with jobs in a speciﬁc geographical location, we

calculated the rate of skill occurrence in the previous

6 months at the given job location. After normalizing

the rates, we use a simple decay function to compute

the new importance score, which combines the pre-

vious importance scores and the new rates with more

weight on the new rates.

Labour Market Information Driven, Personalized, OER Recommendation System for Lifelong Learners

Table 1: User Properties.

Property Values Note

Selected Job

Existing

Jobs

selected

by users

Skills-Levels

[0..100]

for

Skills

determined

by users

Personal

Information

Location

Gender

Education

entered

by users

Pref Resources

[0..100] for

Resources

higher tendency

→higher value

Pref Length [0..100]

preferred long

and

preferred short

Pref Check [0..100]

prefer assured

→closer to 100

Pref Accessibility [0..100]

prefer higher

accessibility

→closer to 100

3.3 Recommending OERs

3.3.1 Method for Initializing Learners’

Properties

Table 1 depicts learners’ properties in our OER rec-

ommender prototype. During the initialization of a

new user, we capture known properties entered by

users (i.e. Personal Information, Skill Level List, and

Selected Job), and also a number of properties with-

out values (i.e. Resource scores, Length scores, Qual-

ity scores, and Accessibility scores). To set an ini-

tial value for these unknown properties, we sample

similar users, based on the known properties and use

weighted average (based on similarity) of their prop-

erties as initial values for unknown properties. This

strategy scaffolds the cold start problem of new users.

To sample similar users, we use (1) to compute the

similarity between user i and j where the Similarity

Effect function for user i and j in property k is calcu-

lated as (2).

similarity(i, j) =

∑

k=known properties

sim e f f ect(i, j, k)

100

(1)

sim e f f ect(i, j, k) =

(

equal val(k), same k for i&j

0, otherwise

(2)

Furthermore, the equality value of property k

(equal val(k)), showing the effect of variable k on

We used FastText Library in Python for our classiﬁcation

task (Joulin et al., 2016)

similar behaviour (rating) by users, is calculated

through the following process:

1. We collect user pairs who gave exactly the same

ratings for the same OER in the period

2. Compute the ratio of the number of pairs having

exactly the same value in property k to the number

of all pairs

3. Normalizing the ratios in a way that sum of all the

ratios becomes equal to 100 and the normalized

ratio of k is the Equality Value of k

This process is executed regularly, after deﬁning a

time period (e.g. after every month).

3.3.2 Method for Updating Learners’ Properties

Since we aim to capture learners’ preferences quickly

and provide relevant OERs according to the changes

and improvements in learners’ property values, we

decided to update user properties after each rating ac-

tion on any of the recommended OERs. This is done

by using a real-time updating process that, according

to the rating score and the properties of the recom-

mended OERs (i.e. length, quality, accessibility), up-

dates the properties of the users. As a consequence,

if a learner is satisﬁed (dissatisﬁed) by a given OER,

we will encourage (discourage) the properties (see de-

tails in the next section) of that particular OER for

that learner. For instance, if a user is dissatisﬁed by a

long OER (e.g. it takes 10 week to complete), we will

update the Preferred Long property of the user and

decrease its value in order to provide shorter OERs

in the future. Along the same line, with assigning

positive ratings to accessible OERs, learners can en-

hance their accessibility criterion and increase their

Preferred Accessibility value to receive content with

accessibility support (critical for instance for visually

impaired learners (Elias et al., 2017)).

3.3.3 OER Properties

Table 2 shows OER properties. Based on existing lit-

erature, we selected Level, Length, Quality, and Ac-

cessibility as important properties of OERs (Piedra

et al., 2015; Atenas and Havemann, 2013; Elias et al.,

2018). When assigning a value to a particular OER

property, ﬁrst we extract and order all existing val-

ues assigned to that property, then classify them, and

count the number of classes. Based on the number of

classes, we assign a value between 0 and 100 to that

property. For instance, we take property Level, we ex-

tract 3 values (beginner, intermediate and advanced -

3 classes), and as a result we set the value for beginner

OERs to 0, intermediate OERs to 50, and advanced

OERs to 100.

CSEDU 2020 - 12th International Conference on Computer Supported Education

100

Table 2: OER Properties.

Property Values Note

Resource Repositories

e.g. SkillCommons,

Wisc-Online

Skill

Existing

Skills based on subjects

Author Full Name the provider

URL URL

web address

of OERs

Length [0..100]

how long

and

how short

Level [0..100]

higher level

→closer to 100

Quality [0..100]

more quality

assurance

→closer to 100

Accessibility [0..100]

more accessibility

→closer to 100

Relevance [0..100]

decreased if

deﬁned Irrelevant

3.3.4 Method for Initializing OER Properties

For each OER, we attempt to identify similar OERs,

based on its known properties. For instance, if we

know Skill and Author of a new OER, we identify all

other OERs provided by the same author and the same

skill target, compute their average values, and set the

initial property values accordingly.

3.3.5 Method for Updating OER Properties

Detecting OER properties is a slow process in the be-

ginning, since change happens when users alter their

rating pattern. This happens usually when they are

confronted with new OERs. Therefore, we run the up-

dating process after a speciﬁc time period (e.g. once

each month). To adjust the properties (except Rel-

evance) of each OER at ﬁrst, we collect all related

users and their ratings in the given time period. After-

wards, we compute the property values for the OER

as X in order to minimize (3) using Gradient Descend,

where θ

is the property vector of user i and Y

is the

satisfaction rate of user i.

LossFunction =

∑

i=users

|(θ

∗ X) −Y

| (3)

This strategy of using all recent ratings in updating

OER properties, enhances the diversity in our rec-

ommendations. All learners contribute to calculat-

ing these OER properties (for each OER they stud-

ied) through their individual evaluations. Users can

also rate OERs as irrelevant. As a consequence, the

Relevance property of an OER o is calculated as (4)

where the total recom(o) shows the number of times

that OER o has been recommended to users and ir-

relev count(o) is the number of times that o has been

determined as Irrelevant. Finally, OERs with a Rel-

evance Value less than the average in relation to a

speciﬁc skill, are marked as Irrelevant (for that skill

only), and therefore will not be recommended (for

that skill) anymore.

relevancy(o) =

total recom(o) − irrelev count(o)

total recom(o)

(4)

3.3.6 Recommendation Algorithm

For recommending an OER to a learner, we calcu-

late Cosine Similarity between the properties of can-

didate OERs (which are related to the skill-level of

any user) and the properties of the user. The system

will recommend an OER with the lowest distance be-

tween those two. Since we update user properties in

a real-time process and update OER properties after a

predeﬁned period, for recommending the best match

for a user, we only need to ﬁnd an OER, which has

the closest properties to the user. Furthermore, Rat-

ing Sparsity problem (i.e. users rate only a few num-

ber of OERs) is one of the most important issue when

building recommender systems. In our case users and

OERs have mutual contribution to calculating proper-

ties, which intends to eliminate the effects of Rating

Sparsity. Even if an OER has limited amount of rat-

ings, we can rely on the properties of the learners. On

the basis of their ratings on other (similar) OERs, we

calculate the properties for OERs suffering from Rat-

ing Sparsity.

3.4 OER Recommender Prototype

Learners were confronted with a prototype of our

recommender system in a form of a dashboard

Through this dashboard learners can search for their

current or desired job, display the list of required

skills and set their level of expertise for each skills.

Subsequently, on the learning tab, the dashboard

shows the current expertise levels of the learner, and

the links to the recommended OERs. OERs are or-

dered according to the importance of skills for the

selected job. In case a learner thinks that a recom-

mended OER is not related (Irrelevant) or does not

ﬁnd the content engaging, a new recommendation

could be generated, without changing the expertise

level of the learner. After consulting (learning) a rec-

ommended OER, learners are asked to rate their sat-

You can ﬁnd demo of our prototype from: https://github.

com/rezatavakoli/CSEDU2020

Labour Market Information Driven, Personalized, OER Recommendation System for Lifelong Learners

101

isfaction with that OER. Finally, the dashboard up-

dates the learner’s expertise level and provide an up-

dated recommendation based on the new rating. This

is done until the learner masters all required skills on

the highest level. Figure 1 depicts the building blocks

of our proposed approach.

4 VALIDATION

To validate our proposed approach, we conducted

semi-structured interviews with subject matter ex-

perts in the job areas of Data Science and Mechanical

Engineering. We focused on jobs, which are related

to these areas, and randomly selected 100 job vacan-

cies for Data Scientists and 100 job vacancies for Me-

chanical Engineers from August 2019. Afterwards,

we applied our skill extraction and importance detec-

tion model to select the most important skills in both

occupations. To evaluate recommendations, we in-

vited four university instructors with at least 12 years

of teaching and 13 years of industrial experience and

eight PhD students with a minimum teaching experi-

ence of 1 year and a minimum industrial experience

of 2 years for a semi-structured interview

. Partici-

pants gave feedback on our prototype with regards to

its general objectives, logic, and potential contribu-

tion to individual learning. Each interviewee had to

go through the following protocol:

1. Learning about the research problem and the pro-

posed approach - 15 minutes

2. Work with our prototype - 15 minutes

3. Going through a semi-structured interview with

the help of a qualitative questionnaire

- 30 min-

utes

During working sessions with our prototype, partic-

ipants generated more than 150 OER recommenda-

tions. 76.9% of these recommendations were useful

and relevant to participants’ skill levels and proper-

ties. 8.2% of the recommended OERs were signalled

as irrelevant, and in 14.9% of the cases participants

decided to change the recommended OERs. The re-

sults of the interviews are summarised in the follow-

ing three sections.

4.1 Objectives

Interviewees conﬁrmed that there is a potential value

in building a labour market information driven OER

Detailed proﬁles of our interview participants are available

on: https://github.com/rezatavakoli/CSEDU2020

The questionnaire is available on: https://github.com/

rezatavakoli/CSEDU2020

recommender system. Both instructors and PhD

students thought that there are several useful and

high quality OERs available on the Internet, but

ﬁnding them are complicated and time-consuming.

Regarding the skill extraction, participants recom-

mended to use alternative data sources, besides va-

cancy announcements. Student 2 for example sug-

gested that “you should also use other data sources

related to labor market like CVs and available data

about salaries”. Moreover, interviewees thought that

this approach is extremely useful for job-seekers, job-

holders, and people who have clear ideas about their

preferred occupation. However, they were skeptical

about those learners, who want to focus their atten-

tion on a speciﬁc skill only.

4.2 Logic

Participants conﬁrmed that our method to calculate

the importance of particular skills in recent job va-

cancies can potentially help learners to focus on the

most important elements of their current or future job.

However, as it was also suggested by Student 1, a

more intelligent decay function, to combine recent

and previous skill important values might be desir-

able. Regarding the self assessment of learners to set

their initial level of expertise, Instructor 1 suggested

to “introduce basic assessment in a form of technical

or non-technical questions” for each targeted skill.

4.3 Contribution to Learning

Participants emphasized that interacting with learners

in order to recognize their preferences (e.g. recom-

mending OERs based on their previous ratings) is one

of the most important, novel and engaging component

of our proposed approach. Student 5 recommended to

include more properties: “You should capture more

learners’ properties such as language preferences or

type of OERs (e.g. presentation, video).” Moreover,

interviewees were convinced that setting speciﬁc and

personalized goals for each skill in our prototype sys-

tem has a strong and positive effect on the learning

process.

5 CONCLUSION AND FUTURE

WORK

In this paper, we showcased a hybrid OER Recom-

mender system prototype to support individual skill

development, targeting concrete, labour market ori-

ented skills and jobs. For this prototype a skill ex-

traction mechanism has been constructed, which cap-

CSEDU 2020 - 12th International Conference on Computer Supported Education

102

Figure 1: Components of our Labour Market Intelligence (LMI) based OER recommender.

tures skill related sentences in vacancy announce-

ments with balanced accuracy of 88.7%. These dy-

namically generated skills became individual learning

objectives and were connected to OER based learning

contents. Recommendations were generated through

a dashboard, with combining OER and learner prop-

erties. The system prototype was validated with semi-

structured interviews. The initial results showed that

our proposed approach has the potential to aid life-

long learners to construct their individual learning

pathways and progress towards their desired job re-

lated skills. Moreover, participants valued that user

properties were critical, when formulating recom-

mendations.

We consider this study as an important ﬁrst step

(and a promising positive feedback) on our ongoing

research project to empower lifelong learners on the

basis of accurate labour market information. We be-

lieve that by confronting learners with labour market

information, we also support them to develop critical

transferable skills such as the awareness of their own

learning needs, continuous reﬂection on their indi-

vidual learning goals, capacities to (re)design person-

alised curricula, or measurement of learning achieve-

ments. Of course this prototype comes with a num-

ber of limitations (e.g. only two jobs were covered;

content was only received from two OER reposito-

ries, the number of properties for the recommenda-

tion were limited), but we believe it is worthwhile to

invest further effort in this area. As the next step we

plan to expand the context of our investigation with

adding more OER repositories to our system, together

with extracting more properties from users to provide

better recommendations. Moreover, accurate skill de-

composition is another key problem to improve, in

order to get better assessment about users’ expertise

level, and to construct more suitable learning path-

ways for lifelong learners. Finally, we plan to use

(quasi-)experimental designs for further developing

and validating our prototype in a number of use cases.

REFERENCES

Atenas, J. and Havemann, L. (2013). Quality assurance in

the open: an evaluation of oer repositories. INNO-

QUAL: The International Journal for Innovation and

Quality in Learning, 1(2):22–34.

Boselli, R., Cesarini, M., Marrara, S., Mercorio, F., Mez-

zanzanica, M., Pasi, G., and Viviani, M. (2018a).

Wolmis: a labor market intelligence system for classi-

fying web job vacancies. Journal of Intelligent Infor-

mation Systems, 51(3):477–502.

Boselli, R., Cesarini, M., Mercorio, F., and Mezzanzanica,

M. (2018b). Classifying online job advertisements

through machine learning. Future Generation Com-

puter Systems, 86:319–328.

Castello, V., Flores, E., Gabor, M., Guerrero, J., Guspini,

M., Luna, J., Mahajan, L., McGartland, K., Szabo,

I., and Ramos, F. (2014). Promoting dynamic skills

matching: challenges and evidences from the smart

project. In INTED2014 Proceedings, pages 2430–

2438. Citeseer.

Chicaiza, J., Piedra, N., Lopez-Vargas, J., and Tovar-Caro,

E. (2015). A user proﬁle deﬁnition in context of rec-

ommendation of open educational resources. an ap-

proach based on linked open vocabularies. In IEEE

Frontiers in Education Conference, pages 1–7. IEEE.

Chicaiza, J., Piedra, N., Lopez-Vargas, J., and Tovar-Caro,

E. (2017). Recommendation of open educational

resources. an approach based on linked open data.

In Global Engineering Education Conference, pages

1316–1321. IEEE.

Colace, F., De Santo, M., Lombardi, M., Mercorio, F., Mez-

zanzanica, M., and Pascale, F. (2019). Towards labour

market intelligence through topic modelling. In Pro-

ceedings of the 52nd Hawaii International Conference

on System Sciences.

Colombo, E., Mercorio, F., and Mezzanzanica, M. (2018).

Applying machine learning tools on web vacancies for

labour market and skill analysis.

Djumalieva, J. and Sleeman, C. (2018). An open and

data-driven taxonomy of skills extracted from online

job adverts. Developing Skills in a Changing World

of Work: Concepts, Measurement and Data Applied

in Regional and Local Labour Market Monitoring

Across Europe, page 425.

Dufﬁn, J., Muramatsu, B., and Henson Johnson, S. (2007).

Oer recommender: A recommendation system for

open educational resources and the national science

digital library. White paper funded by the Andrew W.

Mellon Foundation for the Folksemantic. org project.

Elias, M., James, A., Lohmann, S., Auer, S., and Wald, M.

(2018). Towards an open authoring tool for accessible

slide presentations. In International Conference on

Computers Helping People with Special Needs, pages

172–180. Springer.

Elias, M., Lohmann, S., and Auer, S. (2017). Ontology-

based representation of learner proﬁles for accessi-

ble opencourseware systems. In International Con-

Labour Market Information Driven, Personalized, OER Recommendation System for Lifelong Learners

103

ference on Knowledge Engineering and the Semantic

Web, pages 279–294. Springer.

Gardiner, A., Aasheim, C., Rutner, P., and Williams, S.

(2018). Skill requirements in big data: A content anal-

ysis of job advertisements. Journal of Computer Infor-

mation Systems, 58(4):374–384.

Ha, K.-H., Niemann, K., Schwertel, U., Holtkamp, P.,

Pirkkalainen, H., Boerner, D., Kalz, M., Pitsilis, V.,

Vidalis, A., Pappa, D., et al. (2011). A novel approach

towards skill-based search and services of open educa-

tional resources. In Research Conference on Metadata

and Semantic Research, pages 312–323. Springer.

Hepp, M. (2007). Possible ontologies: How reality con-

strains the development of relevant ontologies. IEEE

Internet Computing, 11(1):90–96.

Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T.

(2016). Bag of tricks for efﬁcient text classiﬁcation.

arXiv preprint arXiv:1607.01759.

Kanwar, A. and Mishra, S. (2018). Global trends in oer:

What is the future?

Karakatsanis, I., AlKhader, W., MacCrory, F., Alibasic, A.,

Omar, M. A., Aung, Z., and Woon, W. L. (2017). Data

mining approach to monitoring the requirements of

the job market: A case study. Information Systems,

65:1–6.

Khobreh, M., Ansari, F., Fathi, M., Vas, R., Mol, S. T.,

Berkers, H. A., and Varga, K. (2015). An ontology-

based approach for the semantic representation of job

knowledge. IEEE Transactions on Emerging Topics

in Computing, 4(3):462–473.

Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismih

ok, G.,

and Den Hartog, D. N. (2018). Text mining in organi-

zational research. Organizational research methods,

21(3):733–765.

Lam, X. N., Vu, T., Le, T. D., and Duong, A. D. (2008). Ad-

dressing cold-start problem in recommendation sys-

tems. In Proceedings of the 2nd international con-

ference on Ubiquitous information management and

communication, pages 208–211. ACM.

Lopez-Vargas, J., Piedra, N., Chicaiza, J., and Tovar, E.

(2014). Recommendation of oers shared in social me-

dia based-on social networks analysis approach. In

IEEE Frontiers in Education Conference, pages 1–8.

IEEE.

Maer-Matei, M. M., Mocanu, C., Zamﬁr, A.-M., and

Georgescu, T. M. (2019). Skill needs for early career

researchers—a text mining approach. Sustainability,

11(10):2789.

McGill, M. M. (2009). Deﬁning the expectation gap: a

comparison of industry needs and existing game de-

velopment curriculum. In Proceedings of the 4th

International Conference on Foundations of Digital

Games, pages 129–136. ACM.

Piedra, N., Chicaiza, J., L

opez-Vargas, J., and Caro, E. T.

(2015). Seeking open educational resources to com-

pose massive open online courses in engineering edu-

cation an approach based on linked open data. J. UCS,

21(5):679–711.

Ruiz-Iniesta, A., Jimenez-Diaz, G., and Gomez-Albarran,

M. (2014). A semantically enriched context-aware

oer recommendation strategy and its application to a

computer science oer repository. IEEE Transactions

on Education, 57(4):255–260.

Sibarani, E. M., Scerri, S., Morales, C., Auer, S., and Col-

larana, D. (2017). Ontology-guided job market de-

mand analysis: a cross-sectional study for the data

science ﬁeld. In Proceedings of the 13th International

Conference on Semantic Systems, pages 25–32. ACM.

Smith, D. and Ali, A. (2014). Analyzing computer pro-

gramming job trend using web data mining. Issues

in Informing Science and Information Technology,

11(1):203–214.

Sun, G., Cui, T., Beydoun, G., Chen, S., Dong, F., Xu, D.,

and Shen, J. (2017). Towards massive data and sparse

data in adaptive micro open educational resource rec-

ommendation: a study on semantic knowledge base

construction and cold start problem. Sustainability,

9(6):898.

Sun, G., Cui, T., Xu, D., Shen, J., and Chen, S. (2018). A

heuristic approach for new-item cold start problem in

recommendation of micro open education resources.

In International conference on intelligent tutoring sys-

tems, pages 212–222. Springer.

Verma, A., Yurov, K. M., Lane, P. L., and Yurova, Y. V.

(2019). An investigation of skill requirements for

business and data analytics positions: A content anal-

ysis of job advertisements. Journal of Education for

Business, 94(4):243–250.

Wan, S. and Niu, Z. (2018). An e-learning recommendation

approach based on the self-organization of learning re-

source. Knowledge-Based Systems, 160:71–87.

Wowczko, I. (2015). Skills and vacancy analysis with data

mining techniques. In Informatics, volume 2, pages

31–49. Multidisciplinary Digital Publishing Institute.

CSEDU 2020 - 12th International Conference on Computer Supported Education

104