Mastery Profiling Through Entity Linking
To Support Project Team Formation in Higher Education
Alexander M. Olieman and Frank Nack
Informatics Institute, University of Amsterdam, Science Park 904, Amsterdam, Netherlands
Keywords: Collaborative Learning, Computer-Supported Group Formation, Information Extraction, Entity Linking,
Mastery Profiling, Graph Exploration, DBpedia Spotlight, Gremlin.
Abstract: Computer-supported group formation enables educators to assign students to project teams. The focus in
this paper is placed on gathering data about student attributes that are relevant in the context of specific
course projects. We developed a method that automatically produces learner models from existing docu-
ments, by linking students to topics and estimating the levels of skill, knowledge, and interest that students
have in these topics. The method is evaluated in an experiment with student participants, wherein its per-
formance is measured on two levels. Our results demonstrate that it is possible to link students to topics with
high precision, but suggest that estimating mastery levels is a more challenging task.
1 INTRODUCTION
Group-based learning has taken an important role in
curricula across the educational spectrum. One as-
pect of group-based learning that has attracted con-
siderable attention from researchers, is the formation
of groups of learners. Group formation influences
the interactions that group members have, and there-
by affects the results of the learning experience
(Kyprianidou et al., 2011). Poorly formed groups
may suffer from, for example, an unproductive use
of time or incompatible personality types. The way
students are partitioned into groups also raises the
question if they can be assessed fairly, e.g. due to an
unbalanced spread of skills (Livingstone and Lynch
2000).
In higher education, common group forming
methods are student self-selection and random as-
signment. These methods do not necessarily lead to
good learning experiences, but are often the only
practical alternatives for instructors who teach large
numbers of new students each year. Instructors
might lack the necessary information about the stu-
dents to implement a more elaborate group for-
mation process, or face the impracticality of manual-
ly solving a large combinatorial problem (Craig et
al., 2010). This has motivated research towards the
development of tools that can aid instructors in
forming groups, which is known as Computer-
Supported Group Formation (CSGF) (Ounnas et al.
2009).
Regardless of the algorithms that are used, the
criteria that can be used in the group formation pro-
cess are limited by the data that is used to describe
the students. Hence much of the previous work
makes use of student attributes for which standard-
ized tests are available, such as team roles, personal-
ity types, and learning styles (Magnisalis et al.,
2011). Important disadvantages of gathering student
data in this way are the dependence on lengthy ques-
tionnaires, and the need to ask students new ques-
tions when course-specific characteristics are taken
into account.
In computer-supported collaborative learning set-
tings where the majority of learning occurs in a
virtual environment, there are opportunities to gather
relevant data about students continuously. In more
traditional settings, it may instead be viable to use
data from existing resources that describe students,
specifically to model students’ mastery of topics.
Previous suggestions are to use text mining tech-
niques on curricula vitae (CVs), academic tran-
scripts, and personal websites (Ounnas et al., 2009).
The objective of this study is to develop and
evaluate a method which allows existing data
sources that describe students’ mastery levels (e.g.
of knowledge, skills, and interests) to be automati-
cally combined into a learner profile for use in
CSGF algorithms. The scope of group formation
182
M. Olieman A. and Nack F..
Mastery Profiling Through Entity Linking - To Support Project Team Formation in Higher Education.
DOI: 10.5220/0004764001820190
In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 182-190
ISBN: 978-989-758-020-8
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
problems that need to be addressed by this method is
restricted to the domain of team project-based higher
education. To work towards this objective, the main
research question that this paper addresses is formu-
lated as follows:
Can existing data sources that describe stu-
dents’ mastery of topics be fused into a learner
profile, to facilitate computer-supported group
formation?
The remainder of this paper is structured as fol-
lows. In Section 2 we discuss related work. Section
3 serves to briefly describe a further exploration of
the problem domain. In Section 4 we discuss gather-
ing existing resources from student participants.
Based on these resources we produce learner pro-
files, using a method that is described in Section 5.
In Section 6 we discuss our evaluation approach and
results, followed by the conclusion and future work.
2 RELATED WORK
2.1 Computer-supported Group
Formation
CSGF is based on the idea that instructors can assign
students to groups by making explicit educational
criteria according to which groups should be formed
(Craig et al., 2010). The essence of CSGF is: the
synthesis of groups by applying criteria that opti-
mize aspects of each group, by making use of data
about the individual learners (Magnisalis et al.,
2011).
A classification, originating from literature on
team diversity, divides relevant attributes into task-
related (e.g. knowledge, skills, experience) and
relations oriented (e.g. gender, culture, attitude,
social ties) (Jackson et al., 1995). Task-related at-
tributes of individual students are relevant because
they indicate which cognitive resources will be
available in any possible grouping. Relations orient-
ed attributes indicate how group members are ex-
pected to interact. A common approach to recording
task-related attributes is to ask students for their
grades in selected prerequisite courses (Lingard and
Berry, 2002). Another approach is to measure skill
levels for a few domain-specific skills by question-
naire (Winter, 2004).
Most of the criteria according to which groups
should be formed can be classified as homogeneous,
heterogeneous, or apportioned (Craig et al., 2010).
Both homogeneous and heterogeneous criteria are
concerned with the distance between students within
a group for a specific attribute, while apportioned
criteria serve to distribute a specific attribute as
evenly as possible across the groups.
Hoogendoorn (2013) has recently conducted
three field experiments which provide evidence for
the effect of heterogeneity on the performance of
student teams. His results suggest that gender di-
verse teams perform significantly better than male-
dominated teams, and no worse than female-
dominated terms. The effect of ethnic diversity on
team performance is found to be positive for teams
where at least half of the members have different
backgrounds. Diversity in cognitive ability of team
members only shows a positive effect on perfor-
mance when the degree of heterogeneity is moder-
ate. Heterogeneity of cognitive resources is suggest-
ed as an underlying mechanism for the effect of
diversity in ethnicity and cognitive ability.
Other researchers argue for certain criteria with-
out empirical support (e.g. based on expert opinion).
Most arguments are made for heterogeneous criteria
(i.e. complementary fit) on specific attributes, in-
cluding skills, knowledge, abilities (Wells, 2002;
Werbel and Johnson 2001; Wilkinson and Fung
2002), and learning styles (Magnisalis et al., 2011).
Student interests and values should however be
grouped homogeneously (Werbel and Johnson,
2001). Grades in prerequisite courses are most often
apportioned (Craig et al., 2010; Ounnas, 2010).
2.2 Entity Linking
Entity linking (EL) is the information extraction task
of automatically “matching a textual entity mention
[...] to a [knowledge base] entry, such as a Wikipe-
dia page that is a canonical entry for that entity.”
(Rao et al., 2013, p. 96). Three key challenges have
been identified for EL to deal with: name variation,
entity ambiguity, and absence (Dai et al., 2012; Rao
et al., 2013). Name variation entails that an entity
can be referred to by multiple different terms. Entity
ambiguity refers to the issue that a single name
string can match with several distinct entities. The
issue with absence is that when no knowledge base
(KB) entry exists for the entity that is mentioned in
the text, no entity should be returned, rather than the
highest-ranking KB entry.
There are, however, two relevant limitations pre-
sent in the existing work on EL. Most research fo-
cuses explicitly on linking named entities (i.e. enti-
ties referred to by proper names), specifically on
persons, locations, and organizations (Mendes, Dai-
ber, et al., 2011; Rao et al., 2013). Additionally,
many current approaches are evaluated only on Eng-
lish-language texts, with a focus on the news domain
MasteryProfilingThroughEntityLinking-ToSupportProjectTeamFormationinHigherEducation
183
(Mendes et al. 2011; Rao et al., 2013).
DBpedia Spotlight is an open-source system that
can annotate any given input text with DBpedia
resources (i.e. KB entries), which are based on se-
mantic extraction from Wikipedia articles (Mendes,
Jakob, et al., 2011). Several parameters provide the
means to filter annotations according to task-specific
requirements. By default, DBpedia Spotlight is not
specialized towards specific entity types, but it may
be configured to annotate only instances of specific
types, either by selection of classes, or by arbitrary
SPARQL
1
queries (Mendes et al., 2011).
When linking targets are known to have a specif-
ic type, e.g. genes (in the biomedical domain), anno-
tating only those entities is quite straightforward
(Dai et al., 2012). If a domain-specific vocabulary
already contains links to DBpedia or Wikipedia then
one can consider all DBpedia Spotlight candidate
entities, and then check whether the top-ranked
candidate has a corresponding entity in the local
vocabulary (Mendes et al., 2011).
Multiple EL researchers have found it helpful to
include a measure of semantic-relatedness between
entities in the disambiguation process (e.g. Han et
al., 2011). The intuition behind collective entity
disambiguation is that the links between, e.g., Wik-
ipedia articles reflect how closely the corresponding
entities are related, and that texts are more likely to
mention several related entities than entirely unrelat-
ed entities.
Besides using metrics of semantic-relatedness
and disambiguation purposes, it might be feasible to
use them to find additional topics in which a student
has some mastery. For instance, when a student's CV
mentions that she is skilled in technical drawing, we
can infer that she has some skill in drawing in gen-
eral.
3 VIEWS ON FORMING TEAMS
In the group formation literature arguments are made
for the relevance of skills, knowledge, abilities,
interest, and grades. All arguments are, however,
made from the educator's perspective, and infor-
mation about the student's perspective is lacking. We
have therefore surveyed a group of university stu-
dents and asked them about the considerations they
have had while forming project teams in the past.
As in other group formation studies, we recruited
participants from a subpopulation of students who
study the same subject (Lingard and Berry, 2002;
1
SPARQL 1.1 - <http://www.w3.org/TR/sparql11-overview/>
Winter, 2004). All participants were enrolled in the
MSc program Design for Interaction at the Delft
University of Technology (DUT), and were recruited
through a mailing list. The same sample of students
were respondents to the questionnaire as well as
participants in the experiment that is described in
subsequent sections.
The questionnaire was taken by 11 students. We
focused our questions on seven attributes: compe-
tence, education, experience, general ability, inter-
est, knowledge, and skill.
The results of the questionnaire broadly corre-
spond to what was found in the CSGF literature. On
this basis, we decide to include skill and knowledge
in our learner profiles. We also include interest in
our profiles because it is used in team formation
criteria by many instructors (Werbel and Johnson
2001; Kyprianidou et al., 2011). Competence, we
argue, is not a suitable choice because it depends on
specific skills and knowledge.
4 GATHERING EXISTING
RESOURCES
After finishing the initial questionnaire, students
were asked to participate in creating a learner profile
based on existing documents about them. To partici-
pate, they needed to provide access to a project port-
folio, an academic transcript, and temporary access
to their LinkedIn
2
profile, and/or provide the URL to
a personal website. The terms that indicate relevant
attributes need to be recognized in these resources,
and should be linked to a shared vocabulary in
which the learner profiles can be expressed. The
quality of the resulting profiles is evaluated by com-
paring them with a ground truth that is given by the
participants.
The decision to gather academic transcripts,
CVs, and personal websites was motivated by sug-
gestions found in literature (Ounnas et al., 2009).
Although documents in a project portfolio do not
describe students in the same sense as the other
document types do, they can give a more detailed
view of the specific topics a student has engaged
with during previous projects (at least for a human
reader).
Participants’ academic transcripts and LinkedIn
profiles were saved by a sign-up application. The
course descriptions that correspond to the course
identifiers in the academic transcripts were retrieved
2
LinkedIn - <http://www.linkedin.com/>
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
184
through the DUT API
3
. The project portfolios con-
sist of deliverables, such as reports, presentation
slides, and project blogs. The corresponding files
were provided by Shareworks Solutions BV. For the
two participants who provided a website, we saved
the pages manually.
In total, 10 LinkedIn profiles (all English), 190
course descriptions (66 Dutch, 124 English), 54
portfolio documents (2 Dutch, 52 English), and two
websites (both English) were gathered. Participants
were associated with between 15 and 57 documents;
on average with 37 documents. Most course descrip-
tions and some portfolio documents were associated
with more than one participant.
5 PRODUCING LEARNER
PROFILES
Two existing implementations of DBpedia Spotlight
are used to produce annotations that we define as
"links between a phrase in a document and a topic,
which is represented by a DBpedia URI". Subse-
quently, we estimate skill, knowledge, and interest
levels by taking into account surrounding terms of
each annotation, document origins, and annotation
frequencies. The learner profiles that are produced
by this method consist of statements, where a state-
ment is "the relation between a student and a single
topic, which is quantified by three mastery levels.
Finally, the set of statements in each profile is ex-
panded by inference over probabilistic and semantic
relations between topics.
5.1 Vocabulary Selection
and Modification
CSGF differs from the current applications of entity
linking: when links are generated for the readers of
an article, it is assumed that the readers are familiar
with the majority of abstract concepts that are men-
tioned. For CSGF, abstract concepts are mostly
relevant, and people and places less so. To annotate
documents only with topics that are relevant for
CSGF, we test the approach taken in Mendes, Dai-
ber, et al. (2011) and in Wetz et al. (2012), for which
we require an application-relevant subset of DBpe-
dia entities.
We use the LinkedIn "Skills & Expertise" vo-
cabulary
4
LI as a basis. Reasons to choose this
vocabulary are that it is already partially linked to
3
Delft University of Technology API - <http://apidoc.tudelft.nl/>
4
LinkedIn Skills & Expertise - <http://www.linkedin.com/skills/>
Wikipedia, and that it is used daily by thousands of
people to describe their professional abilities. From
LI, we define our vocabulary LI∩.
There are 26 292 topics in ; nearly 70% of ∈
LI, but only 0.7% of all DBpedia resources. The
links between
LI and Wikipedia contain inaccura-
cies. We have manually corrected 40 of such links,
but we estimate that at least 10% of ∈
LI are
needlessly missing a link, or are linked to an incor-
rect Wikipedia article.
LI only links topics with English Wikipedia ar-
ticles, and as such would only include English
identifiers for topics. Since our profiling method
also needs to deal with documents in other lan-
guages, we incorporate alternative topic identifiers
into by using Wikipedia's interlanguage links
5
.
For each ∈ the Dutch identifier  (if available)
is retrieved from the nl.dbpedia.org SPARQL end-
point through the query:
SELECT ? WHERE {? owl:sameAs
<http://dbpedia.org/resource/> .}
There are topics that are mentioned frequently in
all types of the gathered resources, but that are not
relevant for learner profiles. We exclude 56 of such
topics in total from .
5.2 Information Extraction Pipeline
The information extraction process that we employ
lends itself to being described as a data transfor-
mation pipeline.
For each gathered document  , a
DBpedia Spotlight implementation, given a configu-
ration, annotates the content per section. The re-
sources that are returned are filtered with our vocab-
ulary.
In each section we count qualifying terms, which
indicate specific types of mastery, and linearly com-
bine the normalized counts with
, ,  weights that depend
on the document origin . The resulting ,,
score is assigned to each annotation within the sec-
tion, after which the scores are summed per topic for
the entire document. The summed scores are stored
in the document (

) as indications of mastery,
where one 
,,,
.
Hereafter, we select for each profile 
 the

that are associated with by a
document link  , . The indications in the
associated documents are summed per , , result-
ing in 1. .
|
|
mastery levels scores, where one
5
See: <http://en.wikipedia.org/wiki/Help:Interlanguage_links>
MasteryProfilingThroughEntityLinking-ToSupportProjectTeamFormationinHigherEducation
185
,,, for each unique topic that is associ-
ated with . The indications that originate from
course descriptions are, before summation, weighted
by the grade that the student received for the corre-
sponding course.
The maximum , , and scores of the summed
indications differ significantly between origins. For
each origin, the scores are linearly transformed to
the range 0, 100 , because we wish to compare the
accuracy of statements from different origins in our
evaluation. The resulting normalized indications
represent the relative mastery levels per topic for a
single student, i.e., they encode beliefs that the stu-
dent has more mastery in one topic than in another.
For CSGF the aim is to compare the mastery lev-
els per topic between students. A final data trans-
formation is thus needed. Each normalized 
,
is
transformed to its percentile rank (PR) 
,
in the
frequency distribution of the  with the same
,  from all profiles. Finally, statements are saved
as 
,
,.
5.3 Linking Documents to Topics
The first step in our approach to mastery level profil-
ing is to ask for each student: in which topics does
this student have any skill, knowledge, and/or inter-
est? We use entity linking to answer this question
based on the gathered documents, in lieu of more
tailored information extraction techniques. This
enables us to test the hypothesis that:
“From all entities that are mentioned in the doc-
uments associated with a student, a vocabulary can
be used to select the entities that are topics in which
this student has some mastery”.
5.3.1 Annotation Method
Two DBpedia Spotlight implementations (Mendes,
Jakob, et al. 2011; Daiber et al. 2013) are used and
configured to produce annotations in our experi-
ment. The original Information Retrieval-based
implementation, with the default configuration, spots
all phrases in the input text that also occur in a da-
taset of possible surface forms for all DBpedia. It
selects candidate entities for each spotted phrase,
and ranks them according to the prior probability
that the observed phrase refers to the selected candi-
date. The candidates are then re-ranked by querying
a Vector Space Model (VSM), in which entities are
represented by the paragraphs that mention them in
Wikipedia, with the context of the observed. Top-
ranking candidates are the most likely disambigua-
tions.
The newer statistical model uses a generative
probabilistic model for disambiguation. This model
is used to calculate a disambiguation score for entity
, given the spotted phrase and its context , by
combining
,
|
, and . The original phrase
spotting method is used in parallel with a Natural
Language Processing (NLP) method that is not lim-
ited to surface forms that occur in DBpedia. Any
overlap in spotted phrases is resolved, after which
the phrases that fall below a score threshold are
dropped from the annotation process.
In both implementations the topical pertinence of
a candidate for the observed context is indicated by
the disambiguation score. The relative difference in
this score between the first and second ranking can-
didate indicates contextual ambiguity, i.e. how un-
certain it is that the top-ranked candidate is the entity
that is mentioned in this context. The confidence
parameter, which is provided at runtime, applies a
threshold of 1  to candidates' con-
textual ambiguity scores. A second runtime parame-
ter, support, specifies the minimum number of Wik-
ipedia inlinks a candidate resource must have to be
further considered.
We define a third runtime parameter which
chooses between single and multiple candidate fil-
tering. In single candidate filtering, we take the set
of top-ranking entities
that Spotlight produces for
a section and select as topics the entities that occur
in our vocabulary (

∩). In multiple candi-
date filtering we instead initialize
, take
the set of ranked candidate vectors
̅
,̅
,…,,
and from each vector we add the top-ranking topic to
(denoted
∪
, where
:


:
in̅and ).
5.3.2 Exploration of the Parameter Space
To assess the suitability of various configurations for
producing learner models (before the ground truth is
given by the participants), we have manually anno-
tated a small test collection of documents, and have
measured the performance of our annotation method
on this collection.
We use the measures precision, recall, and F-
score (F
ß
) to evaluate the performance of the annota-
tion method. The definitions of these measures are
adapted from the prevailing definitions in entity
linking (Han et al. 2011) to better suit our annotation
task. Let  be the person who is profiled in ,
with the set of associated documents

:,
. 
is the set of all topics
with which

:,
has been annotated.

is the set of all topics in which 
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
186
claims to have some mastery.

|
∩
|
|
|
(1)

|
∩
|
|
|
(2)
1

 

 
(3)
In this work 0.5, which reflects our assumption
that recall is only half as important as precision for
this annotation task.
For the test documents we have had to use

and have created 
. In 
we have only included the topics for which im-
plied some mastery on the part of any student asso-
ciated with .
To find which configurations perform best for
our annotation task, we have performed a parameter
sweep on 3 Dutch and 8 English test documents.
Because we were not able to assign a value to the
threshold , we have instead manipulated a spot
score weight which, when enlarged, increases the
probability that spotted terms are annotated.
With both languages, a clear tradeoff between
precision and recall can be observed. Multiple can-
didate filtering, as expected, results in higher recall,
but lower precision, than single candidate filtering.
Increasing the confidence value causes higher preci-
sion, but lower recall. We found the effect of the
support parameter to be negligible.
Based on these results, we continue our main ex-
periment with the IR-based implementation and
 0.3 for English documents, and the
statistical implementation with a spot score weight
of 0.4 and  0.2 for Dutch documents.
For both languages we use single candidate filtering
and  0.
5.4 Estimating Mastery Levels
To estimate which type(s) of mastery a student has
in a topic, we have selected the descriptions of the
25 most attended courses, and recorded all terms that
imply skill or knowledge (i.e. qualifying terms).
We represent the qualifying terms as sets, and
count for every section that contains annotations
how many terms indicate skill and knowledge.
Stemming is used to also count lexical variations of
the terms. Predefined weights per document origin
are linearly combined with the fractions of skill and
knowledge term counts, to produce a ,, score
per annotation in a section. In the defined weights,
we assume that portfolio documents and websites
indicate each type of mastery, but that course de-
scriptions do not indicate any interest.
In our definition of mastery levels we need to
take into account that the ground truth that we will
use in our evaluation is provided by the participants.
The scale and unit in which mastery levels are ex-
pressed need to be understood by students to allow
them to accurately correct their profiles (Bull and
Kay 2007). In our model a mastery level means that
a student has more knowledge, skill, or interest in a
topic than a percentage of his or her peers. "Paul
(Knowledge, 75) Archery", for example, would
indicate that Paul has more knowledge about archery
than 75% of his peers.
To estimate mastery levels, we use the intuition
that a student will have more mastery in topics that
are mentioned more often in the associated docu-
ments. For the topics that originate from course
descriptions we also incorporate the grade that a
student received and the extent of the course. Each
indication of mastery that originates from a course
description is multiplied by a weight
,
, which is
calculated as:
,
0.5 2 _
11 
,
.
(4)
All indications in the associated documents for a
single student are subsequently summed per , .
Hereafter, the indications are normalized per origin,
but across profiles, to the range 0, 100, so that for
each , there exist maximum , , and scores
with the value 100. This enables us to generate indi-
cations for a fifth origin ALL, in which we have
compensated for differences in annotation frequency
and weighting between topics and origins. The indi-
cations for ALL are generated by summing the exist-
ing indications per profile.
Finally, indications are transformed into state-
ments with the desired semantics by using the fre-
quency distributions of , , and scores, again per
, , over all profiles. Each score is transformed
into a mastery level by calculating its percentile rank
in the corresponding frequency distribution. But
because mastery levels are defined relative to a stu-
dent's peers, we would need frequency distributions
that include all peers.
To compensate for the limited number of partici-
pants, we apply a form of additive smoothing in the
calculation of percentile ranks. Into each frequency
distribution
,
we insert the values 0.0, 1.5

,
, and |
,
|1 evenly spaced values
in between. The percentile ranks of individual indi-
cations are computed from these modified frequency
MasteryProfilingThroughEntityLinking-ToSupportProjectTeamFormationinHigherEducation
187
distributions.
5.5 Expanding Profiles by Inference
Because our vocabulary is not a KB, in the sense
that it contains no information about relationships
between topics, we use DBpedia as a source for
semantic relations between topics. We aim to predict
for each student, on the basis of the topics that are
linked from their profile, in which other topics they
are likely to have some mastery.
We use Gremlin
6
, which implements efficient
graph traversal as described in Rodriguez &
Neubauer (Rodriguez & Neubauer 2011), to traverse
the semantic network of DBpedia. For each topic
(node) in a profile, the dcterms:subject links
are followed to the categories the topic is a member
of, from where skos:broader and
skos:narrower are followed to neighboring
categories, up to two levels outward. At each catego-
ry node that is visited during the traversal, the con-
tained topics are also visited, and the frequencies of
these visits are counted as a side effect. When the
traversal is finished, we take the frequency table,
and store it as a measurement of the relatedness
between the starting node and the visited topics.
Then, to infer which newly found topics should
be included in the profile, we summate the frequen-
cy tables of all topics in the profile. From the result-
ing table, we ignore any topics that are already in the
profile, and take the top-10 related topics that are in
our vocabulary, and the top-10 topics that are not in
our vocabulary.
LinkedIn uses a proprietary algorithm, which
likely incorporates aspects of collaborative filtering,
to display 20 "related skills" on each of the pages
that we used as the basis for our vocabulary. Such
lists of related skills are added into a frequency table
for each profile, and are further treated identically to
the inferences from DBpedia.
6 EVALUATION
The ground truth against which we measure the
performance of the method is provided by 8 partici-
pants. We have provided them with an interface that
allowed them to review and correct their own pro-
file. First they were presented with 184–353 topics
that were extracted from all types of documents
(). Participants were asked to remove all
6
Gremlin - < http://gremlin.tinkerpop.com/ >
topics in which they had no mastery by clicking the
corresponding buttons.
The second step for the participants was to cor-
rect the estimated mastery levels. Here, statements
were presented as boxes (again including the topic
name and description) with three sliders for the skill,
knowledge, and interest level. Due to the large
amount of extracted statements, we randomly omit-
ted 50% of the statements that were based only on
extraction from course descriptions or portfolio
documents.
The third step was similar to the first, except
with 20 inferred topics from DBpedia, and 20 in-
ferred topics from LinkedIn. In the fourth and final
step, the participants were asked to add any topics in
which they had mastery that were missing from their
profile.
It is worth noting that people are prone to over-
and underestimating themselves (Dunning et al.,
2003). This is, however, not a weakness of our ex-
periment in particular. In CSGF it is still quite com-
mon to base a profile of task-related attributes solely
on the information that is provided by the learner in
question.
The measures precision, recall, and F
0.5
-score,
which have been defined in Section 5.3.2, are used
to evaluate the performance of our annotation pro-
cess. We do not average our measures over the pro-
files, but rather take the counts of 
,

, and their intersection per profile, sum the
counts, and then compute precision, recall, and F
0.5
over all profiles. To assess how successful we were
at estimating mastery levels, we test for correlation
between the estimated and actual levels. We use
Pearson's correlation coefficient () as a measure,
and we report on statistical significance at the levels
of 0.10 (
˙
) and 0.001 (*). Because we included a
limited number of inferred topics in the participants'
profiles, we cannot use exactly the same measures as
with the extracted topic links. Instead, we use Preci-
sion at 10, which denotes the fraction of inferred
topics in which the participants claim to have mas-
tery.
Our results (see Table 1) indicate that the combi-
nation of extracted information from all document
origins leads to the most accurate profiles. The pro-
files included a large amount of topics in which the
participants actually had some mastery. Course de-
scriptions are the only type of document that could
be used to produce profiles of similar quality by
itself. Documents from other origins lead to topic
links with a comparable precision, but in a quantity
that is likely not sufficient for application in CSGF.
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
188
Table 1: Performance of the Mastery Profiling Method.
Extracted
Origin Topic Links Levels Corr.
Pr. Re. F
0.5
Sk. Kn. In.
Course
d.
0.859 0.711 0.825 .210* .212* –––
LinkedIn 0.935 0.077 0.290 .060 .108 .047
Portfolio 0.811 0.184 0.482 .220* .058 .073
Website 1.000 0.003 0.013 .853
˙
.696 .700
ALL 0.845 0.860 0.848 .268* .241* .299*
Inferred
In-
Vocabulary
DBpedia
Pr. at 10
LinkedIn
Pr. at 10
Yes 0.825 0.925
No 0.750 0.875
Inference can be used to expand learner profiles with
high precision. The additional topics that are found
by taking into account probabilistic relations be-
tween topics from LinkedIn are more precise than
those that are found from a traversal over the seman-
tic network of DBpedia. It is more accurate, in both
cases, to filter the inferred topics with our vocabu-
lary. A large majority of the inferred topics that are
not in our vocabulary is, however, also correct.
Our method was not able to estimate mastery
levels with the accuracy that is necessary for CSGF.
The estimated mastery levels show a weak but sig-
nificant correlation with the levels that the partici-
pants reported. The differences between origins
suggest that course grades and qualifying terms are
both indicators for mastery levels, but that the num-
ber of sections that is annotated with a given topic is
a worse indicator than we expected.
7 DISCUSSION
Our annotation process has produced results that are
very promising for use in CSGF. It does not rely on
optimizations specific for the field the students are
training in. Instead, it relies upon the configurability
of DBpedia Spotlight and the broad coverage of
professional topics that is used on LinkedIn. Ad-
vantages of keeping the method and implementation
field-agnostic are the reproducibility of the experi-
ments with students of other fields, and a greater
potential to collaborate in the development of the
necessary software.
We found that the method makes mistakes that
may, however, be overcome with domain-specific
optimizations. Abbreviations of field-specific con-
cepts which are commonly used with a different
meaning are not disambiguated correctly. We also
found that the coverage of the vocabulary was too
broad. For example, "Schizophrenia" is in most
disciplines never a main topic.
Our results in estimating mastery levels are less
promising. It is possible that we have used suitable
indicators and that the used data transformations are
not right for this task. A post-hoc analysis of our
results can clarify this matter to some extent. It will
be interesting to see if a method that is based on
machine learning, but uses the same features as we
have, will fare better in future research.
To make further advances in mastery profiling,
we may have to turn to techniques that are outside
the scope of the current method. Portfolio docu-
ments that were the product of teamwork inherently
describe the actions of multiple team members. We
would want to distinguish between individuals, and
discern "who did what". For course descriptions it
holds that not all text indicates what the students will
do or learn. Administrative remarks say something
about the course or about the teacher, but give no
relevant information about the students who have
completed the course. Such mistakes ask for more
focus on textual relations, as is done in Open Infor-
mation Extraction (Etzioni et al. 2011).
8 CONCLUSIONS
In this paper, we have presented a method that pro-
duces learner profiles on the basis of existing docu-
ments that are associated with students. It is able to
link students to a large amount of topics, in which
they have skill, knowledge, and/or interest, with
high precision. We have not yet succeeded in esti-
mating the mastery levels that students have in these
topics. Our method can be used as a baseline in
future experiments that aim to produce learner pro-
files from existing documents. We aim to publish
our current implementation under an open source
license to facilitate this.
Our work is also a demonstration of a novel ap-
plication of entity linking. We have shown that
DBpedia Spotlight can be configured to accurately
annotate course descriptions, portfolio documents,
and websites. A customized vocabulary was used to
filter annotations that are relevant to the mastery
profiling task, and to combine the annotations from
Dutch and English language documents into a single
learner profile. The sets of topics that were extracted
from students' associated documents have been suc-
cessfully expanded by inference over semantic and
probabilistic relationships between topics.
MasteryProfilingThroughEntityLinking-ToSupportProjectTeamFormationinHigherEducation
189
We hope that the work that has been described in
this paper can serve as a starting point for the inclu-
sion of detailed task-related attributes in learner
profiles and, more generally, that it will assist in the
adoption of CSGF in higher education.
ACKNOWLEDGEMENTS
Many thanks go to Wouter Weerkamp for valuable
guidance and feedback; to Flip van Haaren and
Raymond Jelierse at Shareworks for feedback and
technical assistance; to the participants for respond-
ing to the questionnaire and for reviewing their own
learner models; and to Joachim Daiber for clarifying
the configurability of DBpedia Spotlight.
REFERENCES
Bull, S. & Kay, J., 2007. Student Models that Invite the
Learner In: The SMILI:-) Open Learner Modelling
Framework. International Journal of Artificial
Intelligence in Education, 17(2), pp.89–120.
Craig, M., Horton, D. & Pitt, F., 2010. Forming
reasonably optimal groups: (FROG). In Proceedings
of the 16th ACM international conference on
Supporting group work. pp. 141–150.
Dai, H. et al., 2012. From Entity Recognition to Entity
Linking: A Survey of Advanced Entity Linking
Techniques. In The 26th Annual Conference of the
Japanese Society for Artificial Intelligence. pp. 1–10.
Daiber, J. et al., 2013. Improving Efficiency and Accuracy
in Multilingual Entity Extraction. In Proceedings of
the 9th International Conference on Semantic Systems.
Austria, Graz.
Dunning, D. et al., 2003. Why people fail to recognize
their own incompetence. Current Directions in
Psychological Science, 12(3), pp.83–87.
Etzioni, O. et al., 2011. Open information extraction: The
second generation. In Proceedings of the 22nd
international joint conference on Artificial
Intelligence. pp. 3–10.
Han, X., Sun, L. & Zhao, J., 2011. Collective entity
linking in web text: a graph-based method. In
Proceedings of the 34th international ACM SIGIR
conference on Research and development in
Information Retrieval. pp. 765–774.
Hoogendoorn, S.M., 2013. Diversity and team
performance: A series of field experiments. PhD
Thesis. University of Amsterdam. Available at:
http://dare.uva.nl/record/440433.
Jackson, S., May, K. & Whitney, K., 1995. Understanding
the dynamics of diversity in decision-making teams. In
R. A. Guzzo, E. Salas, & Associates, eds. Team
Effectiveness and Decision Making in Organizations.
San Francisco, pp. 204–261.
Kyprianidou, M. et al., 2011. Group formation based on
learning styles: can it improve students’ teamwork?
Educational Technology Research and Development,
60(1), pp.83–110.
Lingard, R. & Berry, E., 2002. Teaching teamwork skills
in software engineering based on an understanding of
factors affecting group performance. In Frontiers in
Education Conference. Boston, MA: IEEE.
Livingstone, D. & Lynch, K., 2000. Group Project Work
and Student-centred Active Learning: Two different
experiences. Studies in Higher Education, 25(3),
pp.325–345.
Magnisalis, I., Demetriadis, S. & Karakostas, A., 2011.
Adaptive and Intelligent Systems for Collaborative
Learning Support: A Review of the Field. IEEE
Transactions on Learning Technologies, 4(1), pp.5–
20.
Mendes, P., Jakob, M., et al., 2011. DBpedia Spotlight:
Shedding Light on the Web of Documents. In
Proceedings of the 7th International Conference on
Semantic Systems (I-Semantics). Austria, Graz.
Mendes, P., Daiber, J., et al., 2011. Evaluating DBpedia
Spotlight for the TAC-KBP Entity Linking Task. In
Proceedings of the TAC-KBP 2011 Workshop.
Gaithersburg, USA.
Ounnas, A., 2010. Enhancing the Automation of Forming
Groups for Education with Semantics
. PhD Thesis.
University of Southampton. Available at:
http://eprints.soton.ac.uk/171641/.
Ounnas, A., Davis, H. & Millard, D., 2009. A framework
for semantic group formation in education.
Educational Technology & Society, 12(4), pp.43–55.
Rao, D., McNamee, P. & Dredze, M., 2013. Entity
linking: Finding extracted entities in a knowledge
base. In T. Poibo et al., eds. Multi-Source,
Multilingual Information Extraction and
Summarization. Berlin, Heidelberg: Springer Berlin
Heidelberg, pp. 93–115.
Rodriguez, M. A. & Neubauer, P., 2011. The Graph
Traversal Pattern. In S. Sakr & E. Pardede, eds. Graph
Data Management: Techniques and Applications. IGI
Global.
Wells, C. E., 2002. Teaching Teamwork in Information
Systems. In E. Cohen, ed. Challenges of Information
Technology Education in the 21 st Century. pp. 1–24.
Werbel, J.D. & Johnson, D. J., 2001. The Use of Person –
Group Fit for Employment Selection: A Missing Link
in Person – Environment Fit. Human Resource
Management, 40(3), pp.227–240.
Wetz, P. et al., 2012. Matching Linked Open Data Entities
to Local Thesaurus Concepts. In Proceedings of the I-
SEMANTICS 2012 Posters & Demonstrations Track.
pp. 6–11.
Wilkinson, I. A. G. & Fung, I. Y. Y., 2002. Small-group
composition and peer effects. International Journal of
Educational Research, 37(5), pp.425–447.
Winter, M., 2004. Developing a Group Model for Student
Software Engineering Teams. MSc Thesis. University
of Saskatchewan. Available at: http://hdl.handle.net/
10388/etd-07052004-140018.
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
190