A Systematic Mapping Applied to MOOC's Study
Alexandre Furtado Fernandes, Jorge Cardoso and Maria José Marcelino
CISUC, Department of Informatics Engineering, University of Coimbra, 3030 290 Coimbra, Portugal
Keywords: MOOC, QoS, QoE, Systematic Mapping.
Abstract: MOOC platforms are Web-based learning environments which allow a global participation on a large scale
and with free access. The paradigm presents itself as a new teaching trend, changing the way education can
be offered and funded worldwide. Many institutions are now investing in this teaching mode. However,
since it is a web-based tool, the connection performance can impact both the way learning is experienced by
the student as well as the operation of the platform. The “Quality of Experience” (QoE) concept has been
widely used to refer to how users describe a service they have used while the “Quality of Service” (QoS)
concept deals with the technical performance parameters that are associated with the connection quality.
This paper starts the process of developing a systematic mapping around MOOC platforms, and QoS and
QoE concepts, aiming to provide an overview of the current state of research on these issues.
1 INTRODUCTION
Starting with the emergence and operation of
MOOC platforms (Massive Open Online Courses),
or online courses, open coursesand mass education,
the need to conduct studies of this new concept of
teaching widens itself. "Massive" means it can be
attended simultaneously by thousands of students;
"open" means that anyone can do it and "online"
means that the courses are Web-based and in a non-
attendance mode. Siemens et al (2010, p. 4) sets this
learning opportunity as being: “An online
phenomenon, gaining strength over the past two
years, a MOOC integrates social network
connectivity, the facilitation of a recognized expert
in a distinct field of study, and a collection of free
access online resources (Siemens et al., 2010, p. 4)".
MOOC has been showing itself as the recent
educational phenomenon. It represents a possible
rupture in higher education in terms of online
teaching and learning for some students (Mota et al.,
2012). Such enthusiasm necessarily requires some
caution, due to the fact that this concept is extremely
recent (about to six years) and is still in the process
of consolidation.
MOOCs are essentially characterized by being
open and scalable courses. “Open” means that
students are eligible to participate even if theyare not
registeredin the educational institution that promotes
the course. This possibility however is associated
with minimum skills requirements on the part of the
participant towards the use of computers as well as
the technological infrastructure, preferably with
broad band internet access that allows navigation
without major problems (Mota et al., 2012). As for
scalability, the courses must have the appropriate
layout to meet an exponential number of
registrations, which may reach thousands every time
they are offered.
Furthermore, the accession of major universities
to this model shows that it is an irreversible trend of
learning. The main examples that may be presented
are the universities of Stanford, Columbia, Duke,
Princeton, among others with the “COURSERA”
platform; “UDACITY”, which was born of an
experiment at Stanford University; universities as
MIT, Harvard, the University of Texas, Berkeley
and Georgetown with the “EDx”, etc.
Moreover, British universities launched the
Future Learn, created by the Open University and
with the participation of Birmingham, the British
Library, the British Museum and several other
institutions.
Liyanagunawardena et al (Liyanagunawardena et
al., 2013)
present an article with a systematic review
of the main MOOC's publications, highlighting
forty-five articles reviewed. Such review was
classified into eight different areas of interest:
introductory, concept, case studies, educational
theory, technology, participant focused, provider
444
Furtado Fernandes A., Cardoso J. and José Marcelino M..
A Systematic Mapping Applied to MOOC’s Study.
DOI: 10.5220/0005483904440449
In Proceedings of the 7th International Conference on Computer Supported Education (CSEDU-2015), pages 444-449
ISBN: 978-989-758-108-3
Copyright
c
2015 SCITEPRESS (Science and Technology Publications, Lda.)
focused, and other, and provides an analysis of the
kinds of publication, year, and authors
.
In order to get a better understanding about the
possibilities and limitations, and deepen the studies
in this new concept of teaching and learning, a study
of Systematic Mapping was conducted (Kai Petersen
et al., 2008, p. 71). This mapping makes a survey of
the state of the art in the research on MOOC
platforms and what has been discussed in the areas
of Quality of Service (QoS) and Quality of
Experience (QoE). According to Drogseth (2005,
p.60) the QoE analyzes user-centric approaches,
while QoS investigates approaches focused on
technology.
Quality of Experience (QoE) is used to describe
how students evaluate a service, and on the other
hand, Quality of Service (QoS) describes the
technical performance parameters which reflect the
quality of an internet connection.
Quality of Experience (QoE) is a concept that
has been extensively explored in the evaluation of
communication and learning of platform based on
internet access systems, but the term has not yet
been broadly defined. Subjective parameters, such as
expectations, emotions, usability and context should
be taken into consideration for its definition.
Soldani et al (2007, p.3) defines QoS as "... the
capacity of the network to provide a service to a
guaranteed level." Already QoE "is how a user
perceives the usability of a service when in use -
how pleased he or she has a service in terms of, for
example, usability, accessibility, retention capacity
and integrity of service" (Soldani et al., 2007, p.3).
In the same line of thought, Moebs (2008) defines
QoE as "the degree to which a system meets the tacit
and explicit expectations of the user for the
experience."
This article is organized as follows. Section 2
describes the methodology used to create the
mapping. Section 3 presents the interpretation of the
results. Section 4 presents the conclusions and
further work.
2 SYSTEMATIC MAPPING
Kitchenham, (2007) elucidates that a study of
systematic mapping is a method that provides an
insight into a particular area of research, in order to
allow quantifying, identifying and analyzing results,
establishing proof of research on a particular theme.
The systematic mapping can also be defined as a
literature survey that identifies which types of
studies can be treated by a systematic review, the
place where they were published, in which databases
they were indexed and what their results are
(Petticrew and Roberts, 2006).
The study was conducted in accordance with the
guidance provided by Petersen et al (2008, p.2) and
the procedure consists of a five-step process (Figure
1): definition of the research issue, research
achievement, sorting, keywords, summary and data
extraction and mapping. Each step of the process
generates specific result (second line of the image in
Figure 1).
Figure 1: Sistematic mapping process.
2.1 Definition of Research Parameters
In a systematic mapping, the research questions
(RQ) define the scope and focus of the work. The
following issues have been identified:
RQ1. What kind of work has beendone in the
MOOC field and in relation to major platforms
that offer courses in this format?
RQ2. What is the current state of research on
QoS and QoE applied to MOOC and education
in general?
The answer to question RQ1 will provide an
overview of what the academic community has been
producing by topic.The RQ2 will provide a vision of
what was produced in relation to the quality of
service and experience applied to educational
platforms and in particular to the platforms that offer
MOOCs.
2.2 Conducting the Research
2.2.1 Search Strategy and Data Source
The major focus of the research was based on
MOOC, that emerged from 2008 with the course
"Connectivism and Connective Knowledge”,
launched by George Siemens and Stephen Downes,
specifically to expand the discussion on the new
theory of learning they created (i.e. the
Connectivism). In consideration of that, every search
ASystematicMappingAppliedtoMOOC'sStudy
445
in this work has been made considering publications
from the year 2008.
Since the initial exploratory period of the study
up to the following stages of the searches, the
strategy adopted was to consult the following
databases for a search of the most relevant articles
and publications: IEEEXplore; ACM Digital
Library; ScienceDirect, because these academic
databases add great number of conference
proceedings publications, studies in science of
human and computer sciences.
2.2.2 Primary Studies
Searches were initially made in the database of
digital bookstore IEEExplore using keywords such
as: "MOOC"; “e-Learning”; "QoS"; "QoE” and
small conjugations for key words like “quality of
service and learning ".
This pilot project totaled 36 keys. The search for
the key words was conducted using predefined parts
of the articles, namely:
Full Text – It was sought the presence of the key
throughout the article.
Summary - It was sought the evidence of the key
only in the summary of the article.
Keywords- It was sought the keywords among
the ones defined by the author.
Title – The existence of the key described in the
article title.
In the end, 144 searches were carried out in the
IEEEXplore library.
The first item described as full-text was not
considered in the assessment criteria for selection of
articles, considering the broad range of information
results culminating in a large number of articles.The
importance of its use at this moment is justified to
permit an initial visualization of the amount of
articles that bring the keys in any part of the body of
the text.
112,543 articles were obtained wherever the
existing terms in the used keys had been located in
any part of the text.
30,920 articles were obtained wherever the key
lies in the summary, the keywords or the title.
Based on the first results obtained in this pilot
study, and to ensure the final results would be valid
and reliable, the research process moved towards a
refinement of the keys.
2.3 Screening of Papers
The following keywords were used: QoS; QoE;
MOOC; UDACITY; COURSERA; EDX; e/m/u/b-
Learning; dropout; waiver; evasion; course; open
course and several synonyms, derived words.
A grouping of keys process was also applied
based on the research issues, which resulted in three
groups:
RG1 - "quality of service" associated with the
term "education".
RG2 - the term MOOC and major platforms.
RG3 - "quality of experience" associated with
"education".
(RG meaning: Research Group)
At this stage of the research, although there were
only 44 simple search keys with few conjugated
terms, such as: “quality of service” and “MOOC”,
“quality of service” and “e-learning” for RG1,
“MOOC” and “online, massive open online course”,
“edx and MOOC”, “courser” and “udacity” and
“mitx” for RG2, “quality of experience and
MOOC”, “quality of experience” and “e-learning”
for RG3, the terms were used in order to enhance the
refinement of the searches in relation to the searches
previously made.
At the end of this phase of searching, with the 3
(three) previously defined key groups with
respectively 9, 27, 8 keys each, we obtained the
figures presented in Table 1:
Table 1: first results.
Still working with the three groups of keys, it was
made a reduction in the number of keys from the
merge of the existing keys in each group.
From that point on, for each of the three groups
(RG1, RG2, RG3), the keywords, their synonyms,
derivatives and related words were grouped with the
identifier OR and with the identifier AND,
generating the following keys:
RG1: ("distance learning" OR "online learning"
OR "distance education" OR "online course" OR
"e-learning" OR MOOC OR "massive open
online course") AND "quality of service");
RG2: (“courser” OR “udacity” OR “edx” OR
“mitx” OR “harvardx”) AND ("massive open
online course" or MOOC);
RG3: ("distance learning" OR "online learning"
CSEDU2015-7thInternationalConferenceonComputerSupportedEducation
446
OR "distance education" OR "online course" OR
"e-learning" OR MOOC OR "massive open
online course") AND ("quality of experience").
2.4 Keywording using Abstracts
2.4.1 Repetition of Articles
The next step was to group the articles by their titles
alphabetically.The existence of repeated articles was
observed, due to the following circumstances:
The same article found in more than one of the
databases consulted;
An article where the key took place both in the
"Summary" field " and in the “keywords" field;
An article found in more than one of the keys.
2.4.2 Selection of Relevant Articles
In order to improve the search results obtained in
digital libraries, it was necessary to set up inclusion
and exclusion criteria of items, taking into account
the research issues defined at the beginning of the
work (Kitchenham, 2007).
Inclusion Criteria
Proposed search keys were used in database
consulting systems, and every article that was found
has been considered as long as it meets the following
criteria: the article relevance regarding the research
questions; periodicals and full papers published in
journals, conferences or symposiums; researches
which associate QoE and QoS with MOOC and/or
educational platforms.
Exclusion Criteria
Articles with titles that clearly show that they were
not related with any of the defined issues were
disregarded; studies which were not written in
English; published studies in editorials, prefaces,
summary of articles, interviews, news and reviews.
Inclusion and Exclusion criteria have been
complied with every article by reading the title and
the summary. Whenever that was insufficient, then
the introduction and/or conclusion of the article was
read.
After the first selection, we obtained 27 articles
that have been reviewed and submitted to the last
phases of the methodology.
3 RESULTS
3.1 Data Extraction and Mapping
Process
Seeking to answer initial questions of research, there
is an evolution in the number of articles that deal
with the subject, and in 2008, 2009 and 2010 there
were three articles published, five in 2011, seven in
2012 and six in 2013. Up to May 2014, the time of
completion of this work, no publication was listed in
the digital libraries, in 2014.
There were 17 articles found in IEEExplore; 2 on
ScienceDirect; 5 in the ACM Digital Library (DL).
We also observed the occurrence of articles that
have been found in more than one digital library: 2
articles in IEEE and ACM DL and 1 in Science and
ACM DL.
It is believed that the greatest number of articles
found on IEEExplore may be due to the fact that it is
a specialized library in the areas of data network and
electronic and the QoS and QoE terms are often
associated with this field of study.
3.2 Systematic Map
The classification proposed by Wieringa et al
(2006), subdivided into categories, which proposes
criteria for the evaluation and classification of
articles was used to determine the type of research
evaluated in each article. The classification
suggested may be considered a general
classification, which may be applied to any kind of
research.
The articles were reconsidered individually and
assigned into the categories that most reflect their
content. Three articles were classified as validation
research; 10 as evaluation research; 5 as suggested
solution; 3 as philosophical work and 4 as
experience work articles (Table 2).
Table 2: Categories x years.
ASystematicMappingAppliedtoMOOC'sStudy
447
3.3 Analysis of Articles
After the classification, it was observed that the
articles address a wide range of topics related to the
proposal of this study, however only two articles that
specifically treat MOOCs have been found and both
are studies of applied MOOC.
The first study describes the production of a
course using MOOC on the Coursera platform and
discusses the improvement of the quality and
productivity of software professionals from the
continuity of studies using the platform (Schmidt et
al., 2013). The second study reports the experience
of teaching through a MOOC and identifies positive
and negative points (Egerstedt, 2013).
Several studies deal with the e-Learning theme
and its quality. Nevertheless, such studies are more
associated with the quality of service in a context
that addresses technical performance parameters
such as network conditions that may alter the
connection, quality of content, network architecture,
multimedia systems for the cellular phones segment
and specific algorithms for network control. Some
works seek alternatives to enhance the performance
of the connection as a solution to achieve the
improvement of e-Learning Services.
The works developed by Moebs (Moebs and
Mcmanis, 2008; Moebs, 2008) identify the factors
that most influence the QoE in the learning
environment and propose a model for measurement
of QoE. They present a proposal with five main
components of QoE: effectiveness, usability,
efficiency, expectations and context. These five
components have been related with the user
experience and therefore they can identify the
factors that most influence the QoE in a learning
environment. Thus, those factors are used to develop
a hypermedia learning system.
Another discussed point was the quality of
educational products (Rossi and Mustaro, 2012). It
was based on quality of service, but the presented
model works as a tool to improve the phases of
development, self-evaluation or certification of
educational products.
Among the studies that specifically address
issues associated with the quality of experience, we
find those that measure, evaluate and validate the
QoE. Menkovski, V. et al (2010) present a
methodology for measuring QoE, associated with
the quality of video received by the user. In another
work, Menkovski, Exarchakos, Georgios, Liotta, A
(2010) present a method of construction of a QoE
assessment model that operates continuously and in
real-time.
Kalliris et al (2011) investigate the
implementation and validation of QoE to streaming
live internet courses.
In every article, QoE was associated with a more
technical parameter, such as video and bandwidth of
the connection.
Some articles investigate the relationship
between QoS and QoE, like Muntean et al (2010)
which draws up a method for the mapping of QoS
parameters in order to improve the QoE of e-
Learning applications for final users.
Wang et al (2012) and Dursun et al (2013) used
the SERVQUAL model to assess the quality of a
distance education service, so that the first work
used a combination with another type of assessment,
SERVPERF.
4 CONCLUSIONS
The literature review work on MOOCs, QoE and
QoS was carried out systematically as a result of the
recent MOOC offers. It is considered that the study
of MOOC and creating a model for evaluating the
Quality of Experience will allow a better use for the
students and will ensure the quality of the courses
offered.
An important aspect of the systematic mapping
conducted revealed that few studies address specific
issues to QoE. Only 2 research studies proposed
specific assessment mechanisms. These numbers
support the importance of further studies in this area.
Not only there was a need of searching
individually QoE, MOOC and about teaching
platforms, but also articles that explain the concepts
and stablish background concerning important
points. Some articles were reached from the
bibliographies of the ones related by MS, others
from searches of individual keys, sometimes
containing a specific term, sometimes the name(s) of
the author(s) in particular.
As future work, we will apply again the
methodology to new databases, for example the B-
On.pt for the period 2014, and develop mechanisms
study to assess how these vectors can affect the
quality of learning in MOOCs.
ACKNOWLEDGMENTS
AFF acknowledges the doctoral scholarship
supported CNPq/CAPES - Programa Ciência sem
Fronteiras CsF (8990/13-1) and authorized
CSEDU2015-7thInternationalConferenceonComputerSupportedEducation
448
retirement by IFSUDESTE MG (477/17.05.2013).
REFERENCES
Drogseth, D. 2005. Business Alignment Starts with
Quality of Experience. Business Communications
Review (March), 60-64.
Dursun, T.; Oskaybaş, K.; Gökmen, C. 2013. The Quality
of Service of The Distance Education. Procedia-Social
and Behavioral Sciences, V. 103, p. 1133-1151.
Egerstedt, M. 2013. Controls for The Masses (Focus on
Education).Control Systems, IEEE, V. 33, N. 4, p. 40-
44.
G. Siemens, D. Cormier, A. McAuley, B. Stewart. 2010.
The MOOC Model for Digital Practice. Acessado em
15/01/2014 em http://davecormier.com/edblog/
wpcontent/uploads/MOOC_Final.pdf.
Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael
Mattsson. 2008. Systematic Mapping Studies in
Software Engineering, in 12th International
Conference on Evaluation and Assessment in Software
Engineering, 2008, pp. 71-80.
Kalliris, G., Dimoulas, C., Veglis, A. A., Matsiola, M.
2011. Investigating Quality of Experience and
Learning (QoE & Qol) of Audiovisual Content
Broadcasting To Learners over Ip Networks. p. 836-
841.
Kitchenham, B. & Charters, S. 2007, Guidelines for
performing systematic literature reviews in software
engineering, Technical Report EBSE-2007-01, School
of Computer Science and Mathematics, Keele
University.
Liyanagunawardena, T. R., Adams, A. A. and Williams, S.
A. 2013. MOOCs: a systematic study of the published
literature 2008-2012.
Menkovski, V.; Exarchakos, Georgios; Liotta, A. 2010.
Online Qoe Prediction. In: Quality of Multimedia
Experience (QoMEX), Second International
Workshop On. p. 118-123.
Menkovski, V. Et al 2010. Quality of Experience Models
for Multimedia Streaming. International Journal of
Mobile Computing and Multimedia Communications
(IJMCMC), V. 2, N. 4, p. 1-20.
Moebs, S., & Mcmanis, J. 2008. A Learner, Is A Learner,
Is A User, Is A Customer-So What Exactly Do You
Mean By Quality Of Experience?. Inproceedings Ah
2008 Workshop Technologies for Mobile and Wireless
Adaptive Elearning Environments.
Moebs, S. A. 2008. A Learner, Is A Learner, Is A User, is
A Customer: Qos-Based Experience-Aware
Adaptation. In: Proceedings of The 16th Acm
International Conference On Multimedia. ACM, p.
1035-1038.
Mota, R.; Inmorato, A. 2012. MOOC, uma revolução em
curso. Jornal da ciência, nov. 2012. Disponível em:
<http://www.jornaldaciencia.org.br/ Detalhe.jsp?id=
851>. Acesso em: fev. 2014.
Muntean, V. H.; Otesteanu, M.; Muntean, G.-M. 2010.
Qos Parameters Mapping For The E-Learning Traffic
Mix In Lte Networks. In: Computational Cybernetics
AND Technical Informatics (ICCC-CONTI),
International Joint Conference On. p. 299-304.
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M.
(2008). Systematic mapping studies in software
engineering. In 12th International Conference on
Evaluation and Assessment in Software Engineering,
Vol. 17, p. 1.
Petticrew, M., & Roberts, H. 2006. Systematic reviews in
the social sciences. A practical guide. European
Psychologist, 11(3), p. 244-245.
Rossi, R.; Mustaro, P. N. 2012. Applying Quality
Approaches on Ict-Based Educational Products.
Proceedings of Informing Science & IT Education
Conference (InSITE). p. 249-264.
Schmidt, D. C.; Mccormick, Z. 2013. Producing and
Delivering A Coursera MOOC on Pattern-Oriented
Software Architecture For Concurrent And Networked
Software. In: Proceedings of the Companion
Publication for Conference on Systems, Programming,
& Applications: Software For Humanity. p. 167-176.
Soldani, D., Li, M., & Cuny, R. (2007). QoS and QoE
management in UMTS cellular systems. John Wiley &
Sons.
Wang, Y. Et al 2012. Evaluating the Quality of Distance
Education Services by Using Modern Information
Technology. In: Services Computing Conference. p.
192-199.
Wieringa, R., Maiden, N. A. M., Mead, N. R. & Rolland,
C. 2006. Requirements engineering paper
classification and evaluation criteria: a proposal and
a discussion, Requir. Eng.11(1). p. 102–107.
ASystematicMappingAppliedtoMOOC'sStudy
449
