![](bg7.png)
gory. However, performing such detections strongly
couples ReLiS to the underlying database used, which
hampers its portability.
The tool logs all major operations performed by
the user, so he can revert them, e.g., remove a spe-
cific set of wrongly imported papers or clear the re-
sults of a screening phase, especially when there are a
lot of disagreements in the study selection among the
reviewers.
5 EVALUATION OF RELIS
We have performed a preliminary informal qualita-
tive evaluation of the correctness of the implementa-
tion and the usefulness of the architecture decisions
in ReLiS. In the context of a graduate course on em-
pirical methods in software engineering at our univer-
sity, all students are required to conduct a SMS on
different research topics as part of their project. In the
Fall 2016 edition, there were in total 8 groups of 3
to 4 students with no previous experience in conduct-
ing SR. Students were asked to use ReLiS for their
SMS project. This allowed us to have test subjects
to verify that the tool was functionally correct. This
also gave us the opportunity to assess whether the
category types, to generate the data extraction forms,
built-in ReLiS are sufficient to cover all eight SMS.
All groups used every type of category, but none ap-
plied advanced constraints.
We were also interested in determining how on-
the-fly installation and configuration of projects was
used. For that, we extended ReLiS to log every in-
stallation of projects, while tracking the modifications
performed in the project-specific model. We noted
that all re-installations were aimed at improving the
classification scheme, since the screening procedure
should be fixed from the beginning and changing it
during the process may bias the SR. Students found
that automatically re-installing their project was very
helpful. This was expected because, during the learn-
ing process, they required several iterations before ar-
riving at the desired classification scheme. The most
positive feedback was that they were able to re-install
a modified project without loosing the information on
papers they had already classified. However, we noted
a larger amount of deactivated categories than antic-
ipated. We discovered that it was mainly because of
misunderstanding the syntax of the DSL.
6 RELATED WORK
There are several tools designed to support the con-
duction of SR that were surveyed in (Marshall and
Brereton, 2013; Al-Zubidy et al., 2017). Some tools
specialize only in some desirable SR features, such
as reference management and text-mining techniques
to help extract relevant data from papers (Felizardo
et al., 2010). Other tools are more specific to how
SR are conducted in software engineering and aim to
support the whole SR process. However, they still
have important features not properly supported. Par-
sifal
2
is the most similar tool to ReLiS. It is an online
tool that allows researchers to conduct only SLR col-
laboratively. However, it only provides simple cat-
egory types for the data extraction forms and does
not support, for instance, dependent lists or subcate-
gories. SLRTOOL (S. Barn et al., 2014) also supports
the whole SR process, but does not allow for multiple
screening phases and requires adding studies manu-
ally. The key difference of all these tools and ReLiS
is the flexibility of modifying the configuration of the
SR procedure on-the-fly, which makes it able to pilot
reviews. It is the only tool that uses MDD to adapt
the tool to a specific SR by customizing the whole SR
process and to generate the most advanced data ex-
traction forms thanks to that DSL.
(Barat et al., 2017) have used MDD to support SR.
They designed a DSL to represent research literature,
with a metamodel to describe the core concepts of the
review and associated process. However, their DSL is
only used to collect the data of the SR process, not to
generate a tool like ReLiS.
Related to the architecture of our approach, CMS
are very generic frameworks that heavily rely on ex-
tensions developed outside the tool environment, as
discussed in Section 4. ReLiS project configuration
models are designed within ReLiS which requires no
client-side installation of any tool, as it runs com-
pletely in a web browser, and projects are installed
automatically. Base platforms to build web applica-
tions (e.g., Joomla and WordPress) support easy in-
stallation of custom extensions. However, their de-
velopment requires the user to install the appropriate
tooling and programming environment. Also, they are
very generic and provide too many options that are
not used in specific application domains, such as SR.
The closest work to ours in using MDD for updat-
ing exiting application with on-the-fly installations is
JooMDD (Priefer et al., 2016). It relies on a DSL
where models get automatically generated as Joomla
extensions, in order to raise the level of abstraction
for web extensions development.
2
https://parsif.al/
MODELSWARD 2018 - 6th International Conference on Model-Driven Engineering and Software Development
558