A SYSTEMATIC LITERATURE REVIEW ON SOFTWARE
PRODUCT LINE QUALITY
Carmen Moraga, Mª Ángeles Moraga, Marcela Genero and Mario Piattini
ALARCOS Research Group, University of Castilla-la Mancha, Paseo de la Universidad 4, 13071, Ciudad Real, Spain
Keywords: Software product line, Quality, Systematic literature review.
Abstract: This paper provides a summary of a systematic literature review (SLR) which was performed to find out
which quality characteristics have been seen as relevant in the literature, in the context of Software Product
Lines (SPLs). This SLR, which encompasses works published between 01/01/1995 and 30/05/2008,
identified 319 papers in 5 digital libraries, as well as 8 papers which were added by experts as grey
literature. After analyzing the papers found, 43 were selected as primary studies. After an analysis of the 43
primary studies, we found that maintainability and reusability are the most relevant quality characteristics in
this context. There is increasing interest in the evaluation and improvement of quality in the development of
software which follows the SPL approach. In spite of this, no quality model specific to SPLs has yet been
found which would meet with a consensus of general approval. For that reason, a standard quality model for
SPLs is needed. Techniques for assessing and improving such characteristics must also be provided.
1 INTRODUCTION
The concept of quality is particularly pertinent in the
context of SPL, because common software
components can, and indeed must, be developed
with a higher level of quality, since they are used in
every single product. That being the case, it evident that
quality in SPLs improves the quality in each product
created. Moreover, quality improvement in an SPL
can increase re-use, and will consequently reduce
effort, enhance product reliability, and shorten time-
to-market.
This paper provides a summary of an SLR
carried out to gather the quality characteristics that
have been considered relevant in the literature
related to the development of SPLs, along with the
techniques that have been used to deal with them. In
order to ensure that it is both systematic and
rigorous, this SLR has been conducted by following
the guidelines provided in (Kitchenham and
Charters, 2007). The complete report of this SLR can
be found at http://alarcos.esi.uclm.es/SLR-
SPLquality.
The paper is organized as follows: in Section 2
the review process is presented in detail. The main
results obtained from the SLR are reported in
Section 3 and finally Section 4 presents some
conclusions and suggestions for further research.
2 REVIEW PROCESS
In this section we describe the activities performed in
the “Planning the review” and “Conducting the review”
phases, based on (Kitchenham and Charters, 2007).
2.1 Planning the Review
In this phase we define the review protocol,
consisting mainly of: formulation of the research
question, selection of sources, definition of the
search string, definition of inclusion or exclusion
criteria and the selection of the information
extraction strategy.
The research question that we wished to address
is: “What are the quality characteristics addressed by
researchers in SPL development, and what techniques
are proposed for dealing with them?”.
We selected the following digital libraries to
perform the searches: IEEE Computer Society, ACM,
Science@Direct, Wiley InterScience, and SCOPUS.
We also considered as grey literature some papers
considered relevant by experts, but which were not
found in the digital sources mentioned above.
The major search terms for building the search
string were: “Product Line”, “Quality” and
“Software”. In addition, to refine the search string,
we selected specific quality characteristics obtained
269
Moraga C., Ángeles Moraga M., Genero M. and Piattini M..
A SYSTEMATIC LITERATURE REVIEW ON SOFTWARE PRODUCT LINE QUALITY.
DOI: 10.5220/0003471302690272
In Proceedings of the 6th International Conference on Software and Database Technologies (ICSOFT-2011), pages 269-272
ISBN: 978-989-8425-77-5
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

from pilot searches we have done beforehand and
some papers provided by experts, which were
considered as “grey literature” in this SLR, as well
as other quality characteristics that in our opinion
can be relevant in this context. The alternative
spellings or terminology related to the major terms
are shown in Table 1.
Table 1: Search string.
Major terms
Alternative terms
Product line
Product families OR product family OR system
families OR system family OR product lines
Quality
Maintainability OR flexibility OR reusability OR
modifiability OR changeability OR functionality
OR usability OR efficiency OR portability OR
suitability OR accuracy OR interoperability OR
security OR “fault tolerance” OR recoverability
OR understandability OR comprehensibility OR
learnability OR operability OR attractiveness OR
“time behaviour” OR “resource utilisation” OR
analysability OR stability OR testability OR
adaptability OR installability OR replaceability
OR complexity OR extensibility OR scalability
OR customisability OR availability OR reliability
OR mobility OR integrability OR safety
Software -------
The relevant information from each paper used to
answer the main research question addressed by this
SLR was obtained from a data extraction strategy,
consisting of the following six criteria: Quality
characteristics, object of study, application domain,
type of proposal, research method and type of support.
2.2 Conducting the Review
Firstly, we found 319 papers. Afterward, we
analysed the title and abstract and selected 76 of
them. Finally, 43 papers were chosen as primary
studies, by applying the inclusion and exclusion
criteria. The classification of the 43 papers included
in this SLR can be found in http://alarcos.
esi.uclm.es/SLR-SPLquality.
3 RESULTS
In the following paragraphs we present an analysis of
the results obtained for each criterion used to extract
the relevant information from the 43 selected papers.
3.1 Quality Characteristics
At the beginning, in the planning of the review 37
quality characteristics were proposed, but only 15 of
these were found in the SLR. 11 other quality
characteristics were obtained in the SLR, but these had
not been proposed previously.
The final set of quality characteristics is the
following: functionality, reliability, usability,
efficiency, maintainability, portability, reusability,
effectiveness, availability, configurability, integrability,
scalability, modularity and applicability.
Regarding to the most important quality
characteristics in the context of SPLs, we shows that
maintainability and reusability are the most relevant
ones, followed by reliability, functionality,
portability and availability.
3.2 Object of Study
Various software artefacts were considered when we
were setting out to define or evaluate quality
characteristics in the context of SPLs. We looked at
software artefacts, i.e. software architecture of the
product line, software products obtained from SPLs
and Others. As (Bass et al., 1998) states, “software
architecture of the product line” is the structure or
structures of the system, which are composed of
software components, the externally visible properties
of those components, and the relationships between
them. A “software product obtained from SPLs”
studies the quality of the products obtained using the
SPL; “Others” includes the papers relating to core
assets, core asset requirements or product line
requirements.
3.3 Application Domain
The common domains are the embedded systems,
which can be found in telecommunications, the
automobile industry, electronic systems, medical
machines and so on. In our study, the most typical is
the telecommunications domain, with 11 papers. The
telecommunications domain is a set of
telecommunications functionalities, which in turn
consists of other domains, such as switching, protocols,
telephony and network. A telecommunications SPL is
a specific set of software systems that provides some
of these functionalities (Clements and Northrop,
2001).
The SPL domain that is in second place as regards
prevalence is the medical one, with 5 papers. In
addition, 3 papers have been classified as “general
product line” (i.e. they are not specific to a specific
domain, because the purpose was not really.
3.4 Type of Proposal
The type of proposals, that is, the type of techniques,
has been classified as:
ICSOFT 2011 - 6th International Conference on Software and Data Technologies
270

 Quality definition: when only the quality
characteristics which are desirable for SPL
are proposed, specified or defined.
 Quality improvement: when the proposal attempts
to improve the quality characteristics.
 Quality assessment using scenarios: when the
quality characteristics are evaluated through
scenarios.
 Quality assessment using metrics: when the
quality characteristics are evaluated by metrics.
 Quality assessment using scenarios and metrics:
when scenarios and metrics are used to
evaluate quality characteristics.
We can conclude that the majority of the papers
considered only propose a list of quality characteristics
which are relevant for evaluating SPL quality. The
most widespread evaluation technique in the field of
SPLs are scenarios and, to a lesser extent, quantitative
metrics.
3.5 Research Method
We have considered five research methods, which
reveal the type of evidence of the proposal; this
shows how much evidence relating to SPL quality
existed. The following research methods are ordered
from their lowest to their highest levels:
 Speculation: the paper describes the quality
characteristics, without presenting any study
or example that would indicate its feasibility
in software projects.
 Example: the paper describes approaches or
proposals, showing their utility by means of
an example.
 Survey: the paper sets out a questionnaire filled
in by experts.
 Experience/industrial report: the paper
describes a real team which is developing
software in industry using the proposed SPL
quality approach.
 Experimentation: the paper evaluates the SPL
quality approach in some detail through an
experimental.
Analyzing the papers, we can conclude that the
level of empirical evidence in the different approaches
proposed in the context of SPL quality is very low. The
majority (44.19%) of the proposals are illustrated only
with examples. Approximately 24% of papers are
supported by experimentation. In this case, all of the
10 papers that were found validate the approaches
presented through the use of case studies.
3.6 Type of Support
The proposal was considered to have a tool when the
authors affirmed that a new tool had been created or
when one or several existing tools was/were able to
support their proposal. 20 (47%) of the 43 papers
selected provided a support tool. The majority of the
papers which did not propose a tool were those which
were classified as “quality definition” in the “type of
proposal” category, i.e. those papers which were
limited to only defining quality characteristics did not
propose any evaluation or improvement techniques.
3.7 Additional Results
Crossing the criteria application domain, research
method, type of proposal and type of support, we
can conclude that:
 The highest number of approaches validated
through experiments exists in the
Telecommunications domain, which is followed by
the “Embedded systems” category.
 In the various Application domains,
approximately the same number of proposals is
supported by a tool as those which are not
supported by a tool.
 The research method used to validate the proposals in
each application domain is almost the same.
 Quality improvement proposals exist only in
the Embedded systems and the General product
line domains.
 In the Telecommunications and Medical
domains, a high number of papers exist that
only define quality characteristics.
Crossing “Quality characteristic” and other criteria we
found that:
 For most of the quality characteristics, the “Object
of study” was “Software architecture of the
product line”. However, for the Effectiveness
characteristic, the main “Object of study” was
“Software product obtained from SPLs” and for
Applicability, it was “Others”.
 For the majority of quality characteristics, the
“Application domain” used to the greatest
extent was “Embedded system industries”.
 For the greater proportion of quality
characteristics, the main “Type of proposal”
was “Quality assessment using scenarios” and
also “Quality definition”. However, for the
Usability and the Applicability characteristics, it
was “Quality assessment using metrics”.
 The “Research method” that enjoyed most
widespread use for the greater part of the
characteristics was “Example”.
A SYSTEMATIC LITERATURE REVIEW ON SOFTWARE PRODUCT LINE QUALITY
271

3.8 Recommendations
In this point, taking into account the results
obtained, as well as our experience, some
recommendations are presented in the context of
SPL quality research. These are:
 A standard quality model for SPLs is needed,
including characteristics and sub-characteristics,
bearing in mind the quality characteristics
researched in the context of SPL. In addition,
techniques for assessing and improving such
characteristics must be provided.
 SPL development must be spread more widely
in the academic context, to make controlled
experiments by students more possible. These
are necessary if we are to find preliminary
results before validating the proposal in
industrial environments.
 A repository of easily accessible examples of
SPLs that have already been developed is
needed. They can be used as examples in the
empirical validation of the approaches proposed for
the evaluation and improvement of SPL quality.
The development of SPLs consumes a great
deal of resources, something which the majority
of researchers lack in their universities. They
consequently find it impossible to validate their
proposals empirically.
 Further empirical validation by performing case
studies or controlled experiments is essential if
the level of evidence in this field is to be
increased. If that were carried out, the
approaches proposed for evaluating or
improving SPL quality could become credible
and would consequently be adopted by
practitioners in industrial environments.
4 CONCLUSIONS AND FUTURE
WORK
In this paper, we have presented a review of the
main quality characteristics considered in the
context of SPL development, by performing an SLR
on the existing literature. We have found 43 primary
studies in 5 digital libraries. The main findings
organised by the selected data extraction criteria are
shown in Table 2.
Table 2: Summary of the main findings.
Criteria
analysed
Main findings
Quality
characteristic
The quality characteristics in the majority of the works
published are maintainability and reusability, followed
by reliability, functionality, portability and availability.
Object of
study
The quality focus of SPL development seems to centre
principally upon SPL architecture.
Application
domain
With regard to the SPL domain, the embedded systems
are extensively published, and are mainly
telecommunications infrastructures, medical machines
and electronic systems. The software and the
architecture of these domains are evaluated.
The Telecommunications industry is the domain most
frequently reported in our SLR.
Type of
proposal
The papers contain several metrics concerning the
quality of a software product line.
Although a high percentage (35%) of the proposals
limit themselves to proposing a list of characteristics,
without showing how to evaluate or improve them,
65% of the papers propose the use of metrics and/or
scenarios in the evaluation of quality, or a method to
improve the quality.
Research
methods
24% of the proposals present an experiment, which
means that the majority of the proposals lack empirical
validity.
Type of
support
Different tools have been defined in the articles, in
particular from the development of their own product
lines.
From the SLR we can conclude that the area of
SPLs is not yet mature and that our results are very
useful for practitioners and researchers who need
quality if they are using SPLs.
Future work is foreseen, which would take on the
building of a quality model for SPLs, supported by a
tool. There would be a validation of the quality
model by means of empirical studies.
ACKNOWLEDGEMENTS
This research has been funded by the following projects:
MEDUSAS (CDTI-MICINN and FEDER IDI-
20090557), ORIGIN (CDTI-MICINN and FEDER
IDI-2010043(1-5)), PEGASO/MAGO (MICINN and
FEDER, TIN2009-13718-C02-01), EECCOO
(MICINN TRA2009_0074), MECCA (JCMM
PII2I09-0075-8394), IMPACTUM (PEII 11-0330-
4414) and VILMA (JCCM PEII 11-0316-2878).
REFERENCES
Bass, L., Clements, P., Kazman, R., 1998. Software
architecture in practice. Boston, MA, USA.
Clements, P., Northrop, L., 2001. Software Product Lines:
Practices and Patterns. Boston: Addison-Wesley.
Kitchenham, B., Charters, S., 2007. Guidelines for performing
systematic literature reviews in software engineering.
Technical Report EBSE-2007-01, School of Computer
Science and Mathematics, Keely University.
ICSOFT 2011 - 6th International Conference on Software and Data Technologies
272