Checklist-based Inspection of SMarty Variability Models

Proposal and Empirical Feasibility Study

Ricardo T. Geraldi

, Edson Oliveira Jr.

, Tayana Conte

and Igor Steinmacher

Informatics Department, State University of Maring

a, Maring

a-PR, Brazil

Institute of Computing, Federal University of Amazonas, Manaus-AM, Brazil

Department of Computing, Federal Technological University of Paran

a, Campo Mour

ao-PR, Brazil

Keywords:

Coding, Empirical Study, Feasibility, SMarty, SMartyCheck, Software Inspection, Software Product Line,

Qualitative Analysis.

Abstract:

Software inspection is a particular type of software review applied to all life-cycle artifacts and follows a

rigorous and well-deﬁned defect detection process. Existing literature deﬁnes several inspection techniques

for different domains. However, they are not for inspecting product-line UML variability models. This paper

proposes SMartyCheck, a checklist-based software inspection technique for product-line use case and class

variability models according to the SMarty approach. In addition, it presents and discusses the empirical fea-

sibility of SMartyCheck based on the feedback from several experts. It provides evidence of the SMartyCheck

feasibility, as well as to improve it, forming a body of knowledge for planning prospective empirical studies

and automation of SMartyCheck.

1 INTRODUCTION

A Software Product Line (SPL) encompasses a set of

products for a particular market or mission (Capilla

et al., 2013). The SPL concept has been consolidated

over the last years as a non-opportunistic reuse strat-

egy (Linden et al., 2007; Capilla et al., 2013).

Nowadays, several organizations handle huge

amounts of information, thus depending on software

applications for a competitive market (Capilla et al.,

2013). Therefore, such organizations need non error-

prone information systems, able to change over spe-

ciﬁc market rules (Mello et al., 2014).

Software inspection is efﬁcient and mostly

adopted by industry (Fagan, 2002; Rombach et al.,

2008). The existing literature provides a few re-

cent studies concerned on the application of inspec-

tion techniques to SPL variability models (Cunha

et al., 2012; Mello et al., 2014). However, such

recently proposed techniques are not for inspecting

UML models. In this context, it is suitable to ap-

ply the Stereotype-based Management of Variability

(SMarty) approach (OliveiraJr et al., 2010), in combi-

nation with a software inspection technique, for re-

ducing potential defects at modeling SPL SMarty-

based variabilities. SMarty has been empirically eval-

uated in the last few years (Marcolino et al., 2014).

An inspection technique that takes as a basis SMarty

models become an interesting tool towards defect

identiﬁcation in early life-cycle SPL artifacts with

variabilities. Therefore, this paper presents the pro-

posal of SMartyCheck, a checklist-based inspection

technique for use case and class SMarty SPL models,

as well as its feasibility study as an empirical qualita-

tive analysis.

The remainder of this paper is organized as fol-

lows: Section 2 presents the background with regard

to SPL variability management and the SMarty ap-

proach, software inspection, defect types and exist-

ing inspection techniques; Section 3 presents SMar-

tyCheck, the proposed inspection technique; Sec-

tion 4, presents the SMartyCheck empirical feasibility

study; Section 5 discusses related work; and Section

6 presents conclusion and directions for future work.

2 BACKGROUND

This section presents essential concepts on variability

management and software inspections.

2.1 The SMarty Variability Approach

A Software Product Line (SPL) is a set of software

268

T. Geraldi R., Oliveira Jr. E., Conte T. and Steinmacher I..

Checklist-based Inspection of SMarty Variability Models - Proposal and Empirical Feasibility Study.

DOI: 10.5220/0005350102680276

In Proceedings of the 17th International Conference on Enterprise Information Systems (ICEIS-2015), pages 268-276

ISBN: 978-989-758-097-0

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

intensive systems that share a common infrastructure

formed by the SPL architecture and essential artifacts

(Linden et al., 2007). Such architecture and artifacts

have common features, the similarities, and variable

elements, the variabilities, in order to provide a means

to derive speciﬁc products for a given domain. Vari-

ability represents how SPL speciﬁc products differ

one another.

In this context, the existing literature presents sev-

eral variability management approaches based on fea-

ture models, UML and domain speciﬁc languages.

The Stereotype-based Management of Variability

(SMarty) approach has been developed OliveiraJr

et al. (2010) and empirically evaluated Marcolino

et al. (2014) for UML-based SPLs.

SMarty is composed of a UML proﬁle, the SMar-

tyProﬁle, and a systematic process, the SMartyPro-

cess, to guide one on applying the SMarty stereotypes

and meta-attributes. SMarty version 5.1 supports use

case, class, component, activity and sequence models.

2.2 Software Inspection

Software inspection is a speciﬁc type of software re-

view (Ciolkowski et al., 2003) applied to artifacts by

means of a systematic and well-planned defect identi-

ﬁcation process (Fagan, 1986). More than 60% of de-

fects can be identiﬁed in early phases of the software

life-cycle

(Boehm and Basili, 2001; Fagan, 1986).

By means of software inspection it is possible to ver-

ify several elements of software artifacts in order to

detect different defect types persistent in software

speciﬁcations (IEEE, 1998b).

Fagan (1986) proposed a software inspection pro-

cess driven by roles, such as, moderator, inspector and

author and activities. The process of defect detection

must be standardized and non-ambiguous for the doc-

ument under revision, as for instance, a requirements

speciﬁcation (IEEE, 1998a). Therefore, Anda and

Sjøberg (2002) propose a taxonomy of defect types

for checklist-based software inspections speciﬁcally

for use cases. Such a taxonomy is composed of the

following defect types: Omissions, Incorrect facts, In-

consistencies, Ambiguities and Extraneous Informa-

tion. In addition, these are the main existing defect

types in the literature, as we can see in Travassos et al.

(1999). Furthermore, the literature improves such a

taxonomy by providing several more studies on defect

types, such as in Hayes et al. (2006), IEEE (2012),

Mello et al. (2014), Travassos et al. (1999) and Cunha

et al. (2012).

Therefore, for defect detection it might be used

inspection techniques, thus, the study of Anda and

http://www.cebase.org/defect-reduction.html

Sjøberg (2002) presents the Checklist-Based Reading

(CBR), which is a non-systematic technique for de-

fect detection. It does not provide directions or mech-

anisms on “how” the inspection should be performed,

but “what” must be inspected through guidance us-

ing a document in a checklist format (Alshazly et al.,

2014).

Furthermore, the CBR technique based on check-

list is one of the most adopted inspection techniques

for analyzing artifacts, as well as for identifying the

most relevant information required for certain review

tasks (Mello et al., 2014). The checklist veriﬁcation

items are simple to follow. The checklist format is a

list of non-ambiguous questions, based on either pre-

vious experiences or history, which must be clearly

answered as “Yes” or “No”. In order to make the

checklist able to be applied, a list of defect types must

be available for the inspector (Alshazly et al., 2014).

3 SMartyCheck

The SMartyCheck technique acts at the Domain En-

gineering activities. SMartyCheck was designed as

an inspection technique with the objective of detect-

ing defects in SMarty SPL use case and class vari-

ability models of SPL by means of a checklist with

20 items. The SMartyCheck technique is composed

of two checklists: one for use case models, and one

for class models. We are aware that the automation of

SMartyCheck is an important issue to contribute for

assuring quality in inspection activities, thus, we are

developing a tool for it.

In order to perform inspections, inspectors should

read the checklist answering each question from a

list of assertive questions, marking a single check per

question, based on their prior knowledge with relation

to the SMartyCheck checklist defect types.

The checklist was developed based on the anal-

ysis and adaptation of defect types extracted from a

systematic mapping study that we carried out from

February to March/2014. From the analysis and clas-

siﬁcation of the 51 most relevant retrieved primary

studies, it was evident the identiﬁcation of several

different studies with distinct defect types taxonomy

proposals, most of them empirically evaluated, that

can be adapted by existing software inspection tech-

niques, such as SMartyCheck.

The taxonomy of defect types for the SMarty-

Check checklist contains 13 defect types, adapted

from the following sources: IEEE (2012), IEEE

(1998b), IEEE (1998a), Travassos et al. (1999), Hayes

et al. (2006), Lamsweerde (2009) and Cunha et al.

(2012).

Checklist-basedInspectionofSMartyVariabilityModels-ProposalandEmpiricalFeasibilityStudy

269

Thus, each type of SMartyCheck defect is de-

scribed as follows, in which is possible to under-

stand “what” to inspect throughout the SMartyCheck

checklist for use case and class models:

1. Ambiguous: use case or class model elements

should have only one interpretation. This defect

type was adapted from IEEE (2012) aiming at de-

tecting redundancy;

2. Anomaly: use case or class model element is as-

sociated to another element in an unusual way.

This defect type was adapted from Cunha et al.

(2012) to the context of SMarty models in order to

detect different elements relationships which are

not allowed;

3. Incomplete: absence of any signiﬁcant use case

or class model elements. It was adapted from

IEEE (2012);

4. Inconsistency: refers to the lack of internal con-

sistency, in which a subset of individual elements

has conﬂicts with use case or class models. This

defect type was adapted from IEEE (2012);

5. Incorrect: any element must meet the predicted

relationships of use case or class models. It was

adapted from IEEE (2012);

6. Intentional Deviation: an use case or class model

element that requires or depend on its relation-

ship to other elements according to SMarty vari-

ability models. Its descripton was interpreted and

adapted from Hayes et al. (2006) to the SMarty-

Check context;

7. Instable: every use case or class model element

has a unique identiﬁer (for instance the meta-

attribure name) indicating its importance or sta-

bility. Such an identiﬁer speciﬁes an element ac-

cording to its singularity and responsibility. This

defect type was adapted from IEEE (1998b,a);

8. Incorrect Fact: contradiction in any use case or

class model element. This defect type was inter-

preted and adapted from Travassos et al. (1999);

9. Extraneous Information: non-necessary pro-

vided information from any use case or class

model elements. Over speciﬁed or duplicated el-

ements in use case ou class models should be de-

tected by this kind of defect, adapted from Travas-

sos et al. (1999);

10. Infeasible: any unachievable element of use case

or class models taking into consideration other

system factors, such as, adding new elements

in SMarty SPL models. This defect type was

adapted from Hayes et al. (2006);

11. Non-modiﬁable: when the structure and the style

of one or more elements of use case or class mod-

els can be maintained by any changes. The el-

ements should be expressed in separate way in

order to be consistent and not redundant. It was

adapted from IEEE (1998b,a);

12. Omission: absence of any element of use case and

class models. This defect ype was interpreted and

adapted from Lamsweerde (2009);

13. Business Rule: the deﬁnition of business rules of

a speciﬁc domain may be incorrectly described or

speciﬁed according to SMarty use case and class

models. Its description was adapted from Lam-

sweerde (2009).

We present an application excerpt of how SMarty-

Check is applied to an SPL, in this case to the Arcade

Game Maker (AGM) (SEI, 2009) use case original

model (Figure 1(a)). Therefore, for illustration pur-

pose, we have inserted defects to the AGM use case

model (Figure 1(b)).

Figure 1: AGM Use Case Model according to SMarty: (a)

with no defects (original) (OliveiraJr et al., 2010) and (b)

with inserted defects.

The following items present the application of

SMartyCheck based on the deﬁned defect types and

checklist items answered ”Yes”:

4 Inconsistency (Incons)

• Incons.1 Is there any use case in the

SPL use case model with the stereotype

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

270

<<variationPoint>> whose number of

speciﬁed variants is larger than deﬁned in

maxSelection the variability associated?

Identiﬁed Defect: The use case Play Selected

Game has four variants. Should have at maxi-

mum three.

6 Intentional Deviation (ID)

• ID.1 Is there any use case in the SPL use

case model which requires the selection of an-

other (<<requires>>) and this another is

not speciﬁed in the SPL?

Identiﬁed Defect: The use case Check Previ-

ous Best Score requires the selection of the use

case Save Score.

8 Incorrect Fact (IF)

• IF.1 Is there any use case in the SPL use case

model which was described incorrectly?

Identiﬁed Defect: The use case Play Pong was

described as Pl Pon.

• IF.2 Is there any use case in the SPL use case

model that it is not be mapped its description?

Identiﬁed Defect: The use cases Play Tetris

and Pl Pon can not be mapped.

11 Non-modiﬁable (Nm)

• Nm.1 Is there any use case in

the SPL use case model with the

stereotype <<variationPoint>>,

in which variants associated

(<<optional>>, <<alternative

OR>>

or <<alternative XOR>>) cannot be

combined or selected in accordance with the

meta-attribute variants?

Identiﬁed Defect: The use case Play Selected

Game has selected variants Play Tetris and Pl

Pong erroneously speciﬁed.

12 Omission (Om)

• Om.1 Is there any use case speciﬁed as manda-

tory in the SPL use case model through the

stereotype <<mandatory>> that is not speci-

ﬁed on the SPL?

Identiﬁed Defect: The use cases Exit Game,

Initialization and Animation Loop are not spec-

iﬁed in the SPL.

13 Business Rule (BR)

• Domain/Business Rules: The system should be

able to allow you to install and uninstall a game,

allows choosing a to game, save the state of the

game, exit the game, deﬁne options and provide

the player a way to save his score and check

your best score previous.

BR.1 The use cases of the SPL use cases model

are not clear with the purpose and the desired

functionalities based on deﬁned domain?

Identiﬁed Defect: The system is not able to

allow exit game, deﬁne the options of game and

provide the player a way to save his score.

4 THE FEASIBILITY STUDY

A qualitative empirical study was carried out aiming

at analyzing the SMartyChek feasibility, then improv-

ing the proposed technique.

The analysis of this qualitative study was based

on the data collected from 8 experts. The majority

of the experts are from software engineering with ten

years of expertise in average. Selection criteria took

into account lecturers and practitioners, who directly

worked with or research on software veriﬁcation and

validation over the past years in different universities

and companies in Brazil.

The study documents were given the experts, thus

aiming to support them while answering the elec-

tronic questionnaires to evaluate the feasibility of

SMartyCheck. We created 6 electronic questionnaires

with 6 defective models, in which 2 questionnaires

(1 checklist with 2 use case models and 1 checklist

with 2 class models) were distributed and electroni-

cally randomly sent to the experts. At the end of the

assignment, we asked the experts to answer an open-

ended questionnaire composed of 7 questions about

the SMartyCheck technique. Experts spent about 90-

120 minutes for performing all given tasks of the

study.

Then, experts answered the questionnaire speciﬁ-

cally related to the characterization of SMartyCheck.

Next items present examples of questions:

In your opinion, the inspection process based on

the SMartyCheck technique is adequate for de-

tecting defects in use case and class SMarty SPL

models? Please, support your answer with justi-

ﬁcation.

In your opinion, the format of the SMartyCheck

checklist clearly, objectively and accurately con-

tributes for inspecting use case and class SMarty

SPL models? Please, support your answer with

justiﬁcation.

The answers received were qualitatively analyzed

using procedures of Grounded Theory (GT) (Corbin

and Strauss, 2008). GT approach is based on the con-

cepts of Coding (Corbin and Strauss, 2008). Coding

allows assigning codes or labels to excerpts of text

(Open Coding), which can be grouped and catego-

rized (Axial Coding) according to an expressed idea

Checklist-basedInspectionofSMartyVariabilityModels-ProposalandEmpiricalFeasibilityStudy

271

for the purpose of elucidate the phenomenon (Corbin

and Strauss, 2008). The Selective Coding was not per-

formed in this study, as we do not intend to formalize

a theory, but evaluate the feasibility of the SMarty-

Check technique.

Analysis included only procedures of Open Cod-

ing and Axial Coding. As a result of such coding,

enabled the creation a conceptual model.

During the coding process, 3 categories emerged

from the data collected from the experts regarding

feasibility of SMartyCheck technique: “Feasible for

Inspection”, “Possible Improvements” and “Automat-

ing SMartyCheck”.

A brief description of each category along with

their related codes are presented, as well as excerpts

from the data collected from the experts and the cat-

egorization of codes to evaluate questions concerning

the feasibility of SMartyCheck.

Feasible for Inspection: this category evaluates

the feasibility of SMartyCheck, providing evidence

that improvements are required. Figure 2 presents a

graphical representation of factors that are part of this

category.

Analyzing the experts answers with regard to

SMartyCheck, such a technique supports the correct-

ness based on the coding Supports Correctness as-

signed in accordance to the excerpt from expert #1:

“...the process is interesting and confers greater de-

gree of correctness to the model.”; and the excerpt

from expert #3: “...supporting the correction of the

models.”. Furthermore, the SMartyCheck technique

detects inconsistencies in SMarty models based on

excerpt from expert #6: “...facilitates inconsisten-

cies mapping.”; and the excerpt from expert #8: “...a

checklist guides the reviewer at identifying possible

inconsistencies of the model.”.

Contributes to

Defect Detection

Seems Better than

Ad hoc Technique

Uses SMarty Stereotypes

Has Coherent Defect Types

Supports Correctness

[CA] Feasible for Inspection

is part

Figure 2: Graphical Representation with the Associations

related to the Feasible for Inspection Category.

We evidenced that the SMartyCheck

Contributes to Defect Detection. This factor was

supported by some quotes from the experts, who

answered “Yes.” to “In your opinion, the format

of the SMartyCheck checklist clearly, objectively

and accurately contributes for inspecting use case

and class SMarty SPL models? Please, support

your answer with justiﬁcation.”. Expert #3 also

answered “Yes, because it allowed to detect different

defect types in the UML models.”, whereas expert

#6 answered “Yes, it is possible to identify process

failures throughout the mapping.”.

In addition, the SMartyCheck technique

Has Coherent Defect Types, according to the ex-

perts. When asked “In your opinion, the defects

types are coherent with the description of the check-

list items of SMartyCheck technique for inspection

SMarty SPL models taking into account use case

and class models? Justify.”, the expert #3 answered:

“Yes, they are coherent as they allowed to analyze

possible errors in different elements and correct such

errors.”; and the excerpt from expert #4: “Yes, I

believe that the defect types are ﬁne segmented.”; and

the excerpt from expert #6: “I believe so, because

they express the problem to be mapped.”; and the

excerpt from expert #7: “Yes... Defect types are clear

and objective. However, they could be grouped and

placed on the natural order.”.

Furthermore, several experts reported that

SMarty stereotypes are important elements to

better performing inspections. Based on this,

Uses SMarty Stereotypes factor emerged, supported

by the following quotes: “Yes, the speciﬁcation of

stereotypes is interesting and think their use is essen-

tial for the correct reading of the model.” [Expert

#1]; “Yes, because they represent the key elements of

variability, variation points and variants.” [Expert

#3]; “Yes, stereotypes are essential because they

present clear and intuitive elements related to the

variability of a SPL.” [Expert #4]; “Yes, they are key

elements for inspection, since it is also based mostly

on these stereotypes.” [Expert #5].

In addition, SMartyCheck is different in com-

parison to the Ad hoc technique. The Ad hoc is

an “informal” technique that does not provide in-

structions, procedures, criteria or systematic mecha-

nisms for inspectors performing the reading of soft-

ware artifacts. Therefore, the Ad hoc technique de-

pends on the viewpoint of the inspector, his knowl-

edge and experience in the detection of defects (Rom-

bach et al., 2008). Thus, the SMartyCheck is bet-

ter in comparison to the Ad hoc based on the cod-

ing Seems Better than Ad hoc technique assigned in

accordance to the expert answer #8: “Very different

when the reviewer has to do its work performing an

ad hoc comparison.”.

Figure 2 presents the representation of the

SMartyCheck feasibility by supporting correctness

(Supports Correctness) of SMarty use case and

class models, detecting inconsistencies and defects

(Contributes to Defect Detection). Further-

more, SMartyCheck is suitable for detecting several

different coherent defect types (Coherent Defect

Types). With regard to the SMarty approach, its

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

272

stereotypes are essential and intuitive for defect de-

tection (Uses SMarty Stereotypes).

Possible Improvements: SMartyCheck requires

possible improvements, because it has some limita-

tions, considering redundancies reported by experts

on the items of the checklist.

SMartyCheck has a few redundancies in

items of the checklist based on the coding

Redundant in the Checklist Items, supported by

quotes extracted from answers of expert #4: “...some

questions seemed to me somewhat recurring, which

can be considered a threat to the validity of this pro-

posal.”; and expert #5: “...I consider the technique

with some redundant inspections.”; and expert #7:

“...there are duplicate and ambiguous questions.”;

and expert #7: “The items of SMartyCheck: Incons.1,

Incom.1, An.1, Incons.3, Incons.2 can be done

automatically by syntax analysis of the model.”.

Furthermore, the SMartyCheck technique

has confused descriptions related to the items

of the checklist. It can be evidenced this

Confusion in the Checklist Items on answers from

expert #1: “...some items have caused a little of

confusion initially.”; and expert #3: “...some of the

questions are formulated to hinder understanding.”.

In addition, experts answered that some defect

types are incoherent, ambiguous and hard to interpret

based on the coding Has Incoherent Defect Types as-

signed in accordance to the expert answer #1: “The

defect identiﬁed as ”Non-modiﬁable” seemed to me

confused as to its description.”; and expert #5: “De-

fect #5 (Incor.2), I do not understand why relating 2

mandatory items is incorrect.”; and expert #7: “In

An.1, it must be clear the type of association, the ques-

tion leads to an incorrect markup. In Incons.1, it must

be clear the implication of this.”.

Automating the SMartyCheck: some ex-

perts suggested the automation of some check-

list items to improve SMartyCheck technique.

Automating SMartyCheck was suggested by expert

#4: “...I believe that automating process of system-

atic development, as in the case of SPL, can aggregate

much in the quality requirement.”.

[CA] Possible

Improvements

Redundant in the

Checklist Items

Confusion in the

Checklist Items

is part

Has Incoherent Defect Types

is part is part

Figure 3: Graphical Representation with the Associations

related to the Possible Improvements Category.

Therefore, it is clear that some adjustments are

necessary to improve the SMartyCheck technique ac-

cording to the issues presented under the categories

“Possible Improvements” and “Automating SMarty-

Check”. Figure 3 presents a graphical representation

of the factors, which are part of the category “Possible

Improvements” and the defect types suggested by the

experts that can be automated in prospective improve-

ments related to the category “Automating SMarty-

Check”.

Figure 3 presents the possible improvements cod-

ings of SMartyCheck, which are due to some of

the checklist items are redundant (Redundant in

the Checklist Items) and confusing (Confusion

in the Checklist Items). Thus, they were im-

proved. Incoherent defects are also present in SMar-

tyCheck as we can see in the Has Incoherent

Defect Types coding. Such issues that form the

Possible Improvements coding were reviewed and

improved.

The following items present the improved check-

list of SMartyCheck. Although such items refers to

use case models, they also serve for class models.

1. Business Rule (BR)

BR.1 The use cases of the SPL use cases model

(with defects) model are not clear with the pur-

pose and the desired functionalities based on de-

ﬁned domain?

2. Incomplete (Incom)

Incom.1 All use cases with <<mandatory>>

(required) are not speciﬁed in the SPL use case

model (with defects)?

3. Inconsistency (Incons)

Incons.1 Is there any use case in the SPL

use case model (with defects) speciﬁed with

the stereotype <<variationPoint>> whose

number of speciﬁed variants is larger (greater)

than deﬁned in maxSelection the variability

(<<variability>>) associated?

Incons.2 Is there any use case in the SPL

use case model (with defects) speciﬁed with

the stereotype <<variationPoint>> whose

number of speciﬁed variants is smaller (mi-

nor) than deﬁned in minSelection the variability

(<<variability>>) associated?

4. Incorrect (Incor)

Incor.1 Is there any use case in the SPL use case

model (with defects) speciﬁed with the stereo-

type <<variationPoint>> which is associ-

ated with a use case in SPL (with defects) that is

not <<alternative OR>>?

5. Incorrect Fact (IF)

IF.1 Is there any use case in the SPL use case

model (with no defects) which was described in-

Checklist-basedInspectionofSMartyVariabilityModels-ProposalandEmpiricalFeasibilityStudy

273

correctly in SPL (with defects)?

IF.2 Is there any use case in the SPL use case

model (with defects) that it is not be mapped its

description in SPL (with no defects)?

6. Ambiguous (Am)

Am.1 Is there any use case in the SPL use case

model (with defects) that your description does

not reﬂect on your real goal?

Am.2 Is there any use case in the SPL use case

model (with defects) in that its name or descrip-

tion is equal to the use case which is associated to

owning a duplicate interpretation?

7. Non-modiﬁable (Nm)

Nm.1 Is there any use case in the SPL use

case model (with defects) speciﬁed with

the stereotype <<variationPoint>>,

in which variants associated

(<<optional>>, <<alternative OR>>

or <<alternative XOR>>) can not be com-

bined or selected in accordance with the variants

already speciﬁed in the meta-attribute variants in

SPL (with no defects)?

8. Anomaly (An)

An.1 Is there any use case in the SPL use case

model (with defects) speciﬁed with the stereo-

type <<alternative OR>> associated with a

use case that is not speciﬁed with the stereotype

<<variationPoint>>?

9. Instable (Ins)

Ins.1 Is there any use case in the SPL use case

model (with defects) speciﬁed with the stereo-

type <<variationPoint>>, in which has

the stereotype associated <<variability>>

whose meta-attribute name equals to other use

cases elsewhere speciﬁed with the stereotype

<<variationPoint>>?

10. Infeasible (Inf)

Inf.1 Is there any use case in the SPL use case

model (with defects) speciﬁed with the stereo-

type <<variationPoint>> whose number of

variants speciifed is not (false) allows you to add

new variants as deﬁned in the meta-attribute al-

lowsAddingVar?

11. Omission (Om)

Om.1 Is there any use case speciﬁed as mandatory

(required) in the SPL use case model (with no de-

fects) through the stereotype <<mandatory>>

that is not speciﬁed on the SPL (with defects)?

12. Extraneous Information (EI)

EI.1 Is there any use case in the SPL use case

model (with defects) speciﬁed besides use cases

existing in the SPL (with no defects)?

EI.2 Is there any use case in the SPL use case

model (with no defects) with its functionality du-

plicated in SPL (with defects)?

13. Intentional Deviation (ID)

ID.1 Is there any use case in the SPL use case

model (with no defects) which requires the selec-

tion of another (<<requires>>) and this an-

other is not speciﬁed in the SPL (with defects)?

ID.2 Is there any use case in the SPL use case

model (with no defects) not which requires selec-

tion of another (<<mutex>>) and this another is

speciﬁed in the SPL (with defects)?

By analyzing the overall obtained results, we can

provide evidence that SMartyCheck is feasible, as it:

(i) supports correctness of models; (ii) explicitly con-

tributes for defect detection; (iii) has coherent defect

types; (iv) correctly uses SMarty stereotypes, making

such a detection intuitive and objective; and (v) seems

to be more appropriate than the ad hoc technique.

The following major threats to the validity of this

empirical study are discussed:

Threats to Internal Validity. Tasks performed

by the experts were conducted in a similar manner,

except for the electronic forms (questionnaires) ran-

domly mailed. Experts were trained on the basics

of SPL, variability and software inspection, beyond

the understanding and comprehension of the SMarty-

Check. 20 to 27 days was required to the experts ﬁll in

the questionnaires, thus, fatigue effects were reduced;

Threats to External Validity. Pedagogical SPLs

used in training sessions (SPL Mobile Media) and

in the empirical study (SPL Arcade Game Maker

(AGM)) can endanger the external validity. There-

fore, we tried to use SMartyCheck and AGM stan-

dardized technical documents. Obtaining practitioner

experts from industry was one of the greatest difﬁcul-

ties encountered in this study;

Threats to Conclusion Validity. The main threat

is related to the sample size, 8 experts, however the

previous knowledge of the experts was signiﬁcant.

Therefore, we intend to perform more empirical stud-

ies with a larger sample;

Threats to Construct Validity. This study was

deﬁned and tested based on a pilot project carried out,

reﬁning the SMartyCheck and AGM instrumentation,

which was adequate to apply to a real study. Although

the level of knowledge required on the concepts of

UML modeling, SPL and software inspection is basic,

experts presented high-level skills.

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

274

5 RELATED WORK

The set of techniques “The Software Product Line In-

spection Techniques (SPLIT)” by Cunha et al. (2012)

is aimed at comparing the SPL requirements docu-

ment, feature model and product map. SPLIT has

contributed to the deﬁnition of defect types, adopted

by SMartyCheck. The study conducted by Anda and

Sjøberg (2002) also has. Such a study presents a tax-

onomy of typical defects of use case models based on

two empirical studies.

The study of Belgamo et al. (2005) proposes

TUCCA, a “Technique for Use Case Model Construc-

tion and Construction-based Requirements Document

Analysis”, which is composed of two reading tech-

niques. In such a work, it was identiﬁed new defect

types in experiments that might contribute to improve

SMartyCheck. These defect types are not explicitly

mentioned in the study of Belgamo et al. (2005).

6 CONCLUSION

This paper presented the proposal of SMartyCheck

technique for inspecting SMarty use case and class

models, as well as an empirical qualitative study,

which provided evidence that SMartyCheck is feasi-

ble. Such a technique contributes to SPL model in-

spections by means of coherent defect types, lead-

ing to correctness of models based on a well-deﬁned

structure supported by SMarty stereotypes.

In fact, the qualitative results helped to improve

SMartyCheck since it was possible to identify defects

that require adjustments and correcting the checklist

items considered ambiguous and difﬁcult to under-

stand. Furthermore, the feedback from subjects pro-

vided evidence that SMaryCheck is feasible, because

stated support correctness SMarty SPL models. Fur-

ther empirical studies must be planned and conducted

in academia and industry to evaluate the accuracy of

SMartyCheck.

As future work, we plan to: (i) allow the inspec-

tion of different SMarty models, such as component,

sequence and activity; (ii) automate, partially or to-

tally, the inspection process by means of a tool for sin-

tax analysis of SMarty models; (iii) combine SMarty-

Check and perspective-based techniques to increase

its inspection power; and (iv) conduct more empirical

studies to reﬁne SMartyCheck, as well as corroborate

and generalize the obtained results.

ACKNOWLEDGEMENTS

CAPES-Brasil for granting R. Geraldi a two-year

master’s degree scholarship. Experts from several

universitites.

REFERENCES

Alshazly, A. a., Elfatatry, A. M., and Abougabal, M. S.

(2014). Detecting defects in software requirements

speciﬁcation. Alexandria Engineering Journal.

Anda, B. and Sjøberg, D. I. K. (2002). Towards an inspec-

tion technique for use case models. In Proc. Int. Conf.

on Software Engineering and Knowledge Engineering

(SEKE), pages 127–134, New York, NY, USA. ACM.

Belgamo, A., Fabbri, S., and Maldonado, J. C. (2005).

TUCCA: Improving the Effectiveness of Use Case

Construction and Requirement Analysis. In Proc.

Int. Symposium on Empirical Software Engineering

(ESEM), pages 257–266. IEEE.

Boehm, B. and Basili, V. R. (2001). Software Defect Re-

duction Top 10 List. IEEE Computer, pages 135–137.

Capilla, R., Bosch, J., and Kang, K.-C. (2013). Systems and

Software Variability Management: Concepts, Tools

and Experiences. Springer Berlin Heidelberg.

Ciolkowski, M., Laitenberger, O., and Bifﬂ, S. (2003). Soft-

ware Reviews: The State of the Practice. IEEE Soft-

ware, 20(6):46–51.

Corbin, J. M. and Strauss, A. L. (2008). Basics of Qualita-

tive Research: Techniques and Procedures for Devel-

oping Grounded Theory. Sage Publications, 3 edition.

Cunha, R., Conte, T. U., de Almeida, E. S., and Maldon-

ado, J. C. (2012). A Set of Inspection Technique on

Software Product Line Models. In Proc. Int. Conf.

on Software Engineering and Knowledge Engineering

(SEKE), pages 657–662.

Fagan, M. (2002). A History of Software Inspections.

Springer Berlin Heidelberg.

Fagan, M. E. (1986). Advances in Software Inspections.

IEEE Transactions on Software Engineering (TSE),

pages 744–751.

Hayes, J., Raphael, I., Holbrook, E., and Pruett, D. (2006).

A Case History of International Space Station Re-

quirement Faults. In Proc. Int. Conf. on IEEE En-

gineering of Complex Computer Systems (ICECCS),

page 10. IEEE Computer Society.

IEEE (1998a). Recommended Practice for Software Re-

quirements Speciﬁcations, Standard 830-1998.

IEEE (1998b). Software Reviews, Standard 1028-1997.

IEEE (2012). System and Software Veriﬁcation and Valida-

tion, Standard 1012-2012.

Lamsweerde, A. (2009). Requirements Engineering: From

System Goals to UML Models to Software Speciﬁca-

tions. John Wiley & Sons.

Linden, F., Schmid, K., and Rommes, E. (2007). Software

Product Lines in Action: The Best Industrial Practice

in Product Line Engineering. Springer.

Checklist-basedInspectionofSMartyVariabilityModels-ProposalandEmpiricalFeasibilityStudy

275

Marcolino, A., OliveiraJr, E., Gimenes, I., and Barbosa, E.

(2014). Empirically Based Evolution of a Variabil-

ity Management Approach at UML Class Level. In

Proc. Int. Conf. Computers, Software & Applications

(COMPSAC), pages 354–363, Vasteras, Sweden.

Mello, R. M., Teixeira, E. N., Schots, M., and Werner, C.

M. L. (2014). Veriﬁcation of Software Product Line

Artefacts: A Checklist to Support Feature Model In-

spections. Journal of Universal Computer Science

(JUCS), 20(5):720–745.

OliveiraJr, E., Gimenes, I., and Maldonado, J. (2010).

Systematic Management of Variability in UML-based

Software Product Lines. Journal of Universal Com-

puter Science (JUCS), 16(17):2374–2393.

Rombach, D., Ciolkowski, M., Jeffery, R., Laitenberger, O.,

McGarry, F., and Shull, F. (2008). Impact of research

on practice in the ﬁeld of inspections, reviews and

walkthroughs. ACM SIGSOFT Software Engineering

Notes, 33(6):26.

SEI (2009). Software Engineering Institute Arcade Game

Maker (AGM) Pedagogical Product Line. Carnegie

Mellon University.

Travassos, G. H., Shull, F. J., Fredericks, M., and Basili,

V. R. (1999). Detecting Defects in Object-Oriented

Designs: Using Reading Techniques to Increase Soft-

ware Quality. In Proc. Int. Conf. (ACM SIGPLAN) on

Object-Oriented Programming, Systems, Languages,

and Applications (OOPSLA), pages 47–56, New York,

USA. ACM Press.

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

276