METHODOLOGICAL GUIDELINES FOR SQA IN

DEVELOPMENT PROCESS

An Approach Based on the SPICE Model

Anna Grimán, María Pérez, Luis Mendoza

Processes and Systems Department – LISI

Universidad Simón Bolívar

Caracas – Venezuela

Keywords: Software Quality, SPICE model, Software Development Methodology, Methodological guidelines.

Abstract: As far as international standards for promoting Software Process Quality are concerned, one of the most

popular and accepted is ISO 15504 (or SPICE model). On the other hand, since a development methodology

must guide the main activities in software development, it is necessary that this one fulfils some Quality

Base Practices to guarantee a high-level product. The purpose of this research is analyzing a set of five

methodologies widely used by developers, to identify its adjustment with respect to the aforementioned

standard. This analysis allowed us: (1) determining the degree of alignment of these methodologies with

respect to the SPICE model, and (2) proposing a synthesis of methodological guidelines, based on the best

practices obtained from these methodologies, that supports the characteristics contained in the studied

standard.

1 INTRODUCTION

In the world of Information Systems, one of the

solutions being currently offered to develop quality

products are the assessment models, both software

product as well as software process. These models

are highly useful because they deliver internationally

accepted measurements. This research is limited to

one of these dimensions: the process. To this

objective, the SPICE model has been used.

According to ISO/IEC (2004), Software Process

Improvement and Capability dEtermination (SPICE)

is a model designed for the assessment of software

processes, which is included into the ISO documents

and has evolved to a draft of the ISO 15504

Standard (Pressman, 2005). Pérez et al. (2001) have

adopted the SPICE Quality Model, and have

modified it to include efficiency and effectiveness

aspects of development process.

With the emergence of standards and models,

and the increasing commitment of organizations to

quality, development methodologies have been

trying to incorporate elements of Quality Control

and Assurance into their processes. This has been

evident for developers who use modern

methodologies with a popularity that has

exponentially grown from their inception.

However, these currently accepted

methodologies do not always support the quality

level proposed by this SPICE model for the software

development process. This leaves a gap that has to

be increasingly filled due to the certification needs

faced by the software developing companies

desiring to compete globally.

This paper presents the analysis of five

methodologies according to base practices and

quality characteristics of the SPICE model

(adapted), to determine their degree of compliance

for each methodology studied. At the same time, we

identified their quality-supporting strengths

according to the model within a set of

methodological guidelines which can be

incorporated into any existing methodology to

improve quality level of the development process.

First, SPICE model is described followed by the

method used for the methodology analysis. Then a

brief description of each methodology under study is

presented. Next, Assessment criteria and feature

scoring for each methodology is described. Then,

analysis of the scores with a summary of the

guidelines derived from this study is presented.

269

Grimán A., Pérez M. and Mendoza L. (2006).

METHODOLOGICAL GUIDELINES FOR SQA IN DEVELOPMENT PROCESS - An Approach Based on the SPICE Model.

In Proceedings of the Eighth International Conference on Enterprise Information Systems - ISAS, pages 269-275

DOI: 10.5220/0002462102690275

 SciTePress

Finally, a report on the evaluation, the conclusions

and recommendations for future research are

described

2 QUALITY MODEL BASIS

A wide range of models is available for quality

assessment of the software development process,

such as: Personal Software Process (PSP)

(Humphey, 1997), CMM (Baltzer et el., 1993),

BOOTSTRAP (Engelbart and Engelbart, 1990) and

Software Process Improvement and Capability

dEtermination –SPICE- (ISO/IEC, 2004). This

research is based on an international standard: ISO

15504 (or SPICE).

SPICE provides a framework for the assessment

of software processes. This framework can be used

by organizations involved in planning, management,

monitoring, control, and improvement of

acquisition, supply, development, functioning,

assessment and support of software (ISO/IEC,

2004).

Each instance of the process is characterized by a

set of five (5) levels of process capability, each one

being an aggregation of the enough practice

assessments belonging to each specific level. Proper

practice assessments are the basis for the assessment

system.

However, SPICE model does not consider the

characteristics inherent to the development of

Software Systems, such as process efficiency and

effectiveness. For this reason, the Quality Model

used for this research is based on a model that

integrates the Systemic Quality approach (Callaos

and Callaos, 1996) with the features present in the

SPICE process model. The model used here has a

complex structure defined by level, where each

higher level is made up of lower level elements

(Pérez et al., 2001). This structure is described

below.

Level 0: Life Cycles. As with the SPICE process

model, three Life Cycles are considered. The inter-

relationship between these cycles guarantees the

quality of the Information Systems development

process. These are: Primary Life Cycle (it is made

up of two categories: Customer–Supplier and

Engineering); Support Life Cycle (only contains the

Support category); Organizational Life Cycle (it is

composed of the Management and Organizational

categories).

Level 1: Category. This model covers five

categories of process, in accordance with SPICE.

These are given Table 1.

Level 2: Processes. Each category has a set of

characteristic processes that define the key areas to

be met to achieve, ensure, maintain and control

quality. Each process has an identifier associated

with it that distinguishes it unequivocally. Table 1

shows the processes associated with each category.

Level 3: Principles. Each process has a Principle

(P) associated, which is defined as an abstract and

generic feature of the organization and serves as an

indicator to determine the levels of quality in the

development of Information Systems.

Level 4: Base Practices. A set of Base Practices

(BP) is defined as a set of guidelines to be

implemented by the organization in order to attain a

principle. It should be noted that it was necessary to

reasonably increase the number of BP present in the

SPICE (ISO/IEC 2004) processes model, in order to

maintain a balance in the dimension of the Systemic

Quality (Callaos and Callaos, 1996). For more

details about this model and how it has been

evaluated through several case studies, see (Pérez et

al., 2001).

Table 1: Processes for each category of model.

Category Processes

Customer-

Supplier (CUS)

CUS.1 System or Product

Acquisition Process, CUS.2 Supply

Process, CUS.3 Requirements

Bidding Process, CUS.4 Operation

Engineering

(ENG)

ENG.1 Development, ENG.2

Software and Systems Maintenance

Support (SUP) SUP.1 Documentation, SUP.2

Configuration Management, SUP.3

Quality Assurance Process, SUP.4

Verification, SUP.5 Validation,

SUP.6 Joint Review, SUP.7

Auditing, SUP.8 Problem Solving

Process

Management

(MAN)

MAN.1 Management, MAN.2

Project Management , MAN.3

Quality Management , MAN.4 Risk

Management

Organizational

(ORG)

ORG.1 Organizational Alignment ,

OGR.2 Change Management,

ORG.3 Process Set-up Process,

ORG.4 Process Evaluation Process,

ORG.5 Improvement, ORG.6

Infrastructure Process , ORG.7

Measurement Process ,O RG.8 Re-

use Process

ICEIS 2006 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION

270

3 RESEARCH APPROACH

In this research, the combination of two DESMET´s

evaluation methods (Kitchenham, 1996) was used to

identify the methodological issues presented in each

development methodology. First, a Feature

Analysis-Screen Mode was carried out. The

evaluation is performed by a single person based on

documentation only. It is the best approach for

screening a large number of methods/tools and is

often used as a first stage in a more complex

evaluation exercise reducing a large number of

possible methods/tool to a short-list of one or two

candidate method/tools that can be evaluated in

more detail. In this case the evaluator is responsible

for: identifying the candidate methods/tools;

devising the assessment criteria (i.e. the features to

be assessed and the judgment scales) with or without

specific help from potential tools users; compiling

information about the method/tools; scoring each

feature for each method/tool; analysing the scores;

presenting a report on the evaluation.

Second, once obtained a set of results

(methodological guidelines), a Qualitative Effects

Analysis was followed to validate them. It provides

a subjective assessment of the quantitative effect of

methods and tools, based on a knowledge base of

expert opinion about generic methods and

techniques. The researcher requests an assessment of

the effects of individual methods and/or the

combined impact of several methods. This is quite a

useful approach because the information held in a

database containing expert opinion can be updated

as and when the results of objective studies become

available (Kitchenham, 1996).

Due to format restrictions, only the main results

obtained after applying these methods are presented

in next sections.

4 CANDIDATES DEVELOPMENT

METHODOLOGIES

A software development methodology is a set of

steps and procedures to be followed to develop a

Software System (Whitten, Bentley and Dittman,

2004). The development methodologies studied and

analyzed in this research are presented below.

Microsoft Operation Framework (MOF) is a

collection of best practices, principles, and models

(Microsoft, 2002). MOF is based on operations such

as software development with a life cycle consisting

of different phases working concurrently. MOF is

composed of three core models: process, team and

risk.

Microsoft Solution Framework (MSF) provides

guidance for planning, building, and deploying a

single project life cycle (Microsoft, 2000). People,

processes and continuous risk management are key

elements, besides technology, to achieve successful

IT projects. MSF uses three models, through which

it implements its principles: Risk Management,

Team Management, and Process Management.

Rational Unified Process (RUP) is a Software

Engineering process that provides guidance for the

allocation of tasks and responsibilities within an

organization. (Kruchten, 2003). It uses an iterative

and incremental method based on the review of

requirements and risks, as well as design,

implementation, and validation, followed by a new

review of these elements until finally the end

product is obtained. The phases on which RUP is

based for this iterative development are as follows:

Inception, Elaboration, Construction and Transition.

Extreme Programming (XP) is a discipline of

software development. In comparison to the heavy

methods, XP is an agile method covering a set of

techniques and common sense principles at extreme

levels (Kendall and Kendall, 2002). It is based on

the definition of four variables applicable to any

software project: Cost, Time, Quality, and Scope.

XP puts particular emphasis on small development

teams that could increase if necessary.

Unified Process (UP) is based on a program

engineering relying on principles approximated to

software development (Jacobson et al., 1999). It

consists of iterative principles, requirements and

architecture base development. UP’s life cycle

consists of four consecutive phases: Inception,

Elaboration, Construction and Transition. There is a

fifth phase, Production, which supplements those

previous ones.

5 ASSESSMENT CRITERIA

This section describes the criteria used to analyze

each one of the methodologies under study.

In this case, the base practices included in each

category associated to a development process

proposed by SPICE (CUS, ENG and MAN)

represent a methodological element (activity,

technique, deliverable, etc.) to be satisfied by the

candidate methodology.

SPICE SUP and ORG categories were not

considered, because they are beyond the scope of the

development methodologies. These categories

contain processes establishing corporative business

goals and developing process goods (values),

products and resources, and support processes,

METHODOLOGICAL GUIDELINES FOR SQA IN DEVELOPMENT PROCESS - An Approach Based on the SPICE

Model

271

whereas development methodologies are focused on

the development process and manage this latter at

the operational level.

The application of criteria established previously

follows the algorithm mentioned below:

1. For each Process in each one of the

aforementioned Categories is there any element

(activity/artefact/role) in the methodology, which

satisfies each one of the Base Practices (BP)

associated at the lowest model level?

2. If there is a methodology that satisfies BP, the

one adding most value to the process will be

selected.

3. For each proposal of any methodological

element, its selection is justified illustrating the

criterion on which this selection was based.

In this sense, 89 criteria were proposed based on

BPs, for examples:

C1: Methodology should identify clearly the

existing resources and entities interacting with the

system and/or software product, to determine its

scope.

C2: Methodology should establish quality goals

for the product and the processes that can be

assessed throughout the project, preferably in a

qualitative fashion, based on the status and the

client-inherent quality requirements

6 SCORING EACH FEATURE

FOR EACH METHODOLOGY

Tables 2 shows a sample of the analysis applied to

each methodology and each BP. It is worth

mentioning that a total of 89 analyses were made,

one for each criterion (BP).

Once this criteria analysis process was applied, the

best practices were obtained (that will be

transformed into guidelines), as well as a summary

for each one of the analyzed methodologies with

respect to the SPICE process they satisfy. Table 3

shows the relationship among them.

Table 2: Sample of the analysis applied to ENG category.

Category: Engineering

(ENG)

Process: ENG.1

Development Process

Criterion: using the scale and

scope of the software system

or product to be developed as

a basic definition of the

activities and tasks required

to develop the product or

system in an effective,

efficient and profitable

fashion

Methodology analysis:

MSF: Vision

Document

UP: Provisional Plan

and Use-Case Model

RUP: Vision

Document.

Supplementary

Specifications. Project

Plan

Elements selected as guidelines: Vision document,

because it is useful to determine the scope of the project

and therefore effectively define the activities needed as

a function of the requirements. The Vision Document

also contains all the system specification offering an

overview of the basic requirements and characteristics

of the project. Provisional Plan, because it can be used

to clarify the requirements related to the initial goals,

including scope determination. Use-Case Model,

because with the captured requirements it is possible to

determine an initial scope that will make it possible to

establish activities. Project Plan, because it defines all

basic activities and tasks to develop the system.

Supplementary Specifications, because they specify

the elements supplementing the contents of the Vision

Document, related to the definition of activities and

tasks.

Table 3: Relationship obtained for the analyzed

methodologies.

Candidate

Method.

Quality Criteria Supported

MSF CUS1, CUS2, CUS3, CUS4,

ENG1, MAN1, MAN2, MAN4

MOF CUS1, CUS2, CUS3, CUS4,

ENG1, ENG2, MAN1,MAN2,

MAN3, MAN4

RUP CUS1, CUS2, CUS3, CUS4,

ENG1, ENG2, MAN1, MAN2,

MAN3, MAN4

UP CUS1, CUS2, CUS3, CUS4,

ENG1, ENG2, MAN1, MAN2,

MAN3, MAN4

XP CUS1, CUS3, CUS4, ENG1,

ENG2, MAN1, MAN2, MAN3

ICEIS 2006 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION

272

Table 4: Examples of Activities by Discipline.

7 ANALYSING THE SCORES

As a final result of the analysis conducted, the

strengths identified to support each one of the base

practices proposed by SPICE have been

summarized. These strengths have been grouped into

a set of guidelines that can easily be introduced in

order to improve the current practices in any

organization (Table 4 shows a sample of guidelines

proposed).

To this objective, they have been organized

according to the development disciplines they

correspond to, presenting for each one the related

SPICE categories, the suggested activity, the artefact

they produce, and the role proposed as responsible.

A total of 34 artefacts were obtained for Analysis

discipline, 27 artefacts were obtained for Design

discipline, 35 artefacts were obtained for

Implementation discipline; and 7 artefacts were

obtained for Maintenance discipline. Figure 1 shows

the percentage corresponding to each methodology

based on their analysis.

According to Figure 1, most of the artefacts

related to the Analysis discipline are represented by

the RUP methodology, reaching 35% with 12

artefacts; UP methodology follows with 27%

corresponding to 9 artefacts; the next methodology

is MSF with 21%, 7 artefacts; the fourth is MOF

methodology (12%, 4 artefacts), and finally, the

proposal of own artefacts referenced by the different

authors. Based on this analysis, it can be concluded

that Rational Unified Process (RUP) is the

methodology that is best identified with the SPICE

elements in the analysis, with respect to the activities

in the analysis.

For Design disciplines, Figure 1 shows a small

difference with respect to the artefacts proposed by

UP an RUP, since the percentage represented by

each one is 22% and 25%, respectively, being RUP

the methodology that is most identified with the

elements of the SPICE Quality Model. It also shows

the presence of the artefacts proposed by XP

methodology with 4%, MSF with 11%, and MOF

and the own artefact proposals taken from different

references with 19%.

For Implementation discipline, according to

Figure 1, the highest value, 40%, corresponds to UP,

meaning that this was the methodology which most

identified with the SPICE elements regarding the

activities performed during implementation. The

second methodology is RUP with 23%; then come

the own proposals of artefacts taken from different

references, 17%, MOF with 14%, and finally MSF

with 6%.

Finally, respect to Maintenance, Figure 1 shows

the absence of the methodologies presented in

previous results such as MOF and MSF, which

means that they did not identified with the SPICE

elements regarding the activities corresponding to

Maintenance. It can also be observed that RUP is the

methodology with the highest value (57%), which

represents more than a half of the artefacts

Discipline Activity that

satisfy the

category

Artefact Responsible

Customer-

Supplier

Functional specifications, Use-Case Model, List of

Characteristics (potential requirements), Vision Document,

Supplementary Specifications, Requirement Matrix, approved

Development Plan, Product Acceptance Plan.

System Analyst,

Architect, Use-Case

Specifier, Use-Case

Engineer.

Engineering

Team Model, Organization Pattern, Project Plan including

tasks network.

Project Leader

Analysis

Management

Metrics Plan (Metrics of the analysis model, the design model,

the source code, and maintenance; and quality measurements

such as: correction, maintenance easiness, integrity and use

easiness); Test Model, Quality Assurance Plan.

Project Leader

Customer-

Supplier

Solution Design Model, Process Model, Use-Case Model,

Exploratory Prototype, Product Acceptance Plan, Project Plan,

Supplementary Specification, Vision Document, Business

Case, Quality Assurance Plan, Test Plan.

System Analyst,

Architect, System

Integrator, Use-Case

Engineer, Use-Case

Specifier

Engineering

Intermediate deliverables of the development components,

Improvement Plan, Iteration Plan, Small Versions.

Component Engineer

Design

Management Corporate Knowledge Basis of Change, Project Plan. Process Engineer

METHODOLOGICAL GUIDELINES FOR SQA IN DEVELOPMENT PROCESS - An Approach Based on the SPICE

Model

273

35,29

25,92

22,85

57,14

26,47

22,22

40,00

28,57

11,76

18,51

14,28

20,58

11,11

5,71

3,70

5,90

18,54

17,16

14,29

ANALYSIS DESIGN IMPLEMENTATION MAINTENANCE

Discipline

Percentage

RUP UP MOF MSF XP Proposed

Figure 1: Artefacts per methodology in each phase.

proposed for this phase and a significant

identification with the SPICE elements it contains.

8 REPORT OF THE

EVALUATION

The SPICE model presents a more or less uniform

distribution of practices related to the disciplines of

Analysis, Design, and Implementation, which have

practices associated to CUS, ENG, and MAN

categories, whereas for Maintenance the practices

observed are associated only to ENG.

RUP participation in almost all disciplines is

significant, meaning that this methodology is the

most aligned with the SPICE Quality model,

followed by its predecessor, UP. However, it is

worth mentioning that MSF has a moderate

participation in the Analysis discipline.

According to our proposal was necessary in all

cases to satisfy the standard completely; this means

that these methodologies are not still fully mature as

far as quality is concerned. The methodological

guidelines presented can be analyzed according to

their strengths and weaknesses: it takes into account

the process efficiency and effectiveness; it makes it

possible to ensure the systemic quality of the

development process; and, since it is open, can be

adapted to the requirements of any organization; a

special emphasis is observed in risk and change

management; the customer facilitates user

involvement; it establishes disciplines and activities,

and therefore can be incorporated into most of the

methodologies based on the classic development

cycle.

However, some of the activities defined could be

divided into other activities to reduce their

complexity, and many artefacts obtained are

indicative of an object-oriented development, since

most methodologies studied here are focused on this

type of development. Finally, when each one of the

proposed activities is developed, it is necessary to

take into account the standards established by the

organization where they are applied, including the

definition of standards for the documents produced

by Methodological Adaptation. In addition,

techniques and tools incorporated into each activity

could be fed back by the organization performing it,

besides being implicit in each one of the proposed

artefacts.

9 CONCLUSIONS

The methodological guidelines presented here are

based on the analysis of different methodologies

(RUP, UP, MOF, MSF, and XP), among which,

according to the results obtained, Rational Unified

Process (RUP) is the most comprehensive regarding

the categories proposed by the SPICE process

ICEIS 2006 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION

274

quality standard. However, with these results it is

not possible to state that any of these methodologies

is 100% aligned with the ISO 15504 quality

standard.

As final result the importance of applying a

methodological orientation in the development

process can be established, since it helps to assure

quality, including activities, artefacts, and roles

associated to each process phase, to fill the

loopholes of the existing methodologies and develop

a strategy of competitive advantages and

international certification.

REFERENCES

Baltzer, A., Daryanani, S., and Díaz S. (1993) Modelo de

Capacidad de Madurez (CMM) del SEI, Madrid,

España: Addison Wesley

Callaos, N. and Callaos, B. (1996) A systemic “systems

methodology”. Proceedings of the 6th International

Conference on Systems Research Informatics and

Cybernetics, Alemania.

Engelbart, D., and Engelbart, C. (1990) Bootstrapping and

the Handbook Cycle, Telematics and Informatics (7:1),

January, pp. 27-32.

Humphey, W. (1997) Introduction to the Personal

Software Process, Cambridge, Massachusetts: Addison

Wesley Longman, Inc.

International Organization for Standardization (2004)

Software Process Assessment, TR 15504, on-line, WG

10: Software Process Assessment,

http://www.sqi.gu.edu.au/spice/.

Jacoboson, I.; Booch, J. and Rumbaugh, J. (1999) El

proceso de desarrollo de software, España.

Kendall, J. and Kendall, K. (2005) System Analysis and

Design. 6

ed. Prentice-Hall

Kitchenham, B. (1996) Evaluating Software Engineering

methods an Tools. ACM SIGSOFT. Software

Engineering Notes, 21, 1

Kruchten, P. (2003) The Rational Unified Process, USA.

Microsoft (2000) Microsoft Solution Framework (MSF),

Retrieved form http://www.microsoft.com/msf

Microsoft (2002) Microsoft Operation Framework

(MOF),. Retrieved from

http://www.microsoft.com/mof

Pérez, M.; Rojas, T.; Mendoza, L. and Grimán, A. (2001)

Systemic Quality Model for System Development

Process: Case Study. Proceedings of Americas

Conference on Information Systems, Boston.

Pressman, R. (2005) Ingeniería del Software: un enfoque

práctico, España.

Whitten, J.; Lonnie, D.; Bentley, L and Dittman, K. (2004)

Systems Analysis and Design Methods, (Kitchenham,

1996).Sixth Edition Irwin McGrawHill NY.

METHODOLOGICAL GUIDELINES FOR SQA IN DEVELOPMENT PROCESS - An Approach Based on the SPICE

Model

275