EVALTOOL

A Flexible Environment for the Capability Assessment of Software Processes

Tomás Martínez-Ruiz, Eduardo León-Pavón, Félix García, Mario Piattini

ALARCOS Research Group, Information Systems and Tecnologies Department

UCLM-INDRA Research and Development Institute, University of Castilla-La Mancha

Paseo de la univrsidad, 4 – 13071 ciudad Real, Spain

Francisco J. Pino

IDIS Research Group, Electronic and Telecommunications Engineering Faculty

University of Cauca, Street 5 # 4 – 70 Popayán, Colombia

Keywords: Software Process Assessment, Software Process Improvement, Process Reference Model, Process

Assessment Model, Process Capability.

Abstract: Software process improvement is an important aspect in achieving capable processes, and so organizations

are obviously concerned about it. However, to improve software process it is necessary to assess it in order

to check its weaknesses and strengths. The assessment can be performed according to a given assessment

process or any other and the processes of the organization can also use one particular process model or any

other. The goal of this work is to provide an environment that allows us to carry out assessments that are in

accord with various different process assessment models, on several process reference models. We have

developed an environment composed of two components; one of these generates the database schema for

storing the process reference model and assessment information and the other one assesses the process with

reference to this information, generating results in several formats, to make it possible to interpret data.

With this environment, assessment of software process is an easy task, whichever assessment process is

used, and regardless of the process model used in the organization.

1 INTRODUCTION

Quality is the most effective way to introduce any

product into the buyers’ market at the present time.

Furthermore, due to the importance of software

products in our daily life, quality is a decisive factor

in guaranteeing that products are able to do their

jobs properly.

Software development organizations know the

importance of this aspect and they are indeed

interested in the quality of software products they

create (Piattini, Garcia, & Caballero, 2006). But the

quality of a product depends on the capability of the

processes in which this product is created. Process

capability is therefore an essential characteristic and

there are two factors in this- one is image and the

other is sheer need. They have to project a positive

image if they are to export the software they produce

and they need to turn their projects into effective and

efficient ones.

Improvement of software processes is the way to

maximise both factors. Three elements are needed

when carrying out a process improvement initiative;

a process improvement method, a process reference

model and a process assessment method (Pino,

García, & Piattini, 2008). In this respect, several

organizations have been working on software

process and their capability, as well as on several

ways to evaluate and improve them.

Figure 1 shows

the quagmire with several process models and

assessment methods, together with their

interrelations.

Martínez-Ruiz T., León-Pavón E., García F., Piattini M. and J. Pino F. (2008).

EVALTOOL - A Flexible Environment for the Capability Assessment of Software Processes.

In Proceedings of the Third International Conference on Software and Data Technologies - ISDM/ABF, pages 73-80

DOI: 10.5220/0001884600730080

 SciTePress

Figure 1: Software process quagmire (www.software.org

/quagmire).

However, given the importance of assessing the

processes before, during and after the improvement

has been performed, several tools have been

developed to help users assess processes. These

tools can carry out repetitive actions, by reducing the

cognitive charge of people involved in the

assessments, and they can perform most of the

management tasks that were done manually.

The diversity of existing software process

models and assessment methods has led to the

development of tools for the evaluation of the

processes of each process model with reference to

each assessment method. Each one of these tools

depends on the appropriate process and assessment

method.

In this paper we present an environment that is

able to evaluate processes following any process

reference model and any assessment method. This

environment provides companies with the technical

support necessary to carry out and store the results

of their assessments in an integrated and consistent

way. And it also avoids the development of specific

tools for assess following each new assessment

method or each new process reference model. With

this environment, assessing software processes can

be conducted automatically and in a flexible way by

using a generic structure and assessment procedures.

This avoids having to use diverse tools to conduct

process assessments according to specific process

reference models and assessment methods.

Furthermore, EVALTOOL environment generates

the information about assessments in a consistent

and simple way, including graphics showing a

summary with the assessment results and it also

helps users to use them.

The rest of the paper is structured as follows. In

Section Two the State of the Art in software process

and tools is analysed. Section Three presents the

EVALTOOL environment and describes its main

components. In Section Four, the use of the

environment is illustrated by means of some

examples. Finally, Section Five contains the

conclusions and future work.

2 STATE OF THE ART

Several process models and assessment models have

been created (See Figure 1) in recent times. Several

tools to apply them have also been developed. In this

Section we summarize the most outstanding models,

and analyze the most widely-used tools.

The Software Engineering Institute of Carnegie

Mellon University has developed the Capability

Maturity Model Integrated (CMMI) (SEI, 2004).

This model is based on CMM (Capability Maturity

Model) and contains the best practices, grouped in

several processes. This process model defines six

maturity levels, which classify organizations in a

range from chaotic level to continuous improvement

level and it can be used in staged or continuous

representations.

CMMI uses the Software CMMI Appraisal

Method for Process Improvement (SCAMPI) (SEI,

2001). This method includes the best assessment

practices and defines three steps to plan and prepare

the assessment, carry it out, and inform of its results.

The International Organization for

Standardization has defined both a process and an

assessment method: ISO 12207 (ISO, 2004a) is a life

cycle process model that defines the main activities

that must be performed during the software

development. It groups these activities into

processes and categorizes the processes. This norm

specifies the life cycle process architecture but not

how to implement it. ISO 15504 (ISO, 2004b,

2004c, 2004d) is the standard for carrying out

assessments. The last version is divided into five

parts and it defines the minimum requirements to

guarantee that the assessment results obtained are

repeatable and consistent. It defines six capability

levels.

There are, moreover, other models oriented to

Small and Medium Enterprises (SMEs). MoProSoft

(Oktaba et al., 2005) is the process model for

Mexican SMEs. It contains nine processes, grouped

into three categories, along with EvalProSoft

(Oktaba et al., 2004), which is its assessment

method. It defines six capability levels, like ISO

15504. The COMPETISOFT project (Oktaba et al.,

2007) defines a process model, an assessment

ICSOFT 2008 - International Conference on Software and Data Technologies

method and an improvement model. These models

are designed to be used by SMEs. Its process model

defines ten processes, grouped into three categories

as an organization hierarchy. The Assessment

method defines six capability levels.

On the other hand, several tools have been

developed to help in the assessing of processes. The

most outstanding ones are based on CMM, CMMI

and ISO 15504.

CMM-Quest

allows us to evaluate the most

important processes of an organization, to determine

strengths and weaknesses. It assigns values to

objectives, but it can not assess process practices.

Appraisal Wizard

is based on SCAMPI, using

CMMI as process model. It offers support for

assigning values to the elements of the process

(practices, objectives, etc). It allows the re-use of the

results of an assessment in another later one. There

is a light version of this tool called Appraisal Lite

Spice 1-2-1

assigns values to base and generic

practices. SPiCE Lite

offers a quick and efficient

way to detect weaknesses and strengths of the

process. It shows results as reports or via web and it

has two running modes.

Appraisal Assistant Beta

offers support for

evaluating the maturity of an organization by the

creation of user defined models, converting results

from one framework to another, and generating

reports about each assessment. Appraisal Assistant is

now in a Beta version.

In literature we can find other assessment tools,

even tools based on spreadsheets. Their functionality

is similar to that of the above tools. The main

drawback of these tools in comparison to the present

work is that each one of these tools uses a specific

process model and a specific assessment method (to

see

Table 1). In the context of the present work our

objective is to provide companies with a flexible

tool so they can assess their software processes by

using different reference models and assessment

methods but by using a single environment which

facilitates comparison of results. Another important

characteristic is that an important development effort

has focused on the user interaction facility, by

creating a usable GUI.

3 EVALTOOL

A Flexible Environment for the capability

assessment of Software Processes has been

developed; it is called EVALTOOL. This

environment has the following characteristics:

 It allows assessment using different assessment

methods (ISO/IEC 15504, SCAMPI,

EvalProSoft).

 It is flexible: it allows the defining and addition

of new processes by Process Reference

Models and new assessment methods, when

the methods and processes are compatible

with the environment core.

 It allows comparisons between the results of

several assessments.

 It stores the models in their repository.

 Its reports show information in diagram form.

 It has a very usable interface, by means of

enriched interfaces.

Table 1: Process models and assessment methods used by

assessment tools.

Tool Reference

Model

Assessment

Method

CMM-Quest CMMI-SE/SW

continuous

ISO/IEC 15504

Appraisal

Wizard

Appraisal Lite

CMM, CMMI-

SE/SW staged

and continuous

SCAMPI

SPiCE 1-2-1

SPiCE Lite

ISO/IEC 15504 ISO/IEC

15504:1998

Appraisal

Assistant Beta

CMMI ISO/IEC 15504,

SCAMPI

EVALTOOL Any Any

To support the assessments using several assessment

methods and several process reference models, we

have defined a generic metamodel (

Figure 2). In this,

the elements defined in outstanding process

reference models such as ISO 12207, CMMI,

COMPETISOFT and respective assessment methods

(ISO 15504, SCAMPI, EvalProSoft), have been

taken into account. The environment uses this

metamodel to guide the assessments. Because of

this, the environment is flexible; that is, the

environment is able to assess any processes (from a

process reference model) that are defined in accord

with this metamodel. That being so, the new process

included in the environment must be based on

conformance of the process reference model with

the metamodel defined in the core of the

environment. In addition, some modules can be

designed to implement the other assessment

methods.

EVALTOOL - A Flexible Environment for the Capability Assessment of Software Processes

Level

levelNumber : int

description : String

state : String

ID : int

<<ORM Pers istable>>

Model

modelName : String

description : String

numberOfLevels : Integer

ID : int

<<ORM Persistable>>

0..n

+Levels

0..n

+Model

contains n levels

AssessmentMark

markName : String

organization : String

responsible : String

ID : int

<<ORM Pers istable>>

0..n

+Model

+AssessmentMark

0..n

From each model

there is n mark s

ProcessType

type : String

description : S tring

ID : int

<<ORM Persistable>>

Process

process Name : String

modelName : String

levelNumber : Integer

ID : int

<<ORM Persistable>>

0.. n

+Process

0.. n

+AssessmentMark

Each mark has n

procces t o assess

+Type

+Process

A process is of

type

Attribute

attributeName : String

description : String

objetive : String

minimunFactorCompletness : float

ID : int

<<ORM Persis t able>>

0..n

+Attributes

0..n

+Level

Each level has n attributes

AssessedAttribute

assess ed : boolean

description : String

ID : int

<<ORM Persistable>>

0..n

+Attribute

+AssessedAttribute

0..n

From each attribute

there is n assess ed

attributes

1..n

+AssessedAttribute

1..n

+Process

Each

assessedAt tribute

belongs a process

FixedQuestion

questionText : String

fac t or : float

ID : int

<<ORM Persistable>>

0.. n

+FixedQuestions

0.. n

+Attribute

An attrib ute can have

filex questions

Question

questionText : String

fac t or : float

ID : int

<<ORM Persis t able>>

0..n

+Question

0..n

+AssessedAtribute

Each assessed attribute

has n questions

Ans werType

type : String

ID : int

<<ORM Pers is table>>

+AnswerType

Each question has

an answer type

0..n

+AnswerType

0..n

Each attribute has an

asnwer type

Answer

answerText : String

fac t or : float

ID : int

<<ORM Pers istable>>

+Answer

Each question

has an answer

0..n

+AnswerType

+Answer

0..n

Eac h answertype

has n asnwers

Figure 2: Generic metamodel of process assessment used

by EVALTOOL.

The environment is prepared to generate and store

the results of each assessment in the same way. So it

is possible to retrieve these data and compare them

with the results of later assessments. In this way it is

easy to see the improvement executed in each

process of the organization quickly, by means of the

comparison of the results of the process assessment.

These results are shown using diagrams, to make

them more understandable. Another feature is that

its interface has been produced using enriched

interfaces, so it is very easy and usable.

The environment is composed of two parts. The

first one manages the process reference models and

the second one applies the assessment methods over

these processes.

Figure 3 shows the relationship

between both parts which are described below. Both

components are linked by the database, which is

used by first one to write the new schemas. These

schemas are used by the evaluation component to

obtain information about the organization that is

assessed and its processes.

3.1 Process Model Management

This component (left side of Figure 3) manages the

process reference models on which the assessments

are conducted. This component supports the

inclusion of flexibility in the environment processes

related to different process reference models. This

part allows the management of models, metamodels

and schemas, by applying QVT transformations. It is

able to generate the schema associated with each

model, to allow the information of each process to

be stored in a way that is compatible with the

environment. The elements of the process (among

others purpose, activities, roles and work products)

stored are used to assess that process.

Figure 3: Components of the environment EVALTOOL.

ICSOFT 2008 - International Conference on Software and Data Technologies

Figure 4: Model application and evaluation part of EVALTOOL interface for SCAMPI and ISO/IEC 15504 assessments.

This part also defines the inverse transformation,

from schema to the associated metamodel. The two

transformations can be carried out using both run

modes, automatically, where models or schemas are

transformed without asking the user. They can also

be customisable, in which the user contributes with

the semantic information to achieve idempotent

transformations between models and schemas.

This part is also able to generate a XMI file with

the information stored in the base. To do this, the

metamodel associated with the schema is obtained

and used to create the XMI:SCHEMA.

The interface of the process model management

application is a desktop application and it is a very

simple one. It offers help to the user and is available

in English and Spanish. A more detailed description

of this component can be found in (Martínez-Ruiz,

García, & Piattini, 2008).

3.2 Model Application and Evaluation

This component allows the definition and

assessment of software process, using the databases

created by the component described previously. It

has been developed using the most advanced

technologies. Because of this, it has a very simple,

intuitive and easy GUI that helps users to manage

the different tool functionalities (

Figure 4).

It has the functionality of defining new

assessment marks based on existing reference

models, already included in the database. An

assessment mark includes processes to be assessed

and their evaluations. From this it is possible to add

processes specific to an organization in order to

evaluate them, with reference to one of the schemas

defined as well as in accord with the metamodel

used by the environment.

EVALTOOL - A Flexible Environment for the Capability Assessment of Software Processes

This evaluation part is further able to assess the

chosen processes by means of answering several

questions. The questions are defined from the

process reference model stored in the database. An

additional feature is that, due to the fact that the

results of each assessment are stored in a common

way, a comparison between them is possible. We

can thereby obtain both the weaknesses and the

strengths of processes (and organization). From

these we know the points in which an improvement

effort needs to be carried out. These are called

improvement opportunities.

From the main menu (left side of

Figure 4) we

can create and assess several marks. It is also

possible to watch a demonstration of the application.

This application of the environment has two

versions; one of them is designed for PC, designed

as a web page and offers all the functionality of

these. The other one is designed to run in a Pocket

PC and offers a subset of this functionality. This

version is only able to answer the assessment

questions about the process, in a dynamic way and

within the scene where the software is developed

and the processes take place. It allows us to do

quick- assessment so the improvement opportunities

can be known very rapidly.

4 EVALTOOL APPLICATION

EXAMPLE

EVALTOOL gives support to the process diagnosis

activity of an improvement model. This environment

has been designed to carry out software process

assessments easily. In

Figure 5 we can see how the

environment is related to the process reference

models, the process assessment methods and the

process improvement models. The aim of all this is

to perform assessments of company process, whose

results can be used to begin a process improvement

cycle.

Prior to assessment in the context of a software

organization, some steps are necessary. First of all,

the goals and benefits of the software process

assessment must be presented to the organization

work forces. Leaders must be in agreement with the

assessment and the employees involved must receive

the qualification about the models defined in the

environment before they use it.

Figure 5: Assessment program with EVALTOOL.

To do an assessment of a previously defined

mark using EVALTOOL, it is only necessary to

select an Evaluate Mark, and to choose the mark.

Then the process can be assessed, by selecting and

answering some questions about it. Questions are

ordered by processes and their attributes. Once the

questions are answered, you can see the assessment

report generated (

Figure 6). The user is also informed

about the process whose capability level is too low.

Figure 6: EVALTOOL assessment results.

The environment stores the results obtained in

each assessment and this information is very useful

in seeing the progress between two assessments of

the same improvement cycle.

Figure 7 shows a

comparative chart of the results of two assessments

carried out in the same improvement cycle.

If the mark is not defined, we can define it by

using the Create Mark. The processes of this mark

must be in line with the process reference models

included in the environment. By means of adding a

mark, a new mark is created, and we can include the

processes of a given organization (

Figure 8) within it.

It is, moreover, possible to add some questions to the

process attributes of each process.

ICSOFT 2008 - International Conference on Software and Data Technologies

Figure 7: EVALTOOL assessment result comparative.

If the processes of the new mark are not in

accord with the process reference models included in

the environment, this process reference model has to

be defined in the environment. By means of the

desktop process model management application, a

compatible schema is created in the database,

thereby storing all the information about each of the

processes of this new process model as well as their

sub-elements.

Figure 8: EVALTOOL adding process questionnaire.

5 CONCLUSIONS AND FUTURE

WORK

The assessment of software processes is used by

software development organizations as a means of

knowing the capability level of processes and of

improving them. To make the task of assessing the

organization processes easier, a flexible environment

for the assessment of the capability of software

processes has been developed. It is called

EVALTOOL.

The environment is designed in such a way as to

be used to evaluate a software process by employing

any assessment process. This flexibility allows

organizations that have a lot of different projects and

different processes to use a single tool to evaluate

them. In addition, the environment offers the

possibility of comparing two capability levels.

Furthermore, the evaluation of software

processes gives the work forces of the organization

the following positive features:

 Most of the questions presented in the

questionnaires help users to have better

knowledge of the changes that are being

carried out, as well as the activities that would

be performed and the difficulties entailed in

them.

 The recommendations which the tool gives on

the activities that are not carried out help them

to provide feedback about their process

improvement.

 The environment gives the improvement-

manager a viewpoint from which to oversee

the process assessed, over a period of time.

The environment is being used to carry out

assessments based on the COMPETISOFT

Assessment Method. Several Iberoamerican

organizations are using them to assess and improve

their processes.

Our future work is to extend this environment by

adding mechanisms to help users to improve their

processes. Based on the results of the assessments, it

is possible to determine the strengths and

weaknesses of each process. That it turns makes it

possible to establish what action is needed for the

improvement of the process.

The environment is currently designed to run

with a SQL Server database. Other future work can

be to adapt it to run with other data sources, such as

ODBC, or XML files.

We might also point out that the environment can

be adapted to be used as a didactic tool. It can offer

support for teaching users how to perform an

assessment using any of the assessment methods and

any process model.

ACKNOWLEDGEMENTS

This work is partially supported by the investigation

into Software Process Lines sponsored by Sistemas

Técnicos de Loterías del Estado S.A. within the

framework of the agreement on the “Innovación del

Entorno Metodológico de Desarrollo y

EVALTOOL - A Flexible Environment for the Capability Assessment of Software Processes

Mantenimiento de Software”, as well as by the

ESFINGE TIN2006-15175-C05-05 projects,

financed by the Spanish Ministry of Science and

Technology, COMPETISOFT, financed by CYTED

(Ciencia y Tecnología para el Desarrollo-

506AC0287), MECENAS (JCCM, Consejeria de

Educación y Ciencia, PBI06-0024), and INGENIO,

financed by the Junta de Comunidades de Castilla-

La Mancha, Consejería de Educación y Ciencia,

PAC08-0154-9262.

REFERENCES

ISO. (2004a). ISO/IEC 12207:2002/FDAM 2. Information

technology - Software life cycle processes. Geneva:

International Organization for Standardization.

ISO. (2004b). ISO/IEC 15504-2:2003/Cor.1:2004(E).

Information technology - Process assessment - Part 2:

Performing an assessment. Geneva: International

Organization for Standardization.

ISO. (2004c). ISO/IEC 15504-3:2003/Cor.1:2004(E).

Information technology - Process assessment - Part 3:

Guidance on Performing an Assessment. Geneva:

International Organization for Standardization.

ISO. (2004d). ISO/IEC 15504-4:2003/Cor.1:2004(E).

Information technology - Process assessment - Part 4:

Guidance on use for process improvement and process

capability determination. Geneva: International

Organization for Standardization.

Martínez-Ruiz, T., García, F., & Piattini, M. (2008).

Meta2Relational: Herramienta para la Gestión de

Modelos de Procesos Software. VII Jornadas

Iberoamericanas de Ingeniería del Software e

Ingeniería del Conocimiento, Guayaquil, Ecuador,

p.251-258.

Oktaba, H., Alquicira, C., Su, A., Martínez, A.,

Quintanilla, G., Ruvalcaba, M., López, F., Rivera, M.,

Orozco, M.J., Fernández, Y., & Flores, M.A. (2005).

Modelo de Procesos para la Industria de Software -

MoproSoft - Versión 1.3, Agosto de 2005. NMX-

059/01-NYCE-2005. Ciudad de México: Organismo

nacional de normalización y evaluación de la

conformidad - NYCE.

Oktaba, H., Alquicira, C., Su, A., Palacios, J., Pérez, C. J.,

López, F., Quintanilla. G., Montero, C., & Calvo. A.

(2004). Método de Evaluación de procesos para la

industria de software - EvalProSoft - Versión 1.1,

Marzo de 2004. NMX-I-006/(01 al 04)-NYCE-2004.

Ciudad de México: Organismo nacional de

normalización y evaluación de la conformidad -

NYCE.

Oktaba, H., Garcia, F., Piattini, M., Pino, F., Alquicira, C.,

& Ruiz, F. (2007). Software Process Improvement in

Small Latin-American Organizations:

COMPETISOFT Project. IEEE Computer, 40(10), pp.

21-28.

Piattini, M., Garcia, F., & Caballero, I. (2006). Calidad de

Sistemas Informáticos. Madrid: Ra-Ma.

Pino, F., García, F., & Piattini, M. (2008). Software

Process Improvement in Small and Medium Software

Enterprises: A Systematic Review. Software Quality

Journal, 16(2), 237-261.

SEI. (2001). Standard CMMI® Appraisal Method for

Process Improvement (SCAMPI), Version 1.1: Method

Definition Document (CMU/SEI-2001-HB-001).

Pittsburgh: Software Engineering Institute (SEI).

SEI. (2004). Capability Maturity Model Integration

(CMMI), Version 1.1 CMMI (CMMI-SE/SW/IPPD/SS,

V1.1) Staged Representation (No. CMU/SEI-2002-

TR-012 ESC-TR-2002-012). Pittsburgh: Software

Engineering Insittute (SEI).

CMM-Quest. Available on: http://www.cmm-quest

.com/english. Accessed in March 2008.

Appraisal Wizard. Available on: http://www.

gantthead.com/sharedComponents/offsite.cfm?link=htt

p%3A%2F%2Fwww%2Eisd%2Dinc%2Ecom.

Accessed in March 2008.

Appraisal Lite. Available on: http://www.gantthead

.com/sharedComponents/offsite.cfm?link=http%3A%2

F%2Fwww%2Eisd%2Dinc%2Ecom. Accessed in

March 2008.

SPICE 1-2-1. Available on: http://www.spice121

.com/english. Accessed in March 2008.

SPICE Lite. Available on: http://www.spicelite

.com/English. Accessed in March 2008.

Appraisal Assistant Beta. Available on: http://

www.sqi.gu.edu.au/AppraisalAssistant/about.html.

Accessed in March 2008.

ICSOFT 2008 - International Conference on Software and Data Technologies