EFFECTS OF EXPERT SYSTEMS IN COMPUTER BASED

SUPPORT FOR CMMI IMPLEMENTATIONS

Ercan Öztemel and Nilgün Gökmen

Tubitak National Research Institute of Electronics and Cryptology, Gebze, Kocaeli, Turkey

Marmara University, Industrial Engineering Department, Istanbul, Turkey

Keywords: CMMI, Knowledge based System, Expert System, Software Quality, TRENSIM, Train Simulator.

Abstract: Computing systems are becoming more complex in very dynamic and uncertain situations. Due to this

complexity, the importance of process focused quality approaches is increasing. Capability Maturity Model

Integration (CMMI) standards and implementation practices were developed to simplify the software

project management and to assure expected quality of the respective software. Realizing the CMMI

systems, building and monitoring the implementation practices require an extensive knowledge and

experience. Organizations receive these mainly through consultants which may become too costly in most

of the cases. Although there have been some computer based support tools available in the market, those

still require human experts to justify the related artifacts. In this study a knowledge-based assistant system

so called “CMMI Assistant” is introduced. The main aim of this tool is to support CMMI implementations

through utilising expert system methodology.

1 INTRODUCTION

Today, companies want to deliver their products

faster, cheaper and more efficiently. They must

ensure the reliability of their products and meet their

customers’ expectation. So they must be able to

manage and control their processes in the best way.

CMMI model is capable of handling customer

requirements in this way especially for software

products. This makes it very popular and surely

increases the demand to implement respective

processes.

CMMI is a reference model that covers the

software development and maintenance activities

applied to both products and services. This model

outlines clear definition of processes with respective

roles and responsibilities, and makes sure that the

software development projects are carried out

without any problem. It has a positive influence on

cost, schedule and quality performance (Hollenbach,

2003). It can provide a path for organizations to

achieve their performance goals.

Besides so many advantages, implementing

CMMI is not easy. It requires extensive knowledge

as well as organizational endurance. A Standard

CMMI Appraisal Method for Process Improvement

(SCAMPI) has been developed to support the

companies to improve and prove that they posses

respective capabilities.

Note that the model includes 22 process areas at

different levels. Each process is defined through a

set of specific and generic goals as well as some

practices. The model requires that all processes for

respective levels to be satisfied. Understanding how

those processes are satisfied requires CMMI

knowledge. This clearly indicates the need for the

experts to be employed for the sake of proper

implementation. But, hiring experts to prepare the

organization is a timely and costly activity.

Literature survey indicates that there are some

studies developing questionnaires to measure CMMI

levels of an organization (Yücalar, 2006). But, the

questions designed for this purpose are mainly

superficial and there is usually no set of

recommendations to yield respective improvements.

Similarly, on the market, there are several computer-

aided assistant systems as discussed in the following

sections of this paper. Although they are beneficial

to understand the model and to perform self CMMI

evaluation, those still requires human experts to

justify the respective artifacts of the processes.

Since knowledge based systems are proven to be

capable of storing and utilizing expert knowledge in

certain domains (MacKinnon, 2009), they may be

425

Öztemel E. and Gökmen N. (2010).

EFFECTS OF EXPERT SYSTEMS IN COMPUTER BASED SUPPORT FOR CMMI IMPLEMENTATIONS.

In Proceedings of the 5th International Conference on Software and Data Technologies, pages 425-430

DOI: 10.5220/0002919304250430

 SciTePress

very well utilized for assisting staff of the

organization intending to implement CMMI

processes. There have been some researches along

this line as well. An intelligent fuzzy agent based on

process and product quality assurance (PPQA) is one

of them (Wang and Lee, 2007). It introduces

ontology for supporting CMMI assessment.

Similarly a methodology system for CMMI

diagnosis and analysis is being developed and

patented by Dagnino (2005).

The paper introduces CMMI Assistant which is a

knowledge based system capable of defining the

CMMI areas where lack of implementation exist and

recommend remedies to overcome the respective

gaps.

2 BRIEF OVERVIEW OF CMMI

Upon request of Department of Defence, CMMI was

developed by several dedicated teams in Software

Engineering Institute (SEI) of Carnegie Mellon

University (CMMI Product Team, 2006).

CMMI Assistant which is the subject of this

study mainly covers “CMMI for Development”

model. It is a reference model that covers the

development and maintenance activities applied to

both software products and services. Note that all

activities are carried out by a set of processes with

respective goals and practices. Each process area

includes three types of components such as required

components (specific and generic goals), expected

components (specific and generic practices),

informative components (sub practices, typical work

products...etc.).

For the sake of better process improvement,

these components are handled on the bases of two

different representations mainly: The continuous

representation and the staged representation.

The continuous representation enables an

organization to select a process area (or group of

process areas) and improve the respective processes.

This representation uses capability levels (CLs)

numbered from 0 to 5 to characterize improvements.

As a starting point, if the organization does not

know where to start and which processes to choose

to improve, the staged representation will be a good

choice as it gives a specific set of processes to

improve at each stage and is mainly used for

certification. Due to this, the staged representation

with maturity levels (MLs) numbered from 1 to 5 to

characterize improvements is considered to be the

main focus of the system proposed in this paper.

The SCAMPI is a well defined and accepted me-

thod used for conducting appraisals. The SCAMPI

family of appraisals includes Class A, Class B, and

Class C appraisal methods. SCAMPI A is the most

rigorous method and the only method that can result

in a rating. In SCAMPI A, the organization should

show that all specific and general practices for the

intended maturity level are implemented correctly.

In this study, all Direct Artifacts (D) and Indirect

Artifacts (I) for process areas were identified and

used for rating the maturity level of the organization

under assessment.

3 COMPUTER-AIDED CMMI

In an organization, understanding conceptually the

CMMI, mapping the software developing or

operational processes to CMMI process areas and

respective practices is a difficult activity which may

not always produce expected results. This requires

several internal assessment activities in order to

trigger improvements and prevents respective gaps.

However, conducting internal assessments before

SCAMPI A is costly and necessitates extensive

labour intensive tasks. Because of these reasons,

several computer- aided CMMI assistant programs

are developed in order to support and map the

organizational processes to respective and expected

CMMI capabilities easily. These programs mainly

provide support for conducting internal assessments

in the organization before SCAMPI A. However,

they still require intervention of human experts or

extensive CMMI and domain knowledge to make

decisions on the available artifacts.

There are several computer-aided assistant

systems such as CMM Quest (HM&S, 2009),

Appraisal Assistant Tool (Griffith University, 2007),

Compass (Vector Consulting, 2009), APEX

(Milman and Joubert, 2009), CMMI Appraisal

Recorder (SE-CURE AG, 2009) and QMIM Tool

(Kelemen et al., 2007) for the implementation of

CMMI. They are extremely important and beneficial

to understand the model and to perform self

evaluation on the process areas. However, it is

important to note that those systems are not

intelligent systems and they can only provide a road

map for the developers to make implementations.

Internal decisions are left to the user. There is still a

need for a system to provide both a road map as well

as set of recommendations for possible

improvements.

These computer-aided systems have common

specifications such as;

ICSOFT 2010 - 5th International Conference on Software and Data Technologies

426

 Mapping the typical work products which should

be demonstrated to the company artifacts is

decided by user.

 The user decides whether the artifacts are direct

or indirect, strong or weak.

 The user decides practices, goals and process

areas characterizations according to the situation

(strong or weak) of the respective artifacts.

These basic capabilities can be performed by the

computer provided that the respective knowledge is

stored within the knowledge base. Expert system

technology presents related methodologies and

background knowledge to create fully automated

expert CMMI advisor so called “CMMI Assistant”.

4 CMMI ASSISTANT

Expert systems are computer systems that are

capable of solving problems as human experts of the

same domain would do when they face the same

problem. An expert system is composed of a

knowledge base, inference engine, and a user

interface which handles the communication between

the user and the system (MacKinnon L. M., 2009).

The knowledge base contains the knowledge and

the respective rules for the expert system. Expert

systems generally represent knowledge in the form

of “IF…. THEN……ELSE …..”

The inference engine draws conclusions through

scanning, filtering and interpreting the knowledge

already stored in the knowledge base. CMMI

Assistant uses Forward Chaining (data-driven

reasoning) (Pomykalski et al., 1999) inference

mechanism. In this case, reasoning starts with

certain facts and certain answer to a certain problem

is sought based on the availability of the facts.

While this system is being developed, knowledge

and experiences of experts, detailed model

information, implementation framework (Kökten,

2007.), some implementation questionnaires

(Carnegie Mellon SEI, 2002), appraisal questions

(Kalaycı, 2007), “lessons learned” of some firms and

the documentation of the Practice Implementation

Indicator Database (PIID) (Kneuper, 2009) are

utilized.

Architecture of CMMI Assistant developed is

shown in the Figure 1. As in traditional expert

systems, the CMMI Assistant has a knowledge base,

an inference engine, and a user interface.

4.1 Knowledge Base

There is a need for domain knowledge which was

stored in so called the knowledge base. Since the

proposed methodology is designed to assist CMMI

ML2 implementation, the knowledge base is

populated with the knowledge related to ML2.

Weighted artifacts list, CMMI ML2 process and

assessment information are placed in this knowledge

base.

4.2 Inference Engine

The inference engine manages to make decisions on

the status of CMMI implementations and provide

directions. The system is mainly based on the

understanding that there are two characterizations

for the artifacts: Strong and Weak. These

characterizations are used by CMMI Assistant to

identify the gaps so that possible remedies can be

recommended to overcome them.

There are five situations for CMMI practices:

 If Direct Artifact (D) and Indirect Artifact (I) are

weak and the system gives a recommendation,

then the practice is not implemented.

 If any of D and I is weak and the system gives a

recommendation, then the practice is partially

implemented.

 If D and I are strong and the system gives a

recommendation, then the practice is largely

implemented.

 If D and I are strong and the system gives no

recommendation, then the practice is fully

implemented.

 If any data is not given to the system, then the

practice is not yet implemented.

If all of the related practices are largely or fully

implemented, the goal is considered to be satisfied.

 OrganizationalCMMI

LevelRating

 Recommendations

Weighted

Artifacts

List

CMMIML2

Process

Information

Assesment

Information

Decision‐maker

Process

Ownersand

Managers

Organizatio‐

nalProcess

Data

Figure 1: Components of CMMI Assistant.

Otherwise in turn it is unsatisfied. If all of the

related goals are satisfied, the process area is

supposed to be satisfied.

EFFECTS OF EXPERT SYSTEMS IN COMPUTER BASED SUPPORT FOR CMMI IMPLEMENTATIONS

427

Forward chaining rules on how to decide

“Configuration Management process area” is

satisfied are given as an example below. Note that,

“SP x.y” indicates x

specific practice of y

specific

goal in a process area. Artifacts order is shown as a,

b, c…etc. Also note that, x in “SG x” indicates x

specific goal in a process area.

Example:

Required Direct Artifacts:

SP 1.1 a: Identified configuration items

SP 1.1 b: Specifications of configuration items

SP 1.1 b.1: unique identifier

SP 1.1 b.2: owner responsible

SP 1.1 c: Specifications of configuration items

SP 1.1 c.1: author

SP 1.1 c.2: document files type

SP 1.1 c.3: programming language

SP 1.1 d: Documented plan for keeping information

about configuration items

These are represented as a rule such as the

following.

Rule 1:

IF SP 1.1 b.1 AND SP 1.1 b.2

THEN SP 1.1 b

IF SP 1.1 c.1 OR SP 1.1 c.2 OR SP 1.1 c.3

THEN SP 1.1 c

IF SP 1.1 a AND SP 1.1 b AND SP 1.1 c

THEN D of SP 1.1 is Strong

ELSE D of SP 1.1 is Weak.

Similarly Indirect Artifacts Expected:

SP 1.1 d: Criteria for choosing configuration items

SP 1.1 d.1: Work products that may be used by

two or more groups

SP 1.1 d.2: Work products that are expected to

change over time either because of errors or

change of requirements

SP 1.1 d.3: Work products that are dependent on

each other in that a change in one mandates a

change in the others

SP 1.1 d.4: Work products that are critical for the

project

SP 1.1 e: Criteria for when placing an configuration

item under Configuration Management System

SP 1.1 e.1: Stage of the project lifecycle

SP 1.1 e.2: When the work product is ready for

test

SP 1.1 e.3: Degree of control desired on the work

product

SP 1.1 e.4: Cost and schedule limitations

SP 1.1 e.5: Customer requirements

These can be coded as the following rule.

Rule 2:

IF SP 1.1 d.1 OR SP 1.1 d.2 OR SP 1.1 d.3 OR SP

1.1 d.4

THEN SP 1.1 d

IF SP 1.1 e.1 OR SP 1.1 e.2 OR SP 1.1 e.3 OR SP

1.1 e.4 OR SP 1.1 e.5

THEN SP 1.1 e

IF SP 1.1 d AND SP 1.1 e

THEN I of SP 1.1 is Strong

ELSE I of SP 1.1 is Weak

According to the artifacts characterizations

respective recommendations are provided such as

“You must identify Plans, Process descriptions,

Product specifications, Requirements, Interface

description, Design data, Drawings as a

configuration item in the Design stage”.

Based on the availability of the artifacts, Rule 3,

4 and 5 can assure practices are implemented to a

certain degree and related goal and process area are

satisfied respectively.

Rule 3:

IF D of SP 1.1 is Weak AND I of SP 1.1 is Weak

THEN SP 1.1 is not implemented

ELSE IF D of SP 1.1 is Weak AND I of SP 1.1 is

Strong OR D of SP 1.1 is Strong AND I of SP 1.1 is

Weak

THEN SP 1.1 is partially implemented

ELSE IF D of SP 1.1 is Strong AND I of SP 1.1 is

Strong

THEN SP 1.1 is largely implemented OR fully

implemented.

All SPs and GPs are evaluated in the same way.

Rule 4:

IF SP 1.1 AND SP 1.2 AND SP 1.3 are largely OR

fully implemented

THEN SG 1 is satisfied

ELSE SG 1 is unsatisfied.

All SGs and GGs are evaluated in the same way.

Rule 5:

IF SG 1 AND SG 2 AND SG 3 are satisfied

THEN CM is satisfied

ELSE CM is unsatisfied.

All process areas are evaluated in the same way.

4.3 User Interface

The user interface has two aspects:

a) Screen design:

It is the interface of the program

with the user in such a way that the user enters

his organizational data and the system provides

respective expert response.

b) Explanation:

The user is informed about the

reason behind the decisions. CMMI Assistant is

designed in such a way that the recommendation

to the user includes the reasons behind the

ICSOFT 2010 - 5th International Conference on Software and Data Technologies

428



Figure 2: CMMI Assistant ABC Organization appraisal result page.

decisions in any case. This would definitely decrease

the complexity of the system design.

5 CASE STUDIES

Two case studies are designed in order to show the

proof of concept for CMMI Assistant. In the first

case, the program developed was implemented in

TUBITAK Marmara Research Center (MRC),

Information Technology Institute (ITI) where

several software projects (small, medium and big

size) are carried out. The program was tested mainly

using the process information from the project called

“TRENSIM” which is a driving simulator project

developed for Turkish Railways.

All required knowledge about ML2 process areas

were collected from the MRC ITI process owners.

Specific forms are designed to acquire the

knowledge expected.

Results show that TRENSIM Project satisfies

ML2 of CMMI model. Performing the respective

analysis for the overall organization on some other

projects, CMMI Assistant decided that the institution

is complying with CMMI ML2 standards. This was

also proved through real life implementation and

TUBİTAK MRC ITI, after successful completion of

a SCAMPI Class A Appraisal on 19 December

2008, attained CMMI-DEV v1.2 Staged

Representation ML 3. (ITI, 2008) This is an

indicator that proves accuracy of CMMI Assistant

appraisal results.

Second case study is carried out in a software

developing company ABC which is located in

TUBITAK technopark. All questions about ML2

process areas were collected from company ABC

and knowledge base of CMMI Assistant is populated

with company specific knowledge as well. After

data and knowledge acquisition process CMMI

Assistant decided that the organization is having

several gaps in complying with the required level of

CMMI implementations of the respected processes.

The organization has to improve itself especially in

terms of creating baselines. The CMMI Assistant

provided several reasons for this possible gap and

recommended some actions for improvements.

The result of organizational appraisal for

company ABC is shown in the Figure 2. Note that

SAM Process area is not applicable in ABC. It is

clearly seen that the organization does not satisfy

ML 2. They have to improve their processes by

taking into account all recommendations provided

by CMMI assistants. The program provided both a

road map and recommended areas to be improved.

6 CONCLUSIONS AND FUTURE

WORK

In this study, CMMI Assistant which is an expert

EFFECTS OF EXPERT SYSTEMS IN COMPUTER BASED SUPPORT FOR CMMI IMPLEMENTATIONS

429

system based computer program was developed in

order to provide required expertise to organizations

willing to go for SCAMPI A. The system proposed

is capable of defining the areas where lack of

implementation exist and recommend actions to

overcome those gaps. The case studies clearly

proved the concept and showed the potential benefits

of the proposed approach.

The study for improving the capabilities of

CMMI Assistant continues. It will be extended to be

able to handle MLs 3, 4 and 5 as well. It is also

considered to add some statistical analysis

capabilities to the system to be able to asses the

improvement level of the respective organization.

Note that, it can also be enriched with the capability

of performing appraisals for continuous

representation in any category.

Further study can also be carried out to embed

learning ability to the system in order to create high

degree of flexibility to the system. With the help of

artificial intelligence technologies, agent-based

systems this may seem to be possible but still needs

further research on defining possible

implementations.

REFERENCES

Carnegie Mellon SEI, 2002. “Implementation of the

CMMI Evaluation Model as E-learning Component”.

http://donner.cs.uni-magdeburg.de:8080/elearning/

index.jsp, (available on 14.04.2009)

CMMI Product Team, 2006. Technical Report. “CMMI

for Development Version 1.2.”, Carnegie Mellon SEI.

Pittsburgh, USA.

Dagnino A., 2007. United States Patent Application

20070150293.“Method and System for CMMI

Diagnosis and Analysis”, Patent Application

Publication. USA.

Griffith University, SQI, 2007. “Appraisal Assistant”,

http://www.sqi.gu.edu.au/AppraisalAssistant/indexFra

meset.html (available on 12.03.09)

HM&S IT-Consulting, 2009. http://www.cmm-

quest.com/english/about.htm (available on 12.03.09)

Hollenbach C., 2003. “Quantitatively Measured Process

Improvements at Nortrop Grumman IT”, Nortrop

Grumman Information Technology.

http://www.sei.cmu.edu/library/assets/2003-CMMI-018.pd

f (available on 14.01.2010).

ITI, 2008. http://www.mam.gov.tr/english/BTE/index.html

(available on 14.01.2010).

Kalaycı O., 2007. “CMMI: Yöneticiler için Doğru

Sorular”, Shamrock Process Improvement and

Innovation. Toronto, Canada, 1

edition.

Kelemen Z. D., Balla K., Boka G., 2007. Conference

Proceedings. “Quality Organizer: A Support Tool in

Using Multiple Quality Approaches”, ICCC.

Kneuper, R., 2009. “PIID and SCAMPI Tool

(PST)”.http://www.kneuper.de/English/PIID-SCAMPI

-Tool/, (available on 12.03.2009).

Kökten D., 2007. Master’s Thesis. “CMMI

Implementation Framework”, İstanbul Technical

University Institute of Science and Technology.

İstanbul, Turkey.

MacKinnon L. M., 2009. “Expert Systems, Databases &

Information Systems”, http://www.macs.hw.ac.uk/~

lachlan/dbislectures/lectures/expert.ppt. (available on

12.03.09)

Malaivongs K., 2009. Presentation. “CMMI

Implementation”. http://www.drkanchit.com/cmm/

CMMI_ImplementationNew.pdf, (available on

12.03.2009).

Milman S., Joubert J., 2009. “Process and Tools for

Mitigating Risks to Improve Maturity Level of CMM

and Migrate to CMMI”, LUXOFT/IBS. Russia.

www.dtic.mil/ndia/2003CMMI/Milman.ppt (available

on 12.03.09)

Pomykalski J. J., Truszkowski W. F., Brown D. E., 1999.

“Expert Systems”, The Wiley Encyclopedia for

Electrical and Electronics Engineering.

http://agents.gsfc.nasa.gov/papers/pdf/es.pdf,

(available on 12.03.09)

SE-CURE AG, 2009.“CMMI Appraisal Recorder”.

http://www.se-cure.ch/Products.html (available on

25.01.10).

Vector Consulting, 2009. http://www.vector-consulting.

de/vc_compass_userinterface_en,,2607.html (available

on 12.03.09).

Wang M. H., Lee C. S., 2007.”An Intelligent Fuzzy Agent

based on PPQA Ontology for Supporting CMMI

Assessment”. Fuzzy Systems Conference, IEEE.

Yücalar, F., 2006. Master’s Thesis. “Süreç Odakli Kalite

Yönetimi Anlayişina Hakim Yazilim Sektöründeki

Firmalarin CMMI Basamakli Modeli ile

Değerlendirilmesi”, Maltepe University Institute of

Science and Technology. İstanbul, Turkey.

ICSOFT 2010 - 5th International Conference on Software and Data Technologies

430