PILOTING SOFTWARE ENGINEERING INSTITUTE’S
SOFTWARE PROCESS IMPROVEMENT IN INFORMATION
SYSTEMS GROUPS
Donald R. Chand
Bentley College, Waltham, MA 02452, USA
Keywords: Capability Maturity Model (CMM), Software En
gineering Institute (SEI)
Abstract: Although the capability maturity model has become an accep
ted basis for software process improvement in
software organizations, its diffusion in IS organizations continues to be slow. This paper describes the
experience of piloting the Software Engineering Institute (SEI) process improvement with six different
information systems (IS) groups. It brings out the perceptions and reactions of IS developers and the
assessment team regarding the SEI approach. This case study shows that the typical organization structure
of IS organizations seems to impede the successful implementation of CMM-like improvement effort. More
significantly, it appears that the CMM-based production process view of software does not match the
product development view of IS work.
1 INTRODUCTION
Although the motivation for CMM (Capability
Maturity Model) was the Federal Government’s
need for a method of objectively and consistently
assessing the ability of potential Department of
Defence (DOD) contractors to develop software in
accordance with modern software engineering
practices, CMM has evolved as an approach for
improving the software development capability of a
software organization. Today the value of CMM
model is well accepted and established in software
groups and software engineering organizations both
globally and in all sectors of the economy. Results
of case studies reported in the literature shows that
CMM-based software process improvement (SPI)
efforts substantially improve quality, cycle time and
productivity. Yet, CMM improvement efforts have
been slow in diffusing in IS organizations. For
example, there are no successful stories of CMM
implementations in IS organization comparable to
Hugh Aircraft, Raytheon or Schulumberger. Failed
attempts of CMM implementations at leading
organizations like Fidelity and Digital Corporation
are not discussed or known. Even the successful
achievement of CMM level 3 at John Hancock’s IS
groups encountered serious setbacks and difficulties.
Thus there is a need to understand why CMM-based
improvements have failed to diffuse in IS
organizations. Using the insights gained from
piloting the SEI improvement process at the
Information Management and Technology (IM&T)
division of Digital Corporation, this paper
extrapolates the factors that appear to impede the
acceptance and success of the CMM-based SEI
improvement process in IS organizations.
2 PROJECT BACKGROUND
In the nineties, the SEI maturity improvement
process was an integral part of Consulting Services
of Digital Corporation. The aim of this program was
to improve the performance of software consulting
groups to make them competitive in the industry. At
that time, the IM&T division was Digital’s internal
IS organization responsible for serving the
technology and systems needs of Digital workers
worldwide. Since IM&T projects and groups were
similar to the consulting services groups, namely,
they are often small and their projects usually run
less than a year, it was decided to port the
Consulting Services/SEI improvement process to the
development groups in IM&T.
A four-member project team was estab
lished to
369
R. Chand D. (2005).
PILOTING SOFTWARE ENGINEERING INSTITUTE’S SOFTWARE PROCESS IMPROVEMENT IN INFORMATION SYSTEMS GROUPS.
In Proceedings of the Seventh International Conference on Enterprise Information Systems, pages 369-372
DOI: 10.5220/0002507203690372
Copyright
c
 SciTePress

pilot the SEI improvement process in six business
applications development groups within IM&T.
Since the CMM road map is complemented with a
well-defined SEI improvement process, the team
was trained in the SEI assessment process. Because
the management structure of SEI targeted
organizations is different from that of a typical IS
organization, the team devised a simplified
assessment process that followed the SEI
improvement process very closely, differing only
where an advantage was obvious.
3 PILOT EXECUTION AND
RESULTS
3.1 Overview phase
We followed the SEI process for the overview phase
by making a one-hour presentation to each
management team of the six IS groups selected for
piloting the SEI improvement process. During our
presentation, we conscientiously solicited the
comments and participation of the management
teams of the six groups. The six groups were in three
different functional areas of IM&T. The four
managers in the first functional area were generally
receptive to the idea of improving the software
development process, and because we sought
feedback they pointed out that the SEI process is
missing areas important to IS groups such as
corporate data management and requirements
gathering. Despite this, they were willing, in
varying degrees, to participate in the pilot. Two of
the managers in this functional area wanted us to
lead the assessment and identification of
opportunities. The other two committed their groups
to doing the entire assessment process at least
through creating an action plan.
Within the second functional area they were
already using a process coaching team approach
with a software team working on a critical
application. This project ran parallel to the process-
coaching project. Two members of our SEI process
improvement team were consultants on the process-
coaching project.
In the third functional area, we made the initial
presentation to a technology group that wanted to be
the change agents within the larger organization.
This seemed like an interesting way of
accomplishing change, but other priorities
intervened for that group and this project did not
materialize.
3.2 Team assignment phase
This phase begins with a kick-off meeting, where the
entire group is introduced to the SEI’s improvement
program. After the kick-off meeting, the process
team is selected. It was difficult for some managers
to assemble their whole group within the time frame
of the study because many staff members were
geographically distributed in different States. For
two teams we had to begin the process team
formation step before holding the kick-off meeting.
It took a significantly long time to appoint some
process teams. With two groups, there was also a
significant wait for the team members’ time to free
up so they could begin the work. There were other
delays during the work. These were due to business
pressures, and, in one case, due to illness.
The target groups averaged about 10 members,
and we worked with two team members from each
group. Half the participants were not senior and/or
experienced software developers, although they
were always people who had expressed interest.
Most were knowledgeable about the process, and
one was a group manager.
3.3 Assessment phase
The procedure we used involved distributing to each
process team member a copy of the Assessment
Questionnaire and the score sheet, a copy of the
accompanying glossary of terms, and a copy of the
follow-up questions to be used during the validation
phase. We read and explained each assessment
question to the process team so that there is one
interpretation of a given question. All groups rated
themselves in the first level, below the second
quartile. In the SEI scoring process, third-level and
above software engineering practices do not count
unless the group has already achieved the second
level. We did find that every group had one or two
level 3 or level 4 practices in place.
We observed that there was widespread concern
that IM&T or their own managers, would try to use
this as an evaluation tool.
Several team members provided thoughtful
comments on the contents of the Assessment
Questionnaire, as had their managers during the
overview phase. Most groups observed that the SEI
assessment instrument does not cover the full range
of IM&T activities. They identified the missing
areas in the SEI questionnaire as interaction with
business, organization considerations,
implementation management, personnel issues, and
audit function.
ICEIS 2005 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION
370

3.4 Opportunity selection phase
The purpose of the opportunity selection phase is to
identify and select which software engineering
practices the software group will improve. In the
SEI process, only those questions that would
advance the organizations to the next higher level
are considered as opportunities. In our pilot study,
since all six groups were at level 1, copies of the
level 1 questions with “no” answers were sent to
each software developer in the group. In preparation
for the next meeting they were asked to rank each
question from 1-5 in both cost and benefit to the
group, and also to give a brief rationale for their
cost/benefit ranking.
The results of this ranking were graphed in a
scatter diagram with benefits along the x-axis and
cost along the y-axis. The primary opportunities to
focus were in the lower right quadrant, indicating the
highest benefit and lowest cost. Opportunities
outside but near this quadrant were marked as
secondary opportunities and they were discussed at
the meeting. Since all groups ranked themselves at
level 1, only 33 level 2 questions that received “no”
answers questions were considered as opportunities.
Four groups reached the stage of selecting
opportunities for their groups to undertake. One
group had elected not to go further with an action
plan, but they presented the results to a staff
meeting. One group selected opportunities and
when the project closed, that group was working on
implementing improvement plans. Another group
was still working on selecting opportunities when
the project closed.
Opportunities were selected according to the
group’s perception of what was most lacking in their
development process. The choices varied widely,
from standard cost estimates, project management to
regression testing practices.
The fourth group did not select any
opportunities, rejecting all the questions as being not
relevant to the way their organization currently has
to do business. Their work is mostly maintenance,
enhancements and changes to a legacy system
written in MUMPS. Their process is informal,
based on arbitrary deadlines imposed by their
customer community.
3.5 Validation phase
We asked to see backup material and sought
evidence in specific current projects. In addition, we
interviewed the developers in the group. The
purpose was to confirm the practices that received a
“yes” are in actual use. One group completed the
validation phase.
3.6 Action plan phase
The purpose of the action plan phase is for the
process team to prioritize the opportunities and
create an action plan. In the pilot study, we
encouraged the process groups to incorporate as
many changes as would be practical to undertake.
One group in this study reached this stage.
4 IS PERCEPTIONS OF CMM
The IM&T managers and developers felt that
• SEI assessment process is valuable
• CMM does not cover key IS development
activities
• CMM is too bureaucratic and rigid
• CMM does not address the true needs of IS
development
SEI Assessment Process is Valuable
The SEI process was well received by the IS
managers. Most recognized the need to change and
felt that it would help improve the way they did their
job. The groups saw benefits from exposure to SEI
approach. They felt that just going through the
Assessment Questionnaire process generated
awareness, ideas and discussion that was very
valuable.
CMM is Not the Whole Solution
Nearly all groups told us that CMM does not address
the full spectrum of IM&T activities. For example,
preparing the business partner for work changes,
training the user community, discussing and
implementing changes in the user process, and
analyzing the project after conclusion are considered
part of the software development process.
Furthermore, in IM&T, audit of the tools, methods
and work activity is common. There is no reference
to a DP auditing group outside the development
group in the SEI process.
Lastly, they felt that most of their work involved
constant interaction with the customer maintaining
and enhancing existing systems and that activity is
not covered in the SEI approach. Specifically, in the
SEI questions the customer is mostly missing.
IM&T customers are responsible for sponsoring the
product, negotiating the requirements, setting the
project scope, and specifying the interfaces.
CMM is bureaucratic and rigid
A few of the key process areas of level 2 do not
apply to IS work such as managing subcontractors,
and requiring organizations to achieve level 2 KPA
before working on level 3 KPA is too rigid. Also,
PILOTING SOFTWARE ENGINEERING INSTITUTE’S SOFTWARE PROCESS IMPROVEMENT IN
INFORMATION SYSTEMS GROUPS
371

the idea of separate SEPG and SQA function was
viewed to be bureaucratic.
CMM does not address the true needs to IS work
Most groups felt that in SEI work it is not clear
whether small enhancement projects and
maintenance work is included in the realm of
software development. In IM&T activities, one
group estimated that 20% of the work is new
development and 80% is maintenance effort.
It was pointed out that personnel turn over is
common. Most groups are not up to full headcount.
When a person leaves, work is re-ordered but
invariably some work just is not done.
5 OUR LEARNING
As members of the process assessment team we
learned that
• CMM is flexible
• Management commitment is more than
management agreement
• Matrix reporting is a barrier to improvement
initiatives
• Organization stability is necessary
• IS development is different from software
development
CMM is flexible
Although all the groups were at level 1, their most
pressing concerns were different. We found that the
SEI model is flexible in allowing each organization
to choose what is most important to them.
Management commitment is more than
management agreement
SEI process emphasizes that any improvement
project must have top-management support and
backing. Apparently, we did not build the full
management support for the pilot effort. As a result,
we did not feel that there was sufficient support from
the IS managers, beyond an initial pep talk an
occasional remote prodding. Part of the problem
was that we did not communicate what was expected
from them. We know now that it is important to
explain to the top management what commitment
and readiness really means.
Matrix reporting is a barrier to improvement
Most groups in IM&T report to two or more
organizations. One is the IM&T headquarters and
the other is at least one business unit. They take
directions from each and attempt to satisfy some
median, usually resulting in mediocre performance.
The priorities of the business partner and IM&T are
often not aligned. For example, IM&T may
mandate to downsize at the same time when the
business partner needs a critical information system.
Improvement requires consistent directions, and
conflicting directions makes improvement extremely
unlikely.
Organizational stability is necessary
When this pilot project was executed, the IM&T
organizations were not stable. Every organization
that we came into contact during this period was in
the process of re-organizing. They had new
managers, new colleagues, new structures, new
work, and new locations. In many cases there had
been more than one re-organization within a year.
One organization reorganized three times in the six
months between our first overview presentation and
the closing of the project.
Resources were reduced often, leaving fewer
resources to do the existing work, and often none to
implement the improvement effort. As a
consequence, process improvement became low
priority, placed on the back burner, and the
improvement projects were not completed.
IS development is different from software
development
The CMM-based SEI approach views software
development as a task-driven, structured effort
driven by known and pre-specified ordering of the
requisite tasks. Therefore, people roles are task-
specific, discrete, specialized and identifiable. In IS
work, the production processes are secondary to
product and the development effort takes place
through the network ties developed by the
developers, users, vendors and customer. In the SEI
paradigm, good process produces good product, in
the IS paradigm good product comes from having
good people. Because the SEI process has a
production process view of software, it does not
match the product development view of IS.
REFERENCES
Paulk, M.C., Bill Curtis, M. B. Chrissis and C. V. Weber,
“Capability Maturity Model, Version 1.1, IEEE
Software, July 1993, pp. 18-27.
ICEIS 2005 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION
372