MEASURING CRITICAL SUCCESS FACTORS

IN ERP PROJECTS

Results from a Case Study in a SME

Jos J.M. Trienekens

Eindhoven University of Technology, Den Dolech 2, Eindhoven, The Netherlands

Pedro van Grinsven

RIS Rotterdam, Brugge 24, 2993 LB Barendrecht, The Netherlands

Keywords: ERP system implementation, success factors, measurement.

Abstract: Over the past decade many organizations are increasingly concerned with the implementation of Enterprise

Resource Planning (ERP) systems. This counts for both large and small and medium sized companies.

Implementation can be considered to be a process of change influenced by different so-called critical

success factors (CSF) of type organizational, technological and human. This paper reports on the

development of a measurement approach for managing CSF in an ERP implementation project in a small

and medium sized company (SME). Critical success factors are being derived from project goals and

subsequently measured in this project to monitor and control the implementation project.

1 INTRODUCTION

Over the past decades a new class of software

applications has emerged: Enterprise Resource

Planning (ERP) systems. These software packages

seek to integrate the complete range of a business’s

processes and data communication patterns on the

basis of one single information and IT architecture

(Klaus et al, 2000). Quality control and assurance

during the implementation of ERP packaged

software has been under-researched, in particular

regarding the identification, definition and validation

of critical success factors (CSF) (Krumbholz and

Maiden, 2001), (Marble, 2003). These factors can be

of different types, such as organizational (e.g. top

management support), human (e.g. communication

attitude, user resistance), technical (e.g. business

process modeling methods and tools), (Stelzer and

Mellis, 1998), (Trienekens et al, 2001). Although

some articles recognize factors that drive success in

ERP implementation, they look at them from

different perspectives and also with different

definitions of “success factors” in mind (Aladwani,

2001), (Amoako-Gyampa and Salam, 2003). Over

the past years several research papers have emerged

that strive at the identification and classification of

CSF (Hoon Nah and Shang Lau, 2001). More

recently research papers have emerged that focus on

the evaluation and validation of CFS relevance in

practice, e.g. the CFS relevance along the different

ERP implementation project phases (Esteves and

Pastor, 2004). This paper strives at taking one step

further: the development of an approach towards

measuring CSF during an ERP implementation

project.

The ERP implementation project that has been

selected is a current project at a small and medium

sized (SME) enterprise, called RIS BV in The

Netherlands. RIS BV is primarily active in the

domain of innovative Traffic Sign Systems. Both

standard products and tailormade systems, i.e.

special projects, are being developed. One of their

most important customers is the Dutch ANWB

(national traffic organisation) for that they are the

first supplier of route and road signs systems along

the Dutch highways. The development process for

these complex projects consists of analysis and

design, manufacturing and implementation. These

processes ask for integration of business data from

different perspectives, e.g. project, financial, supply

etc. An ERP system should support the flow of data

throughout the company and the usage of a central

203

J. M. Trienekens J. and van Grinsven P. (2008).

MEASURING CRITICAL SUCCESS FACTORS IN ERP PROJECTS - Results from a Case Study in a SME.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - DISI, pages 203-209

DOI: 10.5220/0001693302030209

 SciTePress

database concept, so that all involved departments

can exchange information and communicate in an

efficient and effective way.

RIS BV implements the ERP system Navision in

its business processes. This is carried out in a

modular way. The module that is being implemented

currently is the Projects module. The high priority

for implementing this module is a consequence of

the current project planning problems at RIS. The

traditional way of planning by using manually

controlled Gantt charts (e.g. in Excel) is not

effective anymore due to, among others, the high

number of change requests in the complex projects.

On the short term RIS BVwants to improve their

reliability and customer satisfaction regarding the

finalisation of the complex projects in due time. The

Navision Projects module should offer the

opportunity to analyse and control the change

requests in the planning process. On that basis it

should become possible, in a flexible way, to derive

management reports on the financial consequences

of changes in the project.

In section 2 the research framework and

approach is presented that is applied in the case

study. Section 3 reports on the results of the case

study at RIS BV. In section 4 conclusions and

recommendations for further research are given.

2 RESEARCH FRAMEWORK

AND APPROACH

Critical success factors (CSF) have become a

management instrument in the broad area of

software engineering, both in development and

implementation. However, the applications that are

presented in literature are still of a qualitative nature

and do not support implementation managers in

practice with operational and quantitative

instruments. As a consequence most implementation

projects only get little support from these studies.

Therefore the main questions still are:

1. How to control ERP implementation projects in

SMEs on the basis of CSF?

2. Can CSF be measured during implementation

projects in SMEs?

Regarding the management and control of CSF

several attempts have been made to make CSF

operational, e.g. by elaborating the definitions and

by investigating the usefulness of these definitions

for practitioners. Although interesting results have

been gained, previous research focused in particular

on the possibility of applying CSF during ERP

implementation. E.g. in (Esteves et al, 2003) in

particular two CSF are being investigated and only

some attempts have been made to make these CSF

operational. However experiences with measurement

of CSF in real-life ERP implementation projects in

SME is not yet reported.

Based on previous research findings on ERP

implementation it was decided to make use of the set

of CSF definitions of (Esteves and Pastor, 2000) and

the GQM method (van Solingen and Berghout,

1999). Applying GQM in order to determine metrics

has also been studied by (Esteves and Pastor, 2003).

However the experimentation with GQM to monitor

and control CSF in a real-life ERP implementation

project has not yet been reported. Regarding the

usage of CSF it was decided to select already early

in the implementation project a limited set of CSF.

This is in conformance with guidelines for the usage

of metrics in real-life projects, (Kitchenham, 1996).

The ERP implementation project is relatively

small, i.e. restricted to only one ERP module, a time

interval of only three months, and to only 10

participating people. The approach that has been

followed in our research consists of the following

steps:

1. Development of the research framework: CSF

and GQM

2. Applying the GQM approach in a real-life SME

project:

a. Identification of the goals of the

project

b. Selection of a limited number of CSF

c. Development of the measurement

instrument (i.e. questionnaires)

d. Collection of the data

e. Analysis and evaluation of the data

3. Evaluation of the research framework and

approach

2.1 Critical Success Factors and Goal

Question Metric Method

CSF have already been investigated by a number of

researchers, see the introduction of this paper. In this

paper the relatively recent and extensively

researched list of (Esteves and Pastor, 2000) has

been used, see table 1.

ICEIS 2008 - International Conference on Enterprise Information Systems

204

Table 1: CSF in accordance with (Esteves and Pastor,

2000).

CSF

Sustained management

support

Project champion role

Effective change

management

Tactical adequate training

program

Project scope management Appropriate usage of

consultants

Dedicated staff and

consultants

Empowered decision makers

Communication Organizational trust between

partners

Formalised project plan Strategic avoid customization

Project team composition Technological software

configuration

Comprehensive process

reengineering

Legacy systems knowledge

Preventive trouble shooting Adequate ERP version

ERP implementation

strategy

User involvement and

participation

The Goals Question Metrics (GQM) method is

an operational instrument for the development of a

metrics program (Solingen and Berghout, 1999). In

accordance with GQM the goals, e.g. project goals,

have to be stated first in a formalised way.

Subsequently these goals have to be refined by

questions about how these goals should be reached.

This refinement then leads to the identification of

data that are needed. For the collection of these data

operational metrics have to be defined. The GQM

method as applied in our case study consists of four

phases, respectively:

1: Planning the measurement – identifying

project goals and allocating resources

In this phase the project goals are being defined in

close collaboration with the project management and

the overall management of RIS BV. Managers,

project members, quality assurance employees

involved in the implementation project have been

determined and have been allocated to the

measurement approach, e.g. regarding data

collection, analysis and evaluation, and the

development of improvement actions during the

ERP implementation project.

2: Development of the measurement instrument

On the basis of a refinement of the project goals the

relevant CSF that should be monitored and

controlled have to be identified. Subsequently the

data to be collected and analysed, in order to be able

to evaluate the control on the project, have to be

determined. In fact a measurement instrument for

the collection of these data has to be developed.

3: Collecting the data – applyting the

questionnaires

In these phase the data have to be collected.

Questionnaires have to be filled in by the

stakeholders involved in the implementation project.

Important is the distribution of the questionnaires,

the organisation of information sessions to clarify

the goals, and the analysis and feed-back of the

measurement activities.

4: Evaluation of the data

The data has to be analysed in a predefined way, e.g.

by using acceptation levels for the final scores.

Acceptable measurement scales should be defined in

close cooperation with representative stakeholders in

the implementation project.

The combination of CSF with the GQM method

has resulted in a measurement approach that has

been applied in the ERP implementation project.

3 APPLYING THE

MEASUREMENT APPROACH:

CASE STUDY RESULTS

In this section the results of the case study on

measuring CSF during ERP system implementation

are presented. In the following we will first

introduce in section 3.1 briefly the case study

environment. Subsequently we will present the

results of the application of the measurement

approach.

3.1 Case Study at RIS BV

The case study has been carried out during the first

three months of the implementation project in 2007.

The objective was to experiment with questionnaire-

based metrics in a real-life ERP implementation

project and to make a first step towards a well-

monitored and controlled project.

3.1.1 Identification of Project Goals

The goals of the ERP implementation project have

been investigated and discussed with representatives

from the various involved management levels and

potential key users of the Navision application. Both

the strategic goals of the organisation and the local

objectives of the different involved departments

have been taken into account. As main result the

following project goals for the ERP implementation

project have been defined:

- timeliness of finishing the implementation

MEASURING CRITICAL SUCCESS FACTORS IN ERP PROJECTS - Results from a Case Study in a SME

205

- ability of key-users to make use of the ERP

application immediately after implementation

3.1.2 Determination of CSF

Based on the two project goals the list of CSF has

been discussed with the involved stake-holders. In

particular a people dimension and a project

dimension have been identified. This is in

conformance with findings of (Estevez and Pastor,

2000) who identified also the importance of

monitoring and controling in particular human and

project aspects during an ERP implementation

project. From each of the two dimensions several

concerns of the involved stakeholders have been

identified. Regarding the people dimension these

concerns are respectively the CSF 'User-involvement

and participation', and 'Tactical adequate training

program'. Regarding the project dimension in

particular the CSF 'Project champion role' and

'Dedicated staff and consultants' were identified.

3.1.3 Measuring CSF:The Questionnaires

For each of the determined CSF a small number of

questions have been formulated with the involved

stake-holders in order to be able to determine

whether a CSF is under control or not. Regarding

these questions structured answers, i.e. measurement

scales, have been defined. The set of CSF with the

questions and the measurement scales are forming

together the measurement instrument. Regarding the

formulation of questions we explicitly adopted

similar questions as presented in the research of

(Mendoza et al, 2006) In that way we were able to

make a comparison between the results of our case

study and the results of case studies of the

previously mentioned authors. In the following the

four questionnaires for the distinct CSF are

presented.

Table 2: Questionnaire regarding the CSF Project

champion role.

CSF Relevance and motivation

Project

champion

role

The project champion should have the

capabilities to monitor and control the

progress of the project and to take and

communicate adequate decisions.

Questions Scale

Is the progress of the

project under

control?

5 = Always

4 = Almost always

3 = Sometimes

2 = Almost never

1 = Never

Table 2: Questionnaire regarding the CSF Project

champion role. (cont.)

Questions Scale

Are the decisions

being

communicated

during project

meetings?

5 = Very fast

4 = Fast

3 = Average

2 = Slow

1 = Very slow

Table 3: Questionnaire regarding the CSF Tactical

adequate training program.

CSF Relevance and motivation of the CSF

Tactical

adequate

training

program

A training and educational plan is needed to

prepare the management and other

stakeholders in the usage of the ERP

application.

Questions Scale

Is a training and

educational

program available?

5=yes

1=no

Is there sufficient

time available for

the training

program?

5 = Plenty

4 = Sufficient

3 = Average

2 = Little

1 = Very little

Table 4: Questionnaire regarding the CSF User

involvement and participation.

CSF Relevance and motivation of the CSF

User

involvement

and

participation

To ensure the integration of the ERP

application in the business system, both on

the operational process level and on the

management level, representatives from

these levels should be involved in the

implementation project.

Questions Scale

Is there a structured

plan for the meetings

and information

sessions?

5 = Yes

1 = No

Are the invited

representatives

present at the

meetings and

interview sessions?

5 = Always

present (80–

100%)

4 = Almost

always present

(60–80%)

3 = Average

present (40–60%)

2 = Almost never

present (20–40%)

1 = Never (0-

20%)

Are actions

regarding 'not

showing up' at

meetings and

interviews being

executed in-time by

the responsible

persons?

5 = Always

4 = Almost

always

3 = Sometimes

2 = Almost never

1 = Never

ICEIS 2008 - International Conference on Enterprise Information Systems

206

Table 5: Questionnaire regarding the CSF Dedicated staff

and consultants.

CSF Relevance and motivation of the CSF

Dedicated staff

and

consultants

Technical support from the ERP provider is

needed during the whole implementation

project.

Questions Scale

How dependend is the

implementation

project from technical

support?

5 = Very low

4 = Low

3 = Average

2 = High

1 = Very high

Is technical support

specified in the service

level agreement in

sufficient detail?

5 = Yes

1 = No

3.1.4 Data Collection and Analysis

In the first three months of the ERP implementation

project the questionnaires have been filled in by 10

stakeholders from different organizational levels,

respectively end-users, and managers on the tactical

and the operational level. The results are shown in

Table 6.

Table 6: Measurement results CSF.

CSF's, questions and

total score per CSF

Accep-

tation

level

Avera

minus

Score

in % of

max.

Project champion role

3 3,1 0,1

3 3,4 0,4

Total score

Tactical adequate

training program

5 1,0 -4

3 1,4 -1,6

Total score

User involvement and

participation

5 1,0 -4

3 4,5 1,5

Total score

Dedicated staff and

consultants

3 1,6 -1,4

5 5,0 0

100

Total score

For each of the CSF a total score has been

calculated in a similar way as has been done in

(Mendoza et al, 2006). The score of each question is

presented as a percentage of the maximum that

could be reached on the measurement scale. In

Figure 1 the measurement results are presented and

are compared with the measurement results of two

previous ERP implementation project in

organizations A en B (Mendoza et al, 2006

The 'Project champion role' is carried out

satisfactory. However, 'Tactical adequate trainign

program' scored significantly lower in the RIS

implementation project than the reference

organizations A and B. It appeared that 'Tactical

adequate training program' is not well

communicated throughout the organisation and that

the training plan is not yet fully operational. As

correcting action the development of a

communication plan has been defined.

Regarding 'User Involvement and Participation' and

'Dedicated staff and consultants' several weaknesses

have been identified, however the overall scores for

these CSF are sufficient.

4 CONCLUSIONS

This paper shows that literature on CSF can act as a

reference basis for the development of a

measurement approach to monitor CSF in ERP

implementation projects. The case study shows that

a well-defined selection of CSF, derived from

project goals, can be taken as a starting point for the

development of a questionnaire-based measurement

instrument. The relatively small questionnaire could

be applied succesfully in a SME in practice during

an ERP implementation project.

The (intermediate) results of the measurement

are being used by the project management to

monitor the implementation project and to take

adequate actions. Further the (intermediate) results

could be compared with results from other

implementation projects that have been described in

literature. As such a first step has been made

towards benchmarking CSF measurement in ERP-

implementation projects.

Further research is needed regarding the

interrelationships between the explicitly defined

project goals, the selected CSF and the questions

(and measurement scales) of the measurement

approach. Currently the measurement approach is

applied in subsequent phases of the mentioned ERP

implementation project. The results will be used to

further improve the monitoring of the

implementation project as well as the validation and

further improvement of the measurement approach.

MEASURING CRITICAL SUCCESS FACTORS IN ERP PROJECTS - Results from a Case Study in a SME

207

Figure 1: CSF scores in RIS BVand the reference organisations A and B.

REFERENCES

Aladwani A.M., 2001, Change management strategies for

successful ERP implementation, Business Process

Management Journal, Vol. 7 No. 3, pp. 266-275, MCB

University Press, 1463-7154.

Amoako-Gyampah K., A.F. Salam, 2003, An extension of

the technology acceptance model in an ERP

implementation environment, Information &

Management.

Esteves J., J. A. Pastor, 2004, Organizational and

Technological Critical Success Factors Behavior

along the ERP Implementation Phases, in:

Proceedings of the Sixth International Conference on

Enterprise Information Systems, Porto, Portugal, April

14-17.

Esteves J., J. Pastor, J. Casanovas, 2003, A

Goal/Question/Metric research proposal to monitor

user involvement and participantion in ERP

implementation Projects, Information Resources

Management Association Conference (IRMA),

Philadelphia (USA).

Esteves J., Pastor J., 2000, Towards the Unification of

Critical Success Factors for ERP Implementation, 10

Annual BIT conference.

Hoon Nah F.F.,J.L. Shang Lau, 2001, Critical factors for

successful implementation of enterprise systems,

Business Process Management Journal, Vol. 7 No. 3,

pp. 285-296. MCB University Press, 1463-7154.

Kitchenham B..A., 1996, Software metrics: measurement

for software process improvement, NCC Blackwell

Publishers

Klaus H., M. Rosemann, G.G. Gable, 2000, What is ERP?

Information Systems Frontiers, 2:2, 141-162.

Krumbholz M., N. Maiden, 2001, The implementation of

enterprise resource planning packages in different

organisational and national cultures, Information

Systems, 26 , 185-204.

Marble R.P., 2003, A system implementation study

management commitment to project management,

Information & Management 41, 111–123.

Markus M.L., C. Tanis, 2000, The enterprise system

experience – from adoption to success”, in Zmud,

R.W. (Ed.), Framing the Domains of IT Management:

Projecting the Future Through the Past, Pinnaflex

Educational Resources, Inc., Cincinatti, OH, pp. 173-

207.

Mendoza, Pérez, Grimán, 2006, Critical Succes Factors

for Managing Systems Integration, Information

Systems Management, Volume 23, Issue 2 pages

0,00%

20,00%

40,00%

60,00%

80,00%

100,00%

Project

champion

role

Tactical

adequate

training prog.

User

involvement an

Participation

Dedicated staff

and consultants

Ris

ICEIS 2008 - International Conference on Enterprise Information Systems

208

56 - 75

Perez M., Griman A., Mendoza L., Rojas T., 2004, A

Systemic Methodological Framework for IS

Research. Proceedings of the Tenth Americas

Conference on Information Systems, New York.

Solingen R., E. Berghout, 1999, The

Goal/Question/Metric Method, A practical guide for

quality improvement of software development,

McGraw-Hil.

Stelzer D., W. Mellis, 1998, Success Factors of

Organizational Change in Software Process

Improvement, Software Process: Improvement and

Practice 4, 227-250.

Trienekens J.J.M., R.J. Kusters, R. van Solingen, 2001,

Product Focused Software Process Improvement:

Concepts and Experiences from Industry, Software

Quality Journal 9, 269-281.

Wohlin C., A. Amschler Andrews, 2001, Assessing Project

Success Using Subjective Evaluation factors, Software

Quality Journal, 9, 43-70.

MEASURING CRITICAL SUCCESS FACTORS IN ERP PROJECTS - Results from a Case Study in a SME

209