IMPROVING USER SATISFACTION IN THE

POST-IMPLEMENTATION PHASE OF A LARGE-SCALE

INFORMATION SYSTEM

Colman Gantley

Athlone, Co. Westmeath, Ireland

Keywords: Usability, Post-Implementation, Information Systems, Satisfaction.

Abstract: This paper presents a framework for identifying and improving user satisfaction for implemented

information systems. Frequently, the post-implementation phase of a systems lifecycle is ignored and

problems go undetected, which leads to user dissatisfaction. A poorly designed system can become a barrier

for users and often they become reluctant to tolerate and use the system. If users resist working with the

technology, the potential for the system to generate significant organisational performance gains can be lost,

rendering the introduced system a costly mistake. This paper attempts to refine usability beyond just human-

computer interaction (HCI). It introduces the Post-Implementation, Usability Synergy (PIUS) framework,

which focuses on six different elements, including: training, functionality, reliability, working environment

and interfacing, all centred on the actual users of the system, with the aim to enhance user satisfaction

towards the new system.

1 INTRODUCTION

A large-scale information system can enable an

organisation to dramatically improve their business

model, change internal structures and over time

increase profit margins. Due to this, many

organisations around the world have invested a

considerable amount of money in setting up these

systems (Lucas, 2005; Hawking & Stein, 2002).

Although information systems (IS) can bring a

competitive advantage to organisations, the high

failure rate in deploying such systems is a major

worry. Many organisations have suffered from

ineffective IS, for instance: Whirlpool, Irish Health

Authority, Allied Waste Industry, Hershey Foods,

Boeing, Mobile Europe, Applied Materials, Waste

Management Inc, Kellogg’s, Irish Prison Service,

Irish League of Credit Unions and Nestlé (Weston,

2001; Yu, 2005). Past research on advanced IT

deployment identifies the post-implementation phase

as the critical period during which the new system

becomes embedded in the host organisation.

Conducting a post-implementation review (PIR) on

the system during this critical period can highlight

usability flaws and system weaknesses. Repairing

these flaws can ultimately limit the chances of a

large-scale system failure (Halpin, 2003; Kueng,

2002; Nielsen et al., 2001). Organisations have a

tendency to push people and process related issues to

the bottom of the list due to time and cost overruns,

many expecting users to find the new system easy to

learn. Unfortunately, this is usually an over

optimistic hypothesis. The problems causing large-

scale IS failure after a system has gone ‘live’ fall into

multiple categories, including inadequate user

training, poor system reliability, inaccurate business

functionality and shoddily designed user interfaces

(Carroll & Carrithers, 1984; Wixon et al., 1990). A

number of prominently publicised system failures

have highlighted the difficulties involved particularly

in the post-implementation phase of an IS. For

example, in 1997 the Irish Health Authority (IHA)

invested in the Personnel, Payroll and Related

Systems (PPARS) project to manage the

development and implementation of a fully

integrated Human Resource Management system.

Originally, the PPARS project was due to cost €8.8

million. However in 2007, the PPARS project has

yet to be fully integrated, largely because the system

was complicated to use, lacked general training and

failed to do what it was intended. It is estimated it

will cost over €231 million, 25 times the original

estimate to solve these problems (Hunter, 2005;

McGee, 2005; Kennedy, 2005, PPARS, 2005;

388

Gantley C. (2008).

IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE INFORMATION SYSTEM.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - HCI, pages 388-392

DOI: 10.5220/0001713703880392

 SciTePress

INBITE, 2005; Irish Health, 2007). Key reasons for

such failures is information systems development

(ISD) methodologies concern themselves primarily

with pre-implementation activities, rather than the

extended post-implementation phase and often

exclude any usability evaluations or users

involvement after the system goes ‘live’.

There is a growing need for research, which

endeavours to develop a framework that captures and

aggregates usability issues in the post-

implementation phase of large-scale IS. This paper

attempts to bridge a significant gap in ISD research

and endeavours to synthesise a framework that will

both examine and indicate specific usability issues

that may arise within the post-implementation phase

of an IS.

2 POST-IMPLEMENTATION

The post-implementation phase (PIP) occurs after the

system has been developed and has gone live. By

conducting an evaluation after the system during this

stage IT payoffs can be realised against their original

objectives and corrective action can be performed

where necessary. As a consequence of this

intervention, the PIP can serve extremely useful. It

can allow the IS managers gain an understanding of

the new system and can reduce the chances of

expensive failures. A well planned and executed PIR

can assist organisations to address needed changes in

the systems architecture and correct any errors in the

system that went unnoticed in the development stage,

such as an inadequately designed interface

(Nicolaou, 2003; Woodings & Everett, 1999).

Including the users in the evaluation of the system

throughout the PIP can determine whether the system

is easy to use, easy to manage and easy to learn.

Despite the huge importance of this stage, ISD

methodologies largely ignore this critical phase of

the development life-cycle. Past research proves that

very few organisations review their system after it

goes live often expecting users to find the new

system easy to learn. Regrettably, this is usually an

over optimistic assumption (Wixon et al., 1990;

Devaraj & Babu, 2004; Palvia et al., 2001; Nicolaou,

2004; Yu, 2005; Nicolaou, 2003; Woodings &

Everett, 1999; Tallon et al., 2001).

3 USABILITY

The International Standardization Organization

(ISO) established a usability standard that can be

defined as the extent to which a product can be used

by specified users to achieve specified goals with

effectiveness, efficiency and satisfaction (ISO-9241,

1998). Usability is multidimensional encompassing

user attitudes, learn-ability, effectiveness and

flexibility towards the system or product with the

aim to make users quality of life better (John, 1996;

Shneiderman, 1998). Systems with high usability

have ‘natural’ interfaces, are easy to learn, easy to

use and are associated with many positive outcomes,

such as a reduction in errors, a positive user attitude

towards the system and increased system use,

therefore gaining greater productivity for the

company (Fruhling et al., 2005; Shang & Seddon,

2000). While organisations have spent millions

implementing large-scale IS, research indicates that

potential users often unable to use the new system,

and often revert back to previous methods or use the

system incorrectly. Users of IS that have low or poor

usability often get to know and use 20–30% of the

available features, which leads to high productivity

slumps, huge loss of profits for the organisation, low

user morale and often an expensive, unused IS

(Nielsen & Coyle, 2001). Yet, organisations still

overlook the importance of usability particularly in

the PIP of the system development life cycle (Calisir

& Calisir, 2003).

4 CONCEPTUAL FRAMEWORK

A comprehensive conceptual framework is needed to

address a wide range of usability issues in the

extended PIP of a large-scale IS. Figure 1 is a

conceptual post-implementation, usability

framework developed by the author. This framework

is called The Post-Implementation Usability Synergy

(PIUS) framework. It is a working theoretical

framework that is to be informed by a future pilot

study. The key practical implication of this

framework is to enhance the user experience by

identifying any usability issues with the IS. The

framework contains six different parts but the

combination of all these elements will improve user

satisfaction with the new system. The PIUS

framework acts as a synergy. It contains six different

parts but operates as one. Each element has an

importance, but combining all the elements together,

similar to a gestalt, will be greater than the sum of

their individual effects. The PIUS framework was

designed to identify areas of user satisfaction with a

new large-scale IS, from the users point of view. The

framework comprises of the six components, these

include: user interface, business functionality, system

IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE

INFORMATION SYSTEM

389

reliability work practices, and support and training.

All these components are built around the users of

the system. By overlooking one of these elements, it

would have a knock-on affect and reduce the user

satisfaction toward the system. If users were satisfied

with all of these elements, the usability synergy was

complete and user satisfaction would be created.

Fig. 1: The PIUS Framework.

End-Users. The purpose of the PIUS framework is to

help identify areas of user satisfaction or

dissatisfaction users are having with an IS. The best

way to achieve these areas is to know exactly what

the user wants and needs, this can be accomplished

by working closely with the user and establishing

which area(s) of the system s/he is having most

trouble with. The user is the person for who will be

working first-hand with the system. If the user is

having difficulties with a certain part of the system,

satisfaction with the system will inevitably be low. It

is essential that the system is reviewed in the post-

implementation phase from the user’s point of view,

to uncover any issues with the system. By not

considering the thoughts and opinions of the user in

the PIP it is impossible to establish whether the

system has an adequate level of reliability,

interfacing, functionality, or training supplied to

ensure the end- user can work effectively and

efficiently with the new system. For an organisation

trying to successfully implement an IS, the user

(employee) is the most important piece. If they are

having difficulties with any aspect of the system user

satisfaction will be low and the PIUS framework will

not able to piece together properly. The synergy will

be incomplete until all issues have been resolved.

Work Practices/Functionality. The business

functionality of an IS indicates the features and

capabilities the system comprises of, and can be

defined as ‘the requirements necessary to perform

the specific tasks which the users require it to do in

the operational situation’. The functionality of a

large-scale IS can be categorised into two different

components, these include:

• Meeting business requirements

• Meeting user’s requirements

Organisations need to have clear and unambiguous

understanding of the business objectives before the

system is designed, to ensure that the correct

functionality is obtained. Failing to meet the

requirements of the business can cause a great deal

frustration for the people trying to work with the

system. A system that is designed with limited

functionality or high inconsistency can cause

problems for the user and frequently is the main

cause for users rejecting to use the system entirely.

The IS should provide the right information to the

right person, at an accurate time. If the functionality

is inadequate, it does not matter how well the

interface is designed, ultimately the system will

create low user morale and a decrease in

productivity. If the functionality is inadequate,

regardless on how well the interface is designed or

how reliable the software is, eventually the system

will create low user confidence. If this occurs, the

PIUS framework will be incomplete and the

elements will not piece together. User satisfaction

will remain low until the functionality is fixed.

System Reliability.

The reliability of a product is

defined as ‘the measure of its ability to perform its

function when required, for a specified time, in a

particular environment’. A reliable IS must support

the needs of the user in a simple, fast and consistent

way. A reliable system must be able to protect

against unauthorized access to its physical and

logical components. The reliability of an IS can be

categorised into three different areas, these include:

• The systems software

• The systems hardware

• The systems communications (Networks)

A reliable system is one that is capable of operating

without material error, fault, or failure during a

specified period in a specified environment. If a user

cannot access the system or cannot display data s/he

requires, it does not matter how well developed the

systems is users will reject using it. For user

satisfaction to be achieved IS reliability must be

developed effectively. If the system does not perform

as expected, it will cause problems for the person

who is trying to carry out their work. Users get very

frustrated when the system fails and they have to

redo any work they have already completed. A

system with poor reliability can reduce a user’s

satisfaction dramatically, and can affect the work

practices of the user. If this component of the PIUS

framework is missing, the user and the work

practices will be also be affected, resulting in a

dissatisfied user and a poor system.

ICEIS 2008 - International Conference on Enterprise Information Systems

390

User Interface. As far as the user is concerned, the

interface is the product, it is the part of the system,

which the user sees, hears and communicates with.

An interface should provide the user with an easy

and flexible interaction with the system therefore,

preventing users from becoming disorientated, and

assisting them carry out their working objectives

effectively.

The design of the interface must consider the

physical design of the workplace (ergonomics) and

the support documentation (training and support).

Since the functionality of systems is made available

through its user interface, its design has a huge

influence on the usability of the system. The

interface must be developed according to the needs

of the users and the business, and should be designed

after all the business process have been finalised. A

good interface requires deep understanding of the

work practices in the context of the tasks that the IS

will help carry out. It is irrelevant how well

engineered the software code or how sophisticated

the hardware is; a bad interface can ruin an otherwise

excellent system. Ultimately, a systems interface can

have a huge bearing on the training, the work

practices and the user. If this component is poor or

limited the overall PIUS framework will not be

complete and will affect the user’s ability to learn

(training) and to carry out their work appropriately.

Support and Training.

Effective support and training

is crucial when users are being introduced to a new

system. New users of high-function application

systems can become frustrated and confused by the

errors they make in the early stages of learning. Lack

of user training and a failure to fully comprehend

how the IS works, is a major reason why newly

implemented systems fail. A systems support is

made up of documentation, both online and physical

manuals and technical support, offered by

individuals who are experts in the system

architecture. If the support and training component is

missing the PIUS framework will be significantly be

hampered. Poor support and training will have a

knock on affect and can lead to an unproductive and

unsatisfied user. Training should provide the user

with the knowledge required to do their job, if

training is inadequate the user would not be able to

do their job correctly.

Working Environment / Ergonomics.

A large

number of factors play a role in ergonomics; these

include body posture and movement, environmental

factors, information and operations as well as tasks

and jobs. Ergonomics can contribute to the solution

of a large number of social problems, such as

comfort, health and safety and user performance

within the environment of the system. The goals of

ergonomics range from the basic aim of making

work safe through increasing human efficiency, to

the purpose of creating human well-being. A well-

designed environment can produce improved

productivity, efficiency, acceptance and contentment

for the user. The aim of ergonomics is to achieve an

increase in user satisfaction by making their working

life secure. Despite, the importance of human safety

in the workplace, organisations fail to recognise how

closely it relates to their success. Users who find the

environment they are working in too uncomfortable

will have low job satisfaction, and will have poor

productivity disregarding how easy the IS is to use or

how reliable it is.

5 CONCLUSIONS

The PIR and usability evaluations are critical

activities within the ISD life cycle, as far as

information evaluation is concerned. Forcing users to

use and accept systems only leads to failure or user

resistance (Nielsen & Coyle, 2001; Woodings &

Everett, 1999; Willcocks et al., 1997; Ardito et al.,

2004). This paper introduces a conceptual framework

that will ensure ultimate system usability for the

users it was designed to assist. The objective of the

PIUS framework is to achieve ultimate usability for

the user and the best way to produce a usable system

is to know exactly what the user wants and needs.

Working closely with the user and establishing

which area or areas of the system s/he is having

difficulty with can accomplish this. This framework

has been specially designed for systems that have

recently been introduced to the workplace and places

the user at the centre of the evaluation. The

framework should be incorporated into a post-

implementation evaluation of the new large-scale IS.

For user satisfaction to be achieved the organisation

must develop a suitable environment for each user.

Only when each piece of a jigsaw is fitted together

properly can a jigsaw be deemed finished. The PIUS

Framework is no different. For example, if the users

are having difficulties working with the interface, no

matter how reliable or functional the system is, the

PIUS framework will not piece together entirely.

Only when each component of the PIUS framework

is pieced together properly can the large-scale IS

achieve ultimate usability for the users. The PIUS

framework is based on current research in the area of

usability and post-implementation. However, as little

work has been done to validate these issues in a

IMPROVING USER SATISFACTION IN THE POST-IMPLEMENTATION PHASE OF A LARGE-SCALE

INFORMATION SYSTEM

391

holistic way, the next stage is to carry out field

research to verify the various elements of the PIUS

framework and to determine the relative importance

of the various pieces in addressing post-

implementation usability comprehensively.

REFERENCES

Babar, M. A.; Zhu, L.; Verner, J. & Huo, M. (2004).

Software Quality and Agile Methods. 28th Annual

International Computer Software and Applications

Conference. Hong Kong, IEEE Computer Society.

Bradford, G. (2003). 'What's Old is New Again: Training is

the Information Technology Constant', Special Interest

Group on University and College Computing Services,

Vol 14, pp.255 - 258

Calisir, F. & Calisir, F. (2003). 'The relation of interface

usability characteristics, perceived usefulness, and

perceived ease of use to end user satisfaction with

enterprise resource planning (ERP) systems', Elsevier,

Vol 20, pp.505-515

Carroll, J. & Carrithers, C. (1984). 'Training Wheels in a

User Interface', Human Aspects of Computing, Vol 27,

pp.800 - 806

Devaraj, S. & Babu, S. (2004). 'How To Measure The

Relationship Between Training And Job Performance',

Vol 47, pp.63 - 67

Dix, A.; Beale, R.; Finlay, J. & Abowd, G. (1993). Human-

Computer Interaction, Prentice Hall.

Fruhling, A.; Vreede, G.-J. d. & Chakrapani, A. (2005). A

repeatable Collaboration Process for Usability Testing.

Proceedings of the 38th Hawaii International

Conference on System Sciences. IEEE.

Halpin, L. (2003). Chaos and Complexity in the Post-

Implementation Phase of ISD. Physical & Quantitative

Science. Waterford, Waterford Institute of Technology

(WIT).

Hawking, P. & Stein, A. (2002). 'E-skills: The Next hurdle

for ERP implementations', 36th Annual Hawaii

International Conference on System Sciences

(HICSS'03), Vol 8, pp.235

Hix, D. & Hartson, H. R. (1993). Developing User

Interfaces: Ensuring Usability Through Product and

Process, Wiley Professional Computing.

Hunter, N. (2005). Hospital's PPARS warning to HSE.

http://www.irishhealth.com/?level=4&id=7916.

INBITE (2005). PPARS goes PEAR Shaped. INBITE

Conference Series.

Irish-Health (2007). PPARS system 'a nightmare'.

IrishHealth.com. Dublin.

ISO-9241 (1998). Ergonomics requirements for office

work with visual display terminals., Part 11 - Guidance

on Usability.

Jeffries, R.; Miller, J. R.; Wharton, C. & Uyeda, K. M.

(1991). 'User Interface Evaluation in the Real World:

A Comparison of Four Techniques', Association for

Computing Machinery, Vol 1, pp.119 - 124

John, B. E. (1996). 'Evaluating Usability Evaluation

Techniques', ACM Computing Surveys, Vol 28,

Kennedy, J. (2005). Plan or be damned, warns expert on

PPARS. Silicon Republic.com.

Krasner, H. (2004). Ensuring E-Business Success by

Learning from ERP Failures. IT Pro. IEEE.

Kueng, P. (2002). Impact of Workflow Systems on People,

Task and Structure: a post-implementation evaluation.

Fifth European Conference on the Evaluation of

Information Technology.

Lucas, H. C. (2005). Information Technology: Strategic

Decision Making for Managers, John Wiley & Sons.

Lyytinen, K. & Klein, H. (1985). The Critical theory of

jurgen habermas as a basis for a theory of information

systems. WG8.2 Conference: Research Methods in

Information Systems. Manchester UK, Elsevier

Science Publishing B.V.

Nicolaou, A. (2003). Quality of Post Implementation

Review for Enterprise Resource Planning.

Nicolaou, A. I. (2004). 'ERP Systems Implementation:

Drivers of Post-Implementation Success', DSS 2004

Conference Proceedings, Vol 1, pp.589-597

Nielsen, J. (1993). Usability Engineering, Florida, Morgan

Kaufmann.

Nielsen, J. & Coyle, K. P. (2001). Emerging Technology.

CIO Magazine. CIO Magazine.

PPARS (2005). Personnel, Payroll and Related Systems.

IN Health (Ed.) July.

Shang, S. & Seddon, P. (2000). A Comprehensive

Framework for Classifying the Benefits of ERP

Systems. Americas Conference on Information

Systems. Long Beach, California.

Shneiderman, B. (1998). Designing the User Interface:

Strategies for Effective Human-Computer Interaction,

Addison Wesley Longman, Inc.

Tallon, P.; Kraemer, K. L. & Gurbaxani, V. (2001).

'Executives' Perceptions of the Business Value of

Information Technology: A Process-Oriented

Approach', Journal of MIS, Vol 12,

Weston, F. C. (2001). 'ERP Implementation and Project

Management', Production and Inventory Management,

Vol 42, pp.75 - 80

Willcocks, L.; Feeny, D. & Islei, G. (1997). Managing IT

as a Strategic Resource, Shoppenhangers Road,

Maidenhead, Berkshire, SL6 2QL, England, McGraw-

Hill Book Company (Europe).

Wixon, D.; Holtzblatt, K. & Knox, S. (1990). Contextual

design: an emergent view of system design. Human

factors in computing systems: Empowering people.

Seattle, Washington, United States, ACM Press.

Woodings, T. L. & Everett, J. E. (1999). A Methodology

with Quality Tools to support Post Implementation

Reviews. 10th Australasian Conference on Information

Systems. New Zealand, ACIS.

Yu, C.-S. (2005). Causes influencing the effectiveness of

the post-implementation ERP system. Industrial

Management & Data Systems. Emerald Group

Publishing Limited.

ICEIS 2008 - International Conference on Enterprise Information Systems

392