A NOVEL EVALUATION MODEL OF USER ACCEPTANCE OF

SOFTWARE TECHNOLOGY IN HEALTHCARE SECTOR

Noor Azizah K. S. Mohamadali and Jonathan M. Garibaldi

Intelligent Modelling and Analysis (IMA) Research Group

School of Computer Science, University of Nottingham, NG8 1BB, U.K.

Keywords:

Evaluation model, User acceptance, IS success model, UTAUT model, Task-technology ﬁt.

Abstract:

Effective evaluation of healthcare information systems is necessary in order to ensure systems adequately

meet the requirements and information processing needs of the users and healthcare organisations. A number

of studies have proposed evaluation frameworks to aid in evaluation work. However, some of them can be

improved upon. This paper proposes a novel evaluation model to evaluate user acceptance of software tech-

nology within the healthcare sector. It describes the theoretical basis behind the development of the research

model and methodology being employed to validate the model.

1 INTRODUCTION

Information systems play an increasingly important

role in developing the structure and functions of

healthcare industry. Almost a billion dollars are be-

ing spent on the procurement of new technology with

the aim to improve organization performance as well

as the quality of service delivered to the patient. Any

new system implemented is estimated to take approx-

imately 18 months to be fully operationalized (Lee

et al., 2008). Although technology often brings ben-

eﬁts to an organization, sometimes implementations

do fail due to low levels of user acceptance (Southon

et al., 1999). Since the success or failure of system

implementation largely depends on user acceptance

of technology, much research has been carried out to

identify those critical factors that inﬂuence user ac-

ceptance (Yusof et al., 2008; Shaw, 2002; Despont-

Gros et al., 2005; Schaper and Pervan, 2007). Some

authors have proposed frameworks to assist evalua-

tion. However, we believe most of these frameworks

can be improved upon. In this paper, we propose a

novel evaluation model to evaluate user acceptance of

technology. The theoretical signiﬁcance of this work

is that it will use well-established theory in informa-

tion systems to inform the proposed model.

2 THEORETICAL BACKGROUND

In order to build the evaluation model, the following

three-step methodology was adopted:

• A general review of existing information system

models to provide a theoretical basis to the key di-

mension of user satisfaction or acceptance of tech-

nology.

• A general review of existing evaluation frame-

works to identify strengths and limitations of each

framework, if any, to identify improvements.

• A general review of information system evalua-

tion studies in healthcare to highlight studied vari-

ables on user acceptance of technology.

2.1 Review of Theories of User

Acceptance of Technology

There are a number of theories that focus on user ac-

ceptance of technology such as the technology accep-

tance model (Davis, 1989), the theory of reasoned

action, the IS Success Model (DeLone and McLean,

2003), the diffusion of innovation theory, the Uniﬁed

Theory of Technology Acceptance and Use of Tech-

nology (Venkatesh et al., 2003) and many other re-

lated theories. Here, we discuss three theories that we

believe are relevant to our research problem.

392

Azizah K. S. Mohamadali N. and Garibaldi J. (2010).

A NOVEL EVALUATION MODEL OF USER ACCEPTANCE OF SOFTWARE TECHNOLOGY IN HEALTHCARE SECTOR.

In Proceedings of the Third International Conference on Health Informatics, pages 392-397

DOI: 10.5220/0002695703920397

 SciTePress

2.1.1 DeLone and McLean IS Success Model

Since its introduction by (Delone and McLean, 1992),

the model has been widely used, evaluated, validated

and extended in various studies. In 2003, the IS Suc-

cess Model was updated by inclusion of net bene-

ﬁts derived from intention to use and user satisfac-

tion (DeLone and McLean, 2003). The IS Success

Model illustrates clear, speciﬁc dimensions of success

or effectiveness and the relationships between the fac-

tors. Dimensions suggested are system quality, infor-

mation quality, service quality, intention to use, ac-

tual use, user satisfaction and net beneﬁts. (Petter and

McLean, 2009) in his paper has given a detailed de-

scription of each of the constructs. However, the lim-

itation of this model is that it cannot provide expla-

nation as to why the same IT system can be adopted

in different ways, with different effects in various set-

tings (Tsiknakis and Kouroubali, 2008).

2.1.2 Uniﬁed Theory of Technology Acceptance

and Use of Technology (UTAUT)

(Venkatesh et al., 2003) proposed a more complete

model for understanding the acceptance and adoption

of information technology. The UTAUT model con-

tains four core determinants of intention and usage,

while also incorporating moderating variables, gen-

der, age, experience and voluntariness of use, which

may or may not have an inﬂuence on user acceptance

of technology. A description of the constructs deﬁned

by the model are as follows:

• Performance expectancy — “The degree to which

an individual believes that using the system will

help him or her to attain gains in job performance.

Measured such as accomplish task quickly, im-

prove job performance, increase productivity, en-

hance effectiveness.”

• Effort expectancy — “The degree of ease associ-

ated with the use of system. Measured such as

interaction with system clear and understandable,

ﬂexible to iteract, easy to use, etc.”

• Social inﬂuence — “The degree to which an indi-

vidual perceive that important others believe he or

she should use the new system.”

• Facilitating condition — “The degree to which an

individual believes than an organization and tech-

nical infrastructure exist to support the use of a

system.”

The UTAUT model is a relatively new model and

to our knowledge it has not been widely used in the

healthcare domain. Thus, this model will be utilized

in understanding user acceptance of technology in

healthcare domain.

2.1.3 Task-technology Fit (TTF)

Task-technology ﬁt model was developed by (Good-

hue, 1995). TTF theory holds that information tech-

nology is more likely to have a positive impact on its

individual performance and be used if the capabilities

of the technology match the tasks that the user must

perform. The importance of task ﬁt and technology

has been highlighted in many studies (Ammenwerth

et al., 2006; Goodhue et al., 2000; Goodhue, 1995).

The ﬁt between attributes to accomplish certain tasks

is more important that the individual attributes them-

selves. For example, a user with certain IT skills is not

a sufﬁcient requirement for the use of a new system;

rather their skills must be match with the requirement

of the IT system itself (system complexity). In addi-

tion to the ﬁt deﬁned by the model, we believe there

must exist also a ﬁt between the individual and the

organisation. For example, if a particular individual

does not possess the skill to use a new technology,

the organisation needs to provide necessary training.

This will eventually inﬂuence the user to accept the

technology. Thus, other than the ﬁt between individ-

ual and organisation, the ﬁt between organisation and

technology is also equally important and we include

this additional ﬁt in our model.

2.1.4 Observation and Recommendation

Both the IS Success Model and the UTAUT model by

itself is an excellent model. Each is internally sound

and based directly on well-tested attitude/ behaviour

models. Both models deﬁne almost similar depen-

dant constructs, where in IS Success Model it is called

‘intention to use’ or ‘use’, whereas in the UTATUT

model it is called ‘behavioural intention’. However,

for this dependant construct, each model deﬁnes dif-

ferent independent constructs. For example, in the

UTAUT model, behavioural intentions are determined

by performance expectancy, effort expectancy and so-

cial inﬂuence. In the IS Success Model, intention to

use or use is determined by information quality, sys-

tem quality and service quality. We believe that, by

combining all these independent factors, we can bet-

ter represent factors that determine intention to use

or behavioral intention of the user of the system. As

observed also, the weakness of the existing models

is in their lack of task focus (ﬁt) between various

constructs, which contributes to the mixed results in

information technology evaluations studies (Dishaw

and Strong, 1999). Thus, instead of proposing TTF

as an alternative to the IS Success Model or UTAUT

model, we propose to add the strengths of TTF mod-

els to the IS Success Model and UTAUT model to

produce an integrated model which incorporate both

A NOVEL EVALUATION MODEL OF USER ACCEPTANCE OF SOFTWARE TECHNOLOGY IN HEALTHCARE

SECTOR

393

attitudes toward information technology and the ﬁt

between technology functionality and the character-

istics of the tasks that users need to accomplish with

the technology. We believe that these three models

overlap in a signiﬁcant way and, if integrated, it could

provide a better explanation of information technol-

ogy utilization by the users and also a stronger model

than any stand alone model.

2.2 Observation on Early Study of

Existing Evaluation Frameworks

In this section, we brieﬂy analyze several existing

evaluation frameworks. Most of the frameworks were

developed based on inﬂuences from other disciplines

(Chiasson et al., 2007).

• ICT and OTs: A model of information and com-

munication technology acceptance and utilisation

by occupational therapists (Schaper and Pervan,

2007). The framework was developed to exam-

ine acceptance and utilisation issues among health

professionals and was based on the UTAUT model

by (Venkatesh et al., 2003) and ‘Technology Ac-

ceptance (TAM)’ by (Davis, 1989). However, de-

scription of task charateristics is lacking in this

model. We believe on the importance of task char-

acteristics and skills that user possess will eventu-

ally inﬂuence users’ usage of the technology and

this must be evaluated when we address user ac-

ceptance of technology.

• Understanding IT: acceptance by individual pro-

fessional: Towards an integrating view (Yi et al.,

2006). The framework was based on the TAM

model by (Davis, 1989), planned behavior and

innovation diffusion. This research model was

tested in the context of personal digital assistance

acceptance by healthcare processionals. We be-

lieve that the important factor which is the ﬁt be-

tween the technology and the task that the user

needs to perform is important and this ﬁt is lack-

ing in this model.

• HOT-ﬁt: An evaluation framework for health

information systems: human, organization and

technology-ﬁt factors (Yusof et al., 2008). The

proposed framework is constructed so that it is

capable of being useful in conducting a thorough

evaluation study and to help researcher and practi-

tioners to understand the perceived complexity of

health information system evaluation. The model

was developed based on the IS Success Model

and the ‘IT-Organization ﬁt model’. The strength

of this model is inclusion of ﬁt element. How-

ever, we believe more constructs from the UTAUT

model can be incorporated into this model to bet-

ter explain user acceptance of technology.

• FIIT: Fit between Individual, technology and

Task by (Ammenwerth et al., 2006). This pro-

posed framework was believed to provide better

analyses on socio-organizational-technical factors

that inﬂuence IT adoption during implementation.

The model referred to the IS Success Model, the

TAM model and the IT adoption model. Inclusion

of several constructs from UTAUT model, includ-

ing moderating factor, can (we believe) further en-

hance the applicability of this model.

Based on our early observations, we believe each of

these frameworks can be improved upon to address

the issue of user acceptance of technology.

3 THE PROPOSED INTEGRATED

EVALUATION MODEL

The following proposed evaluation framework is de-

veloped after studying the existing models and theory

of user acceptance of technology, and following crit-

ical appraisal of the existing evaluation frameworks.

Three models were integrated to build this model as

shown in Figure 1. The features of this integrated

model are:

• Categorisation of all the independent constructs

under Individual, Organizational and Technology

context.

• The individual factors and their dimensions are

performance expectancy, effort expectancy and

social inﬂuence. These are constructs deﬁned in

the UTAUT model.

• The organisational factor and its dimension is fa-

cilitating condition, from the UTAUT Model.

• The technology factors and their dimensions are

system quality, information quality and service

quality. These are constructs deﬁned in the IS

Success Model

• Incorporation of ﬁt between individual, organi-

sational and technology context. In the original

TTF model, the ﬁt is between task (individual)

and technology. In addition to this, we include

ﬁt between individual and organisation, and be-

tween technology and organisation, which we be-

lieve are also equally important.

HEALTHINF 2010 - International Conference on Health Informatics

394

Table 1: Classiﬁcation of the Factors Associated with the Adoption of New Technology.

Factors/Constructs Measurement Variable Sources

Performance

Expectancy

Work Quicker, Improve Quality of work, Improve communication,

will use if offer advantages, save time, information exchange, clear

interaction and understandable, easy to use, multi-tasking, improved

work effectiveness, etc.

1, 2, 3, 4, 6,

Effort Expectancy Clear, easy to remember, always available, easy to use, usability, has

knowledge to use, improve job performance, enhance job effective-

ness, increase job productivity, user-friendly etc.

3, 4, 5,6, 7, 8,

9, 10

Social Inﬂuence Perception of other, usage beneﬁts social factors, social words, at-

mosphere, etc.

7, 11

Information Quality Relevance and usefulness, efﬁciency, conciseness and completeness,

reliability, observability and result demonstrability, accurate infor-

mation, content, format

10,11, 12

Service Quality Helpdesk support, user documentation, quality of hotline, support,

speed to repair

5, 7

System Quality Speed, response time, provide security, downtime, response time,

require multiple update, stability and usability, data accuracy, data

availability, ﬂexibility, reliability, functionality, connection time, sys-

tem function, content design and device accessibility, etc.

3, 5, 6, 8, 9,

11, 12, 13

Facilitating

Condition

Management support, training, provision of computer support (hard-

ware, software), system availability, network infrastructure, security,

compatibility of technology or technical issues/ appropriateness of

technology implemented, helpdesk, maintenance, support from IS

department, support from vendor, technical support, induction, etc.

2, 5, 6, 7, 8, 9,

13, 14, 15

User Satisfaction Task-support satisfaction, quality of work-life satisfaction, interface

satisfaction, decision-making satisfaction, software satisfaction, sat-

isfaction with system components

2, 12

Net Beneﬁts Improve task performance, efﬁciency and effectiveness (goal

achievement), error reduction

Moderating Factors Experience, age and gender 2, 6, 9, 11, 14,

3.1 Classiﬁcation of Factors: Individual,

Organizational, Technology

According to (Chau and Hu, 2002), technology ac-

ceptance has three dimensions:

1. characteristics of the individual;

2. characteristics of the technology; and

3. characteristics of organisational context.

Several evaluation frameworks categorise factors un-

der these three wide categories (Schaper and Pervan,

2007; Yusof et al., 2008; Lorenzi, 1999). Organisa-

tion factors need to be evaluated because introduction

of new information technology changes the services,

operation and also the structure of the organisation it-

self. Technology needs to be evaluated because it be-

comes part of the system within the organisation in

which it operates. The user or individual is mainly

subjective because they imply how people conceive

their relationship with the technology, and individual

also needs to be evaluated (Southon, 1999).

3.2 Validation of Proposed Evaluation

Model: Phase 1 (Literature Review)

In phase one of our research, in order to validate the

proposed model, we conducted a literature review to

identify factors associated with the adoption of new

technology. All these factors are collected and classi-

ﬁed according to the constructs deﬁned in our model

to test if our deﬁned constructs can incorporate these

factors and we managed to classify the identiﬁed fac-

tors. Table 1 shows the classiﬁcation of the factors

identiﬁed from this review. This indicates that most

of the factors associated with user acceptance of tech-

nology can be represented by our model. This demon-

strates the strength and applicability of our proposed

model.

A NOVEL EVALUATION MODEL OF USER ACCEPTANCE OF SOFTWARE TECHNOLOGY IN HEALTHCARE

SECTOR

395

Integrated Model of User Acceptance of Technology

Individual/ People

Factor

Performance Expectancy

Effort Expectancy

Social Influence

Technology Context

Information Quality

Service Quality

System Quality

Organizational Context

Facilitating Condition

Culture /newly added

Moderating Factors

Age, Gender, Experiences and Voluntariness

Behavioural Intention /

Intention to Use / Use

Behaviour

Net

Benefits

User Satisfaction

Fit

Influence

Figure 1: Proposed Evaluation Model of User Acceptance

of Technology.

Table 2: Reference to Sources.

Key Sources

1 (Otieno et al., 2008)

2 (Despont-Gros et al., 2005)

3 (Tsiknakis and Kouroubali, 2008)

4 (Yi et al., 2006)

5 (Shaw, 2002)

6 (Garﬁeld, 2005)

7 (Martens et al., 2008)

8 (Sicotte et al., 2006)

9 (Ammenwerth et al., 2006)

10 (Chiasson et al., 2007)

11 (Lee et al., 2008)

12 (Yusof et al., 2008)

13 (Heeks, 2006)

14 (Mahmood et al., 2000)

15 (Goodhue et al., 2000)

3.3 Validation of Proposed Model:

Phase 2 (Case Study Strategy)

In phase two of our study, in future, a case study strat-

egy will be employed. The case study will serve two

purposes:

1. to evaluate the adoption factor of new technology

in the context of the phenomena under study;

2. to validate the proposed evaluation model.

A case study will be conducted within the clinical

setting with research students from the Breast Can-

cer Pathology Research Group, based in the Queen’s

Medical Centre, Nottingham. This research centre

has recently purchased a new piece of software known

as Distiller (SlidePath, 2008). The model will be

used as a guideline in the evaluation of this software.

During the evaluation process, we will interview the

users, observe their use of the system and also per-

form document analysis.

4 CONCLUSIONS AND FUTURE

WORK

In this paper, we have proposed an evaluation model

to evaluate user acceptance of software technology

in the healthcare sector. Our proposed model in-

tegrates three very well-known theories of informa-

tion systems to represent factors that inﬂuence user

acceptance of software technology. In ﬁrst phase

of the study, we conducted a literature review to

identiﬁed factors associated with the user acceptance

of the technology. These factors were then clas-

siﬁed according to the constructs proposed in our

model. Through this process, we demonstrated that

our model can better represent factors associated with

technology acceptance. As future work, in order to

further validate our model, we will test the model in

clinical settings with research students by means of a

case study. Findings from ﬁeldwork will (hopefully)

be used to further improve and reﬁne the model. Iden-

tifying what are the factors that inﬂuence successful

implementation of new software technology among

the users is vital since the success and failure of any

new systems to operate in a new environment depends

largely on the acceptance of the users of the system.

ACKNOWLEDGEMENTS

Noor Azizah KS Mohamadali would like to grate-

fully acknowledge the funding received from both the

Public Service Department of Malaysia and from the

International Islamic University of Malaysia that is

helping to sponsor this research.

REFERENCES

Ammenwerth, E., Iller, C., and Mahler, C. (2006). It-

adoption and the interaction of task, technology and

HEALTHINF 2010 - International Conference on Health Informatics

396

individuals: a ﬁt framework and a case study. BMC

Medical Informatics and Decision Making, 6:1–13.

Chau, P. Y. K. and Hu, P. J.-H. (2002). Investigating helath-

care professionals’ decisons to accept telemedicine

technology: an empirical test of competing theories.

Information and Management, 39:297–311.

Chiasson, M., Reddy, M., Kaplan, B., and Davidson, E.

(2007). Expanding multi-disciplinary approaches to

healthcare information technologies: What does infor-

mation systems offer medical informatics? Interna-

tional Journal of Medical Informatics, 76s:S89–S97.

Davis, F. D. (1989). Perceived usefulness, perceived ease of

use, and user acceptance of information technology.

MIS Quarterly, 13:319–339.

Delone, W. H. and McLean, E. R. (1992). Information sys-

tems success: The quest for the dependent variable.

Information Systems Research, 3:60–95.

DeLone, W. H. and McLean, E. R. (2003). The delone

and mclean model of information systems success: A

ten year update. Management Information Systems,

19(4):9–30.

Despont-Gros, C., Mueller, H., and Lovis, C. (2005). Eval-

uating user interactions with clinical information sys-

tems: A model based on human-computer interaction

models. Journal of Biomedical Informatics, 38:244–

255.

Dishaw, M. T. and Strong, D. M. (1999). Extending the

technology acceptance model with task-technilogy ﬁt

constructs. Information and Management, 36:9–21.

Garﬁeld, M. J. (2005). Acceptance of ubiquitous comput-

ing. Information Systems Management, 22:24–31.

Goodhue, D. L. (1995). Understanding user evaluation of

information systems. Management Science, 41:1827–

1844.

Goodhue, D. L., Klein, B. D., and March, S. T. (2000).

User evaluation of is as surrogates for objective per-

formance. Information and Management, 38:87–101.

Heeks, R. (2006). Health information systems: Failure, suc-

cess and improvisation. International Journal of Med-

ical Informatics, 75:125–137.

Lee, T.-T., Mills, M. E., Bausell, B., and Lu, M.-H. (2008).

Two-stage evaluation of the impact of a nursing in-

formation system in taiwan. International Journal of

Medical Informatics, 77:698–707.

Lorenzi, N. M. (1999). Imia working group 13: organi-

zational impact of medical informatics. International

Journal of Medical Informatics, 56:5–8.

Mahmood, M. A., Burn, J. M., Gemoets, L. A., and

Jacquez, C. (2000). Variables affecting information

technology end-user satisfaction: a meta-analysis of

the empirical literature. International Journal of Hu-

man Computer Studies, 52:751–771.

Martens, J. D., van der Weijden, T., Winkes, R. A. G.,

Kester, A. D. M., Geerts, P. J. H., Evers, S. M. A. A.,

and Severens, J. L. (2008). Feasibility and accept-

ability of a computerised system with automated re-

minders for perscribing behaviour in primary care. In-

ternationa Journal of Medical Informatics, 77:199–

207.

Otieno, G. O., Hinako, T., Motohiro, A., Daisuke, K., and

Keiko, N. (2008). Measuring effectiveness of elec-

tronic medical records systems: Towards buidling a

composite index for benchmarking hospitals. Interna-

tional Journal of Medical Informatics, 77:657–669.

Petter, S. and McLean, E. R. (2009). A meta-analytic as-

sessment of the delone and mclean is success model:

An examination of is success at the individual level.

Information and Management, 46:159–166.

Schaper, L. K. and Pervan, G. P. (2007). Ict and ots:

A model of information and communication technol-

ogy acceptance and utilisation by occupational thera-

pists. International Journal of Medical Informatics,

76s:S212–S221.

Shaw, N. T. (2002). Cheats: a generic information commu-

nication technology (ict) evaluation framework. Com-

puters in Biology and Medicine, 32:209–220.

Sicotte, C., Pare, G., Marie-Pierre, and Paccioni, A. (2006).

A risk assessment of two interorganizational clinical

information systems. Journal of the American Medi-

cal Informatics Association, 13:557–566.

SlidePath (2008). Distiller. Available from

http://www.slidepath.com.

Southon, G. (1999). It, change and evaluation: an overview

of the role of evaluation in health services:. Interna-

tional Journal of Medical Informatics, 56:125–133.

Southon, G., Sauer, C., and Dampney, K. (1999). Lesson

from a failed information system initiative: issues for

complex organisations. International Journal of Med-

ical Informatics, 55:33–46.

Tsiknakis, M. and Kouroubali, A. (2008). Organizational

factors affecting successful adoption of innovative

ehelath services: A case study employing the ﬁtt

framework. International Journal of Medical Infor-

matics.

Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D.

(2003). User acceptance of information technology:

toward a uniﬁed views. MIS Quarterly, 27:425–478.

Yi, M. Y., Jackson, J. D., Park, J. S., and Probst, J. C.

(2006). Understanding information technology accep-

tance by individual professionals: Towards an integra-

tive view. Information and Management, 43:350–363.

Yusof, M. M., Kuljis, J., Papzafeiropoulou, A., and Ster-

gioulas, L. K. (2008). An evaluation framework for

health information systems: human, organization and

technology-ﬁt factors (hot-ﬁt). International Journal

of Medical Informatics, 77:386–398.

A NOVEL EVALUATION MODEL OF USER ACCEPTANCE OF SOFTWARE TECHNOLOGY IN HEALTHCARE

SECTOR

397