INCLUDING EXPLICITLY THE QUESTION OF ‘WHICH’
IN EVALUATION STUDIES
Noor Azizah K. S-Mohamadali
1,2
and Jonathan M. Garibaldi
1
1
Intelligent Modelling and Analysis Research Group (IMA), School of Computer Science
University of Nottingham, Nottingham, U.K.
2
Department of Information Systems, Faculty of Information and Communication Technology
International Islamic University Malaysia (IIUM), P.O. Box 10, 50728 Kuala Lumpur, Malaysia
Keywords:
Evaluation purposes, Multi-Criteria Decision Analysis (MCDA), Analytic Hierarchy Process (AHP), Fuzzy
AHP Approach (FAHP), Structural Equation Modeling (SEM).
Abstract:
Existing studies of user acceptance factors try to provide answers to the questions of ‘why’, ‘what’, ‘who’,
‘when’ and ‘how’. In this paper, we propose to include the explicit question of ‘which’ into future evaluation
studies. Two distinct approaches are discussed to address the question of ‘which’. The aim is not to show
which is the best, but rather to demonstrate the potential of some alternative approaches to addressing ‘which’.
1 INTRODUCTION
Effective evaluation of health care information sys-
tems is necessary to ensure systems adequately meet
the requirements and information processing needs of
health care organizations. The issue of user accep-
tance of healthcare technology is often the main fo-
cus of research in evaluation studies (Yi et al., 2006;
Martens et al., 2008). Evaluation outcomes may al-
low decision makers to take appropriate courses of
action. However, to be able to take appropriate ac-
tion, decision makers need to know not only fac-
tors that contribute towards acceptance but also which
among these factors are the most crucial in influenc-
ing user acceptance. Knowing the level of importance
of each of the factors would help decision makers han-
dle the factors appropriately, rather than focusing ef-
fort equally on all. Although there are studies that dis-
cuss the critical success factors (Jen and Chao, 2008;
Wu et al., 2007), a formal methodology to answer the
question of ‘which’ should be explicitly introduced as
a vital aspect of user acceptance studies. In this paper,
we propose several approaches that could help to an-
swer explicitly the question of ‘which’, that is which
is the most influential factor in user acceptance.
2 BACKGROUND
According to (Friedman and Wyatt, 1997), evaluation
is carried out to find answers to the following:
• The question of ‘Why’: This question tries to an-
swer the purpose of evaluation. There are stud-
ies for example concerned with user adaptation
to the new technology which examine specific at-
tributes such as user satisfaction (Schaper and Per-
van, 2007; Yi et al., 2006).
• The question of ‘What’ and ‘Who’: There are
mainly four main major stakeholders interested in
the results of evaluation work. These are the or-
ganization, the user of the system, the developer,
and of course the patients.
• The question of ‘When’: Evaluation can be car-
ried out during any or all three phases within
the system development life cycle, which are
pre-implementation (development), during imple-
mentation, and post-implementation (Yusof and
Papazafeiropoulou, 2008).
• The question of ‘How’: There are two distinct ap-
proaches for evaluation which are the objectivist
approach and the subjectivist approach.
In our previous publications, we have proposed a
novel evaluation model of user acceptance of technol-
ogy factors (K.S-Mohamadali and Garibaldi, 2009;
K.S-Mohamadali and Garibaldi, 2010). In this paper,
we access the proposed factors using methods dis-
cussed below to show how these method could help
to answer an explicit question of ‘which’.
341
Azizah K. S-Mohamadali N. and M. Garibaldi J..
INCLUDING EXPLICITLY THE QUESTION OF ‘WHICH’ IN EVALUATION STUDIES.
DOI: 10.5220/0003706503410344
In Proceedings of the International Conference on Health Informatics (HEALTHINF-2012), pages 341-344
ISBN: 978-989-8425-88-1
Copyright
c
 2012 SCITEPRESS (Science and Technology Publications, Lda.)

3 METHODS
3.1 Multi-criteria Decision Analysis
(MCDA) Techniques
MCDA is a discipline which aims to support decision
makers when they are faced with various conflicting
evaluation items. Various MCDA methods are avail-
able, such as the Analytic Hierarchy Process (AHP),
Goal Programming (GP), Fuzzy AHP, Data Envelop-
ment Analysis (DEA), Multi-attribute utility theory,
Scoring methods, Electra, and many more. An on-line
questionnaire was distributed to the medical schools
of various universities in the UK from January 2011
until March 2011. 38 users responded to the question-
naire. Respondents were asked to indicate the extent
to which they felt the influence of various factors to-
wards their acceptance of medically related software
through pairwise comparison methods using linguis-
tic variables (ranging from 1 = equally important to 9
= Absolutely More Important).
3.1.1 Classical AHP Approach
The basic steps involved in this methodology are as
follows (Zahedi, 1986):
• Step 1: Set up the hierarchy structure by breaking
down the decision problem.
• Step 2: Collect the input data by pairwise com-
parisons of the decision elements according to a
given ratio scale.
• Step 3: Use the ‘eigenvalue’ method to estimate
the relative weights of the decision elements.
• Step 4: Aggregate the relative weights of decision
elements to arrive at a set of ratings for the de-
cision alternative. Synthesize them for the final
measurement of given decision alternatives.
3.1.2 Fuzzy AHP: Chang’s Method
The weights of the factors and sub-factors were re-
calculated according to the Fuzzy AHP methodol-
ogy using the same data set. Fuzzy AHP incorporate
fuzzy values into the AHP method. There are several
variations of fuzzy AHP. One approach is known as
Chang’s extent analysis on fuzzy AHP (Chang, 1996).
The steps of Chang’s extent analysis can be given as
in the following:
Step 1: The value of fuzzy synthetic extent with
respect to the ith object is first defined as
S
i
=
m
∑
j=1
M
j
g
i
⊗
"
n
∑
i=1
m
∑
j=1
M
j
g
i
#
−1
(1)
To obtain
∑
m
j=1
M
j
g
i
, the fuzzy addition operation of m
extent analysis values for a particular matrix is per-
formed such as
m
∑
j=1
M
j
g
i
=
m
∑
j=1
l
j
,
m
∑
j=1
m
j
,
m
∑
j=1
u
j
!
(2)
and to obtain
h
∑
n
j=1
∑
m
j=1
M
j
g
i
i
−1
, the fuzzy addition
operation of M
j
g
i
( j = 1, 2, ...m) is performed such as
n
∑
i=1
m
∑
j=1
M
j
g
i
=
m
∑
j=1
l
j
,
m
∑
j=1
m
j
,
m
∑
j=1
u
j
!
(3)
then the inverse of the vector above is computed, such
as
"
n
∑
i=1
m
∑
j=1
M
j
g
i
#
−1
=

1
∑
n
i=1
u
i
,
1
∑
n
n=1
m
i
,
1
∑
n
i=1
l
j

(4)
Step 2: The degree of possibility of M
2
=
(a
2
, b
2
, c
2
) ≥ M
1
= (a
1
, b
1
, c
1
) is defined as
V (M
2
≥ M
1
) = sup
y≥x
[min(µ
M
1
(x)), µ
M
2
(y))] (5)
and can be expressed as follows:
V (M
2
≥ M
1
) = hgt(M
1
∩ M
2
) = µ
M
2
(d) (6)
=





1 if M
2
≥ M
1
0 if l
1
≥ u
2
l
1
−u
2
(m
2
−u
2
)−(m
1
−l
1
)
otherwise
(7)
To compare M
1
and M
2
we need both the value of
V (M
1
≥ M
2
).
Step 3: The degree of possibility for a convex
fuzzy number to be greater than the convex fuzzy
number M
i
(i = 1, 2, ..., k) can be defined by V (M ≥
M
1
, M
2
, ..., M
k
) = V (M ≥ M
1
) and (M ≥ M
2
... and
(M ≥ M
k
) = minV (M ≥ M
i
),
i = 1, 2, 3, ...k. (8)
Assume that
d
0
(A
i
) = minV (S
i
≥ S
k
) (9)
For k = 1, 2, ..., n; k 6= i. The weight vector is given by
W
0
= (d
0
(A
1
), d
0
(A
2
), ..., d
0
(A
i
))
T
(10)
where A
i
(i = 1, 2, ..., n) are n elements.
Step 4: Via normalization, the normalized weight
vectors
W = (d(A − i), d(A
2
), ...d(A
n
))
T
(11)
where W is a non-fuzzy number.
HEALTHINF 2012 - International Conference on Health Informatics
342

3.2 Structural Equation Modeling
(SEM Technique)
Structural equation modeling (SEM) is a statistical
technique which can be used to test and estimate
causal relations using a combination of statistical data
and qualitative causal assumptions (Hair et al., 2010).
SEM techniques allow for both confirmatory and ex-
ploratory modeling which means it can be used for
both theory testing and theory development. An
online questionnaire was published from December
2010 until April 2011 for a period of four months. In
total we obtained 113 respondents. We used Warp-
PLS version 2.0 software to analyze the data with
the application of a jack-knifing technique to deter-
mine the significance levels for loadings, weights and
path coefficients (Kock, 2011). Partial Least Square
(PLS) path modeling is one of the statistical methods
for SEM which can be used to model the complex re-
lationship of multiple endogenous (independent) and
exogenous (dependent) variables.
4 RESULTS AND DISCUSSION
4.1 Classical AHP
By applying steps 3 and 4, we computed the nor-
malized value followed by local weights for each of
the factors and sub-factors. Once we obtained local
weights for each of the factors and sub-factors, we
then computed the global weights for each of the in-
dividual sub-factors by multiplying the importance of
each factor with those of the sub factors. The results
are shown in Table 1. As can be seen, AHP suggests
Information Quality (IQ), = 0.203, as the most influ-
ential factor of user acceptance.
4.2 Fuzzy AHP
By using formula (1) above, the resultant weight vec-
tor is W
0
= (0.826,1.000,0.534)
T
. After normalization,
the normalized weight vector of each objective with
respect to the individual, technology and organization
factors is obtained as W
goal
= (0.350, 0.424, 0.226).
Using the same steps discussed in above section, we
calculated the weight of each of the factors within
these three broad categories. The resulted weights are
shown in table 1. Fuzzy AHP also suggests that Infor-
mation Quality (IQ), = 0.207, as the most influential
factor.
Table 1: The Global Weight of the Factors.
Factor AHP Fuzzy AHP
PE 0.108 0.140
EE 0.055 0.052
SI 0.021 0.000
ISE 0.152 0.159
SWQ 0.153 0.195
SERQ 0.065 0.022
IQ 0.203 0.207
FC 0.171 0.200
MS 0.075 0.026
4.3 Structural Equation Modeling
Technique
Based on the two-step approach recommended by
(Anderson and Gerbing, 1988), we have analyzed
the measurement model to test the reliability and va-
lidity of the instrument and also analyzed the struc-
tural model to test the proposed relationships. Our
model confirmed both reliability and validity prop-
erties. The structural model indicated the casual re-
lationships among constructs which include the es-
timates of the path coefficient and R-squared value.
Figure 1 exhibits the structural model and the an-
alytical results. As can be seen, the SEM method
assigns Software Quality(SWQ), with an R-squared
value=0.26, as the most important weight which sug-
gest this is the most influential factor.
Figure 1: Loading of the factors using SEM.
5 ENHANCING THE
EVALUATION QUESTIONS IN
EVALUATION STUDIES
In this paper, we have demonstrated two methods
(MCDA and the SEM technique) which can help to
answer the explicit question of ‘which’. This question
INCLUDING EXPLICITLY THE QUESTION OF 'WHICH' IN EVALUATION STUDIES
343

of ‘which’ is explicitly added into existing evaluation
questions as shown in Figure 2. The dotted lines show
the original evaluation questions. The solid line is the
proposed ‘which’ question, which we believe, should
be addressed in every evaluation study.
Why
When
Who
Objective of Evaluation
Which stakeholders perspective
is going to be evaluated?
Which phase in system
development life cycle?
What
How
Which
Aspects of evaluation?
Methods of Evaluations
Which is the most
influential factor?
Figure 2: An Enhanced Evaluation Question of ‘Which’.
6 CONCLUSIONS
Evaluation of the factors that influence user accep-
tance of the software technology is a crucial and im-
portant effort. Originally, evaluation is carried out
to find the answer to five main questions which are
‘why’, ‘what’, ‘who’, ‘when’ and ‘how’. We believe
the question of ‘which’ also needs to be explicitly
addressed and specifically recognised in all evalua-
tion studies. Two distinct approaches are discussed
in this study to determine the weighting between fac-
tors. The approaches presented in this study are not
intended to show which is the best to be used to eval-
uate users’ acceptance factors, but rather to illustrate
some of the various options which are available to be
used to derive weights among evaluation factors and,
hence, to explicitly answer the question of ‘which’.
Future evaluation studies should explicitly incorpo-
rate the ‘which’ question in order to realize the maxi-
mum benefit for the various stakeholders.
ACKNOWLEDGEMENTS
Noor Azizah KS Mohamadali would like to gratefully
acknowledge the funding received from both the Pub-
lic Service Department of Malaysia and the Interna-
tional Islamic University of Malaysia(IIUM) in spon-
soring this research.
REFERENCES
Anderson, J. C. and Gerbing, D. W. (1988). Structural
Equation Modeling in Practice: A Review and Rec-
ommended Two-Step Approach. Psychological Bul-
letin, 103:411–423.
Chang, D.-Y. (1996). Applications of the extent analysis
method on fuzzy AHP. European Journal of Opera-
tional Research, 95:649–655.
Friedman, C. and Wyatt, J. (1997). Evaluation Methods in
Medical Informatics. Springer-Verlag, New York.
Hair, J. F., Black, W. C., Babin, B. J., and Anderson, R. E.
(2010). Multivariate Data Analysis. Pearson.
Jen, W.-Y. and Chao, C.-C. (2008). Measuring mobile pa-
tient safety information system success: An empirical
study. International Journal of Medical Informatics,
77:689–697.
Kock, N. (2011). Using WarpPls in e-collaboration Stud-
ies: Descriptive Statistics, settings, and key analy-
sis results. International Journal of e-Collaboration,
7(2):1–18.
K.S-Mohamadali, N. A. and Garibaldi, J. M. (2009). To-
wards the Development of a Novel Evaluation Frame-
work for Information Systems in Healthcare Sector. In
Second International Joint Conference on Biomedical
Engineering Systems and Technologies, pages 17–24.
K.S-Mohamadali, N. A. and Garibaldi, J. M. (2010). A
Novel Evaluation Model of User Acceptance of Soft-
ware Technology. In 3rd International Joint Confer-
ence on Biomedical Engineering Systems and Tech-
nologies, pages 392–397.
Martens, J., der Weijden, T., Winkes, R., Kester, A., Geerts,
P., Evers, S., and Severens, J. (2008). Feasibility
and acceptability of a computerised system with auto-
mated reminders for prescribing behaviour in primary
care. International Journal of Medical Informatics,
77:199–207.
Schaper, L. K. and Pervan, G. P. (2007). Ict and OTs:
A model of information and communication technol-
ogy acceptance and utilisation by occupational thera-
pists. International Journal of Medical Informatics,
76s:S212–S221.
Wu, J.-H., Wang, S.-C., and Lin, L.-M. (2007). Mobile
computing acceptance factors in the healthcare indus-
try: A structural equation model. International Jour-
nal of Medical Informatics, 76:66–77.
Yi, M. Y., Jackson, J. D., Park, J. S., and Probst, J. C.
(2006). Understanding information technology accep-
tance by individual professionals: Towards an integra-
tive view. Information and Management, 43:350–363.
Yusof, M. M. and Papazafeiropoulou, A. (2008). Investi-
gating evaluation frameworks for health information
systems. International Journal of Medical Informat-
ics, 77:377–385.
Zahedi, F. (1986). The analytic hierarchy process: A Survey
of the Method and its Applications. Interfaces, 16:96–
108.
HEALTHINF 2012 - International Conference on Health Informatics
344