ASSESSING WORKFLOW MANAGEMENT SYSTEMS

A Quantitative Analysis of a Worfklow Evaluation Model

Stephan Poelmans

Department of Commercial Sciences, University-College Brussel

Stormstraat 2, 1000 Brussels, Belgium

Hajo A. Reijers

School of Industrial Engineering, Eindhoven University of Technology

Den Dolech 2, 5600 MB Eindhoven, The Netherlands

Keywords: Workflow Systems, Business Process Management, Usability, Technology Acceptance, Field Research.

Abstract: Despite the enormous interest in workflow management systems and their widespread adoption by industry,

few research studies are available that empirically assess the effectiveness and acceptance of this

technology. Our work exactly aims at providing such insights and this paper presents some of our

preliminary quantitative findings. Using a theory-based workflow success model, we have studied the

impact of operational workflow technologies on end-users in terms of perceived usefulness, end-user

satisfaction and perceived organisational benefits. A survey instrument was used to gather a sample of 246

end-users from two different organizations. Our findings show that the considered workflow applications

are generally accepted and positively evaluated. Using partial least squares analysis, the success model was

well supported, making it a usefull instrument to evaluate future workflow projects.

1 INTRODUCTION

Workflow management systems (WfMSs) (or

Business Process Management systems - BPMSs)

have been important information systems to

automate and increase the efficiency of business

processes for almost two decades. A WfMS is

essentially a set of tools enabling the modelling,

enactment, and monitoring of business processes

(Jablonski, 1996). Workflow functionalities have

also been integrated in other contemporary

enterprise systems such as ERP, and call-centre

applications.

It is clear that a pervasive enterprise system such

as a WfMS, often influencing the daily work of

numerous employees, can only be successful if the

targeted end-users accept the system and experience

a performance growth in their jobs. If the workflow

technology is not designed in a way that meets the

needs of both the end-users and the management, the

consequences might be dreadful, leading to the

existence of shadow systems and a loss of

productivity.

Despite this argument, only a few empirical

studies on the usage and success of operational

WfMSs exist. In this paper we present the first

results of a quantitative evaluation of two workflow

applications in different organisations.

In the next section we give a concise overview of

the extant WFM research and of the literature on the

acceptance and success of information systems. Next

we present our workflow system’s evaluation model.

The model has been validated and analysed, using a

sample of 237 end-users. After analysing the results,

we discuss the conclusions and future research.

2 LITERATURE OVERVIEW

2.1 Workflow Literature

The existing literature on workflow systems can be

subdivided into three research areas. A first research

thread is technology-driven and deals with topics

such as adaptive workflow systems and the

183

Poelmans S. and Reijers H. (2009).

ASSESSING WORKFLOW MANAGEMENT SYSTEMS - A Quantitative Analysis of a Worfklow Evaluation Model .

In Proceedings of the 11th International Conference on Enterprise Information Systems - Databases and Information Systems Integration, pages

183-189

DOI: 10.5220/0002005401830189

 SciTePress

development of intelligent tools to support exception

handling (see Weber, 2008; Casati et al., 1998).

Secondly and more recently, a number of studies

have been conducted using process mining methods

to measure the efficiency impact of workflow

systems on business process indicators such as lead

and throughput time (Van der Aalst et al., 2007).

A third research thread concerns the usability,

and usage of WfMSs. This field has so far received

only limited attention. Some studies point to the

negative impacts of a WfMS, while others report on

succesful projects (see Kueng, 2004; Dourish, 2001;

Bowers et al., 1995). However, as these studies are

restricted to a explorative quantitative analysis of

one case, a systematic comparison is hampered.

Exceptions are Reijers et al. (2007) and

Poelmans (2002) who evaluated and compared

several succesful workflow project in a qualitative

and quantitative way (using a survey).

In the underlying study we put forward a theory-

based, quantitative usability study that includes

several workflow projects and divers end-users. In

particular, we developed and validated an

explanatory workflow evaluation model that can be

applied to other workflow usability studies and even

to other enterprise systems.

2.2 ICT Acceptance and Usability

The use, success, and acceptance of information

systems have been investigated in an overwhelming

amount of studies, using widespread research

models such as the technology acceptance model

(TAM, Davis, 1989), and Delone & Mclean’s IS

success model (henceforth ISS model) (Delone et

al., 2003).

These models focus on the individual end-user

and have been applied to assess a diversity of IT

systems (like ERP systems, GSS systems, e-

commerce systems, etc.) (E.g. Wu et al., 2005;

Delone et al., 2004; Karahanna, 2002).

Whereas the TAM is particularly valid to predict

future acceptance and voluntary usage of ICT, the

ISS model focusses more on the evaluation of

objective system and information characteristics that

can enhance user satisfaction, perceived usefulness,

and individual impacts of an information system

(Wixom et al., 2005). The ISS model does not

necessarily imply (future) usage as a dependent

variable, so it can be used to evaluate both

mandatory and voluntary use of information

systems.

3 RESEARCH MODEL

Since WfMSs determine the collaboration of

employees and often integrate other legacy systems,

an individual employee has no real alternative but to

use the system. Therefore, usage frequency should

be considered as mandatory and has no added value

as a success measure. In this view, we turned to the

ISS model to develop our evaluation model.

Figure 1 presents the model that we used and

validated to measure the acceptance and success of

two workflow applications. The model uses three

general concepts as measures of success: perceived

usefulness, end-user satisfaction and, as an ultimate

dependent variable, perceived organisational

benefits.

In accordance with the ISS framework, our

model presumes that if a WfMS does increase job

performance (perceived usefulness), it will increase

the end-user’s satisfaction. Both measures will

impact the employee’s belief that the WfMS is

suitable for the supported business process (as

measured by organisational benefits).

While the three dependent variables are general

indicators of the acceptance of a WfMS, information

and system quality are multi-faceted constructs that

include design characteristics of an IS. Including

these more specific measures is useful to provide

feedback to the designers or administrators of the

WfMS.

Following the ISS literature, system quality

refers to the quality of the software and hardware. It

is a broad concept, including several facets such as

the ease of use, reliability, flexibility and

responsiveness of an IS (Delone et al., 2003).

Information quality refers to the contents, timeliness

and availability of the information that is provided

by the WfMS. Based on our previous research and

on interviews that we conducted in the projects; we

contend that ‘information quality’, in the context of

a WfMS, is not sufficient as an evaluation

instrument. Typically, workflow technology is used

by diverse types of employees, ranging from

administration personnel to management and other

kinds of end–users. As we noticed in previous

workflow research, some end-users (mostly within

administrative jobs), only use the WfMS as an

application to register their tasks or to insert data

that will be used by other employees along the

business process. For those kinds of users, data entry

facilities are even more important than getting

information out of the system. Usually, a

combination of both was required. As employees

were assigned a case, they had to look up

ICEIS 2009 - International Conference on Enterprise Information Systems

184

information in the workflow system and complete it

with new data. Therefore, we distinguish between

‘input quality’ and ‘output quality’. As far as we

know, this distinction has not been applied before.

In the past, end-user training and support (the

ongoing efforts to help end-users who are working

with the WfMS) have been defined as being a part of

end-user satisfaction measures (see for instance Doll

et al., 1988). We agree however, in accordance with

the TAM and the ISS model, that training and

support are only a means to increase the productivity

and success of the workflow technology. Both

factors are by no means a goal in itself and should

therefore be considered as external factors that can

influence the perceived system or information

quality.

Figure 1: A Workflow Success Model.

4 RESEARCH METHODS

4.1 Sample and Data Collection

The sample that we tested in this study was collected

using an online survey that we administered in the

course of 2007 and 2008. The total sample consists

of 246 end-users of two workflow projects in

different organizations in Europe. Prior to the

administration of the survey, some in-depth

interviews were held with management or IT

personnel. Both workflow applications have been

used for several years. In the first organization, a

workflow application was developed in the ‘TIBCO

BPM Suite’ (formerly known as Staffware), in order

to enact and monitor a communal invoice and order

process. The business process has more than 450

end-users. 108 end-users filled out the survey. In the

second organization, a workflow application was

developed in the WfMS Flower. The application was

used nation-wide (with more than 1200 end-users) to

support a strictly regulated governmental process

that deals with objection and appeal requests from

citizens. At the moment of this writing we dispose of

138 responses.

4.2 Measures

Several items were used per construct. Where

possible, we based ourselves on existing scales and

adapted them if necessary. All items were measured

on a 6-point likert-scale, ranging from 1 (‘totally

disagree’ or ‘not at all’) to 6 (‘totally agree’ or

‘almost always’).

Satisfaction, Perceived Usefulness and Perceived

Organisational Benefits. Perceived organisational

benefits consist of two reflective items, asking the

respondent whether the workflow application

increases the efficiency of the business process in

general.

Perceived usefulness is a concept that stems

directly from the TAM. It measures the degree to

which the workflow application enhances the

employee’s job performance and entails 4 reflective

items.

Contrary to perceived usefulness, end-user

satisfaction is not an instrumental concept, as it

includes a more general attitude towards the IS. We

used two items asking in a general way whether the

employee was satisfied with the provided solution.

System Quality.

System quality is a multi-faceted

concept consisting of dimensions such as: the

reliability, the flexibility, and the ease of use of the

provided hardware and software solution (Delone et

al., 2003).

Reliability was measured using 3 formative

items. (E.g. Is the WfMS available if required? Does

it crash? Does information get lost?)

Flexibility can be regarded as a general

construct, measuring the way in which end-users can

re-configure or adjust an operational IS if required.

In the case of WfMSs, the routing of cases (through

the business process) and the assignment (or

allocation) of cases to end-users are two core

workflow functionalities (see Joosten et al., 1994). If

these features do not fit the tasks at hand or if they

cannot easily be adapted, the resulting workflow

application might indeed result in a bureaucratic

despot. As a result, we measured workflow

flexibility using two constituent constructs:

Allocation and Routing Flexibility. In particular, we

used 3 formative items for each construct. We asked

respondent for instance, to what extent they could

choose the cases they were going to process; to what

extent the routing procedure (backward and forward

ASSESSING WORKFLOW MANAGEMENT SYSTEMS - A Quantitative Analysis of a Worfklow Evaluation Model

185

routing) was fixed and to what extent they could

access cases of colleagues.

Ease of use was taken from the TAM and

measured with 3 reflective items.

In our evaluation model, system quality was

tested as a global second-order factor, but additional

tests, using the constituent factors have also been

performed.

Information Quality. As explained previously and

contrary to the IS success literature, we splitted

information quality into two disjoints factors: input

and output quality.

Input quality is measured using 6 formative

items; output quality is formed with 9 items. The

items concern issues such as: the provided facilities

to insert and retrieve information, and the degree to

which information can be entered and retrieved in a

complete, readable and timely way.

Training and Support. To measure training, we

asked the participants to evaluate specific courses

and workshops that had been organised to help

future and novice users. Next we listed a number of

support facilities (such as help desk, on-line

documentation, etc.). The resulting evaluations were

then used as formative items to measure support and

training. The measures were based on 2 to 6 items,

depending on the case at hand.

The questions can be received from the authors

upon request.

4.3 Analysis Techniques

Next to descriptive statistics, and Anova, we used

PLSGraph and thus partial least squares analysis

(PLS) to test our explanatory model. PLS is less

restrictive than covariance-based structural equation

modelling like LISREL in terms of sample size and

distributional requirements (Chin, 1989). PLS

combines a structural model (i.e. paths between

constructs) with a measurement model (i.e. the

constructs with their items) and has become very

popular in the past 10 years in multiple IS-related

journals.

Convergent validity was evaluated by examining

the factor loadings. With loadings of more than 0.7,

our reflective items exceed the threshold level of 0.5

(Hair et al. 1992). Discriminant validity is achieved

(i) when the items load much higher on their own

latent variable, and (ii) when the square root of each

construct's Average Variance Extracted is larger

than its correlations with other constructs (Chin,

1998). In our sample both conditions are met. The

relevant loadings can be given by the authors upon

request.

5 FINDINGS AND DISCUSSION

5.1 General Results

The mean scores on all the factors are presented in

table 1. System quality (SQ) is represented by its

constituting factors (SQ1 to SQ5).

The mean scores on organisational benefits,

usefulness and satisfaction vary between 3.87 and

4.20 (on a 6-point scale). This means that in general

the end-users evaluate their workflow solution as

(rather) good or satisfying.

For organisational benefits, satisfaction and

usefulness, 74%, 72% and 62% of all the

respondents gave a clear positive evaluation (>=4).

These results confirm that the workflow solutions

have been accepted in general.

We can also state that training and support was

well provided and organised in a sufficient way,

with 75% and 87% of the respondents giving a score

of at least 4.

A first remark concerns perceived usefulness. Its

score is still positive but significantly less than the

scores on satisfaction and organisational benefits.

Based on the interviews, we argue that the workflow

systems have been improved over the years, leading

to a satisfying and appropriate solution. Perceived

usefulness however, appraises also the impact of the

workflow application on an individual’s job

performance. We therefore contend that end-users

were more critical regarding the impact on their own

productivity, believing that the workflow could still

be improved to support their daily individual needs.

Using Anova, table 1 also points to significant

differences between the two projects, whereby the

invoice & order application is better assessed than

the occupation & appeal requests process.

The differences in the two cases occur not only

in the first three general factors, but also in input

quality and some factors of system quality.

Interestingly, training and support have also received

a different score. In the future, we will use additional

factors (such as individual and task characteristics)

and a more detailed analysis to explain the

differences between the two projects.

5.2 Validation of the Success Model

In figure 2, we present the validation of the

workflow success model. Because of missing values,

ICEIS 2009 - International Conference on Enterprise Information Systems

186

Table 1: General Results.

Construct

Mean

(n=246)

S.D.

Positive Scores

(> = 4)

Mean Case1

(n=108)

Mean Case2

(n=133)

Org. Benefits 4.20 1.21 181 (74%) 4.44 3.96*

Satisfaction 4.14 1.11 177 (72%) 4.24 3.84*

Usefulness 3.87 1.20 152 (62%) 4.51 3.56*

Ease of Use (SQ1) 3.96 1.11 152 (62%) 4.33 3.62*

Responsiveness (SQ2) 3.56 0.69 152 (62%) 3.48 3.62

Reliability (SQ3) 4.24 1.04 160 (65%) 4.44 4.07*

Routing Flexibility (SQ4) 4.25 1.22 156 (63%) 4.27 4.22

Allocation Flexibility (SQ5) 4.75 1.03 203 (83%) 4.98 4.56*

Input Quality 3.95 1.03 136 (55%) 4.06 3.77*

Output Quality 4.16 1.11 162 (66%) 4.30 4.23

Training

4.11 0.98 183 (75%) 4.26 3.97*

Support

4.33 1.08 214 (87%) 4.62 4.22*

*: significant difference (Anova-test)

1: 237 respondents; 9 missing values; 2: 241 respondents; 5 missing values

Figure 2: Model Validation, Using PLSGraph.

the entire model was tested with a sample of 237

end-users. Figure 2 shows that the three general

success measures, organisational benefits,

satisfaction and usefulness, have an R-square

varying between 60 and 80%, indicating that these

factors are well explained by the model. Perceived

usefulness strongly influences satisfaction, and both

variables are strong predictors of organisational

benefits. System quality is the best predictor of

usefulness and satisfaction. However, in figure 2,

only ease of use represents system quality. Indeed,

including the other factors - routing and allocation

flexibility, and reliability – and thus using a second-

order factor for system quality, did not improve the

model.

Based on our knowledge of the two projects, we

contend that by the time of our investigation, the two

systems were mature and did not cause major

technical disruptions..

Routing and allocation flexibility may have

caused disorders when the systems were introduced,

but by now, the routing and allocation features did

fit the daily needs of the end-users.

However, as has been stated by Reijers (et al.

2007), there is no free lunch. To a certain extent,

individual end-users had to perform some tasks in

the interest of the organisation or business process as

a whole. Examples of such tasks include the

registration of data and activities, and following

standard procedures (even if this is not always

efficient for the individual end-user). Therefore it

seems that end-users have different opinions and

experiences regarding the efforts that are required to

work with the system. As a result, ease of use

ASSESSING WORKFLOW MANAGEMENT SYSTEMS - A Quantitative Analysis of a Worfklow Evaluation Model

187

(measuring if the workflow is ‘free of effort’),

remains an important factor that should not be

overlooked by management.

As explanatory factors, we used training and

support. Although training is to some extent still an

important factor, the relative importance of support

is considerably higher. This result is not surprising.

Because the WfMSs have been used for some years,

most end-users are familiar with the system.

Consequently, it is at present less relevant whether

or not they received a good training to learn how to

use the system. Support however refers to facilities

(such as a help desk) that remain useful to support

even experienced end-users in their day-to-day

activities.

In order to account for project-specific factors

that we did not measure, we added a case dummy to

the evaluation model. The dummy indicates that the

occupation & appeal process scores significantly

lower on perceived ease of use.

6 CONCLUSIONS AND FUTURE

RESEARCH

Workflow Management Systems (also known as

Business Process Management systems) are systems

that enable the modelling, enactment, and

monitoring of business processes.

Although workflow systems may have a

considerable impact on the way employees

collaborate and perform their tasks, not much

empirical research has been done on their usage and

acceptance.

Existing empirical studies are scarce and in most

cases limited to qualitative and single case studies

that are difficult to compare.

The lack of systematic empirical studies on the

effects of workflow systems may feed speculation.

Whereas some researchers point to the risks of

workflow systems that might behave like

bureaucratic dictators, others have focused on the

(potential) benefits and success of workflow

technologies. To contribute to this debate we present

in this study the first quantitative results of a

systematic, end-user based evaluation of 2 workflow

applications in different organisations. Based on the

Delone & McLean’s IS Success model, we

developed a workflow evaluation model that is

reusable for the assessment of other workflow

applications or even other enterprise systems.

The study uses a sample of 248 end-users to

analyse the evaluation model. One project concerns

the support of a European communal invoice and

order process (using the TIBCO BPM Suite); the

other project entails the automation of a

governmental objection and appeal request process

(using Flower).

In particular, information and system quality are

defined as multi-dimensional concepts that are

supposed to impact end-users’ satisfaction and the

perceived usefulness of workflow systems. The

ultimate success factor is defined as perceived

organisational benefits. End-user training and

support were entered as external factors. Multiple

items were used per construct and scales from the

literature were adapted or completed if required.

Next to descriptive statistics, we used PLSGraph

and thus applied partial least squares (PLS) analysis

to test our explanatory model. After having

performed the necessary validity checks, we found

that the proposed success model has considerable

explanatory power.

In general the workflow systems are accepted

and positively evaluated. Using a scale from 1 to 6,

general measures such as perceived usefulness,

satisfaction and perceived organisational have a

score ranging between 3.87 and 4.22. A great

majority of the end-users (varying between 62% and

74%) give a positive score (at least 4) on all the

success measures.

In the future, we will add more workflow

projects to increase our sample and to validate our

current findings. Next, we will also focus on

additional external factors such as task

characteristics and individual traits. Finally, a task-

technology fit analysis, testing for interaction effects

between workflow flexibility and task and process

characteristics, is a topic that deserves special

attention.

REFERENCES

Aalst, van der, W.M.P. Reijers, H.A., Weijters, A.J.M.M.,

Dongen, van, B.F., Alves de Medeiros, A.K. , Song,

M. & Verbeek, H.M.W., 2007. Business Process

Mining: An Industrial Application. Information

Systems, Vol. 32 (5), pp. 713-732.

Barclay, D., Higgins, C., Thompson, R., 1995. The Partial

Least Squares Approach to Causal Modeling, Personal

Computer Adoption and Use as an Illustration.

Technology Studies, Vol. 2 (2), pp. 285-309.

Bowers, J., Button, G., Sharrock, W., 1995. Workflow

From Within and Without: Technology and

CooperativeWork on the Print Industry Shopfloor. In

Proceedings of the Fourth European Conference on

Computer-Supported Cooperative Work, pp. 51-66.

ICEIS 2009 - International Conference on Enterprise Information Systems

188

Casati, F., Ceri, S., Pernici, B., Pozzi, G., 1998. Workflow

Evolution, Data & Knowledge Engineering.

Chin, W., 1998. Issues and opinion on structural equation

modelling. MIS Quarterly, Vol. 22 (1),pp. 7-16.

Davis, F. D., 1989. Perceived usefulness, perceived ease

of use, and user acceptance of information technology,

MIS Quarterly, Vol.13 (3), pp. 319-340.

Delone W., Mclean, E.R., 2004. Measuring e-Commerce a

Success: Applying the DeLone & McLean Information

Systems Success Model. International Journal of

Electronic Commerce, Vol. 9(1), pp. 31-47.

Delone W., Mclean, E.R., 2003. The DeLone and

McLean Model of Information Systems Success: A

Ten-Year Update, Journal of Management Information

Systems, Vol. 19 (4), pp. 9–30.

Doll, W.J., Torkzadeh, G., 1988. The Measurement of

End-User Computing Satisfaction. MIS Quarterly,

1988, 12 , pp. 258-273.

Dourish.P., 2001. Process descriptions as organizational

accounting devices: the dual use of workflow

technologies. In Proceedings of the ACM 2001 Int.

Conference on Supporting Group Work. New York,

ACM Press, pp. 52-60.

Hair J.F., Anderson R.E., Tatham R.L., Black, W.C.,

1992. Multivariate Data Analysis with Readings, 3rd

edn. Macmillan, New York.

Jablonski S., Bussler, C., 1996. Workflow Management:

Modeling Concepts, Architecture and Implementation.

International Thomson Publishing.

Joosten, S., Aussems, G., Duitshof, M., Huffmejer, R.,

Mulder, E., 1994. WA-12 ; An Empirical Study about

the Practice of Workflow Management. University of

Twente, Enschede, The Netherlands, 2nd edition, 227

Karahanna E., Ahuja M., Srite M., Galvin, J., 2002.

Individual differences and relative advantage: the case

of GSS, Decision Support Systems, Vol. 32, pp. 327–

341.

Kueng, P., Hagen , C., 2004. Increased performance

through business process management: an experience

report from a Swiss bank, in: N

EELY, A. ET AL. (Eds.),

Performance Measurement and Management - Public

and Private, Cranfield, Cranfield University, pp. 1-8.

Poelmans S., 2002. Making Workflow Systems Work. An

Investigation into the Importance of Task-

appropriation Fit, End-user Support and other

Technological Characteristics. PhD Thesis, Catholic

Univeristy Leuven, 238 p.

Reijers, H.A., Poelmans, S., 2007. Re-configuring

Workflow Management Systems to Facilitate a

“Smooth Flow of Work”, International Journal of

Cooperative Information Systems, 16, 2, pp. 155-175.

Weber, B., Reichert, M., Rinderle-Ma, S., 2008. Change

patterns and change support features -enhancing

flexibility in process-aware information systems. Data

and Knowledge Engineering, pp. 438-466.

Wixom B.H., Todd, P. A., 2005. A Theoretical Integration

of User Satisfaction and Technology Acceptance,

Information systems research, Vol. 16 (1), pp. 85-102.

Wu I-L., Wu, K-W., 2005. A hybrid technology

acceptance approach for exploring e-CRM adoption in

organizations, Behaviour & Information Technology,

Vol. 24 (4), pp. 303-316.

ASSESSING WORKFLOW MANAGEMENT SYSTEMS - A Quantitative Analysis of a Worfklow Evaluation Model

189