Towards Systemic Evaluation of the Business Value of IT
Harri Töhönen, Timo Itälä, Marjo Kauppinen
Aalto University, P.O. Box 9210, Espoo, Finland
Tomi Männistö
University of Helsinki, P.O. Box 68, Helsinki, Finland
Keywords: IT business value, evaluation, system dynamics, systemic approach.
Abstract: Evaluating the IT business value is a challenging combination of managing the complexity of value
phenomenon and the complexity of broad IT impacts. This study analyses the focal characteristics of IT
business value evaluation and proposes a research agenda towards systemic evaluation approach. The
systemic approach combines concepts of goal driven perspective for benefits, value as a combination of
benefits and costs, and the lifecycle view of potential and realised value. These concepts are integrated
through system dynamics modelling to understand the IT impact structures and dynamic value creating
behaviour emerging from the structures. Finally, systemic approach should be supported by evaluation
workflow practices that facilitate seamless data retrieval for the evaluation process, and the integration of
evaluation outputs within the organisation.
1 INTRODUCTION
The evaluation of the IT business value is an
existing challenge and, at the same time, the
applications of information technology are becoming
more ubiquitous and integrated in everyday business
context. Fragmental interpretations of IT business
value do not ease these evaluation efforts. IT
business value can be interpreted as effectiveness or
productivity, or it can refer to cost efficiency or
added value.
In order to ensure the desired benefits from the
investments, IT cannot be evaluated only as a black
box and by relying only on economic measures such
as return on investment or net present value (e.g.
Martinsons et al. 1999). IT impacts should be
studied from the diverse viewpoints of the
organisation stakeholders while considering various
indirect and complementary factors (Lee, 2001). As
an investment, IT differs from the traditional
tangible assets. It is not used in a ‘vacuum’ and, as
an evaluation target, IT can be approached as a
socio-technical phenomenon (Palvia et al. 2001).
The evaluation of the IT business value is relevant
during the various phases of the IT lifecycle, from
the investment calculations to the benefit realisation
management during the usage phase, until the
decisions on upgrade or discontinuance.
Many authors promote for integrative and
holistic approach for evaluating IT business value
(e.g. Melville et al. 2004). However, finding the
balance between a generic, widely applicable means
of evaluation and sufficiently detailed frameworks
for providing effective guidance on specific context
remains a challenge (Stockdale and Standing, 2006).
A considerable body of literature is also devoted to
IT business value on industrial and economic level
(e.g. Brynjolfsson & Hitt, 1998) but applicable
solutions to evaluate individual IT systems are
scarce.
The purpose of this paper is to analyse the IT
business value evaluation by reviewing the
challenges and existing approaches/solutions. We
focus on company level and approaches that are
applicable at an individual IT system level. The
performed literature review is guided by the
following research question “How to characterise
the evaluation of IT business value?”. Building on
the evaluation characteristics we propose a schema
163
TÃ˝uhÃ˝unen H., ItÃd’lÃd’ T., Kauppinen M. and MÃd’nnistÃ˝u T.
Towards Systemic Evaluation of the Business Value of IT.
DOI: 10.5220/0005886601630170
In Proceedings of the Fifth International Symposium on Business Modeling and Software Design (BMSD 2015), pages 163-170
ISBN: 978-989-758-111-3
Copyright
c
 2015 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

of conceptual, methods and workflow basis for
further research towards systemic IT business value
evaluation.
The structure of this paper is as follows: Chapter
2 (Research methods) presents the literature review
methods. Chapter 3 (Evaluation of IT business
value) introduces the applicable concepts of value
within business system, and continues with a
synthesis of the main challenges of IT evaluation.
The chapter is finalised by reviewing existing
evaluation approaches. In Chapter 4 (Towards
systemic evaluation), the main findings on IT
business value evaluation are summarised, and the
basis for systemic evaluation approach is discussed.
Chapter 5 (Conclusions) concludes the contributions
of this study.
2 RESEARCH METHODS
This study used a qualitative literature review to
identify the challenges, principles and existing
solutions to IT evaluation. The main body of IT/IS
value evaluation literature was searched from seven
widely used academic databases (including
ProQuest, ScienceDirect, ACM and IEEE Explore).
The searches were completed in November 2011 and
scoped to journal article titles, with keyword
combinations of ‘IS’, ‘IT’, ‘information system’,
‘information technology’, ‘value’, ‘analysis’,
‘evaluation’, ‘measuring’, ‘measurement’. A total of
912 resulted articles were screened based on their
titles and abstracts, after which 53 papers were
selected for a deeper analysis. This analysis focused
on the literature which elaborated the evaluation
aspects at a company as well as at an individual IT
system level. Finally 36 papers were included in the
concluding analysis.
A parallel literature pool for systems thinking and
system dynamics was studied. The core of this
systemic literature covered the nominal text books
from e.g. Sterman (2000) and Meadows (2008), as
well as articles of Journal of System Dynamics
Review.
The data analysis phase utilised the grounded
theory (Corbin and Strauss, 2008). The coding of the
evaluation literature identified for example
problems, benefits and costs, methods and
frameworks. The coded data was further analysed
and higher level data groups were formed. These
groups are elaborated in evaluation challenges and
approaches sections 3.2 and 3.3.
3 EVALUATION OF IT BUSINESS
VALUE
3.1 Value Perspectives
Our unit of analysis is an individual IT system that is
evaluated as part of a company’s business system.
At a general level, the IT business value is defined
as the contribution of IT to the company
performance (Tallon et al. 2000; Melville et al.
2004). Performance may mean effectiveness in
meeting the business system purpose and goals with
the economic worth as the ultimate judgment of
success for the profit making companies. The
economic worth is quantified by measures such as
return on investment (ROI), internal rate of return
(IRR) or paypack time (Martinsons et al. 1999).
However, using only the traditional economic
measures for valuing IT is easily insufficient due to
the broad scope of IT impacts and the attributability
challenges when linking the impacts with benefits
and costs.
In this paper we define value as an outcome of
comparison between benefits and costs. For this,
both benefits and costs have to quantified, but not
necessarily in monetary terms. The relevant units for
the quantification depend on the commensurability
needs of the chosen performance evaluation level.
Obviously, monetary units are widely
commensurable while the index - benefits per costs
relation - can be useful for company internal
purposes.
The IT impacts aggregate and disperse through
various business processes (e.g. Mirani and Lederer,
1998; Melville et al. 2004). In order to understand
the multidimensional impact chains of IT, we should
be able to link value creating factors to each other at
multiple levels:
• Individual: benefits and costs as realised by
the employees utilising IT in their daily
tasks.
• Organisational: benefits and costs as
realised at process level, e.g. process
efficiency as input/output ratio.
• Business: benefits and costs as realised at
business outcome level, e.g. productivity,
sales or profitability, economic worth.
The above mentioned three levels serve as an
example of means-end structure where the lower
level goals in the hierarchy act as the means to
achieve the higher-level goals as ends. This means-
end chain theory is widely applied in customer value
research to understand the structures and factors
Fifth International Symposium on Business Modeling and Software Design
164

affecting the value formation (e.g. Gutman, 1982).
Similar structuring is also exercised within IS
research, for example Benefits Dependency Network
to diagnose IT investment business cases (Peppard et
al. 2007).
In order to better understand the temporal
challenges of valuing IT impacts, next we
investigate ‘locus of value’. In customer value
research, locus of value is used for separating the
benefit realisation as a phenomenon from the locus
of explicitly measuring the reflection of phenomenal
value. Ng and Smith (2012) discuss phenomenal
consciousness (P-C value) vs. access consciousness
(A-C value). P-C value is “creation of value in
context that is phenomenal, the raw experience of
creating value (goodness) in interactions around the
experience” while A-C value exists “in the
perception, introspection and memory (or
imagination) of P-C value before (ex ante) and after
(ex post)”.
From the value evaluation point of view, locus of
value relates to: 1) the delay between P-C and A-C
value, i.e. the delay between benefit realisation in
the context and the benefit and/or value
measurement or quantification, 2) how well we are
able to link the root P-C value to the A-C value that
is evaluated at different levels of the business
system. In IS research, locus of value is discussed
together with the levels of analysis. Within the IT
impact chains, locus of value is considered together
with the question of how well the measures distant
(e.g. economic measures) from the value creation
event can actually address the first order impacts at
individual employee or business process levels
(Barua et al. 1995; Davern and Wilkin, 2010). The
distance between the first order impact and the
measurement point can be both cause-and-effect
structural distance or it can be a time distance as a
delay between the event and its measurement.
Davern and Kauffman (2000) discuss locus of
value within the scope of the IT lifecycle. Locus of
potential value defines the baseline for the expected
value before the IT investment while locus of
realised value is relevant after the investment. Locus
of potential or realised value is not a single spot in
time and place but it occurs at multiple levels of
analysis, being a summation of multiple loci of value
from different levels of analysis, including for
example individual, work group and process levels.
3.2 Challenges of IT Evaluation
In general, the evaluation of IT impacts is described
as ‘complex’ and ‘multidimensional’ (e.g. Lee,
2001). In the next paragraphs we elaborate the main
challenges and rationale behind these broad
descriptions (see Table 1 for the summary).
Focus and Volume of IT impacts
We start with two background factors: the focus
of IT and the nature of business system. Our
evaluation is scoped to a single company with a
specific IT system as an element of a socio-technical
business system. The business system includes other
elements such as organisational structures, tasks and
process hierarchies, goal hierarchies and different
interpretations of value. IT impacts traverse through
the business system, either broadly with wide effects
or with more focused and narrow contributions. The
broadness depends on the interrelation of the IT
usage and focus with company goals and functions:
the closer the focus of IT with strategic and
transformative goals, the broader the IT impacts are
when more employees, their tasks and business
processes are supported by IT. The focus of IT
together with the nature of business system reinforce
the volume of IT impacts. The volume reflects the
high number of IT’s direct and indirect touch points
with its surrounding business system.
Complementarity
Complementary factors are non-IT issues that
affect how well the desired benefits and costs are
realised (e.g. Dedrick et al. 2003). The examples of
complementary factors include management
practices, user skills and process maturity. Due to
the complementary factors, the same IT in different
organisational contexts produces different outcomes
(Davern and Kauffman, 2000). We argue that
complementarity is largely a practical embodiment
of a business system being a socio-technical
phenomenon. Further, the broader the focus of IT
within the business system the more significant is
the role of the complementary factors.
Traceability for causes and effects
The volume of impacts together with
complementarity complicate the traceability of IT
impacts within the business system. The indirectness
of relations between IT’s first order impacts to
business performance grow when the hierarchies and
the length of cause-and-effect structures grow
(Melville et al. 2004). The indirectness is related to
the attributability and accountability issues (e.g.
Marthandan and Tang, 2010) when trying to isolate
IT’s contributions for the higher level business
measures.
Towards Systemic Evaluation of the Business Value of IT
165

Table 1: Challenges in IT business value evaluation.
Challenges Concepts and keywords References
Focus of IT
- Strategic, transformational, informational or transactional
- Focus types e.g. operations or market focus
- Savings vs. added value
Mirani & Lederer (1998), Giaglis et al.
(1999), Dedrick et al. (2003), Gregor et al.
(2006), Tallon et al. (2007)
Nature of business
system
- Socio-technical system
- Organisational structures and layers
- Tasks & Processes, Business processes
- Multilevel perspectives
Hamilton & Chervany (1981), Barua et al.
(1995), Wegen & Hoog (1996), Palvia et al.
(2001), Marthandan & Tang (2010)
Volume of IT
impacts
- Broad impacts
- Multiple benefits & costs
Simmons (1996), Mirani & Lederer (1998),
Kanungo et al. (1999), Irani et al. (2006)
Complementarity
- Contextual interaction
- Conversion contingencies, complementary assets
- Complementary organisational resources & capital
Davern & Kauffman (2000), Lee (2001),
Dedrick et al. (2003), Melville et al. (2004)
Traceability for
causes and effects
- Indirectness
- Attributability, accountability
- Locus of value vs. locus of analysis
Giaglis et al. (1999), Delone & McLean
(2003), Melville et al. (2004), Petter et al.
(2008), Davern & Wilkin (2010),
Marthandan & Tang (2010)
Time & dynamics
- Payback delays
- Evolving effects, dynamic objectives
- Locus of impact vs. measuring delays
- Potential vs. realised benefits
Hamilton & Chervany (1981), Giaglis et al.
(1999), Chan (2000), Peppard et al. (2007),
Davern & Wilkin (2010)
Observability &
measurability
- Intangibility, soft benefits
- Non-monetary, non-quantifiable
- Asset type, IT capital
- Hidden benefits & costs
- Perceived vs. independently observable
Giaglis et al. (1999), Ryan & Harrison
(2000), Irani et al. (2006), Gunasekaran et al.
(2006), Bajaj et al. (2008), Davern & Wilkin
(2010)
Accountability for
business impacts
- Economic, financial or accounting measures
- Black box
Simmons (1996), Martinsons et al. (1999),
Bajaj et al. (2008), Davern & Wilkin (2010),
Marthandan & Tang (2010)
Maturity of
methods &
theories
- Generic applicability vs. effective guidance
- Need for integrative or holistic approach
- Insufficient theoretical frameworks
Giaglis et al. (1999), Gunasekaran et al.
(2006), Stockdale & Standing (2006)
Maturity of
practices
- Benefits overstated
- Ambiguous goals & measures
- Focus on easy measures
- Unavailability of data for ex ante – ex post comparison
Hamilton & Chervany (1981), Ragowsky et
al. (1996), Wegen & Hoog (1996), Peppard
et al. (2007)
Traceability for causes and effects
The volume of impacts together with
complementarity complicate the traceability of IT
impacts within the business system. The indirectness
of relations between IT’s first order impacts to
business performance grow when the hierarchies and
the length of cause-and-effect structures grow
(Melville et al. 2004). The indirectness is related to
the attributability and accountability issues (e.g.
Marthandan and Tang, 2010) when trying to isolate
IT’s contributions for the higher level business
measures.
Time & dynamics
In many cases, IT benefit realisation is delayed
from the cost realisation (Peppard et al. 2007).
Locus of value is dispersed into multiple levels of
the organisation and there are delays between the
value realisation and the evaluation of realised
value. Additionally, IT impacts are not static: IT
itself is upgraded and improved while the
complementary factors and the context around the IT
evolve (Chan, 2000). The goals for IT also evolve
(Hamilton and Chervany, 1981). Time delays
together with dynamic changes bring dynamic
complexity into the business system.
Observability & measurability
The above mentioned socio-technical system
characteristics, complementarity, and delay issues
bring concrete challenges to the quantification and
measuring of IT impacts.
Fifth International Symposium on Business Modeling and Software Design
166

Part of the benefits and costs are easily omitted
from the explicit evaluation because those are
structurally or temporally too far from the first order
IT impacts. Additionally, some of the IT impacts are
so intangible or ‘soft’ that they are not easily
quantified into a measurable form.
Accountability for business impacts
The previously mentioned evaluation challenges
explain why single economic measures are easily too
narrow for covering the value of IT. Attributability
and measurability issues affect the reliability of the
financial measures for giving holistic credit for IT’s
contributions.
Maturity of theories, methods & practices
Many authors recognise that the underlying
theoretical basis for IT value evaluation is scattered.
Holistic and integrative evaluation approach is
requested in order to tackle the complexity and
multidimensional issues (e.g. Giaglis et al. 1999;
Gunasekaran et al. 2006). However, one of the
challenges is to find the proper balance of wide
applicability and practical usefulness with specific
situations (Stockdale and Standing, 2006). This
balancing challenge motivates our study by setting
scalability requirements for the investigated
systemic approach.
Many of the evaluation challenges are rooted in
maturity issues of organisational practices and
evaluation culture. Examples include practices for
collecting evaluation baseline data, defining explicit
goals for IT or managing the evolution of measures
and evaluation frameworks (e.g. Ragowsky et al.
1996; Wegen and Hoog, 1996).
3.3 Approaches for IT Evaluation
In the next paragraphs we present an overview of the
categorised evaluation approaches, starting from
general principles and advancing towards practical
solutions.
Principles for measuring & evaluation
Due to the multidimensionality of IT impacts,
benefits and costs, many studies advice for using
multiple units of analysis. Measurements should
integrate the results from several organisational
levels, and they should utilise both qualitative and
quantitative measures, or perceived and
independently observable measures (e.g. Davern and
Wilkin, 2010). Evaluation should be seen as an
incremental and evolving practice (Giaglis et al.
1999; Chan, 2000), and it should be executed both
before and after the investment decisions (Davern
and Kauffman, 2000).
Benefits & costs classifications
The studies in this category identify and group
common benefit and cost factors of IT. Simmons
(1996) classifies benefits into five types: increased
efficiency, increased effectiveness, added value,
marketable product and development of corporate IT
infrastructure. Gregor et al. (2006) classifies benefit
types into transactional, informational, strategic and
transformational benefits. Irani et al. (2006)
introduce extensive cost taxonomy while Ryan and
Harrison (2000) focus on social subsystem benefits
and costs. These studies can be used as a reference
or checklists when identifying relevant elements for
business system modelling and further evaluation.
Constructs for success or effectiveness
Success or effectiveness constructs propose a
structure for the factors impacting or leading
towards desired goals. Information System Success
Model by DeLone and McLean (2003) is a widely
studied cause-and-effect structure that links IS
quality, usage and satisfied users with organisational
net benefits. Technology Acceptance Model (TAM,
TAM2) elaborate IT impacts and usage at the
individual user’s level of analysis (Davis, 1989;
Venkatesh and Davis, 2000). Other examples of IT
effectiveness or impact constructs are provided by
Grover et al. (1996), Kanungo et al. (1999) and
Gable et al. (2008).
Instead of providing specific checklists for IT
benefits or costs, the constructs in this category aim
to understand the overall role and connections of IT
within the socio-technical business system. As
generic reference models, they give guidance for
modelling systemic structures and
interdependencies. Studies in this category can also
identify complementary factors to be included in
system models (e.g. Larsen, 2003).
Constructs for evaluation process
The studies in this category view the evaluation
process or the framework as a unit of analysis.
Hamilton & Chervany (1991) recognises two types
of evaluation perspectives: 1) goal-driven view
focuses on whether actions produced proper
outcomes and the emphasis is on the results, and 2)
system-resource view focuses on whether things
were executed properly and the emphasis is on the
process and the means. From the systemic
evaluation point of view, both the above mentioned
perspectives should be used when applying means-
end thinking to identify the cause-and-effect
structures.
Stockdale & Standing (2006) introduce Context,
Content, Process (CCP) evaluation framework that
takes a holistic view by asking what is evaluated
Towards Systemic Evaluation of the Business Value of IT
167

(Content), why evaluation is conducted and who
affects the evaluation (Context) and when and how
the evaluation is to be performed (Process).
Benefits Dependency Network (BDN) by
Peppard et al. (2007) links the organisational change
and business goals by using the means-ways-ends
approach. Means cover the IT enablers and enabling
changes which facilitate the ways level for
improving, chancing or giving up something. Ways-
level – the changes – target for business level
benefits in order to satisfy the IT investment goals –
the ends level. BDN is an example of a goal-driven
approach that helps in understanding the business
system through cause-and-effect structures. BDN is
presented as a one-way hierarchy from means
towards higher level ends, thus omitting explicit
feedback mechanisms from the higher level issues
back to the lower levels.
Specific evaluation frameworks/methods
Balanced Score Card based approaches are
proposed for integrative and holistic performance
and evaluation tools for IT/IS (e.g. Martinsons et al.
1999; Bajaj et al. 2008). BSC frameworks provide a
familiar measuring concept for business managers
but by default their hierarchical format do not
support feedback structures from the higher level
elements back to the lower level elements.
Tiernan and Peppard (2004) emphasize a
lifecycle view to the IT benefits management - from
vision to value realisation - and introduce a
mathematical formulation for the vision-to-value
vector.
System dynamics (SD) is used by several authors
to evaluate IT/IS, for example Santos et al. (2008)
combine SD with Multicriteria Decision Analysis
(MCDA) within continuous performance
management process, and Mutschler and Reichert
(2008) introduce SD modelling based EcoPOST cost
analysis framework for process-aware information
systems. Pfahl & Lebsanft (1999) introduce a SD
based integrated measurement, modelling and
simulation (IMMoS) approach in a software
development domain. One of the learnings from
IMMoS trial project is the importance of a goal-
driven top-down approach for scoping and
maintaining the focus for system modelling and
measuring efforts.
4 TOWARDS SYSTEMIC
EVALUATION
The answer for our research question “How to
conceptualise the evaluation of IT business value?”
covered evaluation challenges and solutions from
the IT/IS evaluation literature. The IT business value
evaluation appeared to be a combination of
complexity regarding the multidimensional value
concept itself and the evaluation challenges with the
multilevel IT impacts in the business environment.
Several sources suggest an integrative and holistic
evaluation approach that would cover multiple units
of analysis, would combine tangible and intangible
factors, recognise complementary factors, would be
goal oriented and span the lifecycle of IT business
case.
The above mentioned characteristics set the
ground for a systemic evaluation approach. We
propose a scheme of three tightly coupled building
blocks for structuring further research on systemic
evaluation: conceptual basis, methods basis and
workflow basis.
4.1 Conceptual Basis
The conceptual basis covers the focal concepts of IT
business value evaluation within a business system.
At first, the concepts of goal, benefit, cost (or
sacrifice), IT impact and value has to be
semantically linked together. A business model and
an earning logic are practical concepts that can be
used to set the goals and valuing perspectives for IT
impacts. A (business) process and a service are
examples of concepts used to understand the
execution logic and interconnections of a business
system. In order to support the lifecycle view of IT,
a potential value and a realised value should be
linked with expected and realised benefits and costs.
The further research of the conceptual basis
could produce a metamodel for guiding the
population of case specific system models. While
populating generic metamodels and identifying case
specific system elements and their relationships,
existing IT/IS literature provides rich examples as
summarised in ’Benefits & Costs classification’ and
’Constructs for success or effectiveness’ sections.
4.2 Methods Basis
The methods basis gathers means for visualizing and
modeling the linkage of IT impacts with benefits,
costs and even with commensurable value units. Our
Fifth International Symposium on Business Modeling and Software Design
168

further research relies on systems thinking and
system dynamics. Systems thinking provides
principles for defining system boundaries,
understanding emergent properties and synergism of
the business system elements. System dynamics
(SD) is a set of methods for modelling the system
structures and the dynamic behavior of the system
over time (e.g. Sterman, 2000). SD is scalable from
the qualitative analysis with causal loop diagrams to
quantitative analysis with stock-and-flow diagrams
and mathematical equations.
Qualitative SD provides potential means for the
traceability and feedback analysis of IT impacts by
applying cause-and-effect linking with balancing
and reinforcing feedback loops. Qualitative
modelling reveals the mechanisms behind the
system delays and non-linear behaviour.
Quantitative analysis gives further insights into the
system beviour over time. Simulations can be used
to test various system configurations, to find
leverage points in the system structure, or to perform
sensitivity analysis for the system variables
(Sterman, 2000).
System dynamics methods are used as a ‘glue’
for integrating and modelling the conceptual basis
elements and their interdependencies within the
business system. The actual challenges of for
example defining quantitative measures for
intangible benefits and costs still remain. However,
the recognition of the elements and relations
affecting the value creation is the first step in
ensuring that those factors are not left on their own
but actively monitored and managed during the
lifecycle of IT.
4.3 Workflow Basis
The workflow basis focuses on practical means of
applying systemic evaluation methods and concepts
in a real organisational context. The workflow
practices should facilitate a seamless integration of
the evaluation process and the business system
organisation. How to obtain the required data from
the stakeholders, how to scope and iterate the
modelling, how to extract measures from the models
are all example questions for further empirical
studies.
5 CONCLUSIONS
This paper highlights the focal characteristics of the
IT business value evaluation and proposes systems
thinking and system dynamics as the core of a
systemic evaluation approach. The systemic
approach facilitates integrative perspective into the
IT role within the business system: IT investments
and the usage are seen as a continuous business case.
The further research on systemic evaluation
approach is structured into conceptual, methods and
workflow views. These views are currently utilised
as the authors continue data collection and analysis
of the lessons learned from the six industrial cases
experimenting with systemic evaluation approach.
ACKNOWLEDGEMENTS
The authors thank Digile’s Need4Speed programme
for funding, Qentinel Ltd. & Eero Talonen for
business motivation, and Kari Hiekkanen & Mika
Helenius for background discussions.
REFERENCES
Bajaj, A., Bradley, W., Cravens, K. 2008. SAAS:
Integrating Systems Analysis with Accounting and
Strategy for Ex Ante Evaluation of IS Investments.
Journal of Information Systems, 22(1), 97-124.
Barua, A., Kriebel, C. H., Mukhopadhyay, T. 1995.
Information technologies and business value: An
analytic and empirical investigation. Information
Systems Research, 6(1), 3-23.
Brynjolfsson, E., Hitt, L. M. 1998. Beyond the
productivity paradox. Communications of the ACM,
41(8), 49-55.
Chan, Y. E. 2000. IT Value: The Great Divide Between
Qualitative and Quantitative and Individual and
Organizational Measures. J. Manage. Inf. Syst., 16(4),
225-261.
Corbin, J., Strauss, A. 2008. Basics of qualitative
research: Techniques and procedures for developing
grounded theory. (3rd ed.). Thousand Oaks: SAGE
Publications.
Davern, M., Wilkin, C. 2010. Towards an integrated view
of IT value measurement. International Journal of
Accounting Information Systems, 11(1), 42-60.
Davern, M., Kauffman, R. 2000. Discovering Potential
and Realizing Value from Information Technology
Investments. Journal of Management Information
Systems, 16(4), 121-143.
Davis, F. D. 1989. Perceived usefulness, perceived ease of
use, and user acceptance of information technology.
MIS Quarterly, 13(3), 319-340.
Dedrick, J., Gurbaxani, V., Kraemer, K. 2003. Information
Technology and Economic Performance. ACM
Computing Surveys, 35(1), 1-28.
DeLone, W. H., McLean, E. R. 2003. The DeLone and
McLean Model of Information Systems Success: A
Towards Systemic Evaluation of the Business Value of IT
169

Ten-Year Update. Journal of Management
Information Systems, 19(4), 9-30.
Gable, G., Sedera, D., Chan, T. 2008. Re-conceptualizing
information system success: the IS-Impact
Measurement Model. Journal of the Association for
Information Systems, 9(7), 377-408.
Giaglis, G., Mylonopoulos, N., Doukidis, G. 1999. The
ISSUE methodology for quantifying benefits from
information systems. Logistics Information
Management, 12(1/2), 50-62.
Gregor, S., Martin, M., Fernandez, W., Stern, S., Vitale,
M. 2006. The transformational dimension in the
realization of business value from information
technology. The Journal of Strategic Information
Systems, 15(3), 249-270.
Grover, V., Jeong, S. R., Segars, A. H. 1996. Information
systems effectiveness: The construct space and
patterns of application. Information & Management,
31(4), 177-191.
Gunasekaran, A., Ngai, E. W., McGaughey, R. E. 2006.
Information technology and systems justification: A
review for research and applications. European
Journal of Operational Research, 173(3), 957-983.
Gutman, & Jonathan 1982. A Means-End Chain Model
Based on Consumer Categorization Processes. Journal
of Marketing, 46(2), 60-72.
Hamilton, S., Chervany, N. L. 1981. Evaluating
information system effectiveness-Part I: Comparing
evaluation approaches. MIS Quarterly, 5(3), 55-69.
Irani, Z., Ghoneim, A., Love, P. E. 2006. Evaluating cost
taxonomies for information systems management.
European Journal of Operational Research, 173(3),
1103-1122.
Kanungo, S., Duda, S., Srinivas, Y. 1999. A structured
model for evaluating information systems
effectiveness. Systems Research and Behavioral
Science, 16(6), 495-518.
Larsen, K. R. 2003. A taxonomy of antecedents of
information systems success: variable analysis studies.
Journal of Management Information Systems, 20(2),
169-246.
Lee, S. 2001. Modeling the business value of information
technology. Information & Management, 39(3), 191-
210.
Marthandan, G., Tang, C. M. 2010. Information
technology evaluation: issues and challenges. Journal
of Systems and Information Technology, 12(1), 37-55.
Martinsons, M., Davison, R., Tse, D. 1999. The balanced
scorecard: a foundation for the strategic management
of information systems. Decision Support Systems,
25(1), 71-88.
Meadows, D. H. 2008. Thinking in Systems. London:
Earthscan.
Melville, N., Kraemer, K., Gurbaxani, V. 2004.
Information Technology and Organizational
Performance: An Integrative Model of It Business
Value. MIS Quarterly, 28(2), 283-322.
Mirani, R., Lederer, A. L. 1998. An Instrument for
Assessing the Organizational Benefits of IS Projects.
Decision Sciences
, 29(4), 803-838.
Mutschler, B., Reichert, M. 2008. On modeling and
analyzing cost factors in information systems
engineering. In Proceedings of the Advanced
Information Systems Engineering (pp. 510-524).
Springer Berlin Heidelberg.
Ng, I. C., Smith, L. A. 2012. An integrative framework of
value. Review of Marketing Research, 9, 207-243.
Palvia, S. C., Sharma, R. S., Conrath, D. W. 2001. A
socio-technical framework for quality assessment of
computer information systems. Industrial
Management & Data Systems, 101(5), 237-251.
Peppard, J., Ward, J., Daniel, E. 2007. Managing the
realization of business benefits from IT investments.
MIS Quarterly Executive, 6(1), 1-11.
Petter, S., DeLone, W., McLean, E. 2008. Measuring
information systems success: models, dimensions,
measures, and interrelationships. European journal of
information systems, 17(3), 236-263.
Pfahl, D., Lebsanft, K. 1999. Integration of system
dynamics modelling with descriptive process
modelling and goal-oriented measurement. Journal of
Systems and Software, 46(2), 135-150.
Ragowsky, A., Ahituv, N., Neumann, S. 1996. Identifying
the value and importance of an information system
application. Information & Management, 31(2), 89-
102.
Ryan, S. D., Harrison, D. A. 2000. Considering social
subsystem costs and benefits in information
technology investment decisions: A view from the
field on anticipated payoffs. Journal of Management
Information Systems, 16(4), 11-40.
Santos, S., Belton, V., Howick, S. 2008. Enhanced
performance measurement using OR: a case study.
Journal of the Operational Research Society, 59(6),
762-775.
Simmons, P. 1996. Quality outcomes: determining
business value. IEEE Software, 13(1), 25-32.
Sterman, J. D. 2000. Business Dynamics: Systems
Thinking and Modeling for a Complex World. Boston
McGraw-Hill.
Stockdale, R., Standing, C. 2006. An interpretive approach
to evaluating information systems: A content, context,
process framework. European Journal of Operational
Research, 173(3), 1090-1102.
Tallon, P., Kraemer, K., Gurbaxani, V. 2000. Executives'
Perceptions of the Business Value of Information
technology. Journal of Management Information
Systems, 16(4), 145-173.
Tiernan, C., Peppard, J. 2004. Information Technology: Of
Value or a Vulture?. European Management Journal,
22(6), 609-623.
Wegen, B. V., Hoog, R. D. 1996. Measuring the economic
value of information systems. Journal of Information
Technology, 11(3), 247-260.
Venkatesh, V., Davis, F. D. 2000. A Theoretical Extension
of the Technology Acceptance Model: Four
Longitudinal Field Studies. Management Science
,
46(2), 186-204.
Fifth International Symposium on Business Modeling and Software Design
170