A FRAMEWORK FOR ERP EVALUATION AND SELECTION
USING MACBETH TECHNIQUE
Abdelilah Khaled and Mohammed Abdou Janati Idrissi
ENSIAS University, Avenue Mohammed Ben Abdallah Regragui, Madinat Al Irfane, BP 713 Agdal, Rabat, Morocco
Keywords: ERP, Criteria, Evaluation, MACBETH, MCDM, Metrics.
Abstract: Purchasing an inappropriate Enterprise Resource Planning (ERP) system may prove to be a major reason for
its implementation failure. Recently, given the cost of the investment required to acquire, implement and
operate an ERP system, the interest expressed by academics and practitioners concerning the selection of
measures and evaluation techniques of enterprise systems is highly justifiable. Accordingly, system
selection process is an important step in ERP adoption. This paper intends to elaborate a comprehensive
framework for ERP selection and evaluation. It serves a threefold objective. First, it proposes a structured
methodology in which strategic, functional, technical and managerial goals are considered in the selection
decision. Second, it suggests a classification of the main criteria mentioned in the literature under four
categories. Third, it presents an application of the Measuring Attractiveness by a Categorical Based
Evaluation Technique (MACBETH) that allows the quantification of criteria’s weights, the construction of
utility functions related to each criterion and the evaluation of the candidate ERP solutions.
1 INTRODUCTION
Enterprise Resource Planning (ERP) systems have
been considered from the very outset as a
sophisticated, powerful and advanced type of
software that aim at making an organisation more
integrated, competitive and ideally computerized.
They are comprised of a set of functional modules
that coordinate the whole of the organisation’s
activities. Indeed, in comparison with other simple
software that separately bring solutions to a
relatively well defined requirements, an ERP system
manages in an optimal way a large spectrum of
business domains in order to automate and
collaborate together the various organisational
processes.
In this respect, ERP systems are mainly destined
to achieve operational and strategic goals that
provide an organisation with a competitive edge in
an ever-changing business environment, enhance its
business practices and improve its decision making
processes.
During the last decade, it has been widely
reported that an extensive part of ERP projects fail
to deliver the initial value expected from them. The
success rate is at alarmingly low level and they tend
to be a very heavy burden on the organisations’
budget. Therefore, it is quick to point out that the
complexity of ERP systems and the vagueness of the
purchasing objectives may prove to be a barrier to
their successful implementation. To tackle this issue,
Wu, Tai, Tsai and Lu (2009) suggest that a great
deal of importance must be attached to the selection
stage of the ERP acquisition process. In this regard,
purchasing alternatives should be subject to
relentless questioning.
This paper intends to elaborate a comprehensive
framework for ERP selection. Moreover, it provides
specific advice to organisations when considering
selection criteria. It serves three major objectives.
First, it proposes a structured methodology in which
strategic, functional, technical and managerial goals
are considered in the selection decision. Second, it
suggests a classification of the main criteria
mentioned in the literature under four categories.
Third, it presents an application of the Measuring
Attractiveness by a Categorical Based Evaluation
Technique (MACBETH) that allows the
quantification of criteria’s weights, the construction
of scales related to each criterion and the evaluation
of the different ERP options.
339
Khaled A. and Janati Idrissi M..
A FRAMEWORK FOR ERP EVALUATION AND SELECTION USING MACBETH TECHNIQUE.
DOI: 10.5220/0003422303390342
In Proceedings of the 13th International Conference on Enterprise Information Systems (ICEIS-2011), pages 339-342
ISBN: 978-989-8425-53-9
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

2 ERP SELECTION PROCESS
2.1 Selection Methodology
Selecting an optimal ERP solution is a critical factor
for its successful implementation. In fact, an in-
depth understanding of the underlying business
benefits resulting from ERP adoption should be
highly taken into account during the selection
decision. In this regard, four important goals are
deemed wise to be emphasized when defining the
organisation’s requirements: Strategic, functional,
technical and operational. In this section, we propose
a flowchart for selecting an ERP system (Figure 1).
Figure 1: ERP selection’s flowchart.
2.2 Selection Criteria
In the literature, many authors suggest different sets
of selection criteria related to ERP package itself
(Birdogan and Kemal, 2005) whereas others focus
on criteria dealing with supplier’s support and
technical assistance (Eric, Wang, James and Gary,
2008; Remus, 2007). The list of criteria adopted
differs mainly according to the organisation related
business model and size. This paper suggests a
classification of the main criteria found in the
literature into four subcategories. The following four
subcategories are:
Strategic Criteria: They assess the degree to which
the ERP system supports or constrains the ability to
execute the organisation’s business strategy. To this
end, the ERP system should adapt to the new
business and technology changes that would happen
in the future. It is worth pointing out that the
strategic criteria should reflect the strategic
objectives that each organisation wants to attain. As
examples of strategic criteria, we mention future
anticipation and benefits realization.
Functional Criteria: They assess the functional
coverage of the organisation’s requirements.
Teltumbde (2000) argues that the generic
functionalities implemented in the marketplace
available ERP products couldn’t yet meet all the
industry-specific requirements. However, with the
advances in technology and the maturity reached by
ERP solutions, the basic functionalities are well
covered and slight differences could be made among
them. Hence, the decision maker should focus more
on features that really matter such as coding and
nomenclature capabilities, international operation’s
support and transaction’s history management.
Technical Criteria: They assess the technical
features of an ERP solution assuming that it is
basically an IT system. Examples of these criteria
are: Usability, reliability, interoperability, security,
maintainability and efficiency. Refer to the work of
Liang and Lien (2007) for more details about these
criteria. The authors used the ISO 9126 norm to
define a set of quality criteria.
Managerial Criteria: They evaluate the ERP
system implementation’s methodology; they include
the vendor’s reputation and market share, the
integrator’s support and the adopted project
management methodology (Quality management,
implementation time and total cost of ownership).
2.3 Evaluation Method
Numerous methods have been applied to ERP
selection. They include scoring, ranking,
mathematical optimisation, and multi-criteria
decision making (MCDM) (Wei, Chien and Wang,
2005). The ERP system selection is influenced by
multiple factors such as decision-maker’s
preferences, candidate solutions, and the availability
of limited resources. Hence ERP selection could be
considered as a kind of MCDM problem (Wu,
2008). This paper uses MACBETH (Measuring
Attractiveness by a Categorical Based Evaluation
Technique) to direct how to determine an evaluation
of different ERP solutions. MACBETH is
introduced by (Bana e Costa and Vansnick , 1994)
and allows the quantification of criteria’s weights
ICEIS 2011 - 13th International Conference on Enterprise Information Systems
340

together with the construction of utility functions
related to the criteria. An aggregation function is
then elaborated to evaluate each ERP product. The
advantage of choosing MACBETH among other
techniques is its ability to totally cover the decision
process from the expression of the decision maker’s
preferences to the making of final decision. In
addition, MACBETH allows the systematic
construction of utility functions related to each
criterion which is not widely supported by similar
methods.
The remainder of this section deals with an
explicit formalization of an aggregation function
using MACBETH in the context of independent
criteria.
2.3.1 Definition of Utility Functions
Let C1, C2, C3, and C4 be the set respectively
representing the different strategic, functional,
technical and managerial criteria introduced in the
previous section. Let (A
i
, S) be an ordered set of
values representing the performances of the different
ERP products on each criterion C
i
with i =1…4. The
binary relation “S” defines an order over A
i
. For
each ERP
w
and ERP
z
that have respectively “x”
and “y” as performances on A
i
: x S y means that the
decision maker considers that ERP
w
is at least more
attractive than ERP
z
regarding C
i
.
The purpose of the methodology is to define four
marginal utility functions U
i
as homomorphism
functions from (A
i,
S) to (|R,
≥ ). Each utility
function U
i
is interpreted as a numeric scale
translating the linguistic preferences of the decision
maker on C
i
criterion to real numbers. According to
Grabisch (2005), the aggregation model has to
guarantee the commensurability of utility functions
to be coherent. MACBETH ensures this prerequisite
by incorporating into the model two fictitious values
0
i
and 1
i
related to each criterion. 0
i
is defined as a
neutral value with respect to the criterion C
i
. It is
considered as ‘neither satisfying nor unsatisfying’
particular reference for C
i
. 1
i
is defined as a good
value, which is more attractive to a decision maker
than neutral, and is defined as undoubtedly
satisfying particular reference.
Based on these two references 0
i
and 1
i
, we
distinguish an unattractive ERP on C
i
if it is less
attractive than 0
i
, an attractive ERP on C
i
if it is
more attractive than 0
i
and an outstanding ERP on
C
i
if it is at least as attractive as 1
i
.
Similarly, let
ERP neutral
(0
1
, 0
2
, 0
3
, 0
4
) and
ERP good (1
1
, 1
2
, 1
3
, 1
4
) denote two fictitious ERP
systems that have respectively 0
i
and 1
i
for the four
criteria.
MACBETH is based on a questioning procedure
that allows the construction of utility functions
through a discussion with the decision maker. Let
B = A
∪ {ERP
neutral
, ERP
good
}
A is the initial set of ERP options
(1)
For each x, y
∈
B and for each criterion C
i
the
decision maker is asked to verbally judge the
difference of attractiveness between x and y
regarding C
i
. When judging, a decision maker has to
choose one of the following categories: A0: no
difference; A1: very weak; A2: weak; A3: moderate;
A4: strong; A5: very strong; A6: extreme. However,
if the decision maker is unsure about the difference
of attractiveness between two ERP options, they
may choose the union of several successive
categories among these above.
Utility functions are then obtained by the means
of MACBETH matrix of judgment that are called
the MACBETH basic scales. The consistency of this
matrix is verified during the expression of the
decision maker’s preferences. The M-MACBETH
(http://www.m-macbeth.com/) software implements
this procedure. The numerical scales are extracted
thanks to linear programming. A linear
transformation U
i
for each criterion C
i
is used to
refocus the scales on the two values 0 and 1 as
follows:
U
i
→
α
i
U
i
+
β
i
with
α
i
>0 an
d
β
i
∈|R
U
i
(0
i
) =0 and U
i
(1
i
) =1 for each i∈{1,2,3,4}
(2)
To illustrate this step, Figure 2 shows the
judgement matrix related to three ERP systems
according to the functional criteria. MACBETH
software defines the values taken by the ERP
systems in the current scale column. The same
procedure must be applied to the other criteria in
order to determine the other utility functions.
Figure 2: Judgments matrix related to functional criterion.
2.3.2 Aggregation Function Determination
The global attractiveness of each ERP system is
defined after the determination of the different
A FRAMEWORK FOR ERP EVALUATION AND SELECTION USING MACBETH TECHNIQUE
341

criteria’s weights. The aggregation function adopted
in MACBETH is the weighted mean (WM). In this
regard we have:
ψ
(
U
1
,
U
2
,
U
3
,
U
4
)
=
∑
=
4
1i
i
ν
U
i
|
∑
=
4
1i
i
ν
= 1
(3)
It is easy to notice that the coefficients
i
ν
could
be determined as:
1
ν
=
ψ
(1
1
, 0
2
, 0
3
, 0
4
);
2
ν
=
ψ
(0
1
, 1
2
, 0
3
, 0
4
)
3
ν
=
ψ
( 0
1
, 0
2
, 1
3
, 0
4
);
4
ν
=
ψ
(0
1
,0
2
, 0
3
, 1
4
)
(4)
Let denote by [Strategic], [Functional],
[Operational] and [Managerial] four fictitious ERP
systems having respectively the following values:
(1
1
, 0
2
, 0
3
, 0
4
), (0
1
, 1
2
, 0
3
, 0
4
), (0
1
, 0
2
, 1
3
, 0
4
) and
(0
1
, 0
2
, 0
3
, 1
4
).
By using categories A0 - A6 to judge the
difference of attractiveness between each two
fictitious ERP systems (Figure 3); we obtain a
MACBETH scale which measures the overall
attractiveness of the above fictitious ERP systems.
The values obtained represent the weights attributed
to the four criteria (Figure 4).
Figure 3: Judgement matrix for weight’s determination.
Figure 4: Criteria’s weights.
3 CONCLUSIONS
This paper deals with the selection process of an
ERP system. A stepwise methodology is proposed
along with a set of criteria in order to support the
purchasing organisation in its initial selection stage.
This methodology addresses the evaluation and
selection of ERP products according to four major
categories of criteria. Besides, MACBETH is used to
quantify the attractiveness and repulsiveness of the
different ERP options. Hence, this paper provides a
comprehensive framework that represents a useful
means for organisations to evaluate and select an
ERP system.
REFERENCES
Bana e costa, C. A. and Vansnick, J. C. (1994). A
theoretical framework for Measuring Attractiveness by
a Categorical Based Evaluation Technique
(MACBETH). Proceedings of XIth International
Conference on Multicriteria Decision Making, 15-24.
Birdogan, B. and Kemal, C. (2005). Determining the ERP
package-selecting criteria: the case of Turkish
manufacturing companies. Business Process
Management Journal, 11(1), 75–86.
Eric, T. G., Wang, S. P. S., James, J. J. and Gary, K.
(2008). The consistency among facilitating factors and
ERP implementation success: a holistic view of fit.
The Journal of Systems and Software, 81(9), 1609–
1621.
Grabisch, M. (2005). A constructive approach to
multicriteria decision making. Traitement du Signal,
22(4),102.
Liang, S. and Lien, C. (2007). Selecting the Optimal ERP
Software by Combining the ISO 9126 Standard and
Fuzzy AHP Approach. Journal of Contemporary
Management Research, 3(1), 23-44.
Remus, U. (2007). Critical success factors for
implementing enterprise portals: a comparison with
ERP implementation. Business Process Management
Journal, 13(4), 538–552.
Teltumbde, A. (2000). A framework for evaluating ERP
projects. International Journal on Production
Research, 38(17), 4507-4520.
Wei, C. C., Chien, C.F. and Wang, M. J. (2005). An AHP-
based approach to ERP system selection. International
Journal of Production Economics, 96(1), 47–62.
Wu, W. W. (2008). A hybrid approach to IT project
selection. WSEAS Transactions on business and
Economics, 5(6), 361-371.
Wu, J. H., Tai, W. C., Tsai, R. J. and Lu, I. Y. (2009).
Using multiple variables decision-making analysis for
ERP selection. International Journal of
Manufacturing Technology and Management, 18(2),
228–241.
ICEIS 2011 - 13th International Conference on Enterprise Information Systems
342