Modified, Stakeholders Perspective based DEA Approach
in IT and R&D Project Ranking
Dorota Kuchta
1
, Dariusz Skorupka
2
, Artur Duchaczek
2
and Magdalena Kowacka
2
1
Faculty of Computer Science and Management, Wrocław University of Technology,
Smoluchowskiego Street 25, 50-372 Wrocław, Poland
2
Faculty of Management, The General Tadeusz Kosciuszko Military Academy of Land Forces,
Czajkowskiego Street 109, 51-150 Wroclaw, Poland
Key
words: Data Envelopment Analysis, Project, Project Stakeholder, Project Input, Project Output.
Abstract: The state of art of Data Envelopment Analysis applied to IT and R&D projects evaluation and ranking is
presented. Then the role and importance of project stakeholders is discussed. It is shown that this role is not
taken into account in the original DEA method applied to project evaluation and ranking. Thus a modification
of the DEA method is proposed, in which the stakeholders perspective plays a crucial role. An example
illustrates the modified method itself and its advantages.
1 INTRODUCTION
The Data Envelopment Analysis (DEA) has been
used for years in the evaluation and comparison of
production units, where outputs (produced goods) are
generated from inputs (raw resources and labour)
(Charnes et al., 1978). The term of production units
used in the DEA method has been generalised, so that
bank divisions and hospital units can be evaluated,
ranked and compared too. The inputs in such cases
are labour hours and budgets, the outputs - the
services rendered, the number of patients or clients
served etc. However, in recent years the method has
been used among others to evaluate, compare and
rank projects, where inputs and outputs may be of a
different nature than in case of production units, even
in the generalized sense. In case of projects,
especially R&D or IT projects, the inputs and above
all outputs may be of a qualitative nature - for
example, an important output of a project may be the
customer satisfaction. Still, even in such a case the
DEA method has turned out to useful in project
evaluation and ranking - the relevant literature will be
discussed in Section 2.
However, in the DEA method applied to projects
there is an inherent mistake. The DEA in its original
form (see Section 2) maximises the ratioweighted
outputs sum/weighted inputs sum” for each project in
turn, while the decision variables are the weights of
inputs and outputs. The idea is that each project has
the right to “decide” which weights to use with
respect to the inputs and outputs in order to show
itself in the most positive light. Thus, the weights may
take any nonnegative value, the only condition is that
the ratio “weighted output/weighted input” is
maximal. However, it has been overlooked that
projects are seen differently by different stakeholders
(see Section 3). Thus, if the idea is to give to each
project the possibility to show itself in a positive light,
it should also be possible for each project to choose
the best perspective among those of different
stakeholders and in the eyes of individual
stakeholders certain sets of inputs and outputs do not
count. As it will be shown in Section 4, the possibility
to consider various views of various decision makers
is not present in the original DEA method. We will
introduce it in Section 4. In Section 5 it will be
illustrated by means of a computational example.
2 DEA METHOD APPLIED TO
PROJECT RANKING
As mentioned above, the DEA method has been
recently used to evaluate and compare various units,
which may be production units, banks, hospital units,
and, last but not least, projects. The basis for the
comparison are inputs used and outputs generated by
158
Kuchta, D., Skorupka, D., Duchaczek, A. and Kowacka, M.
Modified, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking.
In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 2, pages 158-165
ISBN: 978-989-758-187-8
Copyright
c
2016 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
the units in question. If there are 1 units, each of
them is evaluated by a separated fractional
programming model (which may be reduced to a
linear one), where the unit being evaluated is
numbered as the 0-th unit. The model is formulated
as follows (Charnes et al., 1978).



→
(1)



1,1,…,
(2)

0,
0
(3)
where
 1, ,  are inputs of the k-th object,
0,…, and
 1, , are its outputs,
1,…, and
 1, , are decision
variables and at the same time weights of the,
respectively, inputs and outputs, chosen from the
point of view of the 0-the unit - in order to maximise
its performance, defined in (1).
Each unit becomes in turn the 0-th unit. The value
of the objective function of the fractional
programming problem (1)-(3) formulated when the k-
th unit becomes the 0-th unit will be denoted as
and will be considered to be the performance of the k-
th unit. The units are than ranked in the decreasing
order of
 0, . ,.
The ratios
0,….,
correspond to the
situation when each unit has the right to choose the
inputs and outputs weights is such a way that the ratio
“weighted sum of outputs/weighted sum of inputs” as
for it as high as possible, and the only constraints are
(2) and (3).
Recently the DEA method has been often applied
to projects, especially to IT and R&D projects. In the
following we will show which inputs and outputs are
chosen for projects when they are compared by means
of the DEA method.
For IT projects (understood as projects
concerning the development, installation and use of
computer systems and applications (American
Heritage Dictionaries, 2014) the authors use the
following inputs:
cost of the project, duration of the project,
number of employees needed - the actual
and the planned one (Gusmao and Costa,
2012; Asosheh et al., 2010);
green dollar cost (expenses paid to entities
outside the organisation), brown dollar cost
(internal expenses, the cost of personnel
used), project actual duration, potential risk
(potential loss to the corporation) (Sowlati et
al., 2005).
labour, other expenses, duration (Wray and
Mathieu, 2008)
The outputs for IT projects have been grouped by
various authors in several categories (Sowlati et al.,
2005; Gusmao and Costa, 2012; Wray and Mathieu,
2008):
financial perspective (cost reduction of the
organisation in which the product of the IT
project has been implemented, profit
increase of the organisation, green dollar
benefits (profit to the organisation or
reduction of expenses paid outside the
organisation), brown dollar benefits
(reduction of internal expenses, reduction of
the cost of personnel used);
internal business perspective (a better
control of internal processes, their increased
security achieved thanks to the product of
the project in question);
customer/stakeholder perspective (increased
customer satisfaction, a higher compliance
with needs of stakeholders) - the role of
various stakeholders is here underlined;
learning perspective (benefits consisting in
the fact that thanks to the project the
organisation has achieved new skills);
uncertainty perspective (here the risks linked
to the use of the project product are meant:
the higher the risks, the lower the output of
the project);
the complexity of the product of the project;
intangible benefits (to individual members
of the organisation or to organisational units
of the organisation or to the whole
organisation realising the project);
software production related outputs:
number of function points realised, number
of lines of code.
R&D projects (R&D, i.e. Research and
Development, comprises creative work undertaken
on a systematic basis in order to increase the stock of
knowledge, including knowledge of man, culture and
society and the use of this stock of knowledge to
devise new applications (Frascati Manual, 2002))
have also been subject to evaluation and ranking by
means of DEA method, The following inputs have
been used in the literature (Eilat et al., 2008; Revilla
et al., 2003; Yuan and Huang, 2002):
project cost (budgeted and actual):
the full time equivalent of highly trained
personnel (managers, engineers and
scientists, holding PHD, master, bachelor
degree) used for the realisation of the
project;
Modified, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking
159
total organisation revenues;
total organisation R&D Budget, Total
number of corporate employees
total number of organisation employees;
It has to be underlined that the last three inputs
listed above refer not to the project being evaluated,
but to the whole organisation implementing the
project.
As for the outputs, the following ones have been
used for R&D projects (Eilat et al., 2008; Revilla et
al., 2003; Yuan and Huang, 2002):
discounted cash flow generated by the
project;
performance improvement achieved thanks
to the project;
customer satisfaction with the product of the
project;
congruence with the strategy of the
organisation realising the project;
synergy with other projects realised by the
organisation;
project team satisfaction;
the number of team members trained in
project management thanks to the project
realisation;
probability of technological and commercial
success of the project product;
technical gap size covered by the project
product;
the newness of the technology used;
the complexity of market activities needed
to commercialize the project product;
new scientists gained by the organisation
thanks to the project;
total income generated by the project;
number of patents and copyrights gained
thanks to the project;
number of dissertations worked out thanks
to the project;
number of reports issued thanks to the
project;
number of technology innovations worked
out thanks to the project;
number of seminars organised thanks to the
project;
number of technology transfers resulting
from the project.
To sum up the state of the art of the DEA
application to IT and R&D projects evaluation and
ranking, it has to be said that the inputs and outputs
used are often of a qualitative nature and their value
has to be given by an expert. Another important thing
is that the inputs and outputs used are not of the same
importance to each project stakeholder. For example,
the team satisfaction, new dissertations and new skills
of the project team may be of a high importance to the
persons responsible of the scientific development of
the organisation members, but it will be of no
importance to the financial manager of the
organisation. For the latter the project cost and the
revenues generated by it will be much more
important.
The problem of the diversity of views of different
project stakeholders and of their importance is treated
in the following section.
3 PROJECT STAKEHOLDERS
AND THEIR ROLE
Project stakeholders (project stakeholder is an
individual, group, or organization who may affect, be
affected by, or perceive itself to be affected by a the
project (PMI, 2013)) are often very diversified, and
often have contradictory expectations with respect to
the project.
This statement can be justified by the list of
stakeholders of a certain R&D project, financed by an
external institution, which was realised be a team at a
university with one of the authors of the present paper
as project manager. It has to be underlined that the list
was not completely identified before the project start.
It is only once the project was closed that the
complete list of the stakeholders was known. This
complete list is as follows:
the members of the project team;
the project manager;
the accounting department;
the project management department;
the financial manager of the university;
the scientific manager of the department.
The members of the project team and the project
manager wanted to attain the project goal, which
consisted in elaborating a new accounting system, but
its achievement depended on the information given
by the accounting department. The accounting
department did not want a new accounting system, so
they did not cooperate, as for them the fulfilment of
the project goal meant more work and more
challenges. In this situation the financial manager of
the university was interested only in one thing: that
the project is not found to be a complete failure by the
financing institution, so that the university does not
have to pay back the project budget. The aim of the
project manager was then to find a substitute goal
which would be accepted by the financing institution.
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
160
The team members were not very motivated to fulfil
the substitute goal, but they did it, in order to help the
project manager. The scientific manager of the
department wanted to have new publications in good
journals and he did not care the realisation of which
goal these publications will present. The project
management department wanted to have all the
reports in due times, so that they could register them
and send to the financing institution. They did not
care much about all the problems which caused that
the project was in danger.
Finally the project was accepted with the substitute,
much more moderate goal, so that the university did
not have to pay any money back and in all official
documents the project is qualified as a success,
although in the eyes of the project team and the
project manager it was a failure, partially due to the
wrongful identification of project stakeholders and
their views. Above all the role of the accounting
department was not identified properly.
The project stakeholders are important. The have
been becoming more and more present in the project
success perception. In the past it was considered that
a project is successful if it meets the specification
(scope), cost (budget) and time (deadline). Today
most authors expand this definition substantially,
introducing other project success measures. The
nature of these extensions can be summarised by the
following statement: “There have to be two groups of
project success measures: objective measures (such
as time or cost) and subjective measures (such as the
satisfaction of different project stakeholders)” (Chan
et al., 2002; Chan et al., 2004). Subjective measures
are necessary, because the perception of project
success depends strongly on the assessor (Davis,
2014).
That is why project stakeholders management is
considered as very important. There are many
proposals of methods to identify and manage
stakeholders, both for projects in general (Hartono et
al., 2014; Missonier and Loufrani-Fedida, 2014) and
for IT (Sudevan et al., 2014) and R&D projects
(Broom et al., 2013; Geeson et al., 2015).
In the existing DEA approaches to project
evaluation and ranking, the stakeholders and their
varying views have not been taken into account. All
the possible inputs and outputs are allowed in model
(1)-(3). Each project tends to maximize its
performance by choosing the weights which
maximize function (1) and are selected among all
non-negative values of the decision variables (3).
However, not all the inputs and outputs are important
for each stakeholder. In many cases the situation will
be as follows: for each stakeholder only a certain
subset of inputs and a certain subset of outputs will
matter, the other outputs and inputs will have weights
equal to zero from the outset. If the philosophy of the
DEA approach consists in letting each project present
itself in the best light, each project should also have
the right to choose the stakeholder for which it has the
best performance. And in some cases the decision
maker will want to asses and rank projects taking
various stakeholders into account. The original DEA
method does not make it possible. The proposal
formulated in Section 4 covers this gap.
Our proposal will allow to asses and rank projects
taking into account for example the following
stakeholders and their diversified views:
1) for IT projects:
a) a customer representative, representing the
upper management level, interested in one
input: the project price (linked to the project
budget), and in three outputs: customer
satisfaction, the risk linked to the project
product and the product complexity;
b) the software producer representative,
representing the upper management level,
interested in one input: the green dollar cost,
and in on one output: the green dollar
benefits;
c) another representative of the same software
producer, representing a lower management
level, interested in one input - the number of
employees that were involved in the project,
and in three types of output: the skills that
the employees gained thanks to the project,
other intangible benefits the employees
gained thanks to the project, intangible
benefits gained through the project team as a
whole;
d) another representative of the same software
producer, representing an upper
management level, interested in three inputs:
the project cost, the project duration, the
potential risk linked to the project
realisation, and in two outputs: the customer
satisfaction and the compliance of the
project with customer strategy - which will
mean chances for more contracts in the
future.
2) For R&D projects:
a) the financing institution, interested in one
output: the budget, and in two types of
outputs: the number of publication in high
quality journals and the commercialisation
chances of the product;
b) the dean of a university faculty at which the
project is realised, interested in two types of
Modified, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking
161
input: the full time equivalent of personnel
with various degrees used in the project and
the project duration, and in three types of
output: the numbers of publications, the
number if dissertations worked out and
published, the number of high quality
personnel members employed thanks to the
projects;
c) The financial manager of the institution at
which the project is realised, interested in
one type of input - the time his/her
employees had to spend to help to prepare
the project application and in one output: the
part of the project budget that will be left at
the university disposal.
Of course, the above examples are theoretic,
although partially based on the authors’ experience.
They are not meant to show what the corresponding
stakeholders should be interested in, but what they are
often interested in, even if it is not correct. The main
goal of the above examples is to show that various
stakeholders, also those who are important in the
organisation which realises the project or for which
the project is realised, may have substantially
different views about what the significant inputs and
outputs of a project are. The modified DEA method
will make it possible to take this into account.
4 MODIFIED DEA METHOD
Let 
1,2,,
be the set of the indices of all
possible inputs that may be considered for the set of
projects which are to be evaluated and ranked, and

1,2, ,
the set of the indices of all
outputs. There are K+1 projects to be evaluated and
ranked. Let 
1,2, ,
be the set of the
indices of all the stakeholders that have been
identified for the projects in question.
For the l-th stakeholder (∈) there are given two
sets: 
⊂ and 
⊂. These sets represent the
indices of the inputs and outputs taken by the l-th
stakeholder into consideration. It follows that from
the point of view of the l-th stakeholder it holds:
0 and
0
(4)
for all ∈
and  
where 
and
are 
and
from (3) when the k-
the project becomes the 0-th project for problem (1)-
(3), k=0,…K.
We can thus for each project solve L+1 problems:
one identical to problem (1)-(3), where the individual
stakeholders are not taken into account, one for each
stakeholder (∈). The stakeholders linked
problems will be of the following form (for each ∈
):


→


1, 1,…,
(5)
0,
0
0 and
for all ∈
and  
where each project in turns becomes the 0
th
project.
For each project k=0,1,…K we would then have
L+1 evaluations:
, being the value of the objective
function of the problem (1)-(3), representing the best
possible project evaluation when all inputs and
outputs are put in the same box and treated in the
same way, and
, ∈, being the objective function
of problem (5), when the k-th becomes the 0
th
project,
and representing the best possible project evaluation
when only the inputs and outputs important to the l-
th stakeholder are taken into account.
These values can be then interpreted in several
ways. We can aggregate them to a final project
ranking for example as follows (
, k=0,1,…K
stands for the final ranking of the k-th project):


,
, 
(6)
or we can define 
as a weighted sum of
and
,.
We adopt here the formula (6), where the basic
idea of the DEA method is retained: each project can
present itself in the best possible way, by choosing
only the weights of the inputs and outputs, but also
the stakeholder who would put it in a good position.
But any method of aggregating values
,
, is
better than the original approach, in which only
values
are calculated., because the original method
does not allow to take into account the view of even
the key stakeholders. Using the original DEA method
we may rank lowly some projects which are in fact
good, because they would be highly appreciated by
the key stakeholders whose opinion is crucial for our
organisation. In the next section we will illustrate the
proposed approach by an example.
5 COMPUTATIONAL EXAMPLE
Let is consider 10 R&D projects, whose all possible
inputs and outputs (for all possible stakeholders) are
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
162
given in Table 1. All the inputs names in the Table 1
should be accompanied by words “used in the
project” and all the output names by words
“generated by the project”. The values of the inputs
and outputs have been given by experts and it has
been made sure that they are commeasurable.
Problem (1)-(3), thus the original DEA method,
gives the results shown in Table 2. The original DEA
method would thus give us the following ranking of
the projects: P5, P7, P8, P4, P6, P1, P2, P9, P3, P10.
Let us now consider two key stakeholders (L=2).
We have the following information:

4,5,

1,2,3,4,5
;

={1,2,3,4,5},
1,2,3
.
This means that the first stakeholder disregards
inputs other than the number of assistants and
associate professors engaged in the project and the
second stakeholder disregards among outputs the
number of patents and the number of scientific
degrees generated be the project.
Table 1: All possible inputs and outputs for 10 example projects.
Inputs and outputs names
Inputs and outputs values
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Inputs
a1
budget [mln. EURO] 0.2 0.3 0.5 0.9 0.1 0.6 0.2 0.8 0.8 0.1
a2
duration [months] 12 12 24 36 36 12 24 36 24 24
a3
N
b of full professors 2 2 2 1 3 2 2 3 1 1
a4
N
b of associate professors 5 5 6 8 2 3 3 6 4 5
a5
N
b of assistant professors 3 8 2 4 6 9 2 4 2 3
Outputs
b1
N
b of high quality publications 2 1 1 2 5 2 2 3 1 0
b2
b of international conferences presentations 10 12 10 11 13 9 8 8 5 15
b3
N
b of monographs 1 2 1 1 0 2 2 1 0 1
b4
N
b of patents 0 0 1 1 0 1 1 2 0 0
b5
N
b of scientific degrees and titles 2 1 2 3 1 1 2 2 1 3
Table 2: The results of the original DEA for the example projects.
Input and output weights
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Inputs
a1 0.606 0.909 0.000 0.000 0.145 0.000 0.545 0.000 0.000 1.173
a2 0.066 0.029 0.000 0.000 0.000 0.084 0.000 0.026 0.000 0.000
a3 0.000 0.000 0.000 1.003 0.000 0.000 0.000 0.397 1.356 0.575
a4 0.000 0.000 0.000 0.000 0.000 0.000 0.012 0.000 0.000 0.000
a5 0.000 0.000 2.030 0.000 0.000 0.000 0.031 0.044 1.162 0.000
Ouputs
b1 0.303 0.000 0.000 0.535 0.014 0.000 0.027 0.000 1.227 0.000
b2 0.000 0.015 0.406 0.000 0.000 0.050 0.000 0.000 0.323 0.116
b3 0.000 0.379 0.000 0.000 0.000 0.201 0.210 0.000 0.000 0.000
b4 0.000 0.000 0.812 0.267 0.000 1.205 0.172 1.499 0.000 0.016
b5 0.545 0.000 0.000 0.334 0.000 0.000 0.000 0.000 0.000 0.114
Objective function value 1.867 1.512 1.200 2.333 5.000 2.050 3.098 1.308 0.772 3.000
Table 3: The results of problem (5) for l=1.
Input and output weights
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Inputs
a1 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
a2 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
a3 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
a4 0.070 0.425 0.010 0.010 0.660 1.000 0.029 0.220 0.010 0.082
a5 0.871 0.010 1.000 0.985 0.010 0.010 0.053 0.169 1.000 0.368
Outputs
b1 0.186 0.000 0.000 0.000 1.000 0.000 0.023 0.000 0.318 0.000
b2 0.197 0.070 0.203 0.000 0.000 0.158 0.000 0.000 0.174 0.123
b3 0.000 0.368 0.000 0.000 0.000 0.499 0.257 0.000 0.000 0.000
b4 0.000 0.000 0.403 0.000 0.000 0.755 0.000 1.000 0.000 0.000
b5 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.000 0.000 0.000
Objective function value 0.792 0.714 1.183 0.746 3.623 1.029 2.909 1.000 0.583 1.216
Modified, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking
163
Table 4: The results of problem (5) for l=2.
Input and output weights
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Inputs
a1 0.000 1.000 0.000 0.000 1.000 0.000 0.168 0.000 0.000 1.000
a2 0.134 0.032 0.000 0.000 0.000 0.130 0.000 0.000 0.000 0.000
a3 0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.390 1.000 0.343
a4 0.000 0.000 0.000 0.000 0.012 0.303 0.110 0.000 0.000 0.000
a5 0.336 0.000 1.000 0.000 0.000 0.000 0.194 0.660 0.849 0.016
Ouputs
b1 0.672 0.010 0.193 0.481 0.068 1.000 0.033 1.000 0.896 0.010
b2 0.269 0.015 0.199 0.046 0.010 0.010 0.010 0.010 0.236 0.090
b3 0.201 0.415 0.010 0.313 0.010 0.980 1.000 0.010 0.010 0.010
b4 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
b5 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Objective function value 1.615 1.489 1.098 1.779 3.790 1.637 2.854 0.811 0.769 2.781
Table 5: Position of each project in the three rankings.
ranking P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Original DEA 6 7 9 4 1 5 2 3 8 10
Stakeholder 1 7 9 4 8 1 5 2 6 10 3
Stakeholder 2 5 7 8 6 1 4 2 9 10 3
Table 6: Position of each project in the final ranking on the basis of (6).
Ranking P1 P2 P3 P4 P5 P6 P7 P8 P9 P10
Value (6) 1,867 1,512 1,200 2,333 5,000 2,050 3,098 1,308 0,772 3,000
Final ranking 6 7 8 4 1 5 2 9 10 3
Table 3 and 4 contain the results of the application
of problem (5) for l=1 and 2.
If we considered just the view of the first
stakeholder (Table 3), we would have the following
ranking of the projects: P5, P7, P10, P3, P6, P8, P1,
P4, P2, P9, which is different than the one obtained
by means of the original DEA method (especially if
we consider e.g. project P10).
The view of the second stakeholder alone would
give still another ranking - P5, P7, P10, P4, P6, P1,
P2, P3, P8, P9 - which is shown in Table 4
.
In Table 5 the positions of each project in the three
rankings is shown. This may be a valuable
information, for example, we can see that some
projects have a very stable position (P5, P7, P6), but
in case of some projects the evaluation changes
considerably (P10, P8). The influence of the
stakeholders may be high.
In Table 6 the aggregated values (6) are shown.
This ranking shows the best situation of each project
according to the choice of weights and the
stakeholders and it might be a useful tool for the
project evaluation. The projects at the top of the final
ranking are certainly good in the eyes of some
stakeholders. For example, if we had used the original
DEA method, project P10 would have been rejected.
But this project seems to satisfy two key stakeholders.
Rejecting it would mean disregarding the key
stakeholders of our organisation, which is decisively
wrong.
6 CONLUSIONS
In this paper we propose a modification of the DEA
method which can be used to evaluate projects. The
core of our proposal is the possibility to include in the
DEA model the perspectives of various project
stakeholders. Project stakeholders, especially the key
ones, cannot be neglected in project evaluation and
selection. The original DEA method used for project
evaluation disregarded them.
The proposal, combined with project stakeholders
management methods and with careful project inputs
and outputs identification and evaluation, may be
useful in project selection problems. But of course
real world cases are needed to prove it.
REFERENCES
American Heritage Dictionaries, 2014. The American
Heritage Student Science Dictionary. Boston, MA:
Houghton Mifflin Harcourt Publishing Company.
Asosheh, A., Nalchigar, S., Jamporazmey, M., 2010.
Information technology project evaluation: An
ICEIS 2016 - 18th International Conference on Enterprise Information Systems
164
integrated data envelopment analysis and balanced
scorecard approach, Expert Systems with
Applications, vol. 37, no. 8, pp. 5931-5938.
Broom, M., Brady, B., Kecskes, Z., Kildea, S., 2013. World
Café Methodology engages stakeholders in designing a
Neonatal Intensive Care Unit, Journal of Neonatal
Nursing, vol. 19, no. 5, pp. 253-258.
Chan, A.P.C., Scott, D., Chan, A.P.L, 2004. Factors
affecting the success of a construction project. “Journal
of Construction Engineering and Management”, vol.
130 no. 1, pp. 153–155.
Chan, A.P.C., Scott, D., Lam, E.W.M., 2002. Framework
of success criteria for design/build projects, Journal of
Management in Engineering, vol. 18, no. 3, pp. 120-
128.
Charnes, A., Cooper, W.W., Rhodes, E., 1978. Measuring
efficiency of decision making units. Eur. J. Opl. Res. 2,
429444.
Davis, K., 2014. Different stakeholder groups and their
perceptions of project success, International Journal of
Project Management, vol. 32, no. 2, pp. 189-201.
Eilat, H., Golany, B., Shtub, A., 2008. R&D project
evaluation: An integrated DEA and balanced scorecard
approach, Omega, vol. 36, no. 5, pp. 895-912.
Frascati Manual: Proposed Standard Practice for Surveys
on Research and Experimental Development, 6th
edition, www.oecd.org/sti/frascatimanual, 6th edition,
2002, Pages: 266, ISBN 978-92-64-19903-9.
Geeson, N., Quaranta, G., Salvia, R., Brandt, J., 2015.
Long-term involvement of stakeholders in research
projects on desertification and land degradation: How
has their perception of the issues changed and what
strategies have emerged for combating desertification?,
Journal of Arid Environments, vol. 114, pp. 124-133.
Gusmao, A.P.H., Costa, A.P., 2012. Evaluation of it/is
outsourcing projects using the dea methodology. MCIS
2012 Proceedings. Paper 24.
http://aisel.aisnet.org/mcis2012/24.
Hartono, B., Sulistyo, S.R., Praftiwi, P.P. , Hasmoro, D.,
2014. Project risk: Theoretical concepts and
stakeholders' perspectives, International Journal of
Project Management, vol. 32, no. 3, pp. 400-411.
Missonier, S., Loufrani-Fedida, S., 2014. Stakeholder
analysis and engagement in projects: From stakeholder
relational perspective to stakeholder relational
ontology, International Journal of Project Management.
Project Management Institute (PMI), 2013. Managing
change in organizations: A practice guide. Newtown
Square, PA: Author.
Revilla, E., Sarkis, J., Modrego, A., 2003. Evaluating
performance of public - private research
collaborations: A DEA analysis, Journal of the
Operational Research Society, vol. 54, no. 2, pp. 165-
174.
Sowlati, T., Paradi, J.C., Suld, C., 2005. Information
systems project prioritization using data envelopment
analysis, Mathematical and Computer Modelling, vol.
41, no. 11-12, pp. 1279-1298.
Sudevan, S., Bhas, M., Pramod, K.V., 2014. A typology of
stakeholder identification methods for projects in
software industry, 14th Middle Eastern Simulation and
Modelling Multiconference, MESM 2014 - 4th
GAMEON-ARABIA Conference, GAMEON-
ARABIA 2014, pp. 5.
Wray, B., Mathieu, R., 2008. Evaluating the performance
of open source software projects using data
envelopment analysis, Information Management and
Computer Security, vol. 16, no. 5, pp. 449-462.
Yuan, B., Huang J.-N., 2002. Applying Data Envelopment
Analysis to Evaluate the Efficiency of R&D Projects —
A Case Study of R&D in Energy Technology. in:
Technology Commercialization. pp 111-134.
APPENDIX
The research was partially supported by the National
Science Center in Poland, grant no. 260084
"Research projects success and failure factors".
Modified, Stakeholders Perspective based DEA Approach in IT and R&D Project Ranking
165