Performance Metrics Based on Data Envelopment Analysis for

Evaluating Multi-Objective Linear Programming Solution Methods

Javier E. Gómez-Lagos

, Marcela C. González-Araya

and Luis G. Acosta Espejo

Doctorado en Sistemas de Ingeniería, Faculty of Engineering, Universidad de Talca, Campus Curicó,

Camilo a Los Niches, km 1, Curicó, Chile

Departament of Industrial Engineering, Faculty of Engineering, Universidad de Talca, Campus Curicó,

Camino a Los Niches km 1, Curicó, Chile

Departamento de Ingeniería Comercial, Universidad Técnica Federico Santa María, Avenida Santa María 6400,

Vitacura, Santiago, Chile

Keywords: Multi-Objective Linear Programming, MOLP Performance Metrics, MOLP Solution Methods,

Data Envelopment Analysis, Slack Based Measure Model, Super-efficiency DEA Model.

Abstract: Three performance metrics based on data envelopment models are proposed for evaluating MOLP solution

methods. Every proposed metric is associated to a one category, being these categories the cardinality,

accuracy, and diversity. In addition, the proposed metrics are classified as unary or binary. The cardinality

and accuracy metrics are estimated using a DEA model based on the slack based measure model, while the

diversity metric is calculated using the super-efficiency DEA model. The proposed metrics were applied to

compare two sets of solutions for a MOLP tactical harvest planning model, that were obtained using two

strategies of a MO-GRASP algorithm. The results show that the metrics allow discriminating between the

MOLP solution methods and, moreover, to select one.

1 INTRODUCTION

The multi-objective linear programming (MOLP) is

an area of operations research where many practical

problems have being addressed, as transport (Demir

et al., 2014), agriculture (Varas et al., 2020),

manufacturing (Mirzapour Al-E-Hashem et al.,

2011), location (Karatas & Yakıcı, 2018), among

others. Usually, these MOLP models are difficult to

solve (Deb, 2014). Therefore, exact and heuristic

methods have been proposed for solving them.

However, selecting a suitable solution method is not

a simple task. In this way, different performance

metrics have been proposed for analysing the

solutions obtained by these methods. Regarding this

issue, Riquelme et al. (2015) carried out a literature

review about the performance metrics for evaluating

MOLP solution methods, classifying them in three

categories: cardinality, accuracy, and diversity.

Cardinality represents the number of non-dominated

solutions found by a MOLP solution method.

https://orcid.org/0000-0003-1149-1434

https://orcid.org/0000-0002-4969-2939

Accuracy refers to the convergence of the non-

dominated solution to the Pareto frontier. Thus, it

represents the distance between every non-dominated

solution with the theoretical Pareto frontier

(Riquelme et al., 2015). Diversity considers the

distribution and spread of the non-dominated

solutions. The distribution considers the relative

distance among the non-dominated solutions, and the

spread corresponds to the range of the objective

function values covered by the non-dominated

solutions. It is important to mention that, in every

category, different metrics have been proposed

(Audet et al., 2021; Riquelme et al., 2015).

Furthermore, Riquelme et al. (2015) also classified

the performance metrics into unary and binary. A

metric is unary if the non-dominated solutions are

obtained by only one solution method. On the other

hand, a metric is binary if the non-dominated

solutions are obtained by two solution methods.

In the literature, data envelopment analysis (DEA)

models have been used for estimating MOLP

Gómez-Lagos, J., González-Araya, M. and Acosta Espejo, L.

Performance Metrics Based on Data Envelopment Analysis for Evaluating Multi-Objective Linear Programming Solution Methods.

DOI: 10.5220/0011660200003396

In Proceedings of the 12th International Conference on Operations Research and Enterprise Systems (ICORES 2023), pages 151-157

ISBN: 978-989-758-627-9; ISSN: 2184-4372

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

151

performance metrics. In Bal & Satoglu (2019), the

BCC model (Banker et al., 1984) was used as a metric

for evaluating the performance of Pareto optimal

solutions obtained by the augmented epsilon

constraint method 2. This solution method was

applied to a MOLP model with four objective

functions, aiming to improve the coordination of an

appliance supply chain. Hong & Jeong (2019) used a

CCR (Charnes et al., 1978) for evaluating the

solutions obtained by the weighting method. This

method was used for solving a MOLP model with five

objective functions, which sought to determine

strategic decisions for a facility location–allocation

problem.

In this study, three performance metrics based on

DEA model are proposed for evaluating MOLP

solution methods. Every metric is associated to a

category of cardinality, accuracy, and diversity,

respectively, and can be classified as unary or binary.

This article is divided as follows: Section 2

describes the applied DEA models and the procedure

for calculating every metric. Section 3 presents the

results of this study, while Section 4 summarizes the

conclusions.

2 MATERIAL AND METHODS

The DEA models used for assessing different

performance metrics of MOLP solution methods, as

the associated procedure for applying them, are

presented in this section. As mentioned previously,

the categories considered in this analysis are:

cardinality, accuracy, and diversity. The proposed

cardinality and accuracy metrics are estimated using

a DEA model based on the slacks-based measure

proposed by Tone (2001). On the other hand, the

proposed diversity metric is calculated using the

super efficiency DEA model developed by Andersen

& Petersen (1993). The characteristics for selecting

these DEA models and the way that they can be used

for estimating every metric are detailed in the

following sub-sections.

2.1 Applied DEA Models

The common nomenclature of parameters and

decision variables used in the applied DEA models

are defined in Table 1 and Table 2, respectively. In

this definition, the MOLP nature of the DEA

assessment is considered.

Table 1: Parameters of the DEA models.

Paramete

Definition

𝑚

Number of objective functions to be

minimized in the MOLP model.

𝑠

Number of objective functions to be

maximized in the MOLP model.

𝑛

Number of solutions obtained by a MOLP

solution metho

𝑥



Value of the minimized objective function

i obtained by solution j, where 𝑖

1,…,𝑚, 𝑗  1,…,𝑛.

𝑦



Value of the maximized objective function

r obtained by solution j, where 𝑟

1,…,𝑠, 𝑗  1,…,𝑛.

𝑗



Evaluated solution in every execution of

the applied DEA models.

The different decision variables used in the

applied DEA models are described in Table 2.

Table 2: Decision variables of the DEA models.

Variable Definition

𝑆





Slack of the minimized objective function i,

where 𝑖1,…,𝑚

𝑆





Slack of the maximized objective function r,

where 𝑟1,…,𝑠

𝜆



Intensity of the solution j for establishing the

target in the Pareto frontier of the evaluated

solution.

𝑡

Auxiliary variable that represents a positive

scalar used for the model linearization.

𝐴



Auxiliary variable for the model

linearization. It is a binary variable, where

𝐴



1 if 𝜆



𝑡; 𝐴



0 otherwise, ,𝑗

1,…,𝑛.

𝜃

Proportional reduction of the minimized

objective functions obtained by solution 𝑗



2.1.1 DEA Model Based on the Slacks-based

Measure of Efficiency (INT-SBM)

The proposed DEA model is based on the slacks-

based measure model (SBM) developed by Tone

(2001). This author presented a non-linear model,

which was linearized using the linear transformation

proposed by Charnes & Cooper (1962). The SBM

model was selected because it does not require inputs

in an output-oriented model, or it does not require

outputs in an input-oriented model. In this way, it can

evaluate solution methods that solve MOLP problems

that have only maximization objective functions or

only minimization objective functions. Moreover, the

ICORES 2023 - 12th International Conference on Operations Research and Enterprise Systems

152

non-dominated solutions that belong to the Pareto

frontier will obtain a score equals to one, while the

dominated solutions will obtain a score greater than

zero and lower than one. This score represents the

closeness to the Pareto frontier.

The DEA model formulated in this study differs

from SBM linear model because it considers binary

variables. For this reason, it is called Integer-SBM

(INT-SBM), and it evaluates solutions located in the

non-convex region of the Pareto frontier. These

solutions will obtain a score equals to one.

The proposed DEA model must be executed for

every solution j, where j

represents the evaluated

solution in a specific model execution.

The INT-SBM formulation is:

Minimize 𝜉





𝑡

𝑚



𝑆





𝑥











(1)

Subject to

1𝑡

𝑠



𝑆





𝑦











(2)

𝑡𝑥





𝑥



𝜆







𝑆





, 𝑖  1,…, 𝑚,

(3)

𝑡𝑦





𝑦



𝜆







𝑆





, 𝑟  1,… ,𝑠,

(4)

𝜆







𝑡,

(5)

𝐴







1,

(6)

𝜆



𝑡



1𝐴





𝑗

1,…,𝑛,

(7)

𝜆



𝑡



1𝐴





𝑗

1,…,𝑛,

(8)

𝐴



∈



0,1



𝑗

1,…,𝑛,

(9)

𝜆



0,

𝑗

1,…,𝑛,

(10)

𝑆





0,𝑖1,…,𝑚,

(11)

𝑆





0,𝑟1,…,𝑠, (12)

𝑡0. (13)

The objective function (1) estimates the efficiency

score based on the minimization of slacks associated

to the minimized objective functions obtained by

solution 𝑗



. Constraint (2) allows linearizing the

objective function, making the expression related to

the maximized objective function slacks equals to a

constant value. Constraints (3) and (4) calculate the

slacks of minimized and maximized objective

functions, respectively. Constraint (5) establishes the

convexity of the efficient frontier associated to the

variable returns to scale Tone (2001). Constraints (6)

to (8) allow linearizing 𝜆



𝐴



, which are decision

variables. Finally, constraints (9) to (13) establish the

nature of the decision variables.

In the INT-SBM model,

𝜉





corresponds to the

efficiency score of the evaluated solution 𝑗



, which

varies between zero and one. In this case, 𝜉





= 1

represents a non-dominated solution.

2.1.2 Super-Efficiency DEA Model

Andersen & Petersen (1993) proposed the super-

efficiency DEA model for ranking all the evaluated

units according to their efficiency score. This

efficiency score could be greater than one, for an

input-oriented model, or lower than one, for an

output-oriented model. These values are possible

because the data of every evaluated solution 𝑗



are not

considered in the observed data of the DEA model for

determining the DEA efficient frontier.

In this study, an input-oriented super-efficiency

DEA model (SE-DEA) was used, aiming to improve

the discrimination among the non-dominated

solutions. It is important to mention that the model

orientation does not vary the identification of the

super-efficient solutions’ set. In addition, the SE-

DEA model must be executed for every solution j,

where j

corresponds to the evaluated solution in a

specific model execution.

The SE-DEA formulation is:

Minimize𝛿





𝜃

(14)

Subject to

𝜆



𝑥











𝜃𝑥





,𝑖1,…,𝑚,

(15)

𝜆



𝑦











𝑦





, 𝑟1,…,𝑠,

(16)

𝜆











1,

(17)

𝜆



0,

𝑗

1,…,𝑛,

(18)

Performance Metrics Based on Data Envelopment Analysis for Evaluating Multi-Objective Linear Programming Solution Methods

153

𝜃 free (19)

The objective function (14) minimizes the

proportional reduction of the minimized objective

functions obtained by solution 𝑗



. Constraint (15)

establishes that the proportional reduction of the

minimized objective functions obtained by solution

𝑗



must be greater or equal than the composed target

in the efficient frontier (left hand of the constraint).

Constraints (16) estimates that the maximized

objective functions obtained by solution 𝑗



must be

lower or equal than the composed target in the

efficient frontier (left hand of the constraint).

Constraint (17) imposes the convexity of the efficient

frontier, which is associated to the variable returns to

scale (Banker et al., 1984). Constraints (18) and (19)

establish the nature of the decision variables.

In the SE-DEA model

, 𝛿





corresponds to the

efficiency score of the evaluated solution 𝑗



. This

efficiency score, differently from a traditional BCC

input-oriented model (Banker et al., 1984), could

achieve values greater than one or even the model

could be infeasible. The necessary and sufficient

conditions for infeasibility of SE-DEA models when

variable returns to scale are considered (constraint

17), are presented in the study of Seiford & Zhu

(1999). Consequently, a solution 𝑗



that is an extreme

point of the Pareto efficient frontier will have a

𝛿





value greater than one or the associated model could

be infeasible.

In the following sub-section, the performance

metrics for evaluating MOLP solution methods and

the steps for implementing them using the formulated

DEA models are described.

2.2 Performance Metrics for

Evaluating MOLP Solution

Methods

As mentioned previously, the considered categories

for evaluating MOLP solution methods are

cardinality, accuracy, and diversity. The proposed

metrics in every category and the steps for calculating

them are presented as follows.

2.2.1 Cardinality Metric - CM

The cardinality metric (CM) represents the

domination degree of the solutions obtained by a

MOLP method. For this reason, it is a unary metric,

using the information of a unique solution set. In this

study, it is calculated using the INT-SBM model,

where data of all the solutions



𝑆



obtained by a

solution method are evaluated. It is important to

highlight that the dominated and non-dominated

solutions obtained by a solution method are

considered as observed data of the model. The

following steps must be carried out for obtaining the

cardinality metric CM.

Step 1: Execute the INT-SBM model for every

solution of set 𝑆. In this step, a vector Ξ is obtained,

which corresponds to the vector of 𝜉



, the efficient

measure of the INT-SBM model for every solution i

of the set 𝑆.

Step 2: Calculate the efficiency average of vector Ξ.

This value will correspond to the cardinality metric

CM.

The cardinality metric CM is greater than zero,

and lower than or equal to one. A value equal to one

means that it does not exist any solution dominated

by other in the set 𝑆. On the other hand, a value close

to zero means that few non-dominated solutions exist

in the set 𝑆.

2.2.2 Accuracy Metric - AC

The accuracy metric (AC) represents the domination

degree of one MOLP solution method over other

MOLP solution method. Furthermore, it is a binary

metric because it needs two sets of non-dominated

solutions for making the comparison. In this study,

for estimating the accuracy metric AC, the INT-SBM

model and the metafrontier approach, proposed by

O’Donnell et al. (2008), are used together. The

metafrontier approach allows classifying the non-

dominated solutions into different groups. In this

way, two sets of non-dominated solutions, 𝑆



and 𝑆



obtained by two different solution methods, are

compared. The following steps must be carried out for

estimating the proposed accuracy metric AC.

Step 1: Execute the INT-SBM model for every non-

dominated solution of set 𝑆



. In this step, a vector Ξ



is obtained, which corresponds to the vector of 𝜉



the efficient measure of the INT-SBM model for

every non-dominated solution i of the set 𝑆



Step 2: Execute the INT-SBM model for every non-

dominated solution of set 𝑆



. In this step, a vector Ξ



is obtained, which corresponds to the vector of 𝜉



the efficient measure of the INT-SBM model for

every non-dominated solution i of the set 𝑆



Step 3: Execute the INT-SBM model for every

solution belonging to the union of sets 𝑆



and 𝑆



. In

this step, a vector Ξ



is obtained, which corresponds

to the vector of 𝜉



, the efficient measure of the INT-

ICORES 2023 - 12th International Conference on Operations Research and Enterprise Systems

154

SBM model for solution i belonging to the union of

sets 𝑆



and 𝑆



Step 4: Separate the efficiency vector Ξ



in two sets,

efficiencies scores of solutions from set 𝑆



(Ξ





), and

efficiencies scores of solutions from set 𝑆



(Ξ





Step 5: Calculate the efficiency averages of vectors



, Ξ



, Ξ





, and Ξ





, individually.

Step 6: Make the difference between the efficiency

averages of vectors Ξ



and Ξ





, which corresponds to

, and between Ξ



and Ξ





, which corresponds to

Step 7: Calculate the minimum value between AC

and AC

. This value will correspond to the accuracy

metric AC.

It is important to notice that AC is greater than or

equal to zero, and lower than one. Moreover, the

solution method with the minimum value AC will be

the best method, meaning that this method obtains a

lower number of dominated solutions than the other

solution method.

2.2.3 Diversity Metric - DM

The diversity metric DM evaluates a change in the

Pareto frontier when a new solution is added. This is

a unary metric because it uses the information of a

unique solution set. In this study, the diversity metric

DM is calculated using the SE-DEA model. In this

model, the non-dominated solutions



𝑁𝑆



obtained

by a MOLP method are evaluated. The following

steps must be carried out for obtaining DM.

Step 1: Execute the SE-DEA model for every

solution of the set 𝑁𝑆. A vector Δ is obtained, which

corresponds to the vector of 𝛿



, that is, a vector of the

efficiency score obtained by the SE-DEA model for

every solution i of the set 𝑁𝑆.

Step 2: Identify the subset of 𝑁𝑆 that corresponds to

extreme solutions. These solutions are those that in

the step 1 obtained a 𝛿



value greater than one or the

respective SE-DEA model is infeasible. This subset,

denominated 𝐸𝑆, defines the Pareto frontier.

Step 3: Calculate DM using equation (20).

𝐷𝑀 

𝐸𝑆

𝑁𝑆

(20)

The diversity metric DM is greater than zero, and

lower than or equal to one. A value close to zero

means that most of the solutions are a linear

combination of extreme solutions in 𝐸𝑆. A value

equals to one means that all the solutions are not a

linear combination of other extreme solutions in 𝐸𝑆.

In this way, the best value for DM is one.

3 RESULTS

In this section, for calculating the proposed metrics,

the solutions obtained by two MOLP methods are

used. The solutions were obtained for a MOLP model

based on the tactical harvest planning model proposed

by Gómez-Lagos et al. (2021), where the same case

study used in this article was analysed. In this MOLP

model, the first objective corresponds to the harvest

costs’ minimization (Z

); the second objective

corresponds to the harvest days’ minimization (Z

);

and the third objective corresponds to the harvest fruit

in the optimal conditions’ maximization (Z

). The

two applied MOLP methods are two solution

strategies of the MO-GRASP algorithm (algorithms a

and b) (Martí et al., 2015).

Executing 1000 times every MO-GRASP

algorithm for solving the MOLP model, two sets of

solutions were obtained, 𝑆



and 𝑆



; one set of 1000

solutions for every algorithm. In Figure 1, the trade-

off between the objective function values obtained by

the set 𝑆



are represented. The first trade-off

corresponds to Z

and Z

; the second, Z

and Z

; and

the third, Z

and Z

Figure 1: Objective function values of set 𝑆



A conflict between the objective functions can be

observed in Figure 1 because when an objective

function improves, the other deteriorates.

Performance Metrics Based on Data Envelopment Analysis for Evaluating Multi-Objective Linear Programming Solution Methods

155

Figure 2: Objective function values of set 𝑆



Figure 2 represents the trade-off between the

objective function values obtained by the set 𝑆



The conflict between the objective functions is

also observed in Figure 2.

Table 3 summarizes metrics calculated for both

sets of solutions, 𝑆



and 𝑆



. In this way, it can be

observed that the set 𝑆



obtains the best value for the

cardinality metric CM (0.980). Regarding the

accuracy metric AC, the set 𝑆



again achieve the best

value (0.999), meaning that main of solutions of 𝑆



are not dominated by the solutions of 𝑆



. Finally, for

the diversity metric DM, both sets obtain low values.

However, the set 𝑆



obtains the best value (0.131),

meaning that around 13% are extreme-efficient

solutions, that is, define the Pareto frontier.

Table 3: Values of the performance metrics for 𝑆



and 𝑆



Set of

Solutions

CM AC DM

𝑆



0.980 0.999 0.101

𝑆



0.977 0.962 0.131

From the values presented in Table 3, it could be

suggested to select the algorithm a for solving the

MOLP model because it has a best performance in the

binary metric AC, and in the unary metric CM.

Furthermore, for the unary metric DM, around 10%

the solutions obtained by the algorithm a are extreme-

efficient, close to DM obtained by the algorithm b.

4 CONCLUSIONS

In this study, three performance metrics based on

DEA models for evaluating MOLP solution methods

were proposed. The considered metrics are associated

to cardinality, accuracy, and diversity categories. A

procedure for calculating every metric is presented

and applied to a real case study, where two MO-

GRASP algorithms were compared. Therefore, the

two sets of solutions obtained by every algorithm

were used for estimating the metrics. For the

cardinality and accuracy metrics based on the INT-

SBM efficiency score, it was possible to discriminate

among the non-dominated solutions, independently

of the frontier region where they were located. For the

diversity metric based on the SE-DEA efficiency

score, it was possible to identify the non-dominated

solutions that determine the Pareto frontier. In this

way, these metrics allow discriminating between the

MOLP solution methods and even to select one.

For future research, it could be interesting to

explore DEA models where zero or negative values

can be incorporated in the set of solutions. In addition,

new DEA models could be explored in order to

identify the non-dominated solutions located in non-

convex regions of the Pareto frontier.

ACKNOWLEDGEMENTS

DSc Marcela González-Araya would like to thank

FONDECYT Project 1191764 (Chile) for its financial

support. Ms Javier Gómez-Lagos would like to thank

CONICYT PFCHA/BECA DE DOCTORADO

NACIONAL/2019 under Grant 21191364 for its

financial support.

REFERENCES

Andersen, P., & Petersen, N. (1993). A Procedure for

Ranking Efficient Units in Data Envelopment Analysis

(Vol. 39, Issue 10). https://about.jstor.org/terms

Audet, C., Bigeon, J., Cartier, D., le Digabel, S., & Salomon,

L. (2021). Performance indicators in multiobjective

optimization. In European Journal of Operational

Research (Vol. 292, Issue 2, pp. 397–422). Elsevier B.V.

https://doi.org/10.1016/j.ejor.2020.11.016

Bal, A., & Satoglu, S. I. (2019). The use of data

envelopment analysis in evaluating pareto optimal

solutions of the sustainable supply chain models.

Procedia Manufacturing, 33, 485–492. https://

doi.org/10.1016/j.promfg.2019.04.060

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some

models for estimating technical and scale inefficiencies

ICORES 2023 - 12th International Conference on Operations Research and Enterprise Systems

156

in data envelopment analysis. Management Science,

30(9), 1078–1092. https://doi.org/https://doi.org/10.

1287/mnsc.30.9.1078

Charnes, A., Cooper, W., & Rhodes, E. (1978). Measuring

the efficiency of decision making units. In Company

European Journal of Operational Research (Vol. 2).

Charnes, A., & Cooper, W. W. (1962). PROGRAMMING

WITH LINEAR FRACTIONAL FUNCTIONALS.

Naval Research Logistics Quarterly, 181–186.

https://doi.org/https://doi.org/10.1002/nav.3800090303

Deb, K. (2014). Multi-objective optimization. In Search

Methodologies: Introductory Tutorials in Optimization

and Decision Support Techniques, Second Edition (pp.

403–450). Springer US. https://doi.org/10.1007/978-1-

4614-6940-7_15

Demir, E., Bektaş, T., & Laporte, G. (2014). The bi-

objective Pollution-Routing Problem. European

Journal of Operational Research, 232(3), 464–478.

https://doi.org/10.1016/j.ejor.2013.08.002

Gómez-Lagos, J. E., González-Araya, M. C., Soto-Silva,

W. E., & Rivera-Moraga, M. M. (2021). Optimizing

tactical harvest planning for multiple fruit orchards

using a metaheuristic modeling approach. European

Journal of Operational Research, 290(1), 297–312.

https://doi.org/10.1016/j.ejor.2020.08.015

Hong, J. D., & Jeong, K. Y. (2019). Combining data

envelopment analysis and multi-objective model for the

efficient facility location–allocation decision. Journal

of Industrial Engineering International, 15(2), 315–

331. https://doi.org/10.1007/s40092-018-0294-2

Karatas, M., & Yakıcı, E. (2018). An iterative solution

approach to a multi-objective facility location problem.

Applied Soft Computing Journal, 62, 272–287.

https://doi.org/10.1016/j.asoc.2017.10.035

Martí, R., Campos, V., Resende, M. G. C., & Duarte, A.

(2015). Multiobjective GRASP with path relinking.

European Journal of Operational Research, 240(1),

54–71. https://doi.org/10.1016/j.ejor.2014.06.042

Mirzapour Al-E-Hashem, S. M. J., Malekly, H., &

Aryanezhad, M. B. (2011). A multi-objective robust

optimization model for multi-product multi-site

aggregate production planning in a supply chain

under uncertainty. International Journal of Production

Economics, 134(1), 28–42. https://doi.org/10.1016/j.

ijpe.2011.01.027

O’Donnell, C. J., Rao, D. S. P., & Battese, G. E. (2008).

Metafrontier frameworks for the study of firm-level

efficiencies and technology ratios. Empirical

Economics, 34(2), 231–255. https://doi.org/10.1007/

s00181-007-0119-4

Riquelme, N., von Lücken, C., & Barán, B. (2015,

December 16). Performance metrics in multi-objective

optimization. Proceedings - 2015 41st Latin American

Computing Conference, CLEI 2015.

https://doi.org/10.1109/CLEI.2015.7360024

Seiford, L. M., & Zhu, J. (1999). Infeasibility of super-

efficiency data envelopment analysis models. INFOR

Journal, 37(2), 174–187. https://doi.org/10.1080/

03155986.1999.11732379

Tone, K. (2001). A slacks-based measure of efficiency in

data envelopment analysis. European Journal of

Operational Research, 130(3), 498–509. https://

doi.org/10.1016/S0377-2217(99)00407-5

Varas, M., Basso, F., Maturana, S., Osorio, D., & Pezoa, R.

(2020). A multi-objective approach for supporting wine

grape harvest operations. Computers and Industrial

Engineering, 145(January), 106497. https://doi.org/

10.1016/j.cie.2020.106497

Performance Metrics Based on Data Envelopment Analysis for Evaluating Multi-Objective Linear Programming Solution Methods

157