GROUP DECISION SYSTEMS FOR RANKING AND SELECTION
An Application to the Accreditation of Doping Control Laboratories
Xari Rovira, N
´
uria Agell
ESADE Universitat Ramon Llull, Av. Pedralbes 62, Barcelona, Spain
M
´
onica S
´
anchez, Francesc Prats
Universitat Polit
`
ecnica de Catalunya, Dept MA2, Jordi Girona, 1-3, Barcelona, Spain
Montserrat Ventura
Institut Municipal d’Investigaci
´
o M
`
edica (IMIM, IMAS), Doctor Aiguader, 80, Barcelona, Spain
Universitat Aut
`
onoma de Barcelona (UAB), Facultat de Medicina, Doctor Aiguader, 80, Barcelona, Spain
Keywords:
Qualitative reasoning, Group decision-making, Goal programming.
Abstract:
This paper presents a qualitative approach for representing and synthesising evaluations given by a team of
experts involved in selection or ranking processes. The paper aims at contributing to decision-making analysis
in the context of group decision making. A methodology is given for selecting and ranking several alternatives
in an accreditation process. Patterns or alternatives are evaluated by each expert in an ordinal scale. Qualitative
orders of magnitude spaces are the frame in which these ordinal scales are represented. A representation for the
different patterns by means of k-dimensional qualitative orders of magnitude labels is proposed, each of these
standing for the conjunction of k labels corresponding to the evaluations considered. A method is given for
ranking patterns based on comparing distances against a reference k-dimensional label. The proposed method
is applied in a real case in External Quality Assessment Schemes (EQAS) for Doping Control Laboratory
contexts.
1 INTRODUCTION
In group decision-making processes, the ranking or
selection of patterns depends on the judgments given
by the evaluators in the group. The group decision
method presented in this work is specially suitable
when aiming at an evaluation via qualitative ordinal
descriptions.
This work is a step further in terms of previous
works in the area of multi-criteria decision making
(Agell et al, 2006; Keeney, 1993) when information
is represented by absolute orders of magnitude la-
bels, such as ”very bad”, ”bad”, ”acceptable”, ”good”
and ”very good”. The proposed method is based on
the representation of the initial evaluators’ judgments
of each pattern via a k-dimensional label, which can
be seen as a hyper-rectangle, and its synthesis by
means of the distance to a reference k-dimensional
label. This in turn is based on a qualitative generalisa-
tion of a type of goal programming method known as
the reference point method for vectorial optimisation
and decision-making support (Gonza, 2001; Kallio,
1980).
In general, reference point methods for optimi-
sation in R
n
choose the points at a shorter distance
from a previously fixed reference point as the opti-
mal alternative (the ”goal” to be reached) (Romero,
2001; Romero et al, 2001; Wier, 1980). In this
work, ranking in the set of existing patterns is per-
formed by selecting not a predetermined fixed refer-
ence k-dimensional label, but a ”realistic” reference
for the problem to be solved: with respect to the nat-
ural order, the proposed reference hyper-rectangle is
the supreme of the set of available patterns, guar-
82
Rovira X., Agell N., Sánchez M., Prats F. and Ventura M. (2007).
GROUP DECISION SYSTEMS FOR RANKING AND SELECTION - An Application to the Accreditation of Doping Control Laboratories.
In Proceedings of the Ninth International Conference on Enterprise Information Systems - AIDSS, pages 82-87
DOI: 10.5220/0002389300820087
Copyright
c
SciTePress
anteeing consistency with the order between hyper-
rectangles. The distances between patterns and their
supreme give the ranking of patterns directly. More-
over, any problem of selection or accreditation of pat-
terns can be solved by using this ranking together with
a suitable threshold.
The proposed methodology may be of interest to
very different areas. Specifically, applications in can-
didate assessments (students in learning processes, re-
cruiting processes) as well as project management (ar-
chitectural, civil engineering and business projects)
can be considered. It may also be of interest in
decision-making processes in areas such as finance
and marketing.
The proposed method is applied in the evaluation
of data from an External Quality Assessment Scheme
(EQAS) of accredited doping control laboratories.
Laboratories involved in doping control have to be
accredited by both the World Anti-Doping Agency
(WADA) and by the national accreditation body fol-
lowing the ISO17025 quality standard (World Anti-
Doping Agency, 2004). This regulatory system is jus-
tified, on the one hand, to insure the highest quality
in analytical standards in benefit of the athlete, and,
on the other hand, to harmonise quality standards in
laboratories regularly involved in legal disputes con-
cerning positive test results (i.e. a substance banned
by sport authorities). WADA accreditation is granted
after the evaluation of analytical data provided by lab-
oratories by an experts’ committee based on the in-
formation given to each expert, the judgments of each
evaluator and a further synthesis of these judgments.
Section 2 presents some features related to the
qualitative models of absolute orders of magnitude,
and Section 3 provides a qualitative representation
of patterns in a partially-ordered set. Section 4 in-
troduces the new group decision-making process by
definig a total order in the set of patterns in such a way
that the set of labels corresponding to the available
alternatives becomes a chain (ranking). The consis-
tency property for the ranking method is established.
In Section 5, the application to accreditation of dop-
ing control laboratories is given. Lastly, conclusions
and open problems are presented.
2 ABSOLUTE ORDERS OF
MAGNITUDE MODELS
The one-dimensional absolute orders of magnitude
model (Trave, 2003) works with a finite number of
qualitative labels corresponding to an ordinal scale of
measurement. The number of labels chosen to de-
scribe a real problem is not fixed, but depends on the
characteristics of each represented variable.
Let’s consider an ordered finite set of basic labels
S
= {B
1
,...,B
n
}, each one of them corresponding to
a linguistic term, in such a way that B
1
< ... < B
n
,
as for instance “very bad” < “bad” < “acceptable” <
“good” < “very good”.
The complete universe of description for the Or-
ders of Magnitude Space OM(n) is the set S:
S = S
{[B
i
,B
j
] |B
i
,B
j
S
,i < j},
where the label [B
i
,B
j
] with i < j is defined as the set
{B
i
,B
i+1
,...,B
j
}.
The order in the set of basic labels S
induces a
partial order in S defined as:
[B
i
,B
j
] [B
r
,B
s
] B
i
B
r
and B
j
B
s
. (1)
with the convention [B
i
,B
i
] = {B
i
} = B
i
. This relation
is trivially an order relation in S, but a partial order,
since there are pairs of non-comparable labels.
3 PATTERN REPRESENTATION
IN A PARTIALLY ORDERED
SET
In the proposed group decision-making problem, each
pattern is characterized by the judgments of k eval-
uators, and these evaluations are given by means of
qualitative labels belonging to an orders of magni-
tude space. So, each pattern is represented by a k-
dimensional label, that is to say, a k-tuple of labels.
Let S be the orders of magnitude space with the
set of basic labels S
.
The set of k-dimensional labels or patterns’ repre-
sentations E is defined as:
E = S×
k
... ×S =
=
{
(E
1
,...,E
k
) | E
i
S i = 1,...k
}
. (2)
Each k-dimensional label E = (E
1
,...,E
k
) is a set
of k qualitative labels (each one associated to an ex-
pert judgment) that define a pattern in such a way that,
on every component, the relation E
r
E
s
means that
E
s
represents better results than E
r
.
This order in S is extended to the Cartesian prod-
uct E:
E = (E
1
,...,E
k
) E
= (E
1
,...,E
k
)
E
i
E
i
, i = 1,...,k. (3)
E < E
, that is to say, E E
and E 6= E
, means
that pattern E
is better than pattern E.
This order relation in E is partial, since there are
pairs of non-comparable k-dimensional labels.
GROUP DECISION SYSTEMS FOR RANKING AND SELECTION - An Application to the Accreditation of Doping
Control Laboratories
83
E E
E E
, E
E
Figure 1: The partial order in E.
4 THE GROUP
DECISION-MAKING PROCESS
The proposed method for ranking or selection among
the existing patterns consists in :
Fixing a distance d in E.
Building a reference label
˜
E: a proposition of con-
sistency determines that it has to be the supreme,
with respect to the order , of the set of patterns
which are to be ranked.
Assigning to each k-dimensional label E the value
d(E,
˜
E); so, the patterns will be totally ordered as
a chain.
This chain giving the ranking of the patterns di-
rectly. In an accreditation process, this chain,
along with a threshold, solves the problem.
After this process, if a subset of the patterns is at
the same distance to
˜
E, the same algorithm being
applied to this set, just beginning at the second
point.
4.1 A Distance in the Pattern
Representation Set
A method for computing distances between k-
dimensional labels is presented.
The first step involves codifying the labels in S by
a location function (Agell et al, 2006). Through this
function, each element E
h
= [B
i
,B
j
] in S is codified by
a pair (l
1
(E
h
),l
2
(E
h
)) of integers: l
1
(E
h
) is the oppo-
site of the number of basic elements in S
that are be-
tween B
1
and B
i
, that is to say, l
1
(E
h
) = (i 1), and
l
2
(E
h
) is the number of basic elements in S
that are
between B
j
and B
n
, i.e., l
2
(E
h
) = n j. This pair of
numbers permits each element in S, where all differ-
ent levels of precision are considered, to be ”located”.
This “location” can be extended to any pattern de-
fined by k orders of magnitude labels; the extension
to the set of k-dimensional labels E is:
L(E
1
,...,E
k
) = (l
1
(E
1
),l
2
(E
1
),...l
1
(E
k
),l
2
(E
k
)) (4)
which provides the relative position of a k-tuple of
qualitative labels with respect to the basis of E.
Then, a distance d between labels E,E
in S is de-
fined via any metric R in R
2k
and their codifications:
d(E, E
) = d((E
1
,...,E
k
),(E
1
,...,E
k
)) =
q
(L(E) L(E
))
t
R(L(E) L(E
)). (5)
This function inherits all properties of the distance in
R
2k
.
4.2 Ranking of the Patterns
Starting from a distance d in E and a reference k-
dimensional label
˜
E, a total order E will be defined in
E, in such a way that the set of labels E
1
,...,E
l
cor-
responding to the available patterns becomes a chain
E
i
1
E ··· E E
i
l
, and so a ranking of the patterns is es-
tablished.
4.2.1 A Total Order in E
Let
˜
E E be a k-dimensional label and let us call it
the reference label. Let d be the distance defined in E
in Section 4.1. Then the following binary relation in
E:
E E
d(E
,
˜
E) d(E,
˜
E) (6)
is a pre-order, i.e., it is reflexive and transitive.
This pre-order relation induces an equivalence re-
lation in E by means of:
E E
[E E
, E
E]
d(E
,
˜
E) = d(E,
˜
E). (7)
Then, in the quotient set E/ the following rela-
tion between equivalence classes:
class(E) E class(E
) E E
d(E
,
˜
E) d(E,
˜
E) (8)
is an order relation. It is trivially a total order.
In this way, given a set of patterns E
1
,...,E
l
,
these can be ordered as a chain with respect to their
proximity to the reference label: class(E
i
1
) E · · · E
class(E
i
l
).
If each class(E
i
j
), j = 1,...l, contains only the la-
bel E
i
j
, the process is finished and we obtain the rank-
ing E
i
1
· · · E
i
l
. If there is some class(E
i
j
) with
more than one label, then the same process is applied
to the set of the labels belonging to class(E
i
j
), and
continued until the final ranking E
m
1
· · · E
m
l
is
obtained.
ICEIS 2007 - International Conference on Enterprise Information Systems
84
4.2.2 Consistency of the Ranking Method
The method of ranking via a distance to a reference
label previously selected is really necessary when the
order relation does not provide a total order in the
set of available patterns.
When the set {E
1
,...,E
l
} is totally ordered with
respect to , that is to say, when a priori the patterns
are already ranked E
i
1
··· E
i
l
, then the proposed
method (via choosing of a suitable reference label)
has to reproduce the same ranking. This means that
the method has to be consistent.
Formally, the method will be consistent if the ref-
erence label
˜
E is selected in such a way that:
(E
1
,...,E
l
E)(E
1
··· E
l
=
E
1
E ··· E E
l
) (9)
This requirement is equivalent to the following:
(E,E
E)(E E
= EE E
) (10)
In effect, (9) = (10) is obvious. Reciprocally, if
(10) is satisfied, then when E
1
··· E
l
it suffices
to apply (10) to each pair E
i
E
i+1
.
Before establishing the way of choosing the refer-
ence label
˜
E, in order for property of consistency to
be accomplished, let us compute the supreme of a set
of hyper-rectangles with respect to the partial order
introduced in Section 3.
Given any E
1
,...,E
l
, let E
sup
be the supreme of
the set {E
1
,...,E
l
}, i.e., the minimum label in E
which satisfies E
i
E
sup
,i = 1,··· ,l.
Its computation is as follows:
Let E
r
= (E
r
1
,...,E
r
k
), with E
r
h
= [B
r
i
h
,B
r
j
h
] for all
h = 1,...,k, and for all r = 1,...,l. Then
E
sup
= sup{E
1
,...,E
l
} = (
˜
E
1
,...,
˜
E
k
),
where
˜
E
h
= [max{B
1
i
h
,...,B
l
i
h
},max{B
1
j
h
,...,B
l
j
h
}], (11)
(see Figure 2).
E
1
E
2
E
3
sup{E
1
,E
2
,E
3
}
Figure 2: The supreme of a set {E
1
,...,E
l
}.
Proposition 3 (of consistency) The ranking method is
consistent in the above sense if and only if, for any set
of patterns E
1
,...,E
l
, the reference label
˜
E is chosen
as the supreme of the set {E
1
,...,E
l
}.
Proof. If the label associated to any set of labels
is its supreme, statement (10) is trivial, because if
E E
then E
= sup{E,E
} =
˜
E and d(E
,
˜
E) = 0
d(E,
˜
E), that is to say, EE E
.
To prove that it is necessary to choose the supreme
to assure the consistency, it suffices to present as a
counterexample of (10) the case in which
˜
E is not
the supreme. The easiest of these consists of the pair
E,E
, with E E
and
˜
E = E. It is clear that E/E E
.
5 AN APPLICATION TO
ACCREDITATION OF DOPING
CONTROL LABORATORIES
There is considerable concern over the abuse of drugs
by athletes trying to improve their performance. Dop-
ing in sport is becoming increasingly sophisticated
and sampling methods and testing procedures vary
from one country to another. The absence of harmon-
isation in this area has led to an increasing number of
doping accusations being contested (Warmuth, 2002).
Establishing an external quality assurance scheme for
all Doping Control Laboratories is necessary to in-
crease the legal weight behind drugs tests and reduce
the number of positive drug tests being challenged in
the courts (Donike, 1992). Experimental data evalu-
ated in the present report were generated in the frame-
work of the EU funded project ALADIN 2002 (An-
alytical Laboratories for Antidoping Control: Inter-
national Network for External Quality Assessment),
coordinated by the Institut Municipal d’Investigaci
´
o
M
`
edica (IMIM) in Barcelona, in close collaboration
with the IOC laboratories in London (United King-
dom), Cologne (Germany) and Oslo (Norway), that
aims to develop the external quality assurance scheme
that would meet these needs. The group decision sys-
tem developed in this work is applied to summarise
the opinion of experts and to order the laboratories
involved in the study according to their quality.
5.1 Description of Data
Nine independent experts evaluated analytical data
provided by laboratories when analyzing samples
containing a banned substance (nandrolone) in sev-
eral rounds of the EQAS spread in four consecutive
years, according to their expertise. A total of 105 re-
ports generated by laboratories, were evaluated. Eval-
uations of each analytical report were rated as fol-
lows: Unacceptable; Insufficient; Sufficient; Good;
and Very Good. No intermediate scores were pos-
sible. All experts were management staff from four
GROUP DECISION SYSTEMS FOR RANKING AND SELECTION - An Application to the Accreditation of Doping
Control Laboratories
85
European IOC/WADA accredited anti-doping control
laboratories (King’s College London, United King-
dom; Hormone Laboratory in Oslo, Norway; Ger-
man Sports University in Cologne, Germany; and In-
stitut Municipal d’Investigaci
´
o M
`
edica in Barcelona,
Spain). All of the experts have excellent skills in the
analysis and evaluation of doping control samples and
they have more than ten years’ experience in this an-
alytical field.
Each laboratory has initially been considered as
a 9-tuple of basic qualitative labels belonging to an
OM(5), where the basic elements are: B1= Unac-
ceptable, B2= Insufficient, B3= Sufficient, B4= Good
and B5= Very Good.The reference point -the ”opti-
mal laboratory”- is chosen as the maximum basic la-
bel assigned to all laboratories for each expert. Then
the distance, i.e., level of quality (LQ) from each pat-
tern to the optimal is used to rank the laboratories in
the sample. As shown in Section 4, LQ established a
total order in the set of laboratories, where the smaller
LQ is, the better The quality of the laboratory. As an
example, seven of the patterns are shown in Table 1, in
which distances to the optimal laboratory (LQ) and a
refined order (RLQ) are included in the last columns.
As can be seen in Table 1, some patterns are
equidistant to the “optimal laboratory”. To avoid co-
incidence between distances as much as possible, and
to be able to discriminate laboratories, the group deci-
sion process is reapplied in each group with the same
level of quality, by defining a new ad-hoc reference
point. In that sense, a refined order of the initial set of
patterns (RLQ) is obtained. These can be seen in the
last column of Table 1 for the example considered.
It is important to notice that, although in this case
experts have been asked to evaluate each laboratory
with basic labels, this group decision methodology
can be considered with evaluations including differ-
ent levels of precision and even missing values.
5.2 Experiment and Results
According to the ratings assigned to the 105 reports
by the team of experts, these have been first ordered
by LQ values. The distribution of the results obtained
is shown in Figure 3.
As it can be seen in Figure 3, there are 14 pairs
of indistinguishable laboratories with respect to their
distance to the ”optimal laboratory”, 7 groups of three
laboratories with the same LQ, 4 groups with four
laboratories equally ranked and 3 groups of five of
them. As an example, it can be considered the case
given in table 1 by the laboratories coded as #64, #65,
#55,#17,#70. These ve laboratories have the same
LQ equal to 5,4, so they form one of the three groups
of five laboratories previously described. In a sec-
ond step, the 28 groups in which laboratories are not
distinguished are internally re-ranked and scored with
the RLQ value. Following the former example, table
1 shows the new values allowing a better distinction
in terms of their position to the ”optimal laboratory”.
This second step produces a final order, making it pos-
sible to rank almost all the laboratories in the sam-
ple. At this point, if the group decision desired is a
binary classification (in an accreditation process) ex-
perts will decide the LQ value to be considered as the
threshold.
Stem width: 1,00
Each leaf: 1 case (s)
Figure 3: Distribution of LQ values.
6 CONCLUSION AND FUTURE
RESEARCH
This paper proposes a methodology that synthesises
evaluations given by an experts’ committee. Evalua-
tions are considered in an ordinal scale, for this rea-
son a set of labels describing orders of magnitude is
considered. A group decision method is given in or-
der to rank the patterns based on comparing distances
to a reference k-dimensional label. The methodology
presented allows, on the one hand, the ordinal infor-
mation given by experts on the specific application to
be handled without previous normalisation, and, on
the other, the methods of “goal programming” to be
generalised without the need for previous knowledge
of the ideal goal.
The results applied to a real case show the appli-
cability of the methodology. In the experiment a set
of laboratories are evaluated by a group of experts
ICEIS 2007 - International Conference on Enterprise Information Systems
86
Table 1: Level of quality and Refined level of quality.
code Exp1 Exp2 Exp3 Exp4 Exp5 Exp6 Exp7 Exp8 Exp9 LQ RLQ
105 B
4
B
5
B
3
B
5
B
5
B
4
B
5
B
5
B
5
3,4
74 B
5
B
4
B
5
B
4
B
4
B
3
B
5
B
5
B
5
3,7
45 B
4
B
5
B
4
B
4
B
4
B
4
B
4
B
4
B
4
4,0
64 B
4
B
5
B
4
B
5
B
4
B
3
B
5
B
3
B
3
5,4 2,8
65 B
5
B
5
B
4
B
4
B
4
B
3
B
5
B
3
B
3
5,4 2,8
55 B
3
B
4
B
4
B
4
B
4
B
4
B
3
B
4
B
4
5,4 4
17 B
4
B
4
B
4
B
4
B
4
B
4
B
3
B
3
B
4
5,4 4
70 B
4
B
4
B
4
B
4
B
4
B
3
B
3
B
4
B
4
5,4 4,5
that provide their individual inputs for some poste-
rior manipulation. It is important to point out that
the methodology presented allows imprecision in the
evaluations given by the experts to be considered.
As a future work the design of an automatic sys-
tem to perform the group decision process described
will be implemented. In addition, within the AURA
project framework, it is planned to build up an arti-
ficial intelligence application that includes a learning
machine able to interpret and evaluate the laborato-
ries’ analytical reports and a software tool to imple-
ment the presented methodology.
ACKNOWLEDGEMENTS
This research has been partially supported by the
AURA research project (TIN2005-08873-C02-01 and
TIN2005-08873-C02-02), founded by the Spanish
Ministry of Science and Information Technology,
and by the EU Growth Programme (ALADIN 2002,
G7RT-CT-2000-05022). The authors would like to
thank Dr. Rafael de la Torre and Dr. Rosa Ventura at
Institut Municipal d’Investigaci
´
o M
`
edica (Barcelona)
for compiling the data used in this study and for their
helpful discussions and suggestions.
REFERENCES
Agell, N., S
´
anchez, M., Prats, F. & Rovira, X., 2006. Us-
ing Orders of Magnitude in Multi-attribute Decision-
making. In Proc. of the 20th International Workshop
on Qualitative Reasoning . Hanover, New Hampshire,
15-19.
Donike, M., 1992. Accreditation and Reaccreditation of
Laboratories by the IOC Medical Commission. In
First International Symposium on Current Issues of
Drug Abuse Testing.
Gonz
´
alez Pach
´
on, J., & Romero L
´
opez, C., 2001. ”Aggre-
gation of Partial Ordinal Rankings. An Interval Goal
Programming Approach”. Computers and Operations
Research. 28, 827-834.
Kallio, M., Lewandowski, A., & Orchard-Hays, W., 1980.
An Implementation of the Reference Point Approach
for Multi-Objective Optimization. In WP-80-35,
IIASA, Luxemburg.
Keeney, R.L., & Raiffa, H., 1993. Decisions with multi-
ple objectives preferences and value trade-offs. Cam-
bridge University Press.
Romero L
´
opez, C., 2001. Extended Lexicographic Goal
Programming: A Unifying Approach. The Interna-
tional Journal of Management Science, 29, 63-71.
Romero L
´
opez, C., Tamiz, M., & Jones, D., 2001. Com-
ments on Goal Programming, Pompromise Program-
ming and Reference Point Method Formulations:
Linkages and Utility Interpretations-A Reply. Journal
of the Operational Research Society, 52, 962-965.
Trav
´
e-Massuy
`
es, L., & Dague, P., 2003. Mod
`
eles et raison-
nements qualitatifs. Ed. Lavoisier, Hermes Science,
Paris, France.
Warmuth, R
¨
atsch, G., Mathieson, M., Liao, J., & Lemmen,
C., 2002. ”Active Learning in the Drug Discovery
Process”. Advances in Neural Information Processing
Systems, 14. MIT Press.
Wierzbicki, A.P., 1980. The Use of Reference Objec-
tives in Multiobjective Optimization. In G. Fandel, T.
Gal (eds.): Multiple Criteria Decision Making; The-
ory and Applications, Lecture Notes in Economic and
Mathematical Systems. 177 Springer-Verlag, Berlin-
Heidelberg, 468-486.
World Anti-Doping Agency, 2004. The World Anti-Doping
Code. International Standard for Laboratories (ISL).
GROUP DECISION SYSTEMS FOR RANKING AND SELECTION - An Application to the Accreditation of Doping
Control Laboratories
87