Towards the Layered Evaluation of Interactive Adaptive Systems
using ELECTRE TRI Method
Amira Dhouib
1
, Abdelwaheb Trabelsi
2
, Christophe Kolski
3
and Mahmoud Neji
1
1
Miracl Laboratory, Faculty of Economics and Management Sciences, University of Sfax, B.P. 1088, Sfax 3000, Tunisia
2
College of Computation and Informatics, Saudi Electronic University, Dammam, Saudi Arabia
3
LAMIH-UMR CNRS 8201,University of Valenciennes and Hainaut-Cambrésis, Valenciennes, France
Keywords: Decision Process, Evaluation, Interactive Adaptive System, Multi-criteria Decision Analysis,
ELECTRE TRI.
Abstract: The layered evaluation of interactive adaptive systems has to consider many evaluation methods. The best
evaluation method to be used for individual layers depend on many parameters such as the evaluation
criteria, the stage of the development cycle, and the characteristics of the layer under consideration. This
paper presents a decision model for selecting the appropriate evaluation methods for individual layers of the
interactive adaptive system. Our proposal is based on one multi-criteria method, namely ELECTRE TRI
method. The proposed decision model is applied to determine the suitable evaluation methods for an
adaptive hypermedia system.
1 INTRODUCTION
Nowadays Interactive Adaptive Systems (IAS) are
omnipresent in many application domains such as
education, tourism, and e-commerce (Brusilovsky et
al., 1996; Goren-Bar et al., 2005; Goy et al., 2006).
The evaluation of these systems is an important part
of their development process. Several studies have
reported the advantages of the layered evaluation
approach for interactive adaptive systems
(Brusilovsky et al., 2001; Weibelzahl, 2001;
Paramythis et al., 2010). By applying this approach,
each layer of adaptation is assessed individually
where feasible (Paramythis et al., 2010). A layer of
adaptation refers to a particular step in the
adaptation process of IAS (Paramythis et al., 2010).
In every layer, different evaluation methods can be
applied. It is essential to choose the appropriate
evaluation methods for individual layers in
particular evaluation constraints (Paramythis et al.,
2010). The literature has identified numerous
evaluation methods for interactive adaptive systems
(Gena, 2005; Gena and Weibelzahl, 2007; Velsen et
al., 2008; Mulwa et al., 2011; Dhouib et al., 2016a).
Some of them, like user-as-wizard (Masthoff, 2006),
are specific for the IAS field. The diversity of the
evaluation methods makes the choice of the best
ones in particular settings a difficult task. For
instance, evaluators need to understand the
suitability of each evaluation method in a particular
situation (Ferré and Bevan, 2011; Dhouib et al.,
2016a). They also have to consider different criteria
such as the availability of stakeholders, the system
development phase, the characteristics of layers, etc.
Using a particular Multi-Criteria Decision Analysis
(MCDA) method in the choice of the suitable
evaluation methods decision process is an important
strategy to deal with the presence of numerous
criteria.
In the interactive adaptive system literature, there
are few research studies that address the question of
the choice of evaluation methods for the individual
layers. Paramythis et al. (Paramythis et al., 2010) for
example, presented a framework that guides the
layered evaluation of IAS. The proposed framework
represents a revised version of the previous layered
frameworks, mainly those of (Paramythis et al.,
2001; Weibelzahl and Lauer, 2001; Brusilovsky et
al, 2004). The authors propose the evaluation criteria
related to every layer of adaptation and the methods
to be applied for their evaluation. Another study was
presented by (Dhouib et al., 2016b) in which
Analytic Hierarchy Process was used for the
selection of the best evaluation methods. In spite of
the major progress in IAS research, we still lack a
Dhouib, A., Trabelsi, A., Kolski, C. and Neji, M.
Towards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method.
DOI: 10.5220/0006437901630170
In Proceedings of the 12th International Conference on Software Technologies (ICSOFT 2017), pages 163-170
ISBN: 978-989-758-262-2
Copyright © 2017 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
163
decision model that deal the problem of selection of
suitable evaluation methods for individual layers
with a sorting problematic. Then, this paper presents
a novel decision model that assigns evaluation
methods to each layer of adaptation. This kind of
decision problem is known as a sorting problem
under the MCDA approach (Roy, 1996). A large
number of MCDA sorting methods are available in
the literature. In this research, we adopt one MCDA
method, namely the ELECTRE TRI method. This
MCDA method is considered as one of the most
commonly used MCDA methods for sorting
alternatives into predefined categories. More details
about the ELECTRE TRI method can be found in
(Yu, 1992; Roy and Bouyssou, 1993).
The rest of this paper is structured as follows.
First, we start by a description of the layered
evaluation of interactive adaptive systems and the
MCDA method adopted in this study (Section 2).
Afterward, we present the decision model that
guides IAS evaluators in the selection of appropriate
evaluation methods for individual layers (Section 3).
Following that, we illustrate our proposal with a case
study related to an adaptive hypermedia system in
order to validate it (Section 4). Finally, we present
some conclusions and future work (Section 5).
2 STATE OF THE ART
2.1 The Layered Evaluation of
Interactive Adaptive Systems
The main idea behind layered evaluation is to
separate the adaptation process into its components
(layers) and to assess them separately (Paramythis et
al., 2001; Paramythis et al., 2010). In the literature, a
number of layered evaluation approaches have been
proposed (Karagiannidis and Sampson, 2000;
Paramythis et al., 2001; Brusilovsky et al., 2004;
Paramythis et al., 2010, Manouselis et al., 2014).
These approaches differ essentially in the number of
layers identified. Paramythis et al. (Paramythis et
al., 2010) distinguished five layers for interactive
adaptive systems, including (1) collection of input
data in which data about the interaction context and
the user interaction are collected, (2) interpretation
of the collected data in which an interpretation of the
previously collected input data is conducted, (3)
modelling of the current state of the world in which
knowledge about the interaction context is
introduced in IAS' dynamic models, (4) deciding
upon adaptation in which the appropriate adaptation
is selected, and (5) applying adaptation which
reflects the step of introduction of the adaptations in
the user-system interaction. In every layer, different
evaluation methods can be applied in order to
identify in which layer the problem is.
2.2 Multi-criteria Decision Aid
2.2.1 Overview
Various MCDA methods have been developed to
facilitate the decision-making process. MCDA
methods consist of the three major concepts,
including:
The criteria which refer to the factors on which
the decision is based. The identification of
criteria is an essential step in the decision-
making process.
The alternatives which reflect the set of potential
solutions for the decision-making problem.
The preferences between two alternatives (a, b)
that can have three types, including (1)
preference aPb, which means that alternative a is
preferred to b, (2) indifference aIb, which means
that a is indifferent to b, and (3) incomparability
aRb, which means that a is incomparable to b.
The next section presents the adopted MCDA
method in the present research.
2.2.2 ELECTRE TRI
ELECTRE TRI is a sorting multi-criteria method. It
assigns a set of alternatives to predefined ordered
categories C (Yu, 1992; Mousseau and Slowinski,
1998). The assignment of alternatives into categories
is done by means of a comparison of these
alternatives with the profiles representing the
frontiers between categories. ELECTRE TRI assigns
alternatives to categories following two consecutive
steps, including (1) construction of an outranking
relation aSb
h, and (2) exploitation of the relation
aSb
h.
The ELECTRE TRI method builds an index σ(a,
b
h
) that represents the degree of credibility of the
assertion aSb
h
(where σ(a, b
h
) ͼ [0,1]). In order to
determine this index, the following items should be
calculated:
The partial concordance index C
j
(a,b
h
)
C
j
(a, b
h
) =
0ifg
b
g
a
p
b
1ifg
b
g
a
q
b
Otherwise
(1)
ICSOFT 2017 - 12th International Conference on Software Technologies
164
The global concordance index
C(a, b
h
)=
∑
a,b
h
∈
∑
∈
(2)
The discordance index
D
j
(a, b
h
)=
0ifg
b
g
a
p
b
1ifg
b
g
a
v
b
otherwise
(3)
The credibility index σ(a, b
h
) of the outranking
relation
σ(a, b
h
) =
Cj
(a, b
h
) .
∏
,
,
∈
(4)
Where, F¯ = { j ∈ F : d
j
(a, b
h
) > c(a, b
h
)}
The statement aSb
h
is considered valid if σ(a,b
h
)
≥ λ, where λ ∈ [0.5,1] (Mousseau et al., 2001).
Two assignment procedures can be evaluated
using ELECTRE TRI:
Pessimistic procedure: An alternative a is
assigned to the highest category such that aSb
h−1
.
Optimistic procedure: An alternative a is
assigned to the lowest category C
h
such that b
h
>
a.
More details about the ELECTRE TRI method can
be found in (Yu, 1992; Roy and Bouyssou, 1993;
Mousseau et al., 2001).
The next section describes the decision-making
problem and presents the decision process for the
choice of appropriate evaluation methods for the
layered evaluation.
3 THE PROPOSED DECISION
MODEL
3.1 Problem Definition
A variety of MCDA methods has been proposed in
order to solve the sorting decision problem. None of
these MCDA methods is able to solve all types of
decision-making situations (Guitouni and Martel,
1998). In order to identify the most appropriate one,
an analysis of the different MCDA methods is
conducted. In this analysis, we focus essentially on
the characteristics of the decision problem.
As already presented, the layered evaluation
consists in evaluating every layer of adaptation
independently of the others. Different evaluation
methods can be used for individual layers. Assigning
the alternative evaluation methods to predefined
layers corresponds to one of the problem statements
proposed by Roy (Roy, 1968). The choice of
appropriate evaluation methods in the case of the
layered evaluation can be formulated as a sorting
decision problem.
Given the presence of both quantitative and
qualitative criteria with different types of scales, the
ELECTRE TRI method seems to be an appropriate
MCDA method for the given decision problem. In
addition, ELECTRE TRI presents a powerful
MCDA method that affects the different alternatives
independently of each other. This has a significant
importance in terms of saving computing time when
a varied set of alternatives is presented.
3.2 Decision Process for the Choice of
Appropriate Evaluation Methods
for the Layered Evaluation
As already mentioned, the aim of this research is to
identify the appropriate evaluation methods for the
layered evaluation of interactive adaptive systems.
To this end, we use a specific MCDA method,
namely ELECTRE TRI. The different MCDA
methods need a set of alternatives that corresponds
to the possible solutions. In this study, the
considered alternatives are the evaluation methods.
In IAS literature, there is a variety of evaluation
methods for individual layers. Examples of these
evaluation methods include user-as-wizard
(Masthoff, 2006), focus group (Krueger and Casey,
2009), heuristic evaluation (Magoulas et al., 2003).
Evaluation methods differ in terms of many
criteria. In this study, six criteria, representing the
situation where each evaluation method would be
positioned, are considered. These criteria have
a quantitative and qualitative nature and include:
Layer's input data, which reflect the input data of
the adaptive system’s functionalities to be
evaluated by a layer. The input data can be either
shown to the participants or produced by them
(Paramythis et al., 2010). The evaluation of this
criterion is binary: we use 1 to represent the
shown input data and 0 otherwise;
Layer's output data, which refer to the data
produced by the layers. Like the input data, they
can be either shown or produced by the
stakeholders (Paramythis et al., 2010). This
Towards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method
165
criterion has a binary evaluation, 1 represents the
shown input data and 0 otherwise;
System development phase, which reflects the
moment in which a layer may be evaluated.
Paramythis et al., (Paramythis et al., 2010)
distinguishes three evaluation phases, namely (1)
the specification phase, which occurs when the
general functionality of a layer has been
produced, (2) the design phase, which occurs
when the design of the IAS has been completed
or partially completed, and (3) the
implementation phase which occurs in the
presence of a prototype of the system's
functionality;
Number of evaluators, which reflects the total
number of evaluators involved in the IAS
evaluation process. The evaluation of this
criterion can yield a grade between 0 and N
evaluators;
Number of users, which refers to the total
number of users involved in the layered
evaluation process of the interactive adaptive
systems. The assessment of this criterion can
yield a grade between 0 and N users;
Presence of real users, IAS evaluation can be
applied in the presence of representative or real
users. To make the use of this criterion in the
decision model possible, the assessment made of
this criterion is binary: we use 1 to represent the
presence of real users and 0 otherwise.
It should be noted here that the mentioned criteria
are not exhaustive and that other ones may be
included. In the next step, a performance table is
created. In which every evaluation method is
classified according to the considered criteria. This
classification is carried out through data collected
from different previous research studies such as
(Gena, 2005; Gena and Weibelzahl, 2007;
Paramythis et al., 2010).
Then, a number of technical parameters for
ELECTRE TRI have to be determined, namely the
veto thresholds, the profile limits between
categories, and the importance weights of criteria.
When using ELECTRE TRI method, the evaluator
has to give his/her preferences. It is important to
note that the use of MCDA methods such as
ELECTRE TRI method is based on the preference
relation from the construction of a coherent family
of criteria.
As already stated, two assignment procedures
using ELECTRE TRI method are available, namely
the optimistic and the pessimistic versions. In this
step, an analysis of the usefulness of the results in
each procedure is performed. The analyses
conducted in this step consist in verifying if the
different evaluation criteria that need to be assessed
in each layer are covered by the proposed evaluation
methods cover. In other words, a comparison is
conducted between the evaluation factors to be
assessed in the individual layers and the evaluation
criteria covered. Every evaluation method allows
assessing a number of evaluation factors in the
individual layers of IAS, and depending on the
context of use factors, a number of evaluation
factors must be assessed in the layers. Examples of
these evaluation criteria include transparency,
predictability, timeliness, privacy and trust,
appropriateness of adaptation, and unobtrusiveness.
In the final step, our proposal identifies the
assignment procedure in which the evaluation
factors that should be assessed in the individual
layers are covered. Once the assignment procedure
is identified, the final list of appropriate evaluation
methods will be generated. Figure 1 shows the
proposed decision model.
4 APPLICATION
This section investigates a case study in order to
illustrate the feasibility of the proposed decision
model. The research question addressed is "What are
the best evaluation methods for individual layers of
a specific adaptive hypermedia system?". In this
study, the considered adaptive hypermedia system
assists users in their information-seeking tasks by
presenting information about the vehicles' times
through a Web interface. The system adapts the
interfaces in such a way as to present the relevant
information about the user's destination (Dhouib et
al., 2015).
4.1 Exploration by ELECTRE TRI
Method
The first step corresponds to the identification of the
problem's characteristics. This involves also the
identification of alternative evaluation methods for
interactive adaptive systems and the criteria that
affect the choice of these methods. Table 1
illustrates the alternative evaluation methods
considered in this study. Due to the specificities of
the layered evaluation, the choice of alternative
evaluation methods has to take into account the
formative perspective (Paramythis et al., 2010).
As already stated, the application of the
ELECTRE TRI method needs the consideration of a
number of parameters such as the importance
ICSOFT 2017 - 12th International Conference on Software Technologies
166
weights of criteria. The construction of these
weights importance is carried out through an
elicitation process with the decision maker. Table 2
illustrates the importance weights of the identified
criteria.
Figure 1: Decision process for choosing the appropriate
evaluation methods for the layered evaluation of
interactive adaptive systems (IAS).
The following step continues with an elicitation
of the characteristics needed for the categories. In
our context, the categories represent the layers of
adaptation. The adaptation process of the considered
Table 1: The considered evaluation methods for
interactive adaptive systems.
Number
Evaluation Methods
a1 Focus group
a2 Use
r
test
a3 Heuristic evaluation
a4 Cognitive walkthrough
a5 User-as-wizard
a6 Simulated-users
a7 Co-discovery
a8 Play with layer
a9 Logging use
a10 Thinking-aloud protocol
a11 Interviews
a12 Wizard of Oz
a13 Cross-validation
a14 Data mining
a15 Scenario-based design
a16 Coaching
a17 Contextual design
a18 Retrospective testing
a19 Prototypes
a20 Questionnaires
adaptive hypermedia system is centered on the five
distinct layers of (Paramythis et al., 2010). The
defined layers are collection of input data,
interpretation of the collected data, modelling of the
current state of the world, deciding upon adaptation,
and applying adaptation. These layers have
respectively the following priorities: very high, high,
moderate, low, and very low. Four borders (b1, b2,
b3 and b4), which constitute the limits of the
different layers, are defined. Border b1, for example,
determines the limit between the collection of input
data and interpretation of the collected data layers,
while b2 reflects the limit between interpretation of
Table 2: Considered decision criteria and relative weights.
N
umbe
r
Decision Criteria Wei
g
hts
C1 System development phase 0.23
C2 Layer's input data 0.15
C3 Layer's output data 0.15
C4 Number of users 0.15
C5 Number of evaluators 0.20
C6 Presence of real users 0.12
Towards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method
167
the collected data and modelling of the current state
of the world layers. We use the default value of
ELECTRE TRI for the cut-off level λ, 0.76. This
value gives an intermediate level of strictness to
examination in order to help the assignment of
alternatives into the categories.
In the following stage, the binary relations
defined by aHb
i
are identified, where a represents
the alternatives evaluation method and b
i
the limit
profile between layers. Five steps are conducted,
namely (1) determination of the partial concordance
index, (2) identification of the global concordance
index, (3) calculation of the discordance index, (4)
determination of the credibility index, and (5)
identification of the relations of preference from the
determination of the cut-off level λ.
Table 3: Binary relations between alternatives and the
reference limits profiles.
a
i
Sb1 a
i
Sb2 a
i
Sb3 a
i
Sb4
a1Ib1 a1Sb2 a1Sb3 a1Sb4
a2Ib1 a2Ib2 a2Ib3 a2Sb4
a3Sb1 a3Sb2 a3Sb3 a3Sb4
a4Ib1 a4Ib2 a4Ib3 a4Sb4
a5Ib1 a5Rb2 a5Sb3 a5Sb4
a6Sb1 a6Sb2 a6Sb3 a6Rb4
a7Ib1 a7Ib2 a7Rb3 a7Sb4
a8Sb1 a8Sb2 a8Sb3 a8Sb4
a9Ib1 a9Ib2 a9Ib3 a9Sb4
a10Ib1 a10Ib2 a10Ib3 a10Sb4
a1IIb1 a1IRb2 a1IIb3 a1IIb4
a12Ib1 a12Ib2 a12Ib3 a12Sb4
a13Sb1 a13Sb2 a13Rb3 a13Ib4
a14Sb1 a14Sb2 a14Rb3 a14Ib4
a15Ib1 a15Ib2 a15Sb3 a15Sb4
a16Ib1 a16Ib2 a16Ib3 a16Sb4
a17Ib1 a17Ib2 a17Ib3 a17Sb4
a18Ib1 a18Ib2 a18Ib3 a18Sb4
a19Ib1 a19Ib2 a19Ib3 a19Sb4
a20Ib1 a20Ib2 a20Ib3 a20Sb4
Table 3 presents the binary relations between
alternatives and the reference actions in which the
outranking relations (S), indifference (I), or
incomparability (R) are defined.
4.2 Results and Discussion
The last stage of exploration of the ELECTRE TRI
method consists in assigning the evaluation methods
to the predefined layers. Two allocation procedures
are supported by ELECTRE TRI method. The first
allocation procedure begins with the pessimistic one.
In this procedure, the comparison begins with the
best reference action and proceeds to the action
immediately below until the first profile b
i
which is
outranked by alternative a
i
. After applying the
ELECTRE TRI method, the appropriate evaluation
methods are displayed according to the type of
assignment (i.e., pessimistic or optimistic). Each
evaluation method is compared to the reference
profiles of the layers.
Table 4 shows the final result obtained through
the optimistic and pessimistic versions of ELECTRE
TRI method.
Finally, the evaluator has to compare the
evaluation criteria to be assessed in each layer and
the other ones covered in the proposed evaluation
methods in each assignment procedure. Based on the
information given about the context of use factors,
the evaluation criteria to be covered in the adaptive
system can be determined. It should be noted that
the proposed results are dependent on the considered
evaluation constraints. In this study, the pessimistic
version of ELECTRE TRI is adopted. This version
proposes the most suitable evaluation methods that
cover the different evaluation criteria in each layer.
Table 5 illustrates the appropriate evaluation
methods for individual layers of the considered
adaptive system. Considering the final results
obtained, we can note that in each layer, different
evaluation methods are proposed. These methods are
divergent in their assignment in each layer of
adaptation.
Table 4: Classification results by ELECTRE TRI.
Adaptation layers
Pessimistic
procedure
Optimistic
procedure
Collection of
input data
a13, a14 a8, a15
Interpretation of
the data
a15, a16, a8 a3, a6, a8, a9, a13
Modelling of the
current state of the
world
a1, a3, a6, a12,
a19 a1, a5, a14, a16,
Deciding upon
adaptation
a4, a2, a5, a9, a10,
a17, a20
a4, a2, a11,
a12, a20
Applying
adaptation
a7, a11, a18
a7, a10, a17, a18,
a19
ICSOFT 2017 - 12th International Conference on Software Technologies
168
Table 5: Final proposed evaluation methods.
Adaptation layers Final proposed evaluation methods
Collection of input
data
a13, a14
Interpretation of the
data
a15, a16, a8
Modelling of the
current state of the
world
a1, a3, a6, a12,
a19
Deciding upon
adaptation
a4, a2, a5, a9, a10,
a17, a20
Applying adaptation a7, a11, a18
Every evaluation method is assigned to one layer
according to which this method might be appropriate
to assess the considered evaluation factors of the
considered layer. It is important to examine carefully
the evaluation methods generated and especially to
infer the criteria that really represent the feedback
from evaluator about the context of use factors. In
this study, the ELECTRE TRI method is applied to
the assignment problematic. It aims to allocate each
alternative evaluation method to the appropriate
layers. The different steps of adaptation are defined
a priori by the evaluator. Five adaptation layers are
identified in the adaptation process of the given
adaptive system.
5 CONCLUSION AND FUTURE
WORK
In this research paper, we are interested in proposing
a decision model to guide the layered evaluation of
interactive adaptive systems. The goal is to apply a
multi-criteria decision sorting method in order to
help evaluators in the choice of appropriate
evaluation methods for individual layers. By doing
so, it is possible to assign the best evaluation
methods to the layers of adaptation for particular
evaluation settings. To this end, one MCDA method
is used, namely ELECTRE TRI. The proposed
decision model is applied to determine the suitable
evaluation methods for the individual layers of an
adaptive hypermedia system.
It should be noted that the number of layers may
change from an IAS to another and that not all layers
can be evaluated in isolation in all contexts. In some
cases, it may be necessary to evaluate the layers in
combination (Paramythis et al., 2010). This depends
essentially on some evaluation constraints and the
nature of the IAS. The evaluation also has to
consider the case of the whole adaptive systems in
which there is no distinction between the different
layers. Future directions of this research will then
investigate how to handle the case of the whole
adaptive system and the combination of layers. We
also intend to include other criteria and to test our
proposed model in real evaluation scenarios.
REFERENCES
Brusilovsky, P., Schwarz, E., Weber. G., 1996. Elm-art: an
intelligent tutoring system on world wide web. In
Intelligent Tutoring Systems. Lecture Notes in
Computer Science, Springer Verlag, Berlin, 261–269.
Brusilovsky, P., Karagiannidis, C., Sampson, D. 2004.
Layered evaluation of adaptive learning systems.
International Journal of Continuing Engineering
Education and Lifelong Learning, Burke, R. Hybrid
recommender, 14, (4/5), 402-421.
Dhouib, A., Trablesi, A., Kolski, C. and Neji, M. 2015. A
framework for supporting the choice of usability
evaluation methods for interactive adaptive systems.
15th International Conference on Intelligent Systems
Design and Applications, December, Marrakesh,
Maroc, 289-294.
Dhouib, A., Trabelsi, A., Kolski, C., Neji. M., 2016a. A
classification and comparison of usability evaluation
methods for interactive adaptive systems. 9th
International Conference on Human System
Interactions, Portsmouth, UK, 246–251.
Dhouib A, Trablesi A, Kolski C, Neji M. 2016b. An
approach for the selection of evaluation methods for
interactive adaptive systems using analytic hierarchy
process. IEEE Tenth International Conference on
Research Challenges in Information Science,
Grenoble, France, 1-10.
Ferré, X., and Bevan. N., 2011. Usability planner: a tool to
support the process of selecting usability methods. In
Human-Computer Interaction–INTERACT 2011, ed:
Springer, 652-655.
Goren-Bar D, Graziola I., Rocchi C, Pianesi F, Stock O,
Zancanaro M. 2005. Designing and redesigning an
affective interface for an adaptive museum guide. In:
1st International Conference on Affective Computing
and Intelligent Interaction, Beijing, China. 3784: 939–
946.
Goy, A., Ardissono, L., Petrone. G., 2006. Personalization
in e-commerce applications. In Brusilovsky, P., Kobsa,
A., Nejdl,W. (eds.), The Adaptive Web: Methods and
Strategies of Web Personalization. Lecture Notes in
Computer Science, Springer-Verlag, Berlin
Heidelberg, New York vol. 4321.
Gena, C., Weibelzahl. S., 2007. Usability engineering for
the adaptive web. In: Brusilovsky, P., Kobsa, A.,
Nejdl, W. (eds.). The Adaptive Web: Methods and
Strategies of Web Personalization, Springer, Berlin,
720-762.
Guitouni, A., and Martel, J. M. 1998. Tentative guidelines
to help choosing an appropriate MCDA method.
Towards the Layered Evaluation of Interactive Adaptive Systems using ELECTRE TRI Method
169
European Journal of Operational Research, 109:501–
521.
Krueger, R., Casey, M. 2009. Focus Groups: A Practical
Guide for Applied Research. 4th edn. Sage
Publications, Los Angeles.
Masthoff, J. 2006. The user as wizard: a method for early
involvement in the design and evaluation of adaptive
systems. In: 5thWorkshop on User-Centred Design
and Evaluation of Adaptive Systems at AH’06, Dublin,
Ireland, 460–469.
Magoulas, G. D, Chen, S. Y, and Papanikolaou K. A.
2003. Integrating layered and heuristic evaluation for
adaptive learning environments. In: 2nd Workshop on
Empirical Evaluation of Adaptive Systems at UM2003,
Johnstown, PA, 5-14.
Manouselis, N., Karagiannidis, C., Sampson. D.G., 2014.
Layered evaluation for data discovery and
recommendation systems: an initial set of principles.
IEEE 14th International Conference on Advanced
Learning Technologies (ICALT), 518-519.
Mulwa, C., Lawless, S., Sharp, M., and Wade, V. 2011.
The evaluation of adaptive and user-adaptive systems:
a review. International Journal of Knowledge and
Web Intelligence, IJKWI, 2, 2/3, 138-156.
Mousseau, V, Figueira, J & Naux, JP. 2001. Using
assignment examples to infer weights for ELECTRE
TRI method: Some experimental results. European
Journal of Operational Research, 130(2): 263–275.
Paramythis, A., Weibelzahl, S., Masthoff. J., 2010.
Layered evaluation of interactive adaptive systems:
framework and formative methods. User Modeling
and User-Adapted Interaction. 20, 383-453.
Roy, B., 1968. Classement et choix en présence de points
de vue multiples (la méthode ELECTRE). Revue
Française d'Informatique et de Recherche
Opérationnelle, 8, 1968, 57–75.
Weibelzahl, S., 2001. Evaluation of adaptive systems. In:
8th International Conference on User Modeling.
LNCS, Springer, Berlin, 2109, 292-294.
Weibelzahl, S., Lauer, C. U. 2001. Framework for the
evaluation of adaptive CBR-systems. In. Vollrath, S.
Schmitt, & U. Reimer (Eds.), Experience Management
as Reuse of Knowledge. Proceedings of the 9th
German Workshop on Case Based Reasoning,
GWCBR2001, 254-263.
Yu, W., 1992. ELECTRE TRI: aspects méthodologiques
et manuel d’utilisation. Document du LAMSADE n° 74
LAMSADE Université Paris-Dauphine.
Van Velsen, L., Van der Geest, T., Klaassen, R.,
Steehouder, M. 2008. User-centered evaluation of
adaptive and adaptable systems: a literature review.
The knowledge engineering review, 23, 261-281.
ICSOFT 2017 - 12th International Conference on Software Technologies
170
