![](bg2.png)
The “profile discovering procedure” and its exten-
sion allow us to test the performance of the param-
eters of the recommendation process in order to tune
functions and algorithms. Moreover, it is a suitable in-
strument to compare different recommender systems,
an inconceivable experiment so far.
The outline of this paper is as follows: the next
section describes some evaluation techniques for rec-
ommender systems that have been used in the cur-
rent state-of-the-art. Then, our proposal for evalu-
ating recommender systems, namely what we have
called “profile discovering” is presented in section 3.
Following, its extension for performing multi-agent
collaboration is detailed in section 4. Then, some ex-
perimental results are shown in section 5 and, finally,
section 6 concludes this paper.
2 RELATED WORK
In the current state-of-the-art, recommender systems
use one of the following approaches in order to ac-
quire the results for evaluating the performance of
their systems: a real environment, an evaluation en-
vironment, the logs of the system or a user simulator.
First, results obtained in a real environment with
real users is the best way to evaluate a recommender
system. Unfortunately, only a few commercial sys-
tems like Amazon.com (Amazon, 2003) can show
real results based on their economic effect thanks to
their information on real users.
Second, evaluation environments are an alternative
for some systems to be evaluated in the laboratory by
letting a set of users interact with the system over a
period of time. Usually, the results are not reliable
enough because the users know the system or the pur-
pose of the evaluation. An original approach was ac-
complished by NewT (Sheth, 1994); in addition to the
numerical data collected in the evaluation sessions, a
questionnaire was also distributed to the users to get
feedback on the subjective aspects of the system. The
main problem of the real and the evaluation environ-
ments is that repetition of the experiments, in order to
evaluate different algorithms and parameters, is im-
possible.
Third, the analysis or validation of the logs ob-
tained in a real or evaluation environment with real
users is a common technique used to evaluate rec-
ommender systems. A frequently used technique is
the “10-fold cross-validation technique” (Mladenic,
1996). It consists of predicting the relevance (e.g.,
ratings) of examples recorded in the logs and, then,
comparing them with the real evaluations. These ex-
periments are perfectly repeatable, provided that the
tested parameters do not affect the evolution of the
user profile and the recommendation process. For ex-
ample, the log being validated would be very different
if another recommendation algorithm had been tested.
Therefore, since the majority of the parameters condi-
tion the recommendation process over time, generally,
experiments cannot be repeated.
Finally, a few systems are evaluated through simu-
lated users. Important issues such as learning rates
and variability in learning behaviour across hetero-
geneous populations can be investigated using large
collections of simulated users whose design was tai-
lored to explore those issues. This enables large-scale
experiments to be carried out quickly and also guar-
antees that experiments are repeatable and perfectly
controlled. It also allows researchers to focus on and
study the behaviour of each sub-component of the
system, which would otherwise be impossible in an
unconstrained environment. For instance, Holte and
Yan conducted experiments using an automated user
called Rover rather than human users (Holte and Yan,
1996). NewT (Sheth, 1994) and Casmir (Berney and
Ferneley, 1999) also used a user simulator to evaluate
the performance of systems. The main shortcoming
of this technique is that, at present, it is impossible
to simulate the real behaviour of a user. Users are
far too complicated to predict, at every moment, their
feelings, their emotions, their moods, their anxieties
and, therefore, their actions.
3 “PROFILE DISCOVERING”
In order to solve all the shortcomings of the current
techniques while benefitting from their advantages,
we propose a method of results acquisition called “the
profile discovering procedure” (see Figure 1). This
technique can be seen as an hybrid approach between
real or laboratory evaluation, log analysis and user
simulation.
First of all, it is necessary to obtain as many item
evaluations from real users as possible. It is desir-
able to obtain these user evaluations through a real or
laboratory evaluation although it implies a relatively
long period of time. However, it is also possible and
faster to get the user evaluations through a question-
naire containing all the items which the users have to
evaluate. We call the list of real item evaluations of a
given user A, the A’s real user profile (RU P ).
Once the real user profiles are available, the simula-
tion process; that is the profile discovering procedure
starts. It consists on the following steps:
1. Generation of an initial user profile (U P ) from the
real user profile (RU P , U P ⊂ RU P ).
2. Emulation of the real recommendation process: a
new item (r) is recommended from the U P .
3. Validation of the recommendation:
ICEIS 2004 - INFORMATION SYSTEMS ANALYSIS AND SPECIFICATION
304