![](bg4.png)
Table 3: Improvement over community average.
Algorithm Improvement over community avg.
TMW 7.89 %
KNN 1 7,82 %
KNN 2 7,71 %
Slope One 7,95 %
4 DISCUSSION AND
CONCLUSIONS
A first consideration is that all the algorithms don’t
supply predictions dramatically more accurate than
the community average. Even if it is well known now
(see Netflix prize results) that an improvement of a
few percentage points of accuracy is hard to get, still
in absolute terms the RMSE seems a bit excessive (re-
member that votes range from 0 to 5). Indeed, in all
cases the RMSE is always greater than 0.9, represent-
ing an average error in the order of 20% on the actual
ratings.
A second observation is that our results do not
clearly show which algorithm works better. TMW
provides slightly better results than KNN but their
outcomes are very similar. We find a bit surprising
that KNN using cosine similarity performs better than
the one based on Pearson Correlation and that Slope
One performs better than both of them. Indeed, in
line with previous works we expected KNN based on
Pearson Correlation to perform the best. We iden-
tify two possible reasons. First, we considered a plain
version of KNN and did not investigate possible im-
provements. Second, TMW only looks for the best
mentor available, instead of carrying out a full neigh-
bourhood formation, which could pose problems in a
sparse dataset.
We suspect that the results obtained in our exper-
iments are due to the structure and the dimension of
the dataset. Indeed, the training data set of Netflix
consists of 100 million ratings provided by over 480
thousand users, on nearly 18 thousand movie titles.
Group Lens provides three datasets, one of 100,000
ratings by 943 users for 1,682 movies, another of 1
million ratings by 6,040 users for 3,900 movies, and a
third of 10 million ratings and 100,000 tags by 71,567
users for 10,681 movies. Jester Joke dataset has 4.1
million by 73,496 users on 100 jokes. On the contrary,
our dataset is sparser, and thus most of the RMSE
analyses that can be found in the literature do not ap-
ply to our case.
The main issue, which arises from our experience,
is that it is not clear which is the minimal dimension
of a dataset to make it a reliable base to build a test
bed. This a very important question in our opinion
and our impression is that it has been underestimated
in the literature. Of course, if a dataset is untrustwor-
thy an alternative consists in using public datasets, but
this may be audacious, because it may be not ideal
to tune a system to recommend places on a dataset
which was originated from a system to recommend
movies. Additionally, even if the collaborative filter-
ing approach works with generic items, new difficul-
ties may occur later when the system is extended with
content based capabilities.
In conclusion, this experiment has shown that
from the practitioner’s point of view, finding the best
algorithm is equivalent to finding a reliable dataset
and test bed, and that this issue has not been addressed
adequately in the literature.
REFERENCES
Aggarwal, C., Wolf, J., Wu, K.-L., and Yu, P. (1999). Hort-
ing hatches an egg: a new graph-theoretic approach
to collaborative filtering. In Fayyad, U., Chaudhuri,
S., and Madigan, D., editors, Proceedings of the fifth
ACM SIGKDD international conference on Knowl-
edge discovery and data mining, pages 201–212, New
York. ACM.
Breese, J., Heckerman, D., and Kadie, C. (1998). Empirical
analysis of predictive algorithms for collaborative fil-
tering. Technical Report MSR-TR-98-12, Microsoft
Research.
Huang, Z., Chung, W., and Chen, H. (2004). A graph model
for e-commerce recommender systems. Journal of the
American Society for Information Science and Tech-
nology, 55(3):259–274.
Lemire, D. and Maclachlan, A. (2005). Slope one predic-
tors for online rating-based collaborative filtering. In
Proceedings of SIAM Data Mining (SDM’05).
Punnen, A. (1991). A linear time algorithm for the max-
imum capacity path problem. European Journal of
Operational Research, 53:402–404.
Roda, F., Liberti, L., and Raimondi, F. (2009). Com-
binatorial optimization based recommender systems.
In Proceedings of the 8th Cologne-Twente workshop
(CTW09) on Graphs and Combinatorial Optimiza-
tion, Paris.
Vozalis, E. and Margaritis, K. (2003). Analysis of rec-
ommender systems algorithms. In Lipitakis, E., ed-
itor, The 6th Hellenic European Conference on Com-
puter Mathematics & its Applications, pages 732–745,
Athens. Athens University of Economics and Busi-
ness.
WEBIST 2011 - 7th International Conference on Web Information Systems and Technologies
606