Predicting Classifier Combinations

Matthias Reif, Annika Leveringhaus, Faisal Shafait, Andreas Dengel


Combining classifiers is a common technique in order to improve the performance and robustness of classification systems. However, the set of classifiers that should be combined is not obvious and either expert knowledge or a time consuming evaluation phase is required in order to achieve high accuracy values. In this paper, we present an approach of automatically selecting the set of base classifiers for combination. The method uses experience about previous classifier combinations and characteristics of datasets in order to create a prediction model. We evaluate the method on over 80 datasets. The results show that the presented method is able to reasonably predict a suitable set of base classifiers for most of the datasets.


  1. Ali, S. and Smith, K. A. (2006). On learning algorithm selection for classification. Applied Soft Computing, 6:119-138.
  2. Asuncion, A. and Newman, D. (2007). UCI machine learning repository. mlearn/MLRepository.html University of California, Irvine, School of Information and Computer Sciences.
  3. Bennett, P. N., Dumais, S. T., and Horvitz, E. (2005). The combination of text classifiers using reliability indicators. Information Retrieval, 8(1):67-100.
  4. Bensusan, H. and Giraud-Carrier, C. (2000a). Casa batl is in passeig de grcia or how landmark performances can describe tasks. In Proc. of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 29-46.
  5. Bensusan, H. and Giraud-Carrier, C. (2000b). Discovering task neighbourhoods through landmark learning performances. In Proc. of the 4th Europ. Conf. on Principles of Data Mining and Knowledge Discovery, pages 325-330.
  6. Bensusan, H., Giraud-Carrier, C., and Kennedy, C. (2000). A higher-order approach to meta-learning. In Proc. of the ECML'2000 workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pages 109-117.
  7. Brazdil, P., Gama, J., and Henery, B. (1994). Characterizing the applicability of classification algorithms using meta-level learning. In Machine Learning: ECML-94, volume 784 of Lecture Notes in Computer Science, pages 83-102. Springer Berlin / Heidelberg.
  8. Brazdil, P. B., Soares, C., and da Costa, J. P. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251-277.
  9. Castiello, C., Castellano, G., and Fanelli, A. M. (2005). Meta-data: Characterization of input features for meta-learning. In Modeling Decisions for Artificial Intelligence, volume 3558, pages 295-304. Springer Berlin / Heidelberg.
  10. Cornelson, M., Grossmann, R. L., Karidi, G. R., and Shnidman, D. (2002). Survey of Text Mining: Clustering, Classification, and Retrieval, chapter Combining Families of Information Retrieval Algorithms using Meta-Learning, pages 159-169. Springer.
  11. Dietterich, T. G. (2000). Ensemble methods in machine learning. In Proc. of the First Int. Workshop on Multiple Classifier Systems, pages 1-15.
  12. Engels, R. and Theusinger, C. (1998). Using a data metric for preprocessing advice for data mining applications. In Proc. of the Europ. Conf. on Artificial Intelligence, pages 430-434.
  13. Gama, J. and Brazdil, P. (1995). Characterization of classification algorithms. In Progress in Artificial Intelligence, volume 990 of Lecture Notes in Computer Science, pages 189-200. Springer Berlin / Heidelberg.
  14. Kitoogo, F. E. and Baryamureeba, V. (2007). Metaknowledge as an engine in classifier combination. International Journal of Computing and ICT Research, 1(2):74-86.
  15. Kohavi, R. and John, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence - Special issue on relevance, 97:273-324.
  16. Kuncheva, L. I. and Whitaker, C. J. (2003). Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181-207.
  17. Michie, D., Spiegelhalter, D. J., and Taylor, C. C. (1994). Machine Learning, Neural and Statistical Classification. Ellis Horwood.
  18. Peng, Y., Flach, P., Soares, C., and Brazdil, P. (2002). Improved dataset characterisation for meta-learning. In Discovery Science, volume 2534 of Lecture Notes in Computer Science, pages 193-208. Springer Berlin / Heidelberg.
  19. Pfahringer, B., Bensusan, H., and Giraud-Carrier, C. (2000). Meta-learning by landmarking various learning algorithms. In Proc. of the 17th Int. Conf. on Machine Learning, pages 743-750.
  20. Reif, M. (2012). A comprehensive dataset for evaluating approaches of various meta-learning tasks. In First International Conference on Pattern Recognition and Methods.
  21. Reif, M., Shafait, F., Goldstein, M., Breuel, T., and Dengel, A. (2012). Automatic classifier selection for non-experts. Pattern Analysis and Applications. 10.1007/s10044-012-0280-z.
  22. Segrera, S., Pinho, J., and Moreno, M. (2008). Informationtheoretic measures for meta-learning. In Hybrid Artificial Intelligence Systems, volume 5271 of Lecture Notes in Computer Science, pages 458-465. Springer Berlin / Heidelberg.
  23. Simonoff, J. S. (2003). Analyzing Categorical Data. Springer Texts in Statistics. Springer Berlin / Heidelberg.
  24. Sohn, S. Y. (1999). Meta analysis of classification algorithms for pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(11):1137 -1144.
  25. Todorovski, L. and Dz?eroski, S. (2003). Combining classifiers with meta decision trees. Machine Learning, 50:223-249.
  26. Vilalta, R., Giraud-Carrier, C., Brazdil, P., and Soares, C. (2004). Using meta-learning to support data mining. International Journal of Computer Science and Applications, 1(1):31-45.
  27. Vlachos, P. (1998). StatLib datasets archive. Department of Statistics, Carnegie Mellon University.
  28. Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computing, 8(7):1341-1390.

Paper Citation

in Harvard Style

Reif M., Leveringhaus A., Shafait F. and Dengel A. (2013). Predicting Classifier Combinations . In Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-8565-41-9, pages 293-297. DOI: 10.5220/0004266602930297

in Bibtex Style

author={Matthias Reif and Annika Leveringhaus and Faisal Shafait and Andreas Dengel},
title={Predicting Classifier Combinations},
booktitle={Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},

in EndNote Style

JO - Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Predicting Classifier Combinations
SN - 978-989-8565-41-9
AU - Reif M.
AU - Leveringhaus A.
AU - Shafait F.
AU - Dengel A.
PY - 2013
SP - 293
EP - 297
DO - 10.5220/0004266602930297