A HYBRID DECISION SUPPORT TOOL - Using ensemble of classifiers

S. B. Kotsiantis, P. E. Pintelas


In decision support systems a classification problem can be solved by employing one of several methods such as different types of artificial neural networks, decision trees, bayesian classifiers, etc. However, it may happen that certain parts of instances’ space are better predicting by one method than the others. Thus, the decision of which particular method to choose is a complicated problem. A good alternative to choosing only one method is to create a hybrid forecasting system incorporating a number of possible solution methods as components (an ensemble of classifiers). For this purpose, we have implemented a hybrid decision support system that combines a neural net, a decision tree and a bayesian algorithm using a stacking variant methodology. The presented system can be trained with any data, but in the current implementation is mainly used by tutors of Hellenic Open University to identify drop-out prone students. However, a comparison with other ensembles using the same classifiers as base learner on several standard benchmark datasets showed that this tool gives better accuracy in most cases.


  1. Bauer, E., Kohavi, R, 1999. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning 36, 105-139.
  2. Blake, C.L., Merz, C.J, 1998. UCI Repository of machine learning databases. Irvine, CA: University of California, Department of Information and Computer Science.
  3. Breiman, L, 1996. Bagging Predictors. Machine Learning 24, 123-140.
  4. Dietterich, T.G, 2001. Ensemble methods in machine learning. In Kittler, J., Roli, F., eds. Multiple Classifier Systems. LNCS Vol. 1857, Springer, 1-15.
  5. Dzeroski, S., Zenko, B., 2002. Is Combining Classifiers Better than Selecting the Best One. ICML 2002: 123- 130.
  6. Frank, E., Wang., Y., Inglis, S., Holmes, G., & Witten, I. H., 1998. Using model trees for classification. Machine Learning 32, 63-76.
  7. Freund, Y., Schapire, R., 1996. Experiments with a New Boosting Algorithm, Proceedings: ICML'96, p. 148- 156.
  8. Jensen, F., 1996. An Introduction to Bayesian Networks. Springer.
  9. Ji, C., Ma, S., 1997. Combinations of weak classifiers. IEEE Transaction on Neural Networks 8, 32-42.
  10. Kotsiantis, S., Pierrakeas, C., Pintelas, P., 2003. Preventing student dropout in distance learning systems using machine learning techniques, Proceedings of Seventh International Conference on Knowledge-Based Intelligent Information & Engineering Systems, Lecture Notes in Artificial Intelligence, Vol. 2774, Springer-Verlag, 267-274.
  11. Mitchell, T., 1997. Machine Learning. McGraw Hill.
  12. Opitz, D., Maclin, R., 1999. Popular Ensemble Methods: An Empirical Study, Artificial Intelligence Research 11, 169-198, Morgan Kaufmann.
  13. Quinlan, J.R., 1993. C4.5: Programs for machine learning. Morgan Kaufmann, San Francisco.
  14. Salzberg, S., 1997. On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach, Data Mining and Knowledge Discovery 1, 317-328.
  15. Schaffer, C., 1993. Selecting a classification method by cross-validation. Machine Learning 13, 135-143.
  16. Schapire, R. E., Freund, Y., Bartlett, P., & Lee, W. S., 1998, Boosting the margin: A new explana-tion for the effectiveness of voting methods. The Annals of Statistics 26, 1651-1686.
  17. Seewald, A. K., Furnkranz, J., 2001. An evaluation of grading classifiers. In Advances in Intelligent Data Analysis: Proceedings of the Fourth International Symposium (IDA-01), pages 221-232, Berlin, Springer.
  18. Seewald, A.K, 2002. How to Make Stacking Better and Faster While Also Taking Care of an Unknown Weakness, in Sammut C., Hoffmann A. (eds.), Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), Morgan Kaufmann Publishers, pp.554-561.
  19. Ting, K., & Witten, I., 1999. Issues in Stacked Generalization, Artificial Intelligence Research 10, 271-289, Morgan Kaufmann.
  20. Turban, E., Aronson, J., 1998. Decision Support Systems and Intelligent Systems, Prentice Hall.
  21. Wang, Y., Witten, I., 1997, Induction of model trees for predicting continuous classes, In Proc. of the Poster Papers of the European Conference on ML, Prague, 128-137.
  22. Witten, I., Frank, E. (2000), Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, San Mateo, 2000.
  23. Wolpert, D., 1992, Stacked Generalization. Neural Networks 5, 241-260.

Paper Citation

in Harvard Style

B. Kotsiantis S. and E. Pintelas P. (2004). A HYBRID DECISION SUPPORT TOOL - Using ensemble of classifiers . In Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 2: ICEIS, ISBN 972-8865-00-7, pages 448-453. DOI: 10.5220/0002603604480453

in Bibtex Style

author={S. B. Kotsiantis and P. E. Pintelas},
title={A HYBRID DECISION SUPPORT TOOL - Using ensemble of classifiers},
booktitle={Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 2: ICEIS,},

in EndNote Style

JO - Proceedings of the Sixth International Conference on Enterprise Information Systems - Volume 2: ICEIS,
TI - A HYBRID DECISION SUPPORT TOOL - Using ensemble of classifiers
SN - 972-8865-00-7
AU - B. Kotsiantis S.
AU - E. Pintelas P.
PY - 2004
SP - 448
EP - 453
DO - 10.5220/0002603604480453