Learning Multi-tree Classification Models with Ant Colony Optimization

Khalid M. Salama, Fernando E. B. Otero

Abstract

Ant Colony Optimization (ACO) is a meta-heuristic for solving combinatorial optimization problems, inspired by the behaviour of biological ant colonies. One of the successful applications of ACO is learning classification models (classifiers). A classifier encodes the relationships between the input attribute values and the values of a class attribute in a given set of labelled cases and it can be used to predict the class value of new unlabelled cases. Decision trees have been widely used as a type of classification model that represent comprehensible knowledge to the user. In this paper, we propose the use of ACO-based algorithms for learning an extended multi-tree classification model, which consists of multiple decision trees, one for each class value. Each class-based decision trees is responsible for discriminating between its class value and all other values available in the class domain. Our proposed algorithms are empirically evaluated against well-known decision trees induction algorithms, as well as the ACO-based Ant-Tree-Miner algorithm. The results show an overall improvement in predictive accuracy over 32 benchmark datasets. We also discuss how the new multi-tree models can provide the user with more understanding and knowledge-interpretability in a given domain.

References

  1. Asuncion, A. and Newman, D. (2007). Machine Learning Repository. URL: www.ics.uci.edu/ mlearn/MLRepository.html.
  2. Boryczka, U. and Kozak, J. (2010). Ant Colony Decision Trees. In 4th International Conference on Computational Collective Intelligence: Technologies and Applications (ICCCI'11), pages 4373-382, Berlin, Heidelberg. Springer.
  3. Boryczka, U. and Kozak, J. (2011). An Adaptive Discretization in the ACDT Algorithm for Continuous Attributes. In 3rd International Conference on Computational Collective Intelligence: Technologies and Applications (ICCCI'11), pages 475-484, Berlin, Heidelberg. Springer.
  4. Breiman, L., Friedman, J., Stone, C., and Olshen, R. (1984). Classification and Regression Trees. Chapman and Hall.
  5. Dorigo, M. and Stützle, T. (2003). The Ant Colony Optimization Metaheuristic: Algorithms, Applications, and Advances. In Handbook of Metaheuristics, volume 57 of OPRMS, pages 250-28, New York, NY, USA. Springer.
  6. Dorigo, M. and Stützle, T. (2004). Ant Colony Optimization. MIT Press, Cambridge, MA, USA.
  7. Garca, S. and Herrera, F. (2008). An Extension on ”Statistical Comparisons of Classifiers over Multiple Data Sets” for all Pairwise Comparisons. Journal of Machine Learning Research, 9:2677-2694.
  8. Han, J. and Kamber, M. (2000). Data Mining: Concepts and Techniques. Morgan Kaufmann, San Francisco, CA, USA, 2nd edition.
  9. Liu, Y.-P., Wu, M.-G., and Qian, J.-X. (2006). Evolving neural networks using the hybrid of ant colony optimization and bp algorithms. In 3rd International Conference on Advances in Neural Networks (ISNN'06), pages 714-722, Berlin, Heidelberg. Springer-Verlag.
  10. Martens, D., Backer, M. D., Haesen, R., Vanthienen, J., Snoeck, M., and Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11:651-665.
  11. Otero, F. and Freitas, A. (2013). Improving the Interpretability of Classification Rules Discovered by an Ant Colony Algorithm. In Genetic and Evolutionary Computation Conference (GECCO-2013), pages 73- 80, New York, NY, USA. ACM Press.
  12. Otero, F., Freitas, A., and Johnson, C. (2009). Handling continuous attributes in ant colony classification algorithms. In IEEE Symposium on Computational Intelligence in Data Mining (CIDM 2009), pages 225-231, New York, NY, USA. IEEE Press.
  13. Otero, F., Freitas, A., and Johnson, C. (2013). A new sequential covering strategy for inducing classification rules with ant colony algorithms. IEEE Transactions on Evolutionary Computation, 17(1):64-74.
  14. Otero, F. E. B., Freitas, A. A., and Johnson, C. G. (2012). Inducing Decision Trees with an Ant Colony Optimization Algorithm. Applied Soft Computing, 12(11):3615-3626.
  15. Parpinelli, R. S., Lopes, H. S., and Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4):321-332.
  16. Quinlan, J. (1993). Programs for Machine Learning. Morgan Kaufmann.
  17. Salama, K., Abdelbar, A., and Freitas, A. (2011). Multiple Pheromone Types and Other Extensions to the Ant-Miner Classification Rule Discovery Algorithm. Swarm Intelligence, 5(3-4):149-182.
  18. Salama, K., Abdelbar, A., Otero, F., and Freitas, A. (2013). Utilizing multiple pheromones in an ant-based algorithm for continuous-attribute classification rule discovery. Applied Soft Computing, 13(1):667-675.
  19. Salama, K. and Freitas, A. (2013a). Ant Colony Algorithms for Constructing Bayesian Multi-net Classifiers. Machine Learning - (Under Review).
  20. Salama, K. and Freitas, A. (2013b). Clustering-based Bayesian Multi-net Classifier Construction with Ant Colony Optimization. In IEEE Congress on Evolutionary Computation (IEEE CEC) (2013), pages 3079-3086, New York, NY, USA. IEEE Press.
  21. Salama, K. and Freitas, A. (2013c). Extending the ABCMiner Bayesian Classification Algorithm. In 6th International Workshop on Nature Inspired Cooperative Strategies for Optimization (NICSO'13), volume 512 of SCI, pages 1-12, Berlin. Springer.
  22. Salama, K. and Freitas, A. (2013d). Learning Bayesian Network Classifiers Using Ant Colony Optimization. Swarm Intelligence, 7(2-3):229-254.
  23. Socha, K. and Blum, C. (2005). Training feed-forward neural networks with ant colony optimization: An application to pattern classification. In 5th International Conference on Hybrid Intelligent Systems (HIS 7805), pages 233-238, Washington, DC, USA. IEEE Computer Society.
  24. Socha, K. and Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing & Applications, 16:235-247.
  25. Socha, K. and Dorigo, M. (2008). Ant colony optimization for continuous domains. European Journal of Operational Research, 185:1155-1173.
  26. Stü tzle, T. and Hoos, H. (1997). MAX-MIN Ant System and local search for the traveling salesman problem. Evolutionary Computation, 1997., IEEE International Conference on, pages 309-314.
  27. Tan, P.-N., Steinbach, M., and Kumar, V. (2005). Introduction to Data Mining. Addison Wesley, 2nd edition.
  28. Witten, I. H. and Frank, E. (2010). Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco, CA, USA, 3rd edition.
Download


Paper Citation


in Harvard Style

Salama K. and Otero F. (2014). Learning Multi-tree Classification Models with Ant Colony Optimization . In Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014) ISBN 978-989-758-052-9, pages 38-48. DOI: 10.5220/0005071300380048


in Bibtex Style

@conference{ecta14,
author={Khalid M. Salama and Fernando E. B. Otero},
title={Learning Multi-tree Classification Models with Ant Colony Optimization},
booktitle={Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014)},
year={2014},
pages={38-48},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005071300380048},
isbn={978-989-758-052-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014)
TI - Learning Multi-tree Classification Models with Ant Colony Optimization
SN - 978-989-758-052-9
AU - Salama K.
AU - Otero F.
PY - 2014
SP - 38
EP - 48
DO - 10.5220/0005071300380048