# Learning Multi-tree Classification Models with Ant Colony Optimization

### Khalid M. Salama, Fernando E. B. Otero

#### Abstract

Ant Colony Optimization (ACO) is a meta-heuristic for solving combinatorial optimization problems, inspired by the behaviour of biological ant colonies. One of the successful applications of ACO is learning classification models (classifiers). A classifier encodes the relationships between the input attribute values and the values of a class attribute in a given set of labelled cases and it can be used to predict the class value of new unlabelled cases. Decision trees have been widely used as a type of classification model that represent comprehensible knowledge to the user. In this paper, we propose the use of ACO-based algorithms for learning an extended multi-tree classification model, which consists of multiple decision trees, one for each class value. Each class-based decision trees is responsible for discriminating between its class value and all other values available in the class domain. Our proposed algorithms are empirically evaluated against well-known decision trees induction algorithms, as well as the ACO-based Ant-Tree-Miner algorithm. The results show an overall improvement in predictive accuracy over 32 benchmark datasets. We also discuss how the new multi-tree models can provide the user with more understanding and knowledge-interpretability in a given domain.

#### References

- Asuncion, A. and Newman, D. (2007). Machine Learning Repository. URL: www.ics.uci.edu/ mlearn/MLRepository.html.
- Boryczka, U. and Kozak, J. (2010). Ant Colony Decision Trees. In 4th International Conference on Computational Collective Intelligence: Technologies and Applications (ICCCI'11), pages 4373-382, Berlin, Heidelberg. Springer.
- Boryczka, U. and Kozak, J. (2011). An Adaptive Discretization in the ACDT Algorithm for Continuous Attributes. In 3rd International Conference on Computational Collective Intelligence: Technologies and Applications (ICCCI'11), pages 475-484, Berlin, Heidelberg. Springer.
- Breiman, L., Friedman, J., Stone, C., and Olshen, R. (1984). Classification and Regression Trees. Chapman and Hall.
- Dorigo, M. and Stützle, T. (2003). The Ant Colony Optimization Metaheuristic: Algorithms, Applications, and Advances. In Handbook of Metaheuristics, volume 57 of OPRMS, pages 250-28, New York, NY, USA. Springer.
- Dorigo, M. and Stützle, T. (2004). Ant Colony Optimization. MIT Press, Cambridge, MA, USA.
- Garca, S. and Herrera, F. (2008). An Extension on ”Statistical Comparisons of Classifiers over Multiple Data Sets” for all Pairwise Comparisons. Journal of Machine Learning Research, 9:2677-2694.
- Han, J. and Kamber, M. (2000). Data Mining: Concepts and Techniques. Morgan Kaufmann, San Francisco, CA, USA, 2nd edition.
- Liu, Y.-P., Wu, M.-G., and Qian, J.-X. (2006). Evolving neural networks using the hybrid of ant colony optimization and bp algorithms. In 3rd International Conference on Advances in Neural Networks (ISNN'06), pages 714-722, Berlin, Heidelberg. Springer-Verlag.
- Martens, D., Backer, M. D., Haesen, R., Vanthienen, J., Snoeck, M., and Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11:651-665.
- Otero, F. and Freitas, A. (2013). Improving the Interpretability of Classification Rules Discovered by an Ant Colony Algorithm. In Genetic and Evolutionary Computation Conference (GECCO-2013), pages 73- 80, New York, NY, USA. ACM Press.
- Otero, F., Freitas, A., and Johnson, C. (2009). Handling continuous attributes in ant colony classification algorithms. In IEEE Symposium on Computational Intelligence in Data Mining (CIDM 2009), pages 225-231, New York, NY, USA. IEEE Press.
- Otero, F., Freitas, A., and Johnson, C. (2013). A new sequential covering strategy for inducing classification rules with ant colony algorithms. IEEE Transactions on Evolutionary Computation, 17(1):64-74.
- Otero, F. E. B., Freitas, A. A., and Johnson, C. G. (2012). Inducing Decision Trees with an Ant Colony Optimization Algorithm. Applied Soft Computing, 12(11):3615-3626.
- Parpinelli, R. S., Lopes, H. S., and Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4):321-332.
- Quinlan, J. (1993). Programs for Machine Learning. Morgan Kaufmann.
- Salama, K., Abdelbar, A., and Freitas, A. (2011). Multiple Pheromone Types and Other Extensions to the Ant-Miner Classification Rule Discovery Algorithm. Swarm Intelligence, 5(3-4):149-182.
- Salama, K., Abdelbar, A., Otero, F., and Freitas, A. (2013). Utilizing multiple pheromones in an ant-based algorithm for continuous-attribute classification rule discovery. Applied Soft Computing, 13(1):667-675.
- Salama, K. and Freitas, A. (2013a). Ant Colony Algorithms for Constructing Bayesian Multi-net Classifiers. Machine Learning - (Under Review).
- Salama, K. and Freitas, A. (2013b). Clustering-based Bayesian Multi-net Classifier Construction with Ant Colony Optimization. In IEEE Congress on Evolutionary Computation (IEEE CEC) (2013), pages 3079-3086, New York, NY, USA. IEEE Press.
- Salama, K. and Freitas, A. (2013c). Extending the ABCMiner Bayesian Classification Algorithm. In 6th International Workshop on Nature Inspired Cooperative Strategies for Optimization (NICSO'13), volume 512 of SCI, pages 1-12, Berlin. Springer.
- Salama, K. and Freitas, A. (2013d). Learning Bayesian Network Classifiers Using Ant Colony Optimization. Swarm Intelligence, 7(2-3):229-254.
- Socha, K. and Blum, C. (2005). Training feed-forward neural networks with ant colony optimization: An application to pattern classification. In 5th International Conference on Hybrid Intelligent Systems (HIS 7805), pages 233-238, Washington, DC, USA. IEEE Computer Society.
- Socha, K. and Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing & Applications, 16:235-247.
- Socha, K. and Dorigo, M. (2008). Ant colony optimization for continuous domains. European Journal of Operational Research, 185:1155-1173.
- Stü tzle, T. and Hoos, H. (1997). MAX-MIN Ant System and local search for the traveling salesman problem. Evolutionary Computation, 1997., IEEE International Conference on, pages 309-314.
- Tan, P.-N., Steinbach, M., and Kumar, V. (2005). Introduction to Data Mining. Addison Wesley, 2nd edition.
- Witten, I. H. and Frank, E. (2010). Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco, CA, USA, 3rd edition.

#### Paper Citation

#### in Harvard Style

Salama K. and Otero F. (2014). **Learning Multi-tree Classification Models with Ant Colony Optimization** . In *Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014)* ISBN 978-989-758-052-9, pages 38-48. DOI: 10.5220/0005071300380048

#### in Bibtex Style

@conference{ecta14,

author={Khalid M. Salama and Fernando E. B. Otero},

title={Learning Multi-tree Classification Models with Ant Colony Optimization},

booktitle={Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014)},

year={2014},

pages={38-48},

publisher={SciTePress},

organization={INSTICC},

doi={10.5220/0005071300380048},

isbn={978-989-758-052-9},

}

#### in EndNote Style

TY - CONF

JO - Proceedings of the International Conference on Evolutionary Computation Theory and Applications - Volume 1: ECTA, (IJCCI 2014)

TI - Learning Multi-tree Classification Models with Ant Colony Optimization

SN - 978-989-758-052-9

AU - Salama K.

AU - Otero F.

PY - 2014

SP - 38

EP - 48

DO - 10.5220/0005071300380048