ON THE SUITABILITY OF NUMERICAL PERFORMANCE MEASURES FOR CLASS IMBALANCE PROBLEMS

Vicente García, J. Salvador Sánchez, Ramón A. Mollineda

Abstract

The class imbalance problem has been reported as an important challenge in various fields such as Pattern Recognition, Data Mining and Machine Learning. A less explored research area is related to how to evaluate classifiers on imbalanced data sets. This work analyzes the behaviour of performance measures widely used on imbalanced problems, as well as other metrics recently proposed in the literature. We perform two theoretical analysis based on Pearson correlation and operations for a 2×2 confusion matrix with the aim to show the strengths and weaknesses of those performance metrics in the presence of skewed distributions.

References

  1. Batuwita, R. and Palade, V. (2009). A new performance measure for class imbalance learning: application to bioinformatics problems. In the 8th ICMLA'09, pages 545-550.
  2. Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145-1159.
  3. Cohen, G., Hilario, M., Sax, H., Hugonnet, S., and Geissbuhler, A. (2006). Learning from imbalanced data in surveillance of nosocomial infection. Artificial Intelligence Medicine, 37(1):7-18.
  4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educ and Psychol Meas, 20(1):37-46.
  5. Daskalaki, S., Kopanas, I., and Avouris, N. (2006). Evaluation of classifiers for an uneven class distribution problem. Applied Artificial Intelligence, 20(5):381-417.
  6. Fatourechi, M., Ward, R., Mason, S., Huggins, J., Schlogl, A., and Birch, G. (2008). Comparison of evaluation metrics in classification applications with imbalanced datasets. In the 7th ICMLA'08, pages 777 -782.
  7. Ferri, C., Hernández-Orallo, J., and Modroiu, R. (2009). An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30(1):27-38.
  8. Folleco, A., Khoshgoftaar, T. M., and Napolitano, A. (2008). Comparison of four performance metrics for evaluating sampling techniques for low quality classimbalanced data. In the 7th ICMLA'08, pages 153-158.
  9. García, V., Mollineda, R. A., and Sánchez, J. S. (2010). Theoretical analysis of a performance measure for imbalanced data. In the 20th ICPR'2010, pages 617-620.
  10. Gu, Q., Zhu, L., and Cai, Z. (2009). Evaluation measures of the classification performance of imbalanced data sets. In the 4th ISICA'09, pages 461-471. Springer-Verlag.
  11. Huang, J. and Ling, C.-X. (2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Trans Knowl Data Eng, 17(3):299-310.
  12. Huang, J. and Ling, C.-X. (2007). Constructing new and better evaluation measures for machine learning. In the 20th IJCAI'07, pages 859-864.
  13. Kamal, A., Zhu, X., Pandya, A., Hsu, S., and Shoaib, M. (2009). The impact of gene selection on imbalanced microarray expression data. In the 1st BICoB'09, pages 259-269.
  14. Kennedy, K., Mac Namee, B., and Delany, S. (2010). Learning without default: A study of one-class classification and the low-default portfolio problem. In the AICS'09, pages 174-187.
  15. Khalilia, M., Chakraborty, S., and Popescu, M. (2011). Predicting disease risks from highly imbalanced data using random forest. BMC Medical Informatics and Decision Making, 11(1):51.
  16. Kubat, M. and Matwin, S. (1997). Addressing the curse of imbalanced training sets: one-sided selection. In 14th ICML, pages 179-186.
  17. Ranawana, R. and Palade, V. (2006). Optimized Precision - a new measure for classifier performance evaluation. In the IEEE CEC'09, pages 2254-2261.
  18. Rijsbergen, C. J. V. (1979). Information Retrieval. Butterworths, London, UK.
  19. Seliya, N., Khoshgoftaar, T., and Van Hulse, J. (2009). A study on the relationships of classifier performance metrics. In the 21st ICTAI'09, pages 59 -66.
  20. Sokolova, M., Japkowicz, N., and Szpakowicz, S. (2006). Beyond accuracy, f-score and roc: A family of discriminant measures for performance evaluation. In the AJCAI'06, pages 1015-1021.
  21. Sokolova, M. and Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Inf Process & Manag, 45(4):427-437.
  22. Sun, Y., Wong, A., and Kamel, M. S. (2009). Classification of imbalanced data: A review. International Journal of Pattern Recognition and Artificial Intelligence, 23(4):687-719.
  23. Tan, P.-N., Kumar, V., and Srivastava, J. (2002). Selecting the right interestingness measure for association patterns. In the 8th ACM SIGKDD'02, pages 32-41.
  24. Weng, C. G. and Poon, J. (2008). A new evaluation measure for imbalanced datasets. In the 7th AusDM'08, pages 27- 32.
Download


Paper Citation


in Harvard Style

García V., Salvador Sánchez J. and A. Mollineda R. (2012). ON THE SUITABILITY OF NUMERICAL PERFORMANCE MEASURES FOR CLASS IMBALANCE PROBLEMS . In Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-8425-98-0, pages 310-313. DOI: 10.5220/0003783303100313


in Bibtex Style

@conference{icpram12,
author={Vicente García and J. Salvador Sánchez and Ramón A. Mollineda},
title={ON THE SUITABILITY OF NUMERICAL PERFORMANCE MEASURES FOR CLASS IMBALANCE PROBLEMS},
booktitle={Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2012},
pages={310-313},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003783303100313},
isbn={978-989-8425-98-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - ON THE SUITABILITY OF NUMERICAL PERFORMANCE MEASURES FOR CLASS IMBALANCE PROBLEMS
SN - 978-989-8425-98-0
AU - García V.
AU - Salvador Sánchez J.
AU - A. Mollineda R.
PY - 2012
SP - 310
EP - 313
DO - 10.5220/0003783303100313