Towards an Understanding of the Misclassification Rates of Machine Learning-based Malware Detection Systems

Nada Alruhaily, Behzad Bordbar, Tom Chothia


A number of machine learning based malware detection systems have been suggested to replace signature based detection methods. These systems have shown that they can provide a high detection rate when recognising non-previously seen malware samples. However, in systems based on behavioural features, some new malware can go undetected as a result of changes in behaviour compared to the training data. In this paper we analyse misclassified malware instances and we investigated whether there were recognisable patterns across these misclassifications. Several questions needed to be understood: Can we claim that malware changes over time directly affect the detection rate? Do changes that affect classification occur in malware at the level of families, where all instances that belong to certain families are hard to detect? Alternatively, can such changes be traced back to certain malware variants instead of families? Our experiments showed that these changes are mostly due to behavioural changes at the level of variants across malware families where variants did not behave as expected. This is can be due to the adoption of anti-virtualisation techniques, the fact that these variants were looking for a specific argument to be activated or it can be due to the fact that these variants were actually corrupted.


  1. Alazab, M., Layton, R., Venkataraman, S., and Watters, P. (2010). Malware detection based on structural and behavioural features of api calls.
  2. Bailey, M., Oberheide, J., Andersen, J., Mao, Z. M., Jahanian, F., and Nazario, J. (2007). Automated classification and analysis of internet malware. In Recent Advances in Intrusion Detection, pages 178-197. Springer.
  3. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2):123-140.
  4. Ceron, J. M., Margi, C. B., and Granville, L. Z. (2016). Mars: An sdn-based malware analysis solution. In 2016 IEEE Symposium on Computers and Communication (ISCC), pages 525-530. IEEE.
  5. Chang, E. Y., Li, B., Wu, G., and Goh, K. (2003). Statistical learning for effective visual information retrieval. In ICIP (3), pages 609-612. Citeseer.
  6. Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3):273-297.
  7. Cuckoo Sandbox (2015). Automated malware analysis - cuckoo sandbox.
  8. Fan, C.-I., Hsiao, H.-W., Chou, C.-H., and Tseng, Y.-F. (2015). Malware detection systems based on api log data mining. In Computer Software and Applications Conference (COMPSAC), 2015 IEEE 39th Annual, volume 3, pages 255-260. IEEE.
  9. Faruki, P., Laxmi, V., Gaur, M. S., and Vinod, P. (2012). Behavioural detection with api call-grams to identify malicious pe files. In Proceedings of the First International Conference on Security of Internet of Things, pages 85-91. ACM.
  10. Ferri, C., Hernández-Orallo, J., and Modroiu, R. (2009). An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30(1):27-38.
  11. Firdausi, I., Lim, C., Erwin, A., and Nugroho, A. S. (2010). Analysis of machine learning techniques used in behavior-based malware detection. In Advances in Computing, Control and Telecommunication Technologies (ACT), 2010 Second International Conference on, pages 201-203. IEEE.
  12. Hansen, S. S., Larsen, T. M. T., Stevanovic, M., and Pedersen, J. M. (2016). An approach for detection and family classification of malware based on behavioral analysis. In 2016 International Conference on Computing, Networking and Communications (ICNC), pages 1-5. IEEE.
  13. Huang, J. and Ling, C. X. (2005). Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3):299-310.
  14. Islam, R., Tian, R., Moonsamy, V., and Batten, L. (2012). A comparison of the classification of disparate malware collected in different time periods. Journal of networks, 7(6):946-955.
  15. Kang, P. and Cho, S. (2006). Eus svms: Ensemble of undersampled svms for data imbalance problems. In International Conference on Neural Information Processing, pages 837-846. Springer.
  16. Khoshgoftaar, T. M., Van Hulse, J., and Napolitano, A. (2011). Comparing boosting and bagging techniques with noisy and imbalanced data. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 41(3):552-568.
  17. Kotsiantis, S. B., Zaharakis, I., and Pintelas, P. (2007). Supervised machine learning: A review of classification techniques.
  18. Kruczkowski, M. and Szynkiewicz, E. N. (2014). Support vector machine for malware analysis and classification. InProceedings of the 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)-Volume 02, pages 415-420. IEEE Computer Society.
  19. Lin, W.-J. and Chen, J. J. (2012). Class-imbalanced classifiers for high-dimensional data. Briefings in bioinformatics, page bbs006.
  20. Lu, Y.-B., Din, S.-C., Zheng, C.-F., and Gao, B.-J. (2010). Using multi-feature and classifier ensembles to improve malware detection. Journal of CCIT, 39(2):57- 72.
  21. Maxwell, K. (2012). Mwcrawler. 0day1day/mwcrawler.
  22. Maxwell, K. (2015). Maltrieve. technoskald/maltrieve.
  23. Miao, Q., Liu, J., Cao, Y., and Song, J. (2015). Malware detection using bilayer behavior abstraction and improved one-class support vector machines. International Journal of Information Security, pages 1-19. Microsoft security intelligence re
  24. Moser, A., Kruegel, C., and Kirda, E. (2007). Limits of static analysis for malware detection. In Computer Security Applications Conference, 2007. ACSAC 2007. Twenty-Third Annual, pages 421-430. IEEE.
  25. Moskovitch, R., Feher, C., and Elovici, Y. (2008). Unknown malcode detectiona chronological evaluation. In Intelligence and Security Informatics, 2008. ISI 2008. IEEE International Conference on, pages 267-268. IEEE.
  26. Offensivecomputing (2015). Open malware.
  27. Peiravian, N. and Zhu, X. (2013). Machine learning for android malware detection using permission and api calls. In 2013 IEEE 25th International Conference on Tools with Artificial Intelligence , pages 300-305. IEEE.
  28. Pektas¸, A., Acarman, T., Falcone, Y., and Fernandez, J.-C. (2015). Runtime-behavior based malware classification using online machine learning. In 2015 World Congress on Internet Security (WorldCIS), pages 166- 171. IEEE.
  29. Pirscoveanu, R. S., Hansen, S. S., Larsen, T. M., Stevanovic, M., Pedersen, J. M., and Czech, A. (2015). Analysis of malware behavior: Type classification using machine learning. In Cyber Situational Awareness, Data Analytics and Assessment (CyberSA), 2015 International Conference on, pages 1-7. IEEE.
  30. Salehi, Z., Sami, A., and Ghiasi, M. (2014). Using feature generation from api calls for malware detection. Computer Fraud & Security, 2014(9):9-18.
  31. Sami, A., Yadegari, B., Rahimi, H., Peiravian, N., Hashemi, S., and Hamze, A. (2010). Malware detection based on mining api calls. In Proceedings of the 2010 ACM symposium on applied computing, pages 1020-1025. ACM.
  32. SCHICK, S. (2016). Security intelligence: Tinba malware watches mouse movements, screen activity to avoid sandbox detection.
  33. Scikit-learn (2013). Scikit-learn: machine learning in python.
  34. Shabtai, A., Moskovitch, R., Feher, C., Dolev, S., and Elovici, Y. (2012). Detecting unknown malicious code by applying classification techniques on opcode patterns. Security Informatics, 1(1):1-22.
  35. Symantec (2013a). Symantec: Symantec security response - virus naming conventions. response/ virusnaming.jsp.
  36. Symantec (2013b). Symantec: W32.sality!dam. response/writeup. jsp?docid=2013-043010-4816-99.
  37. Symantec (2015). Internet security threat report. response/ publications/threatreport.jsp.
  38. Symantec (2016a). Symantec: A-z listing of threats & risks. response/landing/ azlisting.jsp.
  39. Symantec (2016b). Symantec: Trojan.gen. response/writeup. jsp?docid=2010-022501-5526-99.
  40. Tian, R., Islam, R., Batten, L., and Versteeg, S. (2010). Differentiating malware from cleanware using behavioural analysis. In Malicious and Unwanted Software (MALWARE), 2010 5th International Conference on, pages 23-30. IEEE.
  41. Veeramani, R. and Rai, N. (2012). Windows api based malware detection and framework analysis. In International conference on networks and cyber security, volume 25.
  42. Virusshare (2016).
  43. VirusTotal (2015). Virustotal - free online virus, malware and url scanner.
  44. VX Heaven (2016).
  45. Walenstein, A. and Lakhotia, A. (2007). The software similarity problem in malware analysis. In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  46. Wang, C., Pang, J., Zhao, R., and Liu, X. (2009). Using api sequence and bayes algorithm to detect suspicious behavior. In Communication Software and Networks, 2009. ICCSN'09. International Conference on, pages 544-548. IEEE.
  47. Xu, J.-Y., Sung, A. H., Chavez, P., and Mukkamala, S. (2004). Polymorphic malicious executable scanner by api sequence analysis. In Hybrid Intelligent Systems, 2004. HIS'04. Fourth International Conference on, pages 378-383. IEEE.
  48. Yap, B. W., Rani, K. A., Rahman, H. A. A., Fong, S., Khairudin, Z., and Abdullah, N. N. (2014). An application of oversampling, undersampling, bagging and boosting in handling imbalanced datasets. In Proceedings of the First International Conference on Advanced Data and Information Engineering (DaEng2013), pages 13-22. Springer.
  49. Ye, Y., Chen, L., Wang, D., Li, T., Jiang, Q., and Zhao, M. (2009). Sbmds: an interpretable string based malware detection system using svm ensemble with bagging. Journal in computer virology, 5(4):283-293.
  50. Ye, Y., Li, T., Huang, K., Jiang, Q., and Chen, Y. (2010). Hierarchical associative classifier (hac) for malware detection from the large and imbalanced gray list. Journal of Intelligent Information Systems, 35(1):1-20.
  51. Ye, Y., Wang, D., Li, T., Ye, D., and Jiang, Q. (2008). An intelligent PE-malware detection system based on association mining. Journal in Computer Virology, 4(4):323-334.
  52. Zhang, B., Yin, J., Tang, W., Hao, J., and Zhang, D. (2006a). Unknown malicious codes detection based on rough set theory and support vector machine. In The 2006 IEEE International Joint Conference on Neural Network Proceedings, pages 2583-2587. IEEE.
  53. Zhang, B.-y., Yin, J.-p., Hao, J.-b., Zhang, D.-x., and Wang, S.-l. (2006b). Using support vector machine to detect unknown computer viruses. International Journal of Computational Intelligence Research, 2(1):100-104.
  54. Zhao, H., Xu, M., Zheng, N., Yao, J., and Ho, Q. (2010). Malicious executables classification based on behavioral factor analysis. In e-Education, e-Business, eManagement, and e-Learning, 2010. IC4E'10. International Conference on, pages 502-506. IEEE.

Paper Citation

in Harvard Style

Alruhaily N., Bordbar B. and Chothia T. (2017). Towards an Understanding of the Misclassification Rates of Machine Learning-based Malware Detection Systems . In Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, ISBN 978-989-758-209-7, pages 101-112. DOI: 10.5220/0006174301010112

in Bibtex Style

author={Nada Alruhaily and Behzad Bordbar and Tom Chothia},
title={Towards an Understanding of the Misclassification Rates of Machine Learning-based Malware Detection Systems},
booktitle={Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,},

in EndNote Style

JO - Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP,
TI - Towards an Understanding of the Misclassification Rates of Machine Learning-based Malware Detection Systems
SN - 978-989-758-209-7
AU - Alruhaily N.
AU - Bordbar B.
AU - Chothia T.
PY - 2017
SP - 101
EP - 112
DO - 10.5220/0006174301010112