Deep Q-Networks for Imbalanced Multi-Class Malware Classification

Antonio Maci, Giuseppe Urbano, Antonio Coscia

2024

Abstract

Nowadays, defending against malware-induced computer infections represents a key concern for both individuals and companies. Malware detection relies on analyzing the static or dynamic features of a file to determine whether it is malicious or not. In the case of dynamic analysis, the sample behavior is examined by performing a thorough inspection, such as tracking the sequence of functions, also called Application Programming Interfaces (APIs), executed for malicious purposes. Current machine learning paradigms, such as Deep Learning (DL), can be exploited to develop a classifier capable of recognizing different categories of malicious software for each API flow. However, some malware families are less numerous than others, leading to an imbalanced multi-class classification problem. This paper compares Deep Reinforcement Learning (DRL) algorithms that combine Reinforcement Learning (RL) with DL models to deal with class imbalance for API-based malware classification. Our investigation involves multiple configurations of Deep Q-Networks (DQNs) with a proper formulation of the Markov Decision Process that supports cost-sensitive learning to reduce bias due to majority class dominance. Among the algorithms compared, the dueling DQN showed promising macro F1 and area under the ROC curve scores in three test scenarios using a popular benchmark API call dataset.

Paper Citation

in Harvard Style

Maci A., Urbano G. and Coscia A. (2024). Deep Q-Networks for Imbalanced Multi-Class Malware Classification. In Proceedings of the 10th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP; ISBN 978-989-758-683-5, SciTePress, pages 342-349. DOI: 10.5220/0012303800003648

in Bibtex Style

@conference{icissp24,
author={Antonio Maci and Giuseppe Urbano and Antonio Coscia},
title={Deep Q-Networks for Imbalanced Multi-Class Malware Classification},
booktitle={Proceedings of the 10th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP},
year={2024},
pages={342-349},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012303800003648},
isbn={978-989-758-683-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP
TI - Deep Q-Networks for Imbalanced Multi-Class Malware Classification
SN - 978-989-758-683-5
AU - Maci A.
AU - Urbano G.
AU - Coscia A.
PY - 2024
SP - 342
EP - 349
DO - 10.5220/0012303800003648
PB - SciTePress