Profile Hidden Markov Model Malware Detection and API Call Obfuscation

Muhammad Ali, Monem Hamid, Jacob Jasser, Joachim Lerman, Samod Shetty, Fabio Di Troia

2022

Abstract

Profile Hidden Markov Models (PHMM) have been used to detect malware samples based on their behavior on the host system and obtained promising results. Since PHMMs are a novel way of categorizing malware and there is limited research work on such detection method, there is no data on the impact that certain obfuscation techniques have on PHMMs. An obfuscation tool that could weaken PHMM based detection has not yet been proposed. Our novel approach is based on applying PHMM detection by training the machine learning models on API calls that are dynamically extracted from the malware samples, and then attempting to elude detection by the same models using obfuscation techniques. Hence, in our paper, we created a PHMM model trained on API call sequences extracted by running malware in a sandbox, then we tried to undermine the detection effectiveness by applying different state-of-the-art API obfuscation techniques to the malware. By implementing sophisticated API calls obfuscation techniques, we were able to reduce the PHMM detection rate from 1.0, without API call obfuscation, to 0.68.

Paper Citation

in Harvard Style

Ali M., Hamid M., Jasser J., Lerman J., Shetty S. and Di Troia F. (2022). Profile Hidden Markov Model Malware Detection and API Call Obfuscation. In Proceedings of the 8th International Conference on Information Systems Security and Privacy - Volume 1: ForSE, ISBN 978-989-758-553-1, pages 688-695. DOI: 10.5220/0011005800003120

in Bibtex Style

@conference{forse22,
author={Muhammad Ali and Monem Hamid and Jacob Jasser and Joachim Lerman and Samod Shetty and Fabio Di Troia},
title={Profile Hidden Markov Model Malware Detection and API Call Obfuscation},
booktitle={Proceedings of the 8th International Conference on Information Systems Security and Privacy - Volume 1: ForSE,},
year={2022},
pages={688-695},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011005800003120},
isbn={978-989-758-553-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 8th International Conference on Information Systems Security and Privacy - Volume 1: ForSE,
TI - Profile Hidden Markov Model Malware Detection and API Call Obfuscation
SN - 978-989-758-553-1
AU - Ali M.
AU - Hamid M.
AU - Jasser J.
AU - Lerman J.
AU - Shetty S.
AU - Di Troia F.
PY - 2022
SP - 688
EP - 695
DO - 10.5220/0011005800003120