Analyzing a Concurrent Self-Modifying Program: Application to Malware Detection

Walid Messahel, Tayssir Touili

2025

Abstract

We tackle the analysis problem of multi-threaded parallel programs that contain self modifying code, i.e., code that have the ability to reconstruct itself during the execution time. This kind of code is usually used to hide malicious portions of codes so that they cannot be detected by anti-viruses. In (Messahel and Touili, 2024), we introduced a new model called Self Modifying Dynamic Pushdown Network (SM-DPN) to model such programs. A SM-DPN is a network of Self-Modifying Pushdown Systems, i.e., Pushdown Systems that can modify their instructions on the fly during execution. We proposed an algorithm to perform the backward reachability analysis of SM-DPNs. However, in (Messahel and Touili, 2024), no concrete example was provided. In this paper, we go one step further. We consider a case study and show concretely how this approach and this model can be applied to represent and analyse an example of a multi-threaded self modifying code infected with a malware.

Paper Citation

in Harvard Style

Messahel W. and Touili T. (2025). Analyzing a Concurrent Self-Modifying Program: Application to Malware Detection. In Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP; ISBN 978-989-758-735-1, SciTePress, pages 176-182. DOI: 10.5220/0013103900003899

in Bibtex Style

@conference{icissp25,
author={Walid Messahel and Tayssir Touili},
title={Analyzing a Concurrent Self-Modifying Program: Application to Malware Detection},
booktitle={Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP},
year={2025},
pages={176-182},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013103900003899},
isbn={978-989-758-735-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP
TI - Analyzing a Concurrent Self-Modifying Program: Application to Malware Detection
SN - 978-989-758-735-1
AU - Messahel W.
AU - Touili T.
PY - 2025
SP - 176
EP - 182
DO - 10.5220/0013103900003899
PB - SciTePress