The Curious Case of Machine Learning in Malware Detection

Sherif Saad

, William Briguglio

and Haytham Elmiligi

School of Computer Science, Windsor University, Canada

Computing Science Department, Thompson Rivers University, Canada

Keywords:

Malware, Machine Learning, Behaviour Analysis, Adversarial Malware, Online Training, Detector Interpre-

tation.

Abstract:

In this paper, we argue that detecting malware attacks in the wild is a unique challenge for machine learn-

ing techniques. Given the current trend in malware development and the increase of unconventional malware

attacks, we expect that dynamic malware analysis is the future for antimalware detection and prevention sys-

tems. A comprehensive review of machine learning for malware detection is presented. Then, we discuss how

malware detection in the wild present unique challenges for the current state-of-the-art machine learning tech-

niques. We deﬁned three critical problems that limit the success of malware detectors powered by machine

learning in the wild. Next, we discuss possible solutions to these challenges and present the requirements of

next-generation malware detection. Finally, we outline potential research directions in machine learning for

malware detection.

1 INTRODUCTION

Nowadays, computer networks and the Internet have

become the primary tool for spreading and distribut-

ing malware by malware authors. The massive num-

ber of feature-rich programming languages and off-

the-shelf software libraries enable the development of

new sophisticated malware such as botnet, ﬁleless,

k-ary and ransomware. New computing paradigms,

such as cloud computing and the Internet of Things,

expand potential malware infection sites from PC’s to

any electronic device.

To decide if software code is malicious or benign

we could either use static analysis or dynamic analy-

sis. Static analysis techniques do not execute the code

and only examine the code structure and other binary

data properties. Dynamic analysis techniques, on the

other hand, execute the code to observe the execution

behaviors of the code over the network or at endpoint

devices. Some malware detection systems apply only

static or dynamic techniques, and some apply both.

While dynamic malware analysis techniques are not

intended to replace static analysis techniques, recent

unconventional malware attacks (botnet, ransomware,

ﬁleless, etc) and the use of sophisticated evasion tech-

niques to avoid detection have shown the urgent need

of dynamic analysis and the limitations of static anal-

ysis. In our opinion, the use of dynamic and beha-

vioral malware analysis will dominate the next-

generation malware detection systems.

There is a general belief among cybersecurity ex-

perts that antimalware tools and systems powered by

artiﬁcial intelligence and machine learning will be the

solution to modern malware attacks. The number of

studies published in the last few years on malware

detection techniques that leverage machine learning

is a distinct evidence of this belief as shown in sec-

tion 2. In the literature, various malware detection

techniques using machine learning are proposed with

excellent detection accuracy. However, malware at-

tacks in the wild continue to grow and manage to

bypass malware detection systems powered by ma-

chinelearning techniques. This because it is difﬁcult

to operate and deploy machine learning for malware

detection in a production environment or the perfor-

mance in a production environment is disturbing (e.g.,

high false positives rate). In fact, there is a signif-

icant difference (a detection gap) between the accu-

racy of malware detection techniques in the literature

and their accuracy in a production environment.

A perfect malware detection system will detect all

types of malicious software and will never consider a

benign software as a malicious one. Cohen provided a

formal proof that creating a perfect malware detection

system is not possible (Cohen, 1987; Cohen, 1989).

Moreover, Chess and White proved that a malware

528

Saad, S., Briguglio, W. and Elmiligi, H.

The Curious Case of Machine Learning in Malware Detection.

DOI: 10.5220/0007470705280535

In Proceedings of the 5th International Conference on Information Systems Security and Privacy (ICISSP 2019), pages 528-535

ISBN: 978-989-758-359-9

detector with zero false positives is not possible

(M. Chess and R. White, 2000). Selcuk et al. dis-

cussed the undecidable problems in malware detec-

tion in more details (Selcuk et al., 2017). In light of

this, the high levels of accuracy claimed by commer-

cial malware detection systems and some malware de-

tection studies in literature seems questionable.

In this paper, we brieﬂy review the current state

of the art in malware detection using machine learn-

ing approach. Then, we discuss the importance of

dynamic and behavioral analysis based on emerg-

ing malware threats. Next, the shortcomings of the

current machine learning malware detectors are ex-

plained to indicate their limitations in the wild. Fi-

nally, we discuss the possible solutions to improve

the quality of malware detection systems and point

out potential research directions.

2 LITERATURE REVIEW

In recent years, machine learning algorithms have

been used to design both static and dynamic analy-

sis techniques for malware detection. Hassen et al.

proposed a new technique for malware classiﬁcation

using static analysis based on control statement shin-

gling (Hassen et al., 2017). In their work, they used

static analysis to classify malware instance into new

or known malware families. They extracted features

from disassembled malicious binaries and used ran-

dom forest algorithm to classify malware using the

extracted features. Using a dataset of 10,260 malware

instances, they reported up to 99.21% accuracy.

Static analysis has been used to study malwares

that infect embedded systems, mobile devices, and

other IoT devices. Naeem et al. proposed a static

analysis technique to detect IoT malware (Naeem

et al., 2018). The proposed technique converts a mal-

ware ﬁle to a grayscale image and extracts a set of

visual features from the malware image to train an

SVM classiﬁer that could distinguish between mal-

ware families using visual features. Using a dataset

of 9342 samples that belong to 25 malware families,

they reported 97.4% accuracy. Su et al. proposed a

similar technique to classify IoT malware into mal-

ware families using visual features and image recog-

nition (Su et al., 2018). Their approach is very simi-

lar to the one proposed in (Naeem et al., 2018). They

used a one-class SVM classiﬁer and tested their ap-

proach on IoT malwares that infect Linux-like IoT

systems; they reported 94.0% accuracy for detect-

ing malware and 81.8% accuracy for detecting mal-

ware families. Raff et al. proposed a malware detec-

tion technique using static analysis and deep learning

(Raff et al., 2017). The proposed technique achieved

94.0% detection accuracy.

Several works have been proposed to detect An-

droid malware apps using static analysis techniques.

Sahin et al. proposed an Android malware detection

model that uses app permission to detect malicious

apps (OSahn et al., 2018). They used the permissions

required by the app with a weighted distance function

and KNN and Naive Bayes classiﬁer to detect mali-

cious apps. They reported an accuracy up to 93.27%.

Su and Fung used sensitive functions and app permis-

sions to detect Android malware (Su and Fung, 2016).

They used different machine learning algorithms such

as SVM, decision tree, and KNN to build an android

malware detector. They reported an average accuracy

between 85.0% and 90.0%

Collecting and monitoring all malware behaviors

is a complicated and time-consuming process. For

that reason, several works in the literature focused

on collecting partial dynamic behaviors of the mal-

ware. Lim et al. (Lim et al., 2015) proposed a mal-

ware detection technique by analyzing network traf-

ﬁc generated when the malware communicates with

a malicious C&C server such as in the case of bot-

net or ransomware. The proposed technique extracts

a set of features from network ﬂows to present a ﬂows

sequence. The authors used different sequence align-

ment algorithms to classify malware trafﬁc. They re-

ported an accuracy above 60% when analyzing mal-

ware trafﬁc in a real network environment.

Kilgallon et al. applied machine learning and dy-

namic malware analysis (Kilgallon et al., 2017). The

proposed technique gathers register value information

and API calls made by the monitored malware bi-

naries. The collected information is stored in vec-

tor structures and analyzed using a value set analysis

method. Then, they used a linear similarity metric to

compare unseen malware to known malware binaries.

Their experiment showed that the proposed technique

could detect malware with an accuracy up to 98.0%

Omind and Nathan proposed a behavioral-based

malware detection method using a deep belief net-

work (David and Netanyahu, 2015). The proposed

method collected data about malware behaviors from

a sandbox environment. The collected data is API

calls, registry entries, visited websites, accessed ports,

and IP addresses. Then using a deep neural net-

work of eight layers, it generates malware signatures.

These signatures could be used to train malware de-

tectors. In their experiments, they reported up to

95.3% detection accuracy with a malware detector

utilizing the SVM algorithm.

Yeo et al. proposed a new malware detection

method by monitoring malicious behaviors in net-

The Curious Case of Machine Learning in Malware Detection

529

work trafﬁc (Yeo et al., 2018). They designed 35

features to describe malicious trafﬁc of malware in-

stances. They tested several machine learning al-

gorithms including CNN, MLP, SVM, and random

forest. The proposed method achieved an accuracy

above 85% when utilizing CNN or random forest.

Prokoﬁev et al. proposed a machine-learning tech-

nique to detect C&C trafﬁc of infected IoT devices

(Prokoﬁev et al., 2018). The proposed approach used

network trafﬁc features such as port number, IP ad-

dresses, connection duration and frequency. They re-

ported a detection accuracy up to 97.3%. However,

the proposed approach is still relying on traditional

malware analysis methods and will not be able to

work in production IoT deployment as discussed in

(Soliman et al., 2017). Several hybrid malware de-

tection techniques that combine both static and dy-

namic analysis have also been proposed (Martinelli

et al., 2016; Paola et al., 2018). These techniques try

to improve the quality and performance of malware

detection systems by taking advantage of static and

dynamic analysis to build robust malware detection

systems.

3 EMERGING MALWARE

THREATS

With the recent changes in malware development

and the rise of commercial malware (malicious code

rented or purchased), many new challenges are facing

malware analysts that make static analysis more difﬁ-

cult and impractical. These challenges will force an-

timalware vendors to adapt behavioral malware anal-

ysis and detection techniques. In our opinion, there

are two main reasons behind these challenges; the rise

of unconventional computing paradigms and uncon-

ventional evasion techniques. There is a new gener-

ation of malwares that take advantage of unconven-

tional computing paradigms and off-the-shelf soft-

ware libraries written by feature-rich programming

languages. The current state-of-the-art malware anal-

ysis/detection techniques and tools are not effective

against this new generation of malware.

3.1 Unconventional Computing

Paradigms

New computing paradigms and technologies such as

cloud computing, the internet of things, big data, in-

memory computing, and blockchain introduced new

playgrounds for malware authors to develop com-

plex and sophisticated malwares that are almost un-

detectable. Here we describe several recent examples

of new malware threats that are difﬁcult to detect or

analyze using static analysis.

For instance, the Internet of Things (IoT) is an ap-

pealing platform for modern and sophisticated mal-

ware scuh as ransomware. Zhang-Kennedy et al. dis-

cussed the ransomware threat in IoT and how a self-

spreading ransomware could infect an IoT ecosystem

(Zhang-Kennedy et al., 2018). The authors pointed

out that the ransomware will mainly lock down IoT

devices and disable the essential functions of these

devices. The study focused on identifying the attack

vectors in IoT, the techniques for ransomware self-

spreading in IoT, and predicting the most likely class

of IoT applications to be a target for ransomware at-

tacks. Finally, the authors identiﬁed the techniques

the ransomware could apply to lock down IoT de-

vices. Authors in (Zhang-Kennedy et al., 2018) used

a Raspberry to develop a proof of concept IoT ran-

somware that can infect an IoT system. One inter-

esting aspect in (Zhang-Kennedy et al., 2018) is the

need for collaboration or swarming behavior in IoT

ransomware, where the IoT ransomware will spread

as much as possible and then lock down the devices

or lock down the device and then spread.

Miller and Valasek developed a proof-of-concept

for malicious code that infects connected cars and

lockdown key functions (Miller and Valasek, 2015).

For instance, the authors demonstrated the ability for

the malicious code to control the steering wheel of a

vehicle, disable the breaks, lock doors, and shut down

the engine while in motion. Behaving as ransomware,

this real example of a malware that locks and disables

key features in IoT systems (e.g. connected cars)

could have life threatening consequences if the ran-

som is not paid. The study explained a design ﬂow

in the Controller Area Network (CAN) protocol that

allows malicious and crafted CAN messages to be in-

jected into the vehicle CAN channel by a compro-

mised mobile phone that is connected to the vehicle

entertainment unit. It was reported that for some ve-

hicles only the dealership could restore and patch the

vehicle to prevent this attack. Choi et al. proposed a

solution for malware attacks in connected vehicles us-

ing machine learning (Choi et al., 2018). The solution

uses SVM to distinguish between crafted malicious

CAN messages, and benign CAN messages generated

by actual electronic control units (ECU). The model

extracts features from the vehicle ECUs and creates

ﬁngerprints for those ECUs. The ECU ﬁngerprint is

noticeable in a benign CAN message and does not ex-

ist in a malicious message

Azmoodeh et al. discussed a new technique to de-

tect ransomware attacks in IoT systems by monitor-

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

530

ing the energy consumption of infected devices (Az-

moodeh et al., 2018). As a proof of concept, they

studied the energy consumption of infected Android

devices. The devices were infected by a ransomware

with crypto impact. They used different machine

learning models (KNN, SVM, NN, and Random For-

est) to analyze energy consumption data and extract

unique patterns to detect compromised Android de-

vices. They reported a ransomware detection accu-

racy of 95.65%.

In 2015, Karam (INTERPOL) and Kamluk

(Kaspersky lab) introduced a proof of concept

distributed malware that also takes advantage of

blockchain technology (Karam and Kamluk, 2015).

In 2018, Moubarak and et al. provided design and

implementation of a K-ary malware (distributed mal-

ware) that takes advantages of the blockchain net-

works such as Etherum and Hyperledger (Moubarak

et al., 2018). The proposed malware is stored and ex-

ecuted inside blockchain networks and acts as a ma-

licious keylogger. While detecting a K-ary malware

is an NP-hard problem(de Dr

ezigu

e et al., 2006), it

is also complicated to implement a K-ary malware.

However, Mubarak’s works demonstrated the sim-

plicity of K-ary malware development by taking ad-

vantage of blockchain technology as distributed and

decentralized network.

3.2 Unconventional Evasion Techniques

The new generation of malware will use advanced

evasion techniques to avoid detection by antimalware

systems and tools. New evasion techniques imple-

mented by malware authors use new technologies and

off-the-shelf software libraries that enable the design

of sophisticated evasion methods. Antimalware ven-

dors and malware researchers discussed recent exam-

ples of using new antimalware evasion techniques in

the wild.

Fileless malware or memory-resident malware is

the new technique used by malware authors to de-

velop and execute malicious attacks. Fileless mal-

ware resides in device memory and does not leave any

ﬁles on the infected device ﬁle system. This makes

the detection of the ﬁleless malware using signature-

based detection or static analysis infeasible. In addi-

tion, the ﬁleless malware takes advantage of the utili-

ties and libraries that already exist in the platform of

the infected device to complete its malicious intents.

In other words, benign applications and software li-

braries are manipulated by ﬁleless malware to accom-

plish the attack objectives.

Fileless malware attacks and incidents are already

observed in the wild compromising large enterprises.

According to KASPERSKY lab, 140 enterprises were

attacked in 2017 using ﬁleless malwares (Global Re-

search and Analysis Team, KASPERSKY Lab, 2017).

Ponemon Institute reported that 77% of the attacks

against companies use ﬁleless techniques (Ponemon

Institute, Research Report, 2017). Moreover, there

are several signs that ransomware attacks are going

ﬁleless, as discussed in (Magnusardottir, 2018). Be-

sides these signs, there are other reasons in our opin-

ion that conﬁrms that ransomware and other malware

attacks will be ﬁleless. One main reason is the mov-

ing towards in-memory computing.

In recent years, in-memory computing and in-

memory data stores became the ﬁrst backbone and

storage technology for many organizations. Many big

data platforms and data grids (Apache Spark, Redis,

HazelCast, etc.) enable storing data in memory for

performance and scalability requirements. Valuable

data and information is stored in memory for a long

time before moving to a persistent data store. In-

Memory ransomware that encrypts in-memory data

(such as recent transactions, ﬁnancial information,

etc.) present a severe and aggressive attack. This is

because any attempt to reset or report the machine

to remove the ransomware from the device memory

or shutdown the application will result in losing this

valuable data permanently.

The moving towards distributed and decentralized

computing is another reason for the rise of ﬁleless ran-

somware. In distributed and decentralized computing

several nodes and devices are available to store the

in-memory malware, which will increase the life ex-

pectancy of the malware since there will always be a

group of active nodes were the malware could repli-

cate and store itself.

The recent and massive development in machine

learning /artiﬁcial intelligence (aka data science) and

a large number of off-the-shelf machine learning li-

braries enable malware authors to develop advanced

evasion techniques.

Rigaki and Garcia proposed the use of deep learn-

ing techniques to create malicious malware sam-

ples that evade detection by mimicking the behav-

iors of benign applications (Rigaki and Garcia, 2018).

In their work, a proof of concept was proposed to

demonstrate how malware authors could cover the

malware C&C trafﬁc. The authors use a Genera-

tive Adversarial Networks (GANs) to enable malware

(e.g., botnet) to mimic the trafﬁc of a legitimate ap-

plication and avoid detection. The study showed that

it is possible to modify the source code of malware

to receive parameters from a GAN to change the be-

haviors of its C&C trafﬁc to mimic the behaviors of

other legitimate network applications, such as Face-

The Curious Case of Machine Learning in Malware Detection

531

book trafﬁc. The enhanced malware samples were

tested against the Stratosphere Linux IPS (slips) sys-

tem, which uses machine learning to detect malicious

trafﬁc. The experiment showed that 63.42% of the

malicious trafﬁc was able to bypass the detection.

A research team from IBM demonstrated the use

of artiﬁcial intelligence to engineering malware at-

tacks (Kirat et al., 2018). In their study, the authors

proposed DeepLocker as a proof of concept to show

how next-generation malware could leverage artiﬁcial

intelligence. DeepLocker is a malware generation en-

gine that malware author could use to empower tra-

ditional malware samples such as WannaCry with ar-

tiﬁcial intelligence. A deep convolutional neural net-

work (CNN) was used to customize a malware attack

by combining a benign application and a malware

sample to generate a hybrid malware that bypasses

detection by exposing (mimicking) benign behaviors.

Besides that, the malware is engineered to unlock its

malicious payload when it reaches a target (endpoint)

with a loose predeﬁned set of attributes. In the study,

those attributes were the biometrics feature of the tar-

get such as facial and voice features. The malware

uses CNN to detect and conﬁrm target identity, and

upon target conﬁrmation, an encryption key is gen-

erated and used by the WannCry malware to encrypt

the ﬁles on the target endpoint device. The encryption

key is only generated by matching the voice and the

facial features of the target. This means reverse engi-

neering the malware using static analysis is not useful

to recover the encryption key.

4 PRACTICAL CHALLENGES

The new and emerging malware threats discussed in

section 3 provide strong evidence for the need of

adopting dynamic and behavioral analysis to build

malware detection tools. The use of machine learning

is the most promising technique to implement mal-

ware detectors and tools that apply behavioral analy-

sis as shown in section 2. While the use of machine

learning for malware detection has shown promising

results in both static and dynamic analysis, there are

signiﬁcant challenges that limit the success of ma-

chine learning based malware detectors in the wild.

4.1 Cost of Training Detectors

The ﬁrst challenge is the cost of training and updating

malware detectors in production environment. Mal-

ware detection is unlike other domains where ma-

chine learning techniques have been applied success-

fully such as computer vision, natural language pro-

cessing, and e-commerce. Malware instances evolve

and change their behaviors over a short period; some

studies by antimalware vendors reported that a new

malware instance could change its behaviors in less

than 24 hours since it has been released (Gupta et al.,

2009; Allix et al., 2015). This means a frequently

trained machine learning model will become out-

dated. This also means we need to frequently retrain

our malware detectors to be able to detect new and

mutated malware instances. Therefore, adaptability

in machine learning models for malware detection is

a crucial requirement and not just a ancillary capabil-

ity.

Recently, the challenge of adaptability, and scal-

ability of machine learning models for malware de-

tection in the wild has become obvious (Narayanan

et al., 2016). The majority of the work proposed in

the literature have done very little to reduce and op-

timize the feature space to design detectors ready for

early malware detection in a production environment

(Hajmasan et al., 2017). For instance, it is not clear

how the proposed detection methods will scale when

the number of monitored endpoints increase. Un-

like computer vision, natural language processing and

other areas that utilize machine learning, malware in-

stances continue to evolve and change. This mostly

requires retraining machine learning models in pro-

duction, which is an expensive and complicated task.

Therefore, when using machine learning for malware

detection, we need to think differently. New methods

to reduce the cost of retraining malware detectors and

improve the detection quality are urgent.

4.2 Malware Detector Interpretability

Cybersecurity analysts always prefer solutions that

are interpretable and understandable, such as rule-

based or signature-based detection. This is because

of the need to tune and optimize these solutions to

mitigate and control the effect of false positives and

false negatives. Interpreting machine learning mod-

els is a new and open challenge (Shirataki and Ya-

maguchi, 2017). However, it is expected that an in-

terpretable machine learning solution will be domain

speciﬁc, for instance, interpretable solutions for ma-

chine learning models in healthcare are different than

solutions in malware detection (Ahmad et al., 2018).

Any malware detector will generate false posi-

tives, and unless malware analysts can understand and

interpret the reason that a benign application wrongly

classiﬁed as malicious, they will not accept those

black box malware detectors. To our knowledge, no

work in the literature investigated the interpretability

of machine learning models for malware detection.

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

532

4.3 Adversarial Malware

Last but not least, a malware detection system utiliz-

ing machine learning could be defeated (bypassed)

using adversarial malware samples. For instance,

Kolosnjaji et al. showed in (Kolosnjaji et al., 2018)

that by using an intelligent evasion attack they can

defeat the deep learning detection system proposed in

(Raff et al., 2017) by Raff et al. They simply used

their knowledge of how the proposed deep learning

detection system operates and designed a gradient-

based attack as an evasion technique to overcome it.

With adversarial malware, the system detection accu-

racy dropped from 94.0% to almost 50.0%. Machine

learning algorithms are not designed to work with ad-

versarial examples. Grosse et al. demonstrated that

using adversarial malware samples; they could reduce

the detection accuracy of a malware detection sys-

tem that uses static analysis and machine learning to

63.0% (Grosse et al., 2017). They also showed that

adopting anti adversarial machine learning techniques

used in computer vision is not effective in malware

detection. Yang et al. proposed adversarial training as

a solution for adversarial malware (Yang et al., 2017).

They designed a method for adversarial android mal-

ware instances generation. The proposed method re-

quires access to the malware binaries and source code,

besides, it is mainly useful for static malware detec-

tion systems.

5 BRIDGING THE DETECTION

GAP

To overcome the challenges we discussed in section 4,

we propose new solutions to mitigate these challenges

and reduce the gap.

5.1 Disposable Micro Detectors

Current best practices in constructing and build-

ing machine learning models follow a mono-

lithic architecture. In monolithic architecture,

a computationally-expensive single-monolithic (to

build and train) machine learning model is used to

detect malwares. While this architecture or approach

for building machine learning models is successful in

other domains, we believe it is unsuitable for mal-

ware detection given the highly evolving character-

istics of malware instance. We propose a new ap-

proach inspired by microservices architecture. In this

approach, multiple, small, inexpensive, focused ma-

chine learning models are built and orchestrated to

detect malware instances. Each model or detector

is built to detect the behaviors of a speciﬁc malware

instance (e.g., Mirai, WannaCry), or at most a sin-

gle malware family (a group of similar malware in-

stances). Also, each model or detector is built us-

ing features that are similar, such as having the same

computational cost, or unique to the speciﬁc execu-

tion environment. This is because out of the superset

of features designed to detect malware, it is common

that a subset of these features could be more or less

useful to detect a speciﬁc malware instance or fam-

ily. The use of micro (small) and focused detectors

reduce the cost of retraining and deployment in pro-

duction. This is because detectors for new malware

could be trained and added without the need to retrain

existing detectors. In addition, when a malware de-

tector becames outdated as a result of malware evolv-

ing behaviors, the outdated detectors are disposed of

and replaced by new ones. The use of micro-detectors

enables adaptability by design rather than attempting

to change machine learning models and algorithms to

support adaptability.

5.2 Analyst Friendly Interpretation

Adopting sophisticated machine learning techniques

for malware detection in a production environment is

a challenge. This is because most of the time it is not

possible to understand how the machine learning sys-

tems make their malware detection decisions. There-

fore, tuning and maintaining these systems is a chal-

lenge in production and new techniques for malware

analysts to interpret and evaluate the performance of

malware detectors are needed. We propose the use

of evolutionary computation techniques such as ge-

netic algorithms or clonal selection algorithms to gen-

erate an interpretation for black-box machine learn-

ing models such as deep learning. Using evolutionary

computation, we could describe the decisions of mal-

ware detectors using a set of IF-Then rules. The only

information required is the input features the malware

detector uses to make a decision.

The IF-Then rules are useful to explain the behav-

iors that trigger a speciﬁc decision (e.g., malicious

or benign) by the malware detector. Cybersecurity

and malware analyst are comfortable working with

IF-Then rules. These rules will help in understand-

ing the decision made by malware detectors, explain

the scope of the detection, and identify potential over

generalization or overﬁtting that could result in false

positives or false negatives.

It is essential that the IF-Then rules set interpre-

tation of the malware detector to be expressed in raw

malware behaviors and not in machine learning fea-

tures. Machine learning features are most likely un-

The Curious Case of Machine Learning in Malware Detection

533

derstandable by machine learning engineers and ex-

perts. The interpretation should be acceptable to a

malware analyst who does not need to be machine

learning experts.

5.3 Anti Adversarial Malware

To improve the resilience of malware detectors

against adversarial malware, we believe it is essential

to study the effort required by the malware authors

to design an adversarial malware for speciﬁc malware

detectors. For example, what technique a malware au-

thor would use to probe and study a malware detector

in production to design a malware that could bypass

this detector.

Measuring the effort to probe detectors and design

adversarial malware under two main settings is essen-

tial. The ﬁrst setting is black-box, where the malware

authors have minimum knowledge about the malware

detector internal design and the features used by the

machine learning algorithm. The second setting is

white-box, where the malware authors have sufﬁcient

knowledge about the malware detector internal design

and the machine learning algorithm. Training and up-

dating the malware detectors is likely the most efﬁ-

cient solution against adversarial malware. Knowing

the effort needed to evade a malware detector will

help in designing training strategies and policies to

increase the effort required to evade the detectors.

As we mentioned before, Cohen provided a for-

mal proof that creating a perfect malware detection

system is not possible (Cohen, 1987; Cohen, 1989).

We believe that designing a perfect adversarial mal-

ware is not possible. Therefore we expect that using

ensemble-based hybrid machine learning approach

for malware detector will be effective against ad-

versarial malware. It is expected that by creating a

malware detector using an ensemble hybrid machine-

learning approach, the risk of evading detection will

decrease and the effort to design adversarial malware

will increase. A hybrid machine learning model is

when two or more different machine learning algo-

rithms are used to construct the model. In the litera-

ture, adversarial malware samples evade malware de-

tectors that use a single machine learning algorithm

or technique (Yang et al., 2017; Grosse et al., 2017;

Kolosnjaji et al., 2018). In our method, a hybrid ma-

chine learning approach for building a malware de-

tector is an approach to provide a defense-in-depth

model for malware detectors.

6 CONCLUSION

In this paper, we reviewed the current state-of-the-art

in malware detection using machine learning. We dis-

cussed the recent trends in malware development and

emerging malware threats. We argued that behavioral

analysis would dominate the next generation antimal-

ware systems. We discussed the challenges of apply-

ing machine learning to detect malware in the wild

and proposed our thoughts on how we could over-

come these challenges. Machine learning malware

detectors require inexpensive training methods; they

need to be interpretable for the malware analysts and

not only for machine learning experts. Finally, they

need to tolerate adversarial malware by design.

REFERENCES

Ahmad, M. A., Teredesai, A., and Eckert, C. (2018). In-

terpretable machine learning in healthcare. In 2018

IEEE International Conference on Healthcare Infor-

matics (ICHI), pages 447–447.

Allix, K., Bissyand

e, T. F., Klein, J., and Le Traon, Y.

(2015). Are your training datasets yet relevant? In

Piessens, F., Caballero, J., and Bielova, N., editors,

Engineering Secure Software and Systems, pages 51–

67, Cham. Springer International Publishing.

Azmoodeh, A., Dehghantanha, A., Conti, M., and Choo,

K.-K. R. (2018). Detecting crypto-ransomware in

iot networks based on energy consumption footprint.

Journal of Ambient Intelligence and Humanized Com-

puting, 9(4):1141–1152.

Choi, W., Joo, K., Jo, H. J., Park, M. C., and Lee, D. H.

(2018). Voltageids: Low-level communication char-

acteristics for automotive intrusion detection system.

IEEE Transactions on Information Forensics and Se-

curity, 13(8):2114–2129.

Cohen, F. (1987). Computer viruses: Theory and experi-

ments. Computers & Security, 6(1):22 – 35.

Cohen, F. (1989). Computational aspects of computer

viruses. Computers & Security, 8(4):297 – 298.

David, O. E. and Netanyahu, N. S. (2015). Deepsign: Deep

learning for automatic malware signature generation

and classiﬁcation. In 2015 International Joint Confer-

ence on Neural Networks (IJCNN), pages 1–8.

de Dr

ezigu

e, D., Fizaine, J.-P., and Hansma, N. (2006). In-

depth analysis of the viral threats with openofﬁce.org

documents. Journal in Computer Virology, 2(3):187–

210.

Global Research and Analysis Team, KASPERSKY Lab

(2017). Fileless attack against enterprise network.

Grosse, K., Papernot, N., Manoharan, P., Backes, M.,

and McDaniel, P. (2017). Adversarial examples for

malware detection. In Foley, S. N., Gollmann, D.,

and Snekkenes, E., editors, Computer Security – ES-

ORICS 2017, pages 62–79, Cham. Springer Interna-

tional Publishing.

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

534

Gupta, A., Kuppili, P., Akella, A., and Barford, P. (2009).

An empirical study of malware evolution. In 2009

First International Communication Systems and Net-

works and Workshops, pages 1–10.

Hajmasan, G., Mondoc, A., and Cre, O. (2017). Dynamic

behavior evaluation for malware detection. In 2017

5th International Symposium on Digital Forensic and

Security (ISDFS), pages 1–6.

Hassen, M., Carvalho, M. M., and Chan, P. K. (2017). Mal-

ware classiﬁcation using static analysis based features.

In 2017 IEEE Symposium Series on Computational

Intelligence (SSCI), pages 1–7.

Karam, C. and Kamluk, V. (2015). Blockchainware - de-

centralized malware on the blockchain. In Black Hat

ASIA.

Kilgallon, S., Rosa, L. D. L., and Cavazos, J. (2017). Im-

proving the effectiveness and efﬁciency of dynamic

malware analysis with machine learning. In 2017 Re-

silience Week (RWS), pages 30–36.

Kirat, Jiyong, and Stoecklin (2018). Deeplocker concealing

targeted attacks with ai locksmithing.

Kolosnjaji, B., Demontis, A., Biggio, B., Maiorca, D., Gi-

acinto, G., Eckert, C., and Roli, F. (2018). Adversarial

malware binaries: Evading deep learning for malware

detection in executables. CoRR, abs/1803.04173.

Lim, H., Yamaguchi, Y., Shimada, H., and Takakura, H.

(2015). Malware classiﬁcation method based on se-

quence of trafﬁc ﬂow. In 2015 International Con-

ference on Information Systems Security and Privacy

(ICISSP), pages 1–8.

M. Chess, D. and R. White, S. (2000). An undetectable

computer virus. In Proceedings of Virus Bulletin Con-

ference.

Magnusardottir, A. (2018). Fileless ransomware: How it

works & how to stop it? White Paper.

Martinelli, F., Mercaldo, F., Saracino, A., and Visaggio,

C. A. (2016). I ﬁnd your behavior disturbing: Static

and dynamic app behavioral analysis for detection of

android malware. In 2016 14th Annual Conference on

Privacy, Security and Trust (PST), pages 129–136.

Miller, C. and Valasek, C. (2015). Remote exploitation of

an unaltered passenger vehicle. White Paper.

Moubarak, J., Chamoun, M., and Filiol, E. (2018). Devel-

oping a k-ary malware using blockchain. In NOMS

2018 - 2018 IEEE/IFIP Network Operations and

Management Symposium, pages 1–4.

Naeem, H., Guo, B., and Naeem, M. R. (2018). A light-

weight malware static visual analysis for iot infras-

tructure. In 2018 International Conference on Artiﬁ-

cial Intelligence and Big Data (ICAIBD), pages 240–

244.

Narayanan, A., Yang, L., Chen, L., and Jinliang, L.

(2016). Adaptive and scalable android malware de-

tection through online learning. In 2016 International

Joint Conference on Neural Networks (IJCNN), pages

2484–2491.

OSahn, D., Kural, O. E., Akleylek, S., and Kilic¸, E. (2018).

New results on permission based static analysis for an-

droid malware. In 2018 6th International Symposium

on Digital Forensic and Security (ISDFS), pages 1–4.

Paola, A. D., Gaglio, S., Re, G. L., and Morana, M. (2018).

A hybrid system for malware detection on big data. In

IEEE INFOCOM 2018 - IEEE Conference on Com-

puter Communications Workshops (INFOCOM WK-

SHPS), pages 45–50.

Ponemon Institute, Research Report (2017). The 2017 state

of endpoint security risk.

Prokoﬁev, A. O., Smirnova, Y. S., and Surov, V. A. (2018).

A method to detect internet of things botnets. In

2018 IEEE Conference of Russian Young Researchers

in Electrical and Electronic Engineering (EIConRus),

pages 105–108.

Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro,

B., and Nicholas, C. K. (2017). Malware detection by

eating a whole exe. CoRR, abs/1710.09435.

Rigaki, M. and Garcia, S. (2018). Bringing a gan to a knife-

ﬁght: Adapting malware communication to avoid de-

tection. In 2018 IEEE Security and Privacy Work-

shops (SPW), pages 70–75.

Selcuk, A. A., Orhan, F., and Batur, B. (2017). Undecidable

problems in malware analysis. In 2017 12th Inter-

national Conference for Internet Technology and Se-

cured Transactions (ICITST), pages 494–497.

Shirataki, S. and Yamaguchi, S. (2017). A study on in-

terpretability of decision of machine learning. In

2017 IEEE International Conference on Big Data (Big

Data), pages 4830–4831.

Soliman, S. W., Sobh, M. A., and Bahaa-Eldin, A. M.

(2017). Taxonomy of malware analysis in the iot. In

2017 12th International Conference on Computer En-

gineering and Systems (ICCES), pages 519–529.

Su, J., Vasconcellos, V. D., Prasad, S., Daniele, S., Feng,

Y., and Sakurai, K. (2018). Lightweight classiﬁca-

tion of iot malware based on image recognition. In

2018 IEEE 42nd Annual Computer Software and Ap-

plications Conference (COMPSAC), volume 01, pages

664–669.

Su, M.-Y. and Fung, K.-T. (2016). Detection of android

malware by static analysis on permissions and sensi-

tive functions. In 2016 Eighth International Confer-

ence on Ubiquitous and Future Networks (ICUFN),

pages 873–875.

Yang, W., Kong, D., Xie, T., and Gunter, C. A. (2017). Mal-

ware detection in adversarial settings: Exploiting fea-

ture evolutions and confusions in android apps. In AC-

SAC.

Yeo, M., Koo, Y., Yoon, Y., Hwang, T., Ryu, J., Song,

J., and Park, C. (2018). Flow-based malware de-

tection using convolutional neural network. In 2018

International Conference on Information Networking

(ICOIN), pages 910–913.

Zhang-Kennedy, L., Assal, H., Rocheleau, J., Mohamed, R.,

Baig, K., and Chiasson, S. (2018). The aftermath of a

crypto-ransomware attack at a large academic institu-

tion. In Proceedings of the 27th USENIX Conference

on Security Symposium, SEC’18, pages 1061–1078,

Berkeley, CA, USA. USENIX Association.

The Curious Case of Machine Learning in Malware Detection

535