Do You Trust Your Device? Open Challenges in IoT Security Analysis

Lorenzo Binosi

, Pietro Mazzini

, Alessandro Sanna

, Michele Carminati

, Giorgio Giacinto

Riccardo Lazzeretti

, Stefano Zanero

, Mario Polino

, Emilio Coppa

and Davide Maiorca

Politecnico of Milan, Italy

Sapienza University, Italy

University of Cagliari, Italy

LUISS University, Italy

Keywords:

Firmware Analysis, IoT, Security Evaluation.

Abstract:

Several critical contexts, such as healthcare, smart cities, drones, transportation, and agriculture, nowadays

rely on IoT, or more in general embedded, devices that require comprehensive security analysis to ensure

their integrity before deployment. Security concerns are often related to vulnerabilities that result from inad-

equate coding or undocumented features that may create signiﬁcant privacy issues for users and companies.

Current analysis methods, albeit dependent on complex tools, may lead to superﬁcial assessments due to

compatibility issues, while authoritative entities struggle with specifying feasible ﬁrmware analysis requests

for manufacturers within operational contexts. This paper urges the scientiﬁc community to collaborate with

stakeholders—manufacturers, vendors, security analysts, and experts—to forge a cooperative model that clari-

ﬁes manufacturer contributions and aligns analysis demands with operational constraints. Aiming at a modular

approach, this paper highlights the crucial need to reﬁne security analysis, ensuring more precise requirements,

balanced expectations, and stronger partnerships between vendors and analysts. To achieve this, we propose a

threat model based on the feasible interactions of actors involved in the security evaluation of a device, with a

particular emphasis on the responsibilities and necessities of all entities involved.

1 INTRODUCTION

IoT devices are progressively becoming part of our

everyday lives, as they are widely used in various crit-

ical contexts, such as healthcare, smart cities, trans-

portation, agriculture, and several others (European

Cyber Security Organisation - ECSO, 2022). Conse-

quently, third-party manufacturers working on such

devices are signiﬁcantly larger than in other tradi-

tional computing markets (servers, laptops, and desk-

top workstations).

This variety leads to the development of differ-

ent hardware and software platforms for devices with

the same goals, typically featuring ﬁrmware charac-

terized by various levels of complexity (depending

on the target application and the architecture). Un-

fortunately, to keep their products economically com-

petitive, manufacturers need to integrate new features

quickly. They may omit rigorous testing of their plat-

forms, thus possibly ignoring well-established soft-

ware security best practices and leaving undocu-

mented privacy-sensitive functionalities.

This aspect poses a signiﬁcant security concern:

can we trust devices we did not analyze? In particu-

lar, these devices can either embed vulnerabilities that

miscreants can exploit to execute, e.g., unauthorized

code, or they can even hide privacy-breaking func-

tionalities to gather information about users. From

this respect, we can imagine two categories of ven-

dors: (i) benign vendors, who can unintentionally pro-

duce vulnerable code due to the time-the-market pres-

sure; (ii) malicious vendors, whose goal is to embed

hidden functionalities or vulnerabilities deliberately.

Unsurprisingly, analyzing such devices following

standard security evaluation methodologies that rely

on popular reverse engineering techniques is trouble-

some due to the custom platforms used by the devices,

making most state-of-the-art tools fail to provide, in

some cases, even basic information. Hence, a major

question emerges: what do we need to analyze IoT

devices effectively?

This paper considers the scenario where an institu-

568

Binosi, L., Mazzini, P., Sanna, A., Carminati, M., Giacinto, G., Lazzeretti, R., Zanero, S., Polino, M., Coppa, E. and Maiorca, D.

Do You Trust Your Device? Open Challenges in IoT Security Analysis.

DOI: 10.5220/0012856200003767

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 21st International Conference on Security and Cryptography (SECRYPT 2024), pages 568-575

ISBN: 978-989-758-709-2; ISSN: 2184-7711

tional or industrial entity needs to perform a security

evaluation of a device before deploying it in produc-

tion. In particular, we identify two signiﬁcant aspects:

• The investigation of the complex dynamics im-

pacting IoT security evaluations. In particular,

the device manufacturer may adopt anti-analyses

techniques to protect its intellectual property from

competitors and complicate the actions of exter-

nal attackers interested in exploiting the device.

However, such strategies can also make traditional

security evaluation methodologies ineffective and

could be thus seen as hostile practices by a secu-

rity evaluator. In this complex scenario, a signif-

icant research problem is what the manufacturers

could reasonably provide to permit effective se-

curity evaluations while considering their needs.

Reasoning on the minimum requirements for ef-

fective security evaluations is inevitably tied with

the design of state-of-the-art analysis tools.

• The development of novel security evaluation ap-

proaches. First, security evaluators should be

aware of the possible anti-analysis practices a

manufacturer may adopt. Hence, we need to de-

vise tools to detect the adoption of such tech-

niques. Second, assuming the partial cooperation

of a manufacturer (motivated by the economic

incentive of selling its own device), we believe

there is a need to rethink existing security eval-

uation frameworks, making them extensible with

minimal building blocks that could be reasonably

asked the manufacturer when carrying out an IoT

security evaluation.

This paper explores these two signiﬁcant research

directions in more detail. More speciﬁcally, our con-

tributions are the following:

• We describe the actors involved in a security eval-

uation of a device, thus considering the vendors,

the manufacturers, the attackers, the security eval-

uator, and the relationship between them.

• We describe a general workﬂow to conduct a secu-

rity evaluation of a device, pinpointing some anti-

analysis techniques that could be adopted by de-

vice manufacturers and may impact the effective-

ness of the security evaluations.

• We sketch open challenges and future research di-

rections that impose signiﬁcant research efforts by

the scientiﬁc community.

We believe that our work can be helpful to the sci-

entiﬁc community in orienting future research works

and inspiring new ideas to secure the IoT devices

landscape.

Security

Evaluator

Device

Manufacturer

Device

Vendor

External

Attacker

(A)

(B)

(C)

(D)

(E)

Figure 1: Actors that play a role in a security evaluation of a

device. Solid edges (A and B) represent direct interactions,

while dotted edges (C, D, and E) are indirect relationships.

2 IoT SECURITY ANALYSIS:

CONTEXT

IoT, and more in general embedded devices, pose sev-

eral security concerns that historically were less per-

ceived when considering traditional systems. Indeed,

such devices may come with heavily customized

hardware and software platforms developed by dif-

ferent device manufacturers, making it quite hard

to assess their security level using standard security

methodologies and tools. In this heterogeneous land-

scape, several institutional and private entities need to

verify the security of these devices before deploying

them in potentially critical production environments.

In this section, we describe the actors that have a

direct or indirect role in this scenario, the workﬂow

that could be followed during the security evaluation

of the device, and the several issues that could emerge

when carrying out the evaluation.

2.1 Involved Actors

As exempliﬁed by Figure 1, we can identify at least

four major actors with a direct or indirect role in the

security evaluation of an embedded device.

Security Evaluator. This actor is the entity in charge

of performing the security evaluation on behalf of

the device buyer. It directly interacts with the de-

vice vendor (edge A in Figure 1) and, in several cases,

can pose some requirements about the device and the

technical material (documentation, software artifacts,

etc.) needed by the evaluator to test the device. In-

directly, it also interacts with the device manufacturer

(edge E) since the device crucially depends on this ac-

tor. Finally, the security evaluator should often verify

whether the device may contain vulnerabilities (e.g.,

outdated and vulnerable libraries) that could be ex-

ploited by an external attacker (edge D).

Device Vendor. This entity commercially offers a de-

vice in the market. It directly interacts with the manu-

Do You Trust Your Device? Open Challenges in IoT Security Analysis

569

facturer to obtain the device (edge B) and with the se-

curity evaluator when selling the device (edge A). Un-

fortunately, the device vendor does not always have

full knowledge about the internals of a device. Hence,

its cooperation may not be sufﬁcient to have an ac-

curate view of a device. This situation implies that

a vendor could not satisfy some requirements from

the security evaluator, even when cooperation is gen-

uinely offered.

Device Manufacturer. This entity produces the de-

vice and is expected to have full knowledge of the

device’s internals. Notice that, in many cases, the

device manufacturer may rely on other manufactur-

ers (e.g., the System-On-Chip manufacturer). How-

ever, we expect it to be aware of the relevant internal

details of the third-party components. The manufac-

turer may adopt any strategy to protect its own intel-

lectual rights, including making choices or adopting

anti-analysis techniques that may hinder, or at least

slow down, reverse engineering attacks. Moreover,

such anti-analysis techniques could also be adopted to

protect end users since they may slow down external

attackers (edge C) interested in exploiting the device.

However, to keep their product economically compet-

itive, manufacturers often have to quickly integrate

innovative features in response to consumer pressure,

possibly skipping proper testing due to the need to re-

duce production costs. Moreover, such vendors may

reside in quite different countries with respect to the

nationality of the ﬁnal consumers, possibly inducing

a mismatch between regulations and strategic national

interests. Therefore, they could rely on anti-analysis

techniques to protect poorly tested code and hide un-

documented functionalities.

External Attacker. This entity may be interested in

ﬁnding and exploiting a vulnerability on the device.

Such exploitation concerns both the security evaluator

(edge D), which would like to detect any easy-to-spot

ﬂaw, and the device manufacturer (edge C), which can

experience signiﬁcant reputation damage, potentially

harming its business in the long run.

2.2 Security Evaluation Workﬂow

The security evaluation of an embedded device is

a challenging task that may require different ap-

proaches and methodologies depending on the de-

vice under analysis and the level of cooperation with

the device vendor. Various factors signiﬁcantly in-

ﬂuence the evaluation strategy, such as the manufac-

turer’s transparency in sharing ﬁrmware and source

code or using obfuscation and security features to pro-

tect the device. Despite these variations, we identify

three major steps to approach the security evaluation

of embedded devices (Nadir et al., 2022). For the sake

of simplicity, in this paper, we do not consider the

steps of analysis aimed at reverse engineering the de-

vice hardware internals since such evaluation would

most likely be out of reach for most security evalu-

ators. Nonetheless, evaluators indirectly validate the

hardware behaviour by checking the device execution

through dynamic analyses.

Firmware Extraction. The security analysis begins

with acquiring the embedded device’s ﬁrmware. No-

tice that this process is performed even when the

source code is available since the evaluator may need

to check whether the ﬁrmware binary is consistent

with the released source code. In some instances,

manufacturers facilitate the ﬁrmware acquisition pro-

cess by providing the plaintext binary directly on their

websites or enabling access via remote interfaces such

as SSH or Telnet, allowing complete ﬁrmware re-

trieval. However, as stated in the previous section,

this may not always be true. Manufacturers may want

to protect intellectual properties and only provide the

device, making it necessary to extract the ﬁrmware

from the device. Typically, this process involves ex-

tracting the ﬁrmware by dumping the ﬂash memory or

disk storage – depending on the ﬁrmware’s location –

via a debug interface like JTAG or UART, which are

standard in such scenarios. For instance, a common

tool utilized for exploiting this interface is the JTAG-

ulator

, a hardware tool by Grand Idea Studio used to

identify the usually hidden JTAG interface. However,

mechanisms like Readout Protection (RDP), Secure

Boot, and encryption may complicate the extraction

process, often requiring cooperation with the manu-

facturer to access the ﬁrmware successfully. Readout

Protection, for instance, may be used to disable the

JTAG interface altogether, preventing the device from

being reprogrammed without a manual reﬂash.

Static Analysis. This technique involves the analysis

of the source code (when available) and the assem-

bly code of the ﬁrmware to identify potential security

ﬂaws without running the program itself (Costin et al.,

2014; David et al., 2018). Since the availability of the

full source code is quite rare in most scenarios, we fo-

cus our discussion on techniques that operate on the

binary code.

Static analysis of the ﬁrmware begins with dis-

assembling the binary image to distinguish between

code and data and to identify further assembly refer-

ences, function boundaries, data structures, etc. This

initial step is fundamental for the success of static

analysis; a partial or erroneous disassembly process

https://grandideastudio.com/portfolio/security/

jtagulator

SECRYPT 2024 - 21st International Conference on Security and Cryptography

570

may lead to an inconclusive analysis. It is then pos-

sible to leverage static analysis techniques such as

Value-Set Analysis (Balakrishnan and Reps, 2004;

Balakrishnan et al., 2005) or Taint analysis (Schwartz

et al., 2010) to identify memory corruption vulnera-

bilities such as buffer overﬂows. Over the years, many

state-of-the-art approaches have adopted heuristics to

identify vulnerabilities in the ﬁrmware of embedded

devices (Neshenko et al., 2019). Some of them are

collected in frameworks, such as FACT (Fraunhofer

FKIE-CAD, 2017), or EMBA (Secure Firmware,

2022), to perform several analyses simultaneously.

For instance, the CVE Lookup analysis tries to match

publicly disclosed vulnerabilities with binaries within

the ﬁrmware. However, this analysis does consider

single binaries and does not consider the whole pic-

ture of the ﬁrmware. Even if there are already known

vulnerabilities in a few ﬁrmware binaries, they might

not represent real vulnerabilities, especially when the

vulnerable binaries are never executed or the vulnera-

ble functionalities are never used. This happens often

in embedded devices, and simple analysis may pro-

duce false positives. Moreover, vendors usually patch

vulnerabilities on the spot without updating the entire

code base; this choice is mainly due to the adoption of

old toolchains, which do not always support updated

binaries/libraries (Yu et al., 2022). Hence, more ad-

vanced techniques are required. To reduce the num-

ber of false positives, KARONTE (Redini et al., 2020)

leverages multi-binary taint analysis to detect vulner-

abilities following the data ﬂow of relevant ﬁrmware

applications. Moreover, KARONTE considers all

the ﬂows generated via Inter-Process Communication

(IPC) techniques such as fork(), execve(), shared

mmap(), etc. With this approach, KARONTE de-

tected 46 zero-days over 53 ﬁrmware, showing that it

scales regardless of the ﬁrmware size. Albeit promis-

ing, taint analysis has well-known limitations, and it

cannot detect complex vulnerabilities.

Another approach, not only designed for embed-

ded devices, tries to identify function clones (Hu

et al., 2017). This approach is used for both reverse

engineering and vulnerability detection; if a known

vulnerable function is detected, then the application

might be vulnerable.

Finally, other approaches rely on heuristics and

Machine Learning models (Nicolao et al., 2018;

Remigio et al., 2023; Thomas et al., 2017). ELISA

and its extension (Nicolao et al., 2018; Remigio et al.,

2023) present a supervised machine learning frame-

work designed for code discovery in header-less bi-

nary ﬁles, aiding static analysis and reverse engineer-

ing by distinguishing executable instructions from

ﬁrmware without metadata. They use a two-step pro-

cess: ﬁrst identifying the Instruction Set Architec-

ture (ISA) using logistic regression, and then delineat-

ing code boundaries and identifying data within those

boundaries using Conditional Random Fields (CRFs)

in (Nicolao et al., 2018) and lstm-based with heuris-

tics in (Remigio et al., 2023). HumIDIfy (Thomas

et al., 2017) is a system developed to identify un-

documented functionality and authentication bypass

vulnerabilities in ﬁrmware. It utilizes machine learn-

ing by creating proﬁles of expected ﬁrmware behavior

using a Binary Functionality Description Language.

These proﬁles are then compared to the actual behav-

ior of the ﬁrmware in real-time. Signiﬁcant deviations

suggest the presence of hidden functionalities.

Dynamic Analysis. A signiﬁcant limitation of static

analysis is the inability to test and exploit systems

without physical access to the device. Unlike static

analysis, which involves dissecting and reading code,

dynamic analysis allows for the execution of code and

observation of its behavior without detailed knowl-

edge of the program’s internals. Testing the ﬁrmware

on the physical device, from one side, allows the eval-

uator to validate the actual behavior of the device, but,

from the other side, often offers limited inspection ca-

pabilities (e.g., to check the internal execution state)

and does not scale well when only a limited number of

devices is available. Hence, the research community

has often suggested relying on device emulators, such

as QEMU (Bellard, 2005), to run the ﬁrmware im-

ages, making it easier to parallelize the analysis and

integrate advanced vulnerability detection techniques,

such as software fuzzing and symbolic execution. Dy-

namic analysis, however, requires metadata to emu-

late ﬁrmware accurately, which necessitates some un-

derstanding of the ﬁrmware’s architecture. In addi-

tion, dynamic analysis has scalability issues due to

the complex and varied conﬁguration setups needed

for different devices.

Introduced in 2014, Avatar (Zaddach et al., 2014)

is a framework designed for dynamic analysis of em-

bedded devices’ ﬁrmware. It emulates ﬁrmware in-

structions on an external machine while directing I/O

operations to the actual device hardware. Avatar

has been utilized in security applications such as re-

verse engineering, vulnerability discovery, and back-

door detection. It has been tested on diverse hard-

ware types, including a GSM phone, a hard disk boot-

loader, and a wireless sensor node, using QEMU (Bel-

lard, 2005) for ﬁrmware emulation and KLEE (Cadar

et al., 2008) to explore ﬁrmware paths for vulnera-

bilities. However, Avatar’s reliance on real hardware

for vulnerability discovery and the slow execution of

its processes from emulation to hardware interaction

limit its scalability.

Do You Trust Your Device? Open Challenges in IoT Security Analysis

571

In February 2016, Chen et al. introduced Firma-

dyne (Chen et al., 2016), an automatic dynamic anal-

ysis system targeting Linux-based ﬁrmware vulnera-

bilities in network-connected devices. Firmadyne an-

alyzed 23,035 ﬁrmware images, conﬁrming vulner-

abilities in 887 of them. The system automatically

collects ﬁrmware images and metadata from vendor

websites using scripts. It employs a customized ver-

sion of Binwalk for kernel extraction and uses QEMU

to emulate a Linux kernel tailored to each ﬁrmware

image. Firmadyne performs dynamic analyses to de-

tect and exploit vulnerabilities, utilizing the Metas-

ploit framework

for exploitation. Potential improve-

ments for Firmadyne include expanding its analysis

capabilities to non-Linux systems like RTOS and en-

hancing ﬁrmware extraction methods to handle obfus-

cated and encrypted images.

In 2016, Costin (Costin et al., 2016) developed

a fully automated dynamic ﬁrmware analysis frame-

work that employs full system emulation via QEMU,

without needing actual hardware. The framework fo-

cuses on identifying vulnerabilities in Linux-based

systems, especially within embedded web interfaces,

using tools like Arachni, Zed Attack Proxy (ZAP),

and w3af, with additional capabilities for using scan-

ning and exploitation tools such as Nmap, Nessus,

and Metasploit. Despite discovering 225 new vulner-

abilities across 45 ﬁrmware images, improvements in

tool efﬁcacy and emulation accuracy were noted as ar-

eas for enhancement due to inherent automation chal-

lenges in dynamic analysis.

In 2017, Palavicini (Palavicini Jr. et al., 2017)

advanced the analysis of ﬁrmware vulnerabilities in

Industrial Internet of Things devices, particularly

targeting industrial control systems. Employing a

suite of tools including binary analysis tools like

ANGR (Shoshitaishvili et al., 2016) and American

Fuzzy Lop (AFL)

, and virtualization tools like Open-

PLC, Firmadyne (Chen et al., 2016), and QEMU, the

analysis process was divided into three phases: ex-

traction of ﬁrmware for emulation, emulation of ex-

tracted code, and vulnerability analysis using tech-

niques like fuzzing and symbolic execution. This

approach uncovered backdoors, information leakage,

and botnet-capable code.

In 2019, a framework named FIoT (Zhu et al.,

2019) was introduced to address memory corruption

issues in constrained IoT devices. FIoT applies sym-

bolic execution to speciﬁcally test slices of ﬁrmware

that may be prone to buffer overﬂows, highlighting

the resource-intensive nature of symbolic execution

and its unsuitability for large ﬁrmware images. This

https://www.metasploit.com/

https://github.com/google/AFL

led to the discovery of 35 zero-day memory corrup-

tion vulnerabilities in 115 lightweight IoT devices.

In 2022, the framework OFRAK (Red Balloon Se-

curity, 2022) was published by Red Balloon Secu-

rity. Presented at Summercon 2022, OFRAK facili-

tates ﬁrmware patching. This technique modiﬁes the

ﬁrmware and observes its behavior when re-ﬂashes

inside the device, either via JTAG interface or by des-

oldering the memory and performing a manual ﬂash.

For instance, in Linux-based ﬁrmware, an analyzer

may introduce custom code inside /etc/init.d,

which will be run with sudo privileges.

2.3 Breaking Security Evaluations

Both static and dynamic analyses can be thwarted by

various strategies. In the context of embedded de-

vices, one may for example focus on custom or un-

known architectures that are not recognized by mod-

ern disassemblers and decompilers. A typical chal-

lenge in this respect is emulating the ﬁrmware behav-

ior, which would require an in-depth understanding of

the processor architecture and memory management.

The problem is further worsened by the adoption of

unknown formats and undocumented hardware, mak-

ing the detection of executable sections difﬁcult.

Other security measures can directly target the se-

mantics of the code and are directly inspired by mod-

ern anti-analysis techniques in X86-64 or ARM ar-

chitectures. For example, typical static obfuscation

techniques involve removing the symbols from the bi-

nary (i.e., binary stripping) or ﬂattening the control

ﬂow graph. Other techniques that prevent static anal-

ysis include dynamic code loading, where most of the

executable content is de-compressed or decrypted at

runtime, making static analysis inefﬁcient.

Dynamic analysis can also be hindered by eva-

sive techniques (Galloro et al., 2022), e.g., strategies

aimed at (among others) delaying the execution of the

binary (a simple example is the use of the sleep func-

tion) or preventing the execution of debuggers (with

the combined use of fork and ptrace). Other strate-

gies are often seen in Portable Executable (PE) mal-

ware, such as the adoption of TLS callbacks.

Anti-fuzzing techniques (Jung et al., 2019) com-

plicate automated vulnerability detection through

fuzzing. Methods such as checksum validations en-

sure only properly formatted inputs are processed,

thwarting random or malformed data typical of

fuzzing. Timing checks prevent rapid input se-

quences, halting potential fuzzing attempts. Addition-

ally, adaptive context-aware responses obscure sys-

tem logic, making it difﬁcult for fuzzers to predict

outcomes based on inputs. These strategies collec-

SECRYPT 2024 - 21st International Conference on Security and Cryptography

572

tively hinder the effectiveness of fuzzing tools in dis-

covering software vulnerabilities.

3 IoT SECURITY ANALYSIS:

OPEN CHALLENGES

As highlighted in the previous section, IoT and other

embedded devices present unique security challenges

not as evident in traditional systems due to their cus-

tomized hardware and software from various manu-

facturers. Assessing their security with standard tools

is difﬁcult. Consequently, both institutional and pri-

vate entities must rigorously evaluate the security of

these devices before using them in critical environ-

ments. We now review four research directions that

we believe were not already deeply investigated in the

past by the community and are worth consideration.

Devising a Threat Model for IoT Security Evalua-

tion Landscape. As detailed in Section 2, different

key players have a crucial role when considering the

security evaluation of an embedded device, including

the manufacturer, the vendor, the security evaluator,

and any external attacker. To systematically evaluate

the complex dynamics emerging among such play-

ers, we believe that the scientiﬁc community should

propose a threat model able to capture the capabili-

ties of these players, taking into account and putting

into perspective their respective constraints and ob-

jectives. Indeed, different from traditional systems,

several public and private institutions may start to per-

ceive the device manufacturer of an embedded system

as an attacker who may have adopted strategies to

hinder security analysis to hide scarcely tested code

or undocumented functionalities. At the same time,

manufacturers have the right to protect their intellec-

tual property from competitors and adopt any defense

that could thwart external attackers interested in ex-

ploiting the devices. Hence, it would be interesting

to model the capabilities of a device manufacturer by

identifying in more detail the speciﬁc anti-analysis

strategies that could be adopted, the levels at which

such strategies would operate (hardware, software, a

mix of the two), and their consequences from differ-

ent points of view (economical, practical, technolog-

ical, strategic). Investigating such research direction

could help the community determine the right trade-

off between the manufacturers’ needs and the security

evaluators’ needs.

Investigation of Granular but Actionable Security

Analysis Requirements. Several public and private

institutions may have the commercial or legal power

to impose some security analysis requirements when

a vendor proposes a device. For instance, such in-

stitutions may require that, during a security evalua-

tion, the vendor should cooperate by providing tech-

nical documentation, the source code, the compiler

toolchain, a disassembler, a device emulator, and sev-

eral other analysis or reversing tools. However, it

could be unlikely, or even completely unrealistic, to

believe that any security evaluator can impose any

kind of requirement on any vendor. Indeed, manu-

facturers hardly want to disclose technical informa-

tion about their devices or provide their internal de-

velopment tools. Moreover, they are both unable

and unwilling to invest effort in devising tools for re-

versing tasks when their internal workﬂow does not

need them. Finally, since the security evaluators may

not directly interact with manufacturers, imposing re-

quests that vendors cannot satisfy could be unrealistic.

In this scenario, we believe that the scientiﬁc com-

munity should consider three orthogonal aspects:

1. the requirements of the state-of-the-art ap-

proaches for security evaluations;

2. the constraints of the vendor and the needs of the

manufacturer;

3. the operational context of the device, i.e., how

sensitive and critical is the environment where the

device will be deployed.

In particular, the community should evaluate and pro-

vide some clear insights on what could be asked to the

vendor in order to make effective some of the state-of-

the-art tools for security analyses while keeping into

account the operational context of the device. For in-

stance, if a device is deployed into a critical environ-

ment, then it could be reasonable to require the source

code of the software executed by the device, since the

availability of the source code would allow the use of

a large number of powerful state-of-the-art tools. In

contrast, when a device is deployed in a less critical

environment, the request for the source code may be

unrealistic but then it should be clear what other min-

imum requirements could be imposed by a security

evaluator to allow a signiﬁcant security evaluation.

While devising such guidelines would not typi-

cally be the job of the scientiﬁc community, it could

be argued that security analysis requirements are

strictly connected with the state-of-the-art security

analysis approaches. Therefore, the scientiﬁc com-

munity should explore different analysis strategies de-

pending on the requirements that could be realistically

imposed in the real world of security evaluations.

Hence, the deﬁnition of realistic security analysis re-

quirements and the development of security analysis

approaches should be carried out simultaneously.

Devising Modular Security Analysis Frameworks.

Do You Trust Your Device? Open Challenges in IoT Security Analysis

573

A crucial limitation of several existing state-of-the-

art analysis tools for security evaluations is that they

are often not modular and extensible. For instance,

it is still quite hard and expensive for a manufac-

turer to support open source emulators required for

running dynamic analyses, such as software fuzzing

and taint analysis, or to support heavyweight analysis

techniques, such as symbolic execution. This exces-

sive cost results in a barrier that could be exploited

by manufacturers to justify their lack of cooperation.

We thus believe that the scientiﬁc community should

investigate how to make existing security analysis

frameworks, such as KARONTE (Redini et al., 2020),

ELISA (Nicolao et al., 2018), AVATAR (Zaddach

et al., 2014), and Firmadyne (Chen et al., 2016), ex-

tremely modular and extensible, minimizing as much

as possible the effort needed to develop and maintain

the low-level building blocks required by such tools,

thus lowering the bar for device manufacturers.

Reversible Anti-Analysis Techniques. The manu-

facturer should be allowed to integrate anti-analysis

techniques that are targeted at hindering the reversing

tasks carried out by competitors and external attack-

ers. To still allow effective security evaluations from

selected actors, such as a security evaluator, the re-

search community could explore the idea of reversible

anti-analysis techniques that, as their name suggests,

could allow a selected party to make ineffective the

anti-analysis strategies when the manufacturer is will-

ing to (partially) cooperate (possibly, under the re-

quest of a vendor). For instance, in a scenario where

the manufacturer integrates anti-fuzzing techniques

into its code as protection against the fuzzing per-

formed by external attackers, the security evaluator

may ask to the vendor to provide the technical bits

needed to bypass such anti-analysis approach during

the security evaluation. Hence, anti-analysis tech-

niques could be conceived as trapdoor functions that

are easy to reverse when the trapdoor is provided.

4 CONCLUSIONS

This paper highlighted the challenges that may

emerge when carrying out a security evaluation of an

IoT device. Speciﬁcally, we believe there is a need

to systematically analyze security contexts by high-

lighting the involved actors and the technical secu-

rity evaluation workﬂow. Indeed, complex dynam-

ics arise when considering the needs and constraints

of the device manufacturers, the external attackers,

and the security evaluators. For instance, manufac-

turers may adopt anti-analyses techniques to protect

their own devices from competitors and external at-

tackers. However, at the same time, by adopting such

practices, manufacturers may hinder security evalu-

ations, making it unclear whether they are trying to

hide poorly tested code, that may lead to easy ex-

ploitation from an external attacker, or conceal undoc-

umented privacy-risk functionalities. Unfortunately,

existing state-of-the-art security tools do not yet take

into the complexity of such a landscape and often

fall short by either posing unreasonable requirements

(e.g., availability of the source code) or cannot be

easily extended with minimal effort, thus imposing

a strong barrier for manufacturers that are willing to

(partially) cooperate with a security evaluator. We be-

lieve that our work can be a starting point to inspire

further research in this ﬁeld, fostering cooperation be-

tween the scientiﬁc community and device manufac-

turers to secure our device landscape.

ACKNOWLEDGMENTS

This work was partially supported by Project

FARE (PNRR M4.C2.1.1 PRIN 2022, Cod.

202225BZJC, CUP D53D23008380006, Avviso

D.D 104 02.02.2022) and Project SETA (PNRR

M4.C2.1.1 PRIN 2022 PNRR, Cod. P202233M9Z,

CUP F53D23009120001, Avviso D.D 1409

14.09.2022). Both projects are under the Italian

NRRP MUR program funded by the European Union

- NextGenerationEU.

REFERENCES

Balakrishnan, G. and Reps, T. W. (2004). Analyzing mem-

ory accesses in x86 executables. In 13th International

Conference, CC 2004, volume 2985 of Lecture Notes

in Computer Science, pages 5–23. Springer.

Balakrishnan, G., Reps, T. W., Melski, D., and Teitelbaum,

T. (2005). WYSINWYX: what you see is not what

you execute. In Meyer, B. and Woodcock, J., edi-

tors, Veriﬁed Software: Theories, Tools, Experiments,

VSTTE 2005, volume 4171 of Lecture Notes in Com-

puter Science, pages 202–213. Springer.

Bellard, F. (2005). Qemu, a fast and portable dynamic trans-

lator. In Proceedings of the FREENIX Track: 2005

USENIX Annual Technical Conference, April 10-15,

2005, Anaheim, CA, USA, pages 41–46. USENIX.

Cadar, C., Dunbar, D., and Engler, D. R. (2008). KLEE:

unassisted and automatic generation of high-coverage

tests for complex systems programs. In 8th USENIX

Symposium on Operating Systems Design and Imple-

mentation, OSDI 2008. USENIX Association.

Chen, D. D., Woo, M., Brumley, D., and Egele, M. (2016).

Towards automated dynamic analysis for linux-based

embedded ﬁrmware. In 23rd Annual Network and

SECRYPT 2024 - 21st International Conference on Security and Cryptography

574

Distributed System Security Symposium, NDSS 2016,

San Diego, California, USA, February 21-24, 2016.

The Internet Society.

Costin, A., Zaddach, J., Francillon, A., and Balzarotti, D.

(2014). A large-scale analysis of the security of em-

bedded ﬁrmwares. In Fu, K. and Jung, J., editors,

Proceedings of the 23rd USENIX Security Symposium,

San Diego, CA, USA, August 20-22, 2014, pages 95–

110. USENIX Association.

Costin, A., Zarras, A., and Francillon, A. (2016). Au-

tomated dynamic ﬁrmware analysis at scale: a case

study on embedded web interfaces. In Proceedings of

the 11th ACM on Asia Conference on Computer and

Communications Security, pages 437–448.

David, Y., Partush, N., and Yahav, E. (2018). Firmup:

Precise static detection of common vulnerabilities in

ﬁrmware. ACM SIGPLAN Notices, 53(2):392–404.

European Cyber Security Organisation - ECSO

(2022). Technical Paper on Internet of

Things (IoT). https://ecs-org.eu/?publications=

technical-paper-on-internet-of-things-iot. Accessed:

2024-04-27.

Fraunhofer FKIE-CAD (2017). FACT core: Firmware

Analysis and Comparison Tool. https://github.com/

fkie-cad/FACT core. Accessed: 2024-04-24.

Galloro, N., Polino, M., Carminati, M., Continella, A., and

Zanero, S. (2022). A systematical and longitudinal

study of evasive behaviors in windows malware. Com-

put. Secur., 113:102550.

Hu, Y., Zhang, Y., Li, J., and Gu, D. (2017). Binary

code clone detection across architectures and compil-

ing conﬁgurations. In Proceedings of the 25th Interna-

tional Conference on Program Comprehension, ICPC

2017, pages 88–98. IEEE Computer Society.

Jung, J., Hu, H., Solodukhin, D., Pagan, D., Lee, K. H.,

and Kim, T. (2019). Fuzziﬁcation: Anti-Fuzzing

techniques. In 28th USENIX Security Symposium.

USENIX Association.

Nadir, I., Mahmood, H., and Shah, G. A. (2022). A taxon-

omy of iot ﬁrmware security and principal ﬁrmware

analysis techniques. Int. J. Crit. Infrastructure Prot.,

38:100552.

Neshenko, N., Bou-Harb, E., Crichigno, J., Kaddoum, G.,

and Ghani, N. (2019). Demystifying iot security: An

exhaustive survey on iot vulnerabilities and a ﬁrst em-

pirical look on internet-scale iot exploitations. IEEE

Communications Surveys & Tutorials, 21(3):2702–

2733.

Nicolao, P. D., Pogliani, M., Polino, M., Carminati, M.,

Quarta, D., and Zanero, S. (2018). ELISA: eliciting

ISA of raw binaries for ﬁne-grained code and data sep-

aration. In Detection of Intrusions and Malware, and

Vulnerability Assessment - 15th International Confer-

ence, DIMVA 2018, volume 10885 of Lecture Notes in

Computer Science, pages 351–371. Springer.

Palavicini Jr., G., Bryan, J., Sheets, E., Kline, M., and

Miguel, J. S. (2017). Towards ﬁrmware analysis of

industrial internet of things (iiot) - applying symbolic

analysis to iiot ﬁrmware vetting. In Proceedings of the

2nd International Conference on Internet of Things,

Big Data and Security, IoTBDS 2017, pages 470–477.

SciTePress.

Red Balloon Security (2022). Open Firmware Re-

verse Analysis Konsole. https://github.com/

redballoonsecurity/ofrak. Accessed: 2024-04-27.

Redini, N., Machiry, A., Wang, R., Spensky, C., Continella,

A., Shoshitaishvili, Y., Kruegel, C., and Vigna, G.

(2020). Karonte: Detecting insecure multi-binary in-

teractions in embedded ﬁrmware. In 2020 IEEE Sym-

posium on Security and Privacy, SP 2020, San Fran-

cisco, CA, USA, May 18-21, 2020, pages 1544–1561.

IEEE.

Remigio, R., Bertani, A., Polino, M., Carminati, M., and

Zanero, S. (2023). The good, the bad, and the binary:

An lstm-based method for section boundary detection

in ﬁrmware analysis. In 18th Int. Workshop on Se-

curity, IWSEC 2023, Lecture Notes in Computer Sci-

ence. Springer.

Schwartz, E. J., Avgerinos, T., and Brumley, D. (2010).

All you ever wanted to know about dynamic taint

analysis and forward symbolic execution (but might

have been afraid to ask). In 31st IEEE Symposium

on Security and Privacy, SP 2010, 16-19 May 2010,

Berleley/Oakland, California, USA, pages 317–331.

IEEE Computer Society.

Secure Firmware (2022). EMBA - The ﬁrmware secu-

rity analyzer. https://github.com/e-m-b-a/emba. Ac-

cessed: 2024-04-27.

Shoshitaishvili, Y., Wang, R., Salls, C., Stephens, N.,

Polino, M., Dutcher, A., Grosen, J., Feng, S., Hauser,

C., Kr

ugel, C., and Vigna, G. (2016). SOK: (state of)

the art of war: Offensive techniques in binary analy-

sis. In IEEE Symposium on Security and Privacy, SP

2016, San Jose, CA, USA, May 22-26, 2016, pages

138–157. IEEE Computer Society.

Thomas, S. L., Garcia, F. D., and Chothia, T. (2017).

Humidify: A tool for hidden functionality detection

in ﬁrmware. In Detection of Intrusions and Mal-

ware, and Vulnerability Assessment - 14th Interna-

tional Conference, DIMVA 2017, volume 10327 of

Lecture Notes in Computer Science, pages 279–300.

Springer.

Yu, R., Nin, F. D., Zhang, Y., Huang, S., Kaliyar, P., Za-

kto, S., Conti, M., Portokalidis, G., and Xu, J. (2022).

Building embedded systems like it’s 1996. In 29th An-

nual Network and Distributed System Security Sympo-

sium, NDSS 2022, San Diego, California, USA, April

24-28, 2022. The Internet Society.

Zaddach, J., Bruno, L., Francillon, A., and Balzarotti, D.

(2014). AVATAR: A framework to support dynamic

security analysis of embedded systems’ ﬁrmwares. In

21st Annual Network and Distributed System Security

Symposium, NDSS 2014, San Diego, California, USA,

February 23-26, 2014. The Internet Society.

Zhu, L., Fu, X., Yao, Y., Zhang, Y., and Wang, H. (2019).

Fiot: Detecting the memory corruption in lightweight

iot device ﬁrmware. In 18th IEEE International Con-

ference On Trust, Security And Privacy In Computing

And Communications / 13th IEEE International Con-

ference On Big Data Science And Engineering, Trust-

Com/BigDataSE 2019, pages 248–255. IEEE.

Do You Trust Your Device? Open Challenges in IoT Security Analysis

575