Hypervisor Integrity Measurement Assistant

Lars Rasmusson

and Mazdak Rajabi Nasab

SICS Swedish Institute of Computer Science, Box 1263, SE-164 29 Kista, Sweden

Chalmers University of Technology, SE-412 96 Gothenburg, Sweden

Keywords:

Cloud Computing, Hosted Computing, Security, Untrusted Code, Virtual Machine, Machine Code Inspection,

Checksumming.

Abstract:

An attacker who has gained access to a computer may want to run arbitrary programs of his choice, and upload

or modify conﬁguration ﬁles, etc. We can severely restrict the power of the attacker by having a white-list of

approved ﬁle checksums and a mechanism that prevents the kernel from loading any ﬁle with a bad checksum.

The check may be placed in the kernel, but that requires a kernel that is prepared for it. The check may also be

placed in a hypervisor which intercepts the kernel and prevents the kernel from loading a bad ﬁle. Moving the

integrity check out from the VM kernel makes it harder for the intruder to bypass the check. We describe the

implementation of two systems and give performance results. In the ﬁrst implementation the checksumming

and decision is performed by the hypervisor instead of by the kernel. In the second implementation the kernel

computes the checksum and only the ﬁnal integrity decision is made by the hypervisor. We conclude that

it is technically possible to put ﬁle integrity control into the hypervisor, both for kernels without and with

pre-compiled support for integrity measurement.

1 INTRODUCTION

The techniques for building completely secure system

has made signiﬁcant strides as we have seen ﬁrst cor-

rectness proofs for an OS kernel, for C compilers, and

for the gate level implementation of multicore CPUs.

However, even if the kernel is correct, there will likely

always be bugs in application programs, as they are

more diverse, modiﬁed much more frequently, and

generally built by fewer developers, or built by com-

posing code from different sources (libraries, etc.).

Thus, even if the OS kernel is correct, there may still

be be opportunities for intruders to take control of a

computer via the application programs that run on it.

This paper is concerned with how to limit the abilities

of an attacker who has broken in to a computer.

The approach taken here is to prevent an attacker

who has broken into a computer via an application

program from executing arbitrary programs.

The

computer owner creates in advance a list with check-

sums for each ﬁle that the kernel is allowed to load,

and then a mechanism prevents the kernel from load-

In general an attacker has many more options to gain

control, including using the programs that are already avail-

able. However, not being able to run scripts, etc., severely

limits any automated attack.

ing a ﬁle if its checksum does not match with the list.

While this mechanism does not prevent the attacker

from (mis)using existing programs, the mechanism

may, if properly deployed, severely limit the amount

of control an attacker can gain over the computer.

A mechanism to prevent loading of unapproved

ﬁles has recently made its way into the Linux kernel.

IMA, the Linux Integrity Measurement Architecture,

hooks into the mmap system call inside the kernel and

computes the checksum of the entire ﬁle just when it

is about to be memory mapped. If so conﬁgured, IMA

forces mmap to fail if the checksum is not correct,

and thus prevents a user process to access ﬁles with

incorrect checksums.

IMA is built into the kernel and executes inside the

kernel and therefore only works in kernels that have

been prepared with IMA from the beginning. How-

ever, in a hosted virtualized environment, such as in

a cloud computing infrastructure, it may also be de-

sirable to be able to run arbitrary kernels of the cus-

tomer’s choice, while at the same time be able to pro-

tect against intrusion in the customers’ VMs.

Another security complication is that the integrity

of the checksum list must be protected by the kernel.

If the list is kept in the kernel its authenticity can be

stated with a with a digital signature. However, the

Rasmusson L. and Rajabi Nasab M..

Hypervisor Integrity Measurement Assistant.

DOI: 10.5220/0004370500260035

In Proceedings of the 3rd International Conference on Cloud Computing and Services Science (CLOSER-2013), pages 26-35

ISBN: 978-989-8565-52-5

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

public key used to verify the list’s signature must be

protected, and it may therefore be desirable to move

the key out from the VM and into the hypervisor.

We have therefore developed two prototypes to

test different ways to provide hypervisor support for

ﬁle checksumming and policing (or integrity mea-

surements, in IMA words).

The ﬁrst prototype is a hypervisor that is able to

prevent a Guest kernel from loading bad/tampered

ﬁles. At similar places where IMA would intercept,

the hypervisor intercepts the Guest VM execution

near the end of mmap, computes the checksum and

makes the mmap call fail if the checksum is incor-

rect. But, differently from IMA, the ﬁrst prototype

does not require any checksumming code inside the

kernel. The entire checksum computation is moved

into the hypervisor.

The second prototype relies on the kernel to com-

pute the checksum with IMA, and just checks the in-

tegrity of the value in the white-list by verifying the

white-list signature with a public key kept in the hy-

pervisor.

The next section describes the implementation of

the two approaches. Section 3 reports the results of

the performance measurements. In section 4 we dis-

cuss the conclusions that can be drawn from the ob-

servation. In section 5 we discuss related work which

this work is based on, or which addresses a similar

problem, and the paper conclusions are given in sec-

tion 6.

2 HYPERVISOR MEASUREMENT

IMPLEMENTATION

2.1 Approach 1: Hypervisor

Checksumming

2.1.1 Design

One problem with moving the checksumming out

from the kernel is the question of how the hypervisor

can get hold of the content in ﬁles inside the VM. The

hypervisor can’t simply read the ﬁle contents from the

VM’s disk. A kernel can manage several different ﬁle

systems, ﬁles can be encrypted on disk or retrieved

over the network. Therefore the VMM cannot sim-

ply bypass the VM and read the ﬁle contents from

the Guest’s disk before the Guest kernel is allowed to

proceed.

Our approach is to let the hypervisor intercept and

divert the Guest kernel’s execution just before it is

about to return from mmap. Then the hypervisor re-

trieves the ﬁle by tricking the kernel into loading the

ﬁle contents. The hypervisor injects several function

calls into the kernel by modifying the VMs program

counter (EIP) and registers of the kernel thread that

requests the mmap.

Whitelist

mmap

...

kernel_load

...

while not EOF

x=inject syscall

c=checksum(c,x)

r = match(c,whitelist)

set_return_val(r)

Figure 1: In the ﬁrst approach, the hypervisor computes the

checksum.

First the hypervisor makes the thread call kmalloc

to reserves one page of memory inside the kernel’s

address space. This page is mapped into the hypervi-

sor’s memory so the hypervisor can have quick access

to the page contents.

Next the hypervisor retrieves the ﬁle contents by

repeatedly injecting calls to

kernel read

to load the

ﬁle, one page at a time, see ﬁgure1. For each page that

is retrieved, the hypervisor updates the ﬁle checksum.

Finally it calls kfree to release the memory page, ad-

justs mmap’s return value in register EAX as desired,

and restores the EIP to continue the execution as nor-

mal.

Our approach requires that the hypervisor is aware

of which addresses to intercept, and it also needs in-

formation about some of the kernel data structures, in

particular the offsets to the ﬁelds in the structs.

The required kernel symbols and data structures

are listed in table 1. This information can be obtained

from the kernel symbol ﬁle and header ﬁles. Hackers

have other methods for ﬁnding the kernel layout and

data structures in an unknown kernel.

Invoking kernel calls is not side effect free, and

thus not completely invisible to the kernel. Just as

with IMA, when mmap has ﬁnished, the kernel will

have loaded the contents of the entire ﬁle. This has

two effects, the ﬁrst one being that the initial call to

mmap is much slower (since it has to read the entire

ﬁle), and the second one is that subsequents calls to

See section 5 - How to gather offsets & symbol ad-

dresses, in Smashing The Kernel Stack For Fun And

Proﬁt, by Sinan ‘noir’ Eren, Phrack, Volume 0x0b, Is-

sue 0x3c, Phile #0x06 of 0x10, http://www.phrack.com/

issues.html?issue=60&id=6#article

HypervisorIntegrityMeasurementAssistant

Table 1: Kernel symbols and data structures needed by the

hypervisor.

Kernel symbol

process measurement

kernel read

kfree

security file mmap

kmem cache alloc trace

kmalloc caches

destroy inode

integrity inode free

Data structure

struct file

struct path

struct dentry

struct inode

struct super block

read are faster, since some of the blocks will be in the

page cache.

In Linux without IMA, memory-mapping a ﬁle

does not load any of the ﬁles content. It just installs a

handler that detects when the user process tries to read

from the ﬁle and then loads only the requested pages

(by trapping memory page faults). Thus, if only some

bytes in the middle of the ﬁle are read, only the corre-

sponding block will be loaded from disk.

IMA keeps one checksum for the entire ﬁle. This

means that when the ﬁle is mapped, the entire ﬁle

needs to be read in for the checksum to be computed.

This makes mapping large ﬁles slow, and is a substan-

tial way to how Linux currently works.

One can imagine other approaches to checksum-

ming the ﬁles, such as having separate checksums

for different parts of the ﬁle (such as one per disk

block, or one per megabyte). These checksums could

be checked when the blocks are actually loaded into

memory. Then it would not be mmap that fails if the

ﬁle has a bad checksum. Instead the program would

get a memory protection error later, when the virtual

memory system has triggered loading a corrupted part

of the ﬁle.

While the late check is faster, it has its drawbacks.

Failing in the middle of an execution may enable situ-

ations where an attacker may cause a program to start

because the ﬁrst blocks are unmodiﬁed, and then fail

later upon accessing a modiﬁed part of the ﬁle. This

may leave the system in an inconsistent state, which

may potentially be exploited for attacks.

2.1.2 Implementation

In our implementation we are using a hypervisor

which is a modiﬁed version of the Bochs x86 emu-

lator. The original Bochs emulator is implemented in

C++. Each CPU instruction is implemented as an ”in-

struction function,” individual method invocations on

a CPU object. We have modiﬁed the implementation

in two ways.

First we have moved out the CPU instructions into

a separate ﬁle which is compiled into LLVM bit code,

for two reasons. One is that the LLVM is a compiler

toolkit with an intermediate code representation that

is very good for doing optimization passes. We an-

ticipate to be able to speed up execution by optimiz-

ing together the sequences of instructions in a basic

block. The other reason is that we can add very efﬁ-

cient probes into the instruction functions. A test that

would usually require the use of an MMU or CPU

debug functionality may be inlined as regular instruc-

tions into the native code, or even eliminated com-

pletely. The optimization work is not done yet, so the

LLVM step still incurs some additional overhead.

The second way we have modiﬁed Bochs is to

add the integrity mechanism. It adds a test in the

CPU instruction loop to see if the EIP (Instruction

Pointer) has reached the address that we want to in-

tercept. This check is inefﬁcient and costly, and of

the kind that we expect to be able to mostly opti-

mize away with the technique outlined above. If we

have reached the monitored address, we create a con-

text record in the hypervisor’s memory that stores the

kernel thread’s current CPU state, including registers,

stack pointer and an intercept mode state.

By looking at the contents of the register and

traversing the Guest’s memory, we can retrieve infor-

mation from the kernel data structures, such as ﬁle

pointers, ﬁlename, mode bits, ﬁle size, etc. If the ﬁle

should be checksummed according to the policy, we

perform a binary search in a list of previously com-

puted hashes to see if the ﬁle was already checked. If

not, we start the process of checksumming the ﬁle.

To checksum a ﬁle, we have to go through three

modes - allocate - read - free. The kernel thread’s

current mode is stored in the intercept mode state ﬁeld

in the context record above. If the kernel thread is

new, we set it to ’allocate’ mode, modify the CPU

registers to prepare for a call to kmalloc to allocate

one page of kernel memory, and resume computation.

When the CPU returns to the intercepted address,

it could either be because kmalloc has returned, or

due to another thread calling mmap. We distinguish

threads by looking at the value of the stack pointer.

When a thread has returned to the intercepted ad-

dress from kmalloc, i.e. when it is in the ’allocate’

mode, we set the thread’s mode to ’read’. We pre-

pare a SHA1 context in the hypervisor memory, and

set the CPU registers to call

kernel read

, to read

the ﬁrst 4k bytes from the ﬁle into the Guest mem-

ory page that we previously kmalloc:ed, and resume

execution. When the same thread eventually returns

CLOSER2013-3rdInternationalConferenceonCloudComputingandServicesScience

and is in read state, we update the SHA1 checksum,

and if there are more bytes to read, we repeat the call

kernel read

for the next 4k bytes, and so on.

When we have reached the end of the ﬁle, we set

the thread’s mode to ’free’. We set the CPU registers

to call kfree to free the kernel memory page we allo-

cated earlier, and resume execution. When the thread

again reaches the intercepted address and is in free

state, we restore the CPU registers from the context

record, which is then invalidated, store the computed

checksum in the checksum cache, and output it to a

log, before we resume the thread’s computation.

2.2 Approach 2: Hypervisor Approves

Kernel-computed Checksums

2.2.1 Design

The other approach to hypervisor integrity measure-

ment consists of using the hypervisor’s ability to set

break points in the Guest VM. In this approach, we

have used the Xen hypervisor and the gdbsx function-

ality which enables the use of gdb to place break-

points in the Guest VM. In dom0, the debugger,

gdb connects to gdbsx which is only availble to the

priviledged domain, dom0. The measured domain’s

memory is made available to dom0 by Xen’s ability

to map memory pages into multiple domains.

In this approach we have used gdb’s scripting ca-

pability to run the Guest and execute a python script

when a break point is hit. The break point is placed

precisely when the IMA code in the Guest kernel is

computed. Control is passed back to the gdb in dom0

who checks that the signature is on the approved list,

and then resumes execution of the Guest.

As for the ﬁrst approach, placing the break point

requires knowledge of some addresses in the Guest

kernel, but fewer, since the Guest execution is never

diverted, only suspended. However, in approach 2

we require that the kernel is already compiled with

IMA, and the hypervisor only checks that the result-

ing checksum is indeed a ’trusted’ checksum, see

ﬁgure2.

To improve the security attributes of the system,

the VMI application possesses only the public key,

and does not have the private key. The private key is

only used to sign each ﬁle on ﬁlesystem on a trusted

machine when preparing the Guest VM image. This

has impact on the system design. While only the pub-

lic key is needed for signature veriﬁcation, the private

key is used for signing. Essentially whenever a ﬁle is

modiﬁed its signature has to be updated as well. The

set of ﬁles the security function will check depends

on the security policy and can be conﬁgured. There-

Whitelist+sign

mmap

c=ima_checksum

kernel_load

...

r = c in whitelist

&& checksign...

...(whitelist, pubkey)

set_return_val(r)

pubkey

...

Figure 2: In the second approach, the kernel computes the

checksum, and the hypervisor veriﬁes that the whitelist is

genuine.

fore it could be the case that veriﬁcation only happens

for important ﬁles, for instance ﬁles owned by root,

which are expected to be immutable.

2.2.2 Implementation

Kernel debugging is the method used for virtual ma-

chine introspection, VMI, in approach 2, and it is al-

ready incorporated in the Xen VMM. A python script

connects to gdbsx, the gdb server implemented in

Xen, and performs the security function. gdbsx pro-

vides the functionality for the security function to in-

sert breakpoints at arbitrary addresses in Guest VM’s

memory. Whenever CPU hits a breakpoint in the

Guest VM, it generates a breakpoint exception in-

terrupt. Xen hypervisor controls this interrupt. Xen

pauses the Guest VM, and gives control to dom0

which in turn generates a SIGTRAP signal. This

signal is handled by gdb, which triggers our python

script. Using VMI, the python script in dom0 is no-

tiﬁed whenever a ﬁle is accessed in the target DomU

which veriﬁes the ﬁle’s signature by the proper public

key.

To bridge the semantic gap, some initial informa-

tion is provided to gdb. To allow gdb to access vari-

ables and structures in the Guest VM kernel space,

gdb needs to access the compiled directory of the ker-

nel. This allows gdb to know about exact structure of

the kernel. Also, the unzipped version of the kernel

ﬁle has to be available to the gdb. This ﬁle contains

symbol information of the kernel, and matches infor-

mation in the system.map ﬁle.

To enforce internally denying ﬁle access, either a

new module has to be installed or an available mech-

anism in the Linux kernel has to be used. Here we

take advantage of the IMA-Appraisal mechanism in

the kernel. IMA-Appraisal stores the 160-bit SHA1

digest of ﬁles in the extended attribute of a ﬁle, ”se-

curity.ima”. In contrast to storing good digest val-

ues centrally in a database, saving good SHA1 di-

HypervisorIntegrityMeasurementAssistant

gests in the ﬁle inode removes the search delay. IMA-

Appraisal stores valid crypto values in ﬁles extended

attributes to be enforced internally in a VM. In ad-

dition, IMA implements an internal cache which im-

proves the performance.

To handle the problem about updating ﬁles’ sig-

nature, one more security function is designed. The

second security function uses only a secret key to ver-

ify the HMAC-SHA1 digest of ﬁles. HMAC-SHA1

is chosen since this crypto function generates a 160-

bit digest, and IMA uses 160-bit SHA1 digest as well.

Since these two crypto functions generate 160-bit di-

gests, no kernel modiﬁcation is required, while in the

RSA signature veriﬁcation, the Linux kernel should

be modiﬁed to be able to work with 4096-bit RSA sig-

natures. Below the RSA veriﬁcation design is named

VMI-RSA and the HMAC-SHA1 validation design is

named VMI-HMAC respectively.

VMI-HMAC is a low-rate context switching se-

curity function which validates the HMAC-SHA1 di-

gests of ﬁles, in a Guest VM as they are loaded in

memory. The security function runs in dom0, and

uses a secret key. VMI-RSA is a low-rate context

switching security function which veriﬁes the RSA

signature of ﬁles when they are loaded in the mem-

ory of a Guest VM. Veriﬁcation occurs using the

proper public key in dom0. To enhance the security

attributes, dom0 and the security function do not pos-

sess the private key used for initial signing.

The IMA-Appraisal extended attribute, ”secu-

rity.ima”, is 160 bits, while the RSA signature is 4096

bits. For this reason it was decided to design and im-

plement the VMI-HMAC security function that uses

the original 160-bit, and does not change the length of

security.ima. In this model, VMI in Dom0 possesses

a secret key and using that key, it veriﬁes the integrity

of ﬁles and updates their extended attributes.

IMA uses the

ima calc hash

function for cal-

culating the SHA1 digests. This function is

only called by

ima collect measurement

. To

verify the integrity of ﬁles VMI-HMAC inter-

cepts

ima calc hash

function ﬂow. It is possi-

ble to replace the calculated SHA1 digest by in-

serting a breakpoint in either where

ima calc hash

returns or where it is called. We chose to

put it in

ima collect measurement

, which calls

ima calc hash

By inserting a breakpoint after where the

ima

calc hash function is called, it is possi-

ble to replace the ﬁle hash digest, stored in

iint->ima xattr.digest

with the HMAC-SHA1

digest. The HMAC-SHA1 digest is computed by the

VMI application with the secret key. When CPU ex-

ecutes a breakpoint it gives control to the breakpoints

handler which is gdb/gdbsx, and then ﬁnally a python

script is executed, which ﬁrst reads the SHA1 digest

computed by IMA via

iint->ima xattr.digest

Next it computes HMAC value based on the secret

key and the SHA1 digest, and ﬁnally it replaces the

SHA1 digest with the newly computed HMAC-SHA1

digest.

For VMI-RSA we needed to mod-

ify

IMA DIGEST SIZE

in ima.h and

evm ima xattr data.digest

in integrity.h to

hold 4096 bits. These changes do not affect IMA

normal behavior since all variable and structure ﬁelds

are initialized with 0. Since VMI-RSA does not have

access to the private key it cannot update ﬁle check-

sums. But since it has the public key, it can verify the

RSA signature stored in extended security attributes,

by intercepting

ima appraise measurement

just

after it has compared the checksum with the value

stored in the security.ima attribute. IMA simply

performs a byte-by-byte comparison with the security

attribute. VMI-RSA on the other hand will copy over

checksum and the 512 bytes of the security attribute

into dom0 and verify that it contains a valid signature

for the checksum. VMI-RSA then overwrites the

return value of the IMA’s byte-by-byte comparison to

instead report success if and only if the signature was

valid.

3 RESULTS

3.1 Approach 1: Hypervisor

Checksumming

Here we report on the experiments that were made to

study the overhead caused by the hypervisor integrity

measurements.

Since the experimental setup is quite complex

with layers of software executing on top of others, we

performed experiments to ﬁrst determine how much

each layer added to the total execution time. This al-

lows us to ﬁnally determine the cost of the integrity

measurements themselves.

In the experiment we test the effect of our modiﬁ-

cations to the task of memory mapping a single block

of medium sized and large ﬁle. The effect that is ex-

pected is that without integrity measurements, the ex-

ecution time is very quick, even if the ﬁles are large,

but with integrity measurements, either by IMA, the

hypervisor, or both, the execution time should be con-

siderably longer for the ﬁrst access of the ﬁle. Later

accesses to the ﬁle should be faster since the check-

sums are only computed the ﬁrst time the ﬁle is ac-

CLOSER2013-3rdInternationalConferenceonCloudComputingandServicesScience

cessed.

The test machine is a Dell R200, quad-core Intel

Xeon X3360, 2.83GHz with 4 GiB memory, running

Ubuntu 10.04 with Linux kernel 2.6.32. The Guest

VM gets 1 GiB of RAM.

In the ﬁrst experiment we ﬁrst read one block from

ﬁve ﬁles of size 100 MiB and report the execution

time. Then we repeat that experiment 99 times and

report the total execution time.

In the second experiment we ﬁrst read one block

from 20 ﬁles of size 1 MiB and report the execution

time. Then we repeat that experiment 99 times and

report the total execution time.

Each of the experiment was repeated three times

for each system conﬁguration. The OS in the VM

was rebooted between each run to clean the VM page

caches.

Four systems were investigated: chroot with the

loopback mounted ﬁle system, Linux kvm, a regular

Bochs without any modiﬁcations, a Bochs with the

LLVM JIT-compiled byte code, and a Bochs with the

LLVM JIT-compiled byte code that performs integrity

measurements of the Guest VM.

Each of the four systems were run with Linux

IMA either off or on, except for chroot which was

only run with IMA off, because IMA support was

lacking in the Host kernel.

The time measurements are not very accurate for

low values. This is because the VM Guest clocks

drifted very much, and we synchronized the local

clock with NTP before measuring the start and stop

time. This incurred an additive cost of about two

seconds, with some second of variance due to net-

work trafﬁc. This is unimportant for understanding

the qualitative results of our experiments, and more

precision in these measurements will not add to the

understanding of the integrity measurement system

we report here.

The results are shown in table 2.

3.2 Approach 2: Hypervisor Approves

Kernel-computed Checksums

Here we report on the experiments that were made to

study the overhead caused by the hypervisor approv-

ing but not computing the ﬁle checksums. The values

are in table 3.

To determine the sources of execution overhead

for approach 2 we run a set of experiments, similarly

to approach 1. We vary the number of ﬁles and ﬁle

sizes, and in each case we read just the ﬁrst 10 KiB

of each ﬁle in the set using dd. The experiment is

run with or without IMA without any virtualization

(”bare metal”), in Xen dom0 with direct device ac-

Table 2: The table shows the outcomes of three repeated ex-

periments. Execution time in seconds, including NTP over-

head.

Hypervisor Integrity 5 ﬁles 100 MiB 20 ﬁles 1 MiB

check First Next 99 First Next 99

access accesses access accesses

1. chroot

3 3 3 4

3 4 3 5

2 3 2 5

2. kvm

2 4 3 9

3 5 3 8

3. kvm IMA

14 4 3 9

15 5 3 9

4. c++ bochs

1 25 1 74

1 24 2 72

1 24 1 73

5. c++ bochs IMA

675 23 15 68

678 23 15 67

677 23 15 67

6. llvm-bochs

1 29 2 90

1 30 2 89

1 29 2 88

7. llvm-bochs IMA

847 28 19 85

840 30 19 82

838 27 19 83

8. llvm-bochs hypervisor

48 31 3 90

47 30 3 95

45 30 3 95

9. llvm-bochs

IMA +

hypervi-

sor

904 30 21 91

907 30 20 96

923 31 21 95

cess, in domU with indirect device access. Finally we

run the VMI-HMAC and VMI-RSA, and a dummy

VMI that does nothing but trapping to dom0 and re-

turn. The latter is used to estimate the cost of the trap-

ping. As for the approach 1 experiments we use ntp

to synchronize the clocks before and after each round

of the experiment to mitigate clock drift.

The tests are run on an Intel Core i7-2829QM

2.30GHz processor, with Intel VT, 16GB RAM, and

7200 RPM iSCSI disk. The DomU has 12GB RAM.

Fedora 16, 64-bit, is used as OS for all system con-

ﬁgurations. The Linux kernel is 3.2.0-rc1 with the

IMA-Appraisal patches.

4 DISCUSSION

4.1 Approach 1: Hypervisor

Checksumming

Comparing the execution times of the tests in a ch-

rooted environment and a kvm without IMA, we see

as expected that they are about as big for the ﬁrst ac-

cess. The only signiﬁcant cost here is the cost of the

ntpd synchronization which is performed before and

HypervisorIntegrityMeasurementAssistant

Table 3: Performance results for approach 2; the execution time measured in seconds. The average is taken over the last ﬁve

of six runs.

after each synchronization,and it takes approximately

(3+3+2)/3 = 2.7 seconds. After removing the time

for ntp synchronization removed from the next access

times and divide by 500, we can see that each of the

next 99 access to the 5 large ﬁles takes about 1 ms

each for chroot, and also 1 ms for the 20 small ﬁles.

Kvm takes 4 ms per ﬁle for the large ﬁles and 3 ms

per ﬁle for the small ﬁles.

The third line in table 2 shows that running a ker-

nel with IMA enabled in kvm signiﬁcantly increases

execution time in particular for the ﬁrst access to the

ﬁle. Subsequent accesses are not signiﬁcantly af-

fected. In the ﬁrst access, we read 5*100 MiB, so

the cost is 23 ms per MiB for the large ﬁles. For the

small ﬁles the same cost would add 0.5 seconds to the

values, which is within the measurement error.

The fourth line shows the results of running in an

unmodiﬁed bochs emulator. In the experiments the

clock inside the emulator drifted a lot, which is what

forced us to use ntp to set the clocks. This means that

the errors are larger for the bochs measurements. For

instance, the measurement scripts reported about one

second for the ﬁrst accesses to both large and small

ﬁles. Since the clocks are synchronized with ntp be-

fore and after the experiment the error due to drift is

additive rather than multiplicative. Each ﬁle access

takes around 45 ms and 35 ms for the large and small

ﬁles respectively in the experiment. The difference

may be partially explained as a measurement error,

due to a 3 second clock drift per measurement.

Comparing line 4 (c++ bochs) and line 2 (kvm),

we see that c++ bochs is about ﬁve to seven times

slower than kvm for our workload. Comparing line 4

(c++ bochs) and 6 (llvm-bochs), and comparing line

5 (c++ bochs IMA) with line 7 (llvm-bochs IMA) we

see that the llvm-bochs is about 20-25 percent slower

than c++ bochs in all cases.

The overhead of running IMA can be seen in line

5. The initial access is very slow since IMA has to

load and compute the checksum of the entire ﬁle. The

cost is 1.34 seconds per MB for the large ﬁles and

only 0.12 seconds for the small ﬁles. The difference

may be due to the the Host OS being able to cache

the small ﬁles, implying that with bochs, a cache miss

in the Host incurs a ten-fold increase in loading and

checksumming. The next 99 accesses are faster than

their non-IMA counterparts. This can be explained

by recalling that IMA has placed the ﬁle content in

the Guest OS’s page cache, and thus do not have to

retrieve it from the Host OS.

We are now in the position of assessing the over-

head of putting the integrity check into the hypervi-

sor. Line 8 shows that the hypervisor integrity ﬁle is

much faster (47 and 3 seconds for large and small ﬁles

respectively) when compared to having checksum in

side the kernel (676 and 15 seconds) when using the

bochs hypervisor.

Comparing line 8 with line 6 for the next 99 ac-

cesses where no integrity check is made, we see that

the hypervisor integrity check code adds 6 percent to

the execution time. Comparing line 9 with line 7, we

see that the hypervisor integrity check adds 8 percent

to the execution time,

From the above we ﬁnd that the costs for the llvm-

CLOSER2013-3rdInternationalConferenceonCloudComputingandServicesScience

bochs hypervisor integrity check can be attributed in

most part to the use of bochs as hypervisor, since it

incurs a ﬁve to seven times execution overhead over

kvm. To that is added a 20 percent cost due to our

use of a modiﬁed bochs, llvm-bochs. On top of that is

the 8 percent overhead from the hypervisor integrity

check added to arrive from the values in the ”next 99

accesses” columns on line 2 to the values on line 8.

The values in ”First access column” in line 8 (47

seconds) is the result of reading and checksumming

the large ﬁles. This operation added about 12 seconds

to kvm IMA on line 3, making us expect a value of 90

seconds. We currently do not have a good explanation

for why we see a lower value.

We also see a large dip in performance when run-

ning IMA inside any of the bochs hypervisors. This

dip was initially thought to be due to bad interaction

between IMA, bochs and the Host OS, but since the

hypervisor on line 8 is also loading the pages via the

Guest kernel, it must be a bad interaction between

IMA and bochs only.

4.2 Approach 2: Hypervisor Approves

Kernel-computed Checksums

The measured execution time for the second approach

provide some insight into where the costs are. In this

approach we run most of the Guest code natively in

domU using the Intel VT, and only invoke the in-

tegrity functions in dom0 when the break point in

ima calc hash

is reached.

The two big sources of overhead are the kernel’s

and IMA’s bookkeeping when opening ﬁles, and the

actual work in computing the checksum. To esti-

mate these individual costs we chose workloads that

stressed these two sources individually, by having few

or many ﬁles, and a small or large ﬁle set to check-

sum. The execution times can be seen in table 3.

To tease out the individual components, we do a

least-squares ﬁt of the four values for the length of

ﬁrst execution on each line to

time = k

/1000 ∗ #files+ k

∗ gigabytes

where k

is the time (in milliseconds) to handle each

ﬁle, and k

is the time to handle 1 GB of data. This

ﬁt is very coarse, but the results in table 4 show that

IMA adds about 10 seconds per GiB that gets check-

summed.

The bookkeeping cost per ﬁle rises from 1-2 ms

to 50 ms with IMA, which is particularly costly when

there are many ﬁles that need to be checksummed.

HMAC adds another 50 ms per ﬁrst ﬁle access, and

RSA adds yet another 20 ms to that, reaching almost

120 ms per ﬁrst ﬁle access. Subsequent ﬁle accesses

Table 4: Least squares ﬁt of the ﬁrst access to a ﬁle to the

function time = k

/1000∗ A+ k

∗ B where A is the number

of ﬁles accessed, 100 or 20000, and B is the total size of the

set in GB, 1 or 10.

Hypervisor k

, ms/ﬁle k

, s/GiB

bare metal

1.2994 0.41262

IMA 50.965 9.6601

dom0

1.6046 0.34258

IMA 54.203 9.7068

domU

1.8657 0.5719

IMA 4.2216 10.6

breakpoint IMA 43.668 8.7023

HMAC IMA 97.505 8.7085

RSA IMA 118.7 9.5136

do not reach the break point as the ﬁle is marked as

already checksummed by the IMA code in the Linux

kernel.

We note that the estimated time per GiB appears to

have decreased for the last three lines in table 4. This

may indicate that the least squares ﬁt to the data was

not very accurate, and some of the cost due to check-

summing was incorrectly attributed to the triggering

of the breakpoint. However, a re-ﬁt of the data with

set to 10.6 seconds does not signiﬁcantly change

the result, only reducing the least squares estimated

cost per ﬁle with 0.5 ms.

Unintuitively, the cost per ﬁle is very low for the

DomU with IMA. In table 3 the execution time for

DomU IMA in test 2 is only 86 seconds, compared

to 1034 seconds for bare metal IMA. This cannot be

explained by caching effects, since the DomU was

running on a separate disk device. Further discussion

about this anomaly can be found in (Nasab, 2012).

5 RELATED WORK

For virtualization, we have used Bochs (Mihocka and

Shwartsman, 2008), which is an emulator written in

C++ that can run unmodiﬁed kernels on any CPU,

and Xen (Barham et al., 2003), which can use ei-

ther modiﬁed kernels (paravirtualization), or take ad-

vantage of special virtualization support in CPUs that

have it (HVM). By itself Xen only monitors coarse

VM behavior such as what goes in and out via the

virtual devices, memory and CPU utilization rate, not

ﬁne details such as ﬁle content.

Virtualization has been used to lock down Guests

in many ways. Logging and replay techniques were

proposed in (Dunlap et al., 2002). VM introspection

for intrusion detection was proposed in (Garﬁnkel and

Rosenblum, 2003; Riley et al., 2008). XenAccess

HypervisorIntegrityMeasurementAssistant

(Payne et al., 2007) enable introspection of a Xen VM

without being able to interrupt it, which is why we

needed to use gdb-sx instead for our experiments.

iRODS (Wan et al., 2009) is a policy based sys-

tem for controling cloud clients’ access to data in the

Amazon S3 ﬁle system. iRODS enforces policies at

speciﬁc enforement points in user programs, not for

general ﬁles opened by the kernel.

While we are interested in moving IMA-like ﬁle

integrity checks into the hypervisor, the work by

Christodorescu et al. (Christodorescu et al., 2009) use

introspection for verifying the integrity of the kernel

to protect against root kits. SecVisor (Seshadri et al.,

2007) use hardwaresupport to achievethe same thing.

Other approaches focus on preventing the kernel

to only load authorized code (Riley et al., 2008; Litty

and Lie, 2006). That fails to address the issue with

modiﬁed conﬁg ﬁles.

In approach 1 we use machine code to machine

code JIT translation to speed up execution. It was

used early in the Dynamo system (Bala et al., 2000).

PIN (Reddi et al., 2004) and DynamoRIO (Bruening,

2004) are tools that let a user write probes that are dy-

namically injected into an application level program,

not a full OS.

LLVM (Lattner and Adve, 2004) is a compiler

framework that has an intermediate code representa-

tion that enables programmatical modiﬁcation, opti-

mization and JIT compilation at runtime. It is used

in the Binary Code Inliner to produce optimized code

for each translated basic block on the ﬂy.

This work is relevant to the area of cloud comput-

ing. Good surveys of multi-tenant cloud risks include

(Rodero-Merino et al., 2012; Vaquero et al., 2011;

Constandache et al., 2008; Descher et al., 2009; Bald-

win et al., 2009). The proposed solution addresses

the situation where one needs efﬁcient and automatic

methods for detecting and stopping hacked servers

from being controlled by intruders.

6 CONCLUSIONS

We conclude that it is technically possible to put ﬁle

integrity control into the hypervisor, both for kernels

without and with pre-compiled support for integrity

measurement. Both techniques require intercepting

the kernel at the right place and accessing the Guest

VM RAM.

While the presented work has not focused on op-

timizing the execution time, we have still investigated

the sources of execution overhead for two different

ways to make a hypervisor enforce ﬁle integrity on

Guest virtual machines. In the ﬁrst approach the hy-

pervisor injected calls into the Guest kernel to load

the ﬁle when the ﬁle was ﬁrst opened, and the hy-

pervisor computed the ﬁle checksum. In the second

approach, the hypervisor relied on the kernel to com-

pute the checksum, and only matched the checksum

with the white-list.

The implementation of the ﬁrst approach used a

modiﬁed emulator,which made it easy to intercept the

Guest at addesses extracted from the kernel symbol

ﬁle System.map. The main result is that the overhead

of the injection approach is about eight percent for a

small ﬁle set, after the other sources of overhead have

been accounted for.

The use of the emulator was the major source of

overhead, responsible for a ﬁve time slowdown com-

pared to HVM virtualization via KVM. This overhead

may be signiﬁcantly reduced by changing virtualiza-

tion technique. An additional 20 percent overhead

came from the use of a JIT tool chain to produce the

code. This overhead may be removed by using a non-

JITed emulator, but the reason to include the JIT in the

ﬁrst place is that it makes possible emulator optimiza-

tion such as run-time optimizing hot traces, though it

was not implemented in the current prototype.

The second approach used Xen’s gdb debug fa-

cilities to attach python scripts to monitor a Guest

VM. We observed that triggering the break point costs

around 40 ms per triggering, but the number of trig-

gerings is reduced by Guest kernel only computes the

checksum when the ﬁle is not in the ﬁle checksum

cache. The overhead of the approach is between 20

percent for a small ﬁle set and 100 percent for a large

ﬁle set.

REFERENCES

Bala, V., Duesterwald, E., and Banerjia, S. (2000). Dy-

namo: a transparent dynamic optimization system. In

Proceedings of the ACM SIGPLAN 2000 Conference

on Programming Language Design and Implementa-

tion, PLDI ’00, pages 1–12, New York, NY, USA.

ACM. http://doi.acm.org/10.1145/349299.349303.

Baldwin, A., Dalton, C., Shiu, S., Kostienko, K., and Ra-

jpoot, Q. (2009). Providing secure services for a vir-

tual infrastructure. SIGOPS Oper. Syst. Rev., 43:44–

51. http://doi.acm.org/10.1145/1496909.1496919.

Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T.,

Ho, A., Neugebauer, R., Pratt, I., and Warﬁeld, A.

(2003). Xen and the art of virtualization. SIGOPS

Oper. Syst. Rev., 37:164–177. http://doi.acm.org/

10.1145/1165389.945462.

Bruening, D. L. (2004). Efﬁcient, transparent, and com-

prehensive runtime code manipulation. PhD thesis,

Massachusetts Institute of Technology, Cambridge,

MA, USA. http://citeseerx.ist.psu.edu/viewdoc/

summary?doi=10.1.1.68.7639.

CLOSER2013-3rdInternationalConferenceonCloudComputingandServicesScience

Christodorescu, M., Sailer, R., Schales, D. L., Sgan-

durra, D., and Zamboni, D. (2009). Cloud se-

curity is not (just) virtualization security: a short

paper. In Proceedings of the 2009 ACM Work-

shop on Cloud Computing Security, CCSW ’09,

pages 97–102, New York, NY, USA. ACM. http://

doi.acm.org/10.1145/1655008.1655022.

Constandache, I., Yumerefendi, A., and Chase, J. (2008).

Secure control of portable images in a virtual

computing utility. In Proceedings of the 1st

ACM workshop on Virtual machine security, VM-

Sec ’08, pages 1–8, New York, NY, USA. ACM.

http://doi.acm.org/10.1145/1456482.1456484.

Descher, M., Masser, P., Feilhauer, T., Tjoa,

A. M., and Huemer, D. (2009). Retain-

ing data control to the client in infrastructure

clouds. Availability, Reliability and Security,

International Conference on, 0:9–16. http://

doi.ieeecomputersociety.org/10.1109/ARES.2009.78.

Dunlap, G. W., King, S. T., Cinar, S., Basrai, M. A.,

and Chen, P. M. (2002). Revirt: Enabling in-

trusion analysis through virtual-machine logging

and replay. In Culler, D. E. and Druschel,

P., editors, OSDI. USENIX Association. http://

www.usenix.org/events/osdi02/tech/dunlap.html.

Garﬁnkel, T. and Rosenblum, M. (2003). A virtual ma-

chine introspection based architecture for intrusion

detection. In In Proc. Network and Distributed Sys-

tems Security Symposium, pages 191–206. http://

suif.stanford.edu/papers/vmi-ndss03.pdf.

Lattner, C. and Adve, V. (2004). LLVM: A Com-

pilation Framework for Lifelong Program Analysis

& Transformation. In Proceedings of the Interna-

tional Symposium on Code Generation and Optimiza-

tion: Feedback-directed and Runtime Optimization,

CGO ’04, pages 75–, Washington, DC, USA. IEEE

Computer Society. http://llvm.org/pubs/2003-09-30-

LifelongOptimizationTR.pdf.

Litty, L. and Lie, D. (2006). Manitou: a layer-

below approach to ﬁghting malware. In Torrel-

las, J., editor, ASID, pages 6–11. ACM. http://

doi.acm.org/10.1145/1181309.1181311.

Mihocka, D. and Shwartsman, S. (2008). Virtual-

ization without direct execution or jitting - de-

signing a portable vm. In 1st Workshop on

Architectural and Microarchitectural Support for

Binary Translation. http://bochs.sourceforge.net/

Virtualization

Without Hardware Final.pdf.

Nasab, M. R. (2012). Security functions for vir-

tual machines via introspection. Master’s thesis,

Chalmers University, Sweden. http://publications.lib.

chalmers.se/records/fulltext/160810.pdf.

Payne, B. D., Carbone, M., and Lee, W. (2007). Secure and

Flexible Monitoring of Virtual Machines. Computer

Security Applications Conference, Annual, 0:385–

397. http://doi.ieeecomputersociety.org/10.1109/

ACSAC.2007.10.

Reddi, V. J., Settle, A., Connors, D. A., and Cohn, R. S.

(2004). PIN: A Binary Instrumentation Tool for

Computer Architecture Research and Education. In

Proceedings of the 2004 workshop on Computer Ar-

chitecture Education: held in conjunction with the

31st International Symposium on Computer Archi-

tecture, WCAE ’04, New York, NY, USA. ACM.

http://doi.acm.org/10.1145/1275571.1275600.

Riley, R., Jiang, X., and Xu, D. (2008). Guest-transparent

prevention of kernel rootkits with vmm-based mem-

ory shadowing. In Lippmann, R., Kirda, E., and Tra-

chtenberg, A., editors, RAID, volume 5230 of Lecture

Notes in Computer Science, pages 1–20. Springer.

Rodero-Merino, L., Vaquero, L. M., Caron, E., Mure-

san, A., and Desprez, F. (2012). Building safe paas

clouds: A survey on security in multitenant software

platforms. Computers & Security, 31(1):96 – 108.

http://dx.doi.org/10.1016/j.cose.2011.10.006.

Seshadri, A., Luk, M., Qu, N., and Perrig, A. (2007). Secvi-

sor: a tiny hypervisor to provide lifetime kernel code

integrity for commodity oses. In Bressoud, T. C.

and Kaashoek, M. F., editors, SOSP, pages 335–350.

ACM. http://doi.acm.org/10.1145/1294261.1294294.

Vaquero, L. M., Rodero-Merino, L., and Mor´an, D. (2011).

Locking the sky: a survey on IaaS cloud security.

Computing, 91:93–118. http://dx.doi.org/10.1007/

s00607-010-0140-x.

Wan, M., Moore, R., and Rajasekar, A. (2009). In-

tegration of cloud storage with data grids. Com-

puting, (October). https://www.irods.org/pubs/

DICE

icvci3 mainpaper pub-0910.pdf.

HypervisorIntegrityMeasurementAssistant