NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC

PERFORMANCE ON LINUX

Michael G. Iatrou

Department of Electrical and Computer Engineering, University of Patras, GR26504, Patras Greece

Artemios G. Voyiatzis

Department of Electrical and Computer Engineering, University of Patras, GR26504, Patras Greece

Industrial Systems Institute, Building of the Patras Science Park, Stadiou Str, Platani, 26504 Patras, Greece

Dimitrios N. Serpanos

Industrial Systems Institute, Building of the Patras Science Park, Stadiou Str, Platani, 26504 Patras, Greece

Department of Electrical and Computer Engineering, University of Patras, GR26504, Patras Greece

Keywords:

IPsec, Performance, Petworking, Security, Linux.

Abstract:

Virtual Private Network (VPN) connectivity is a necessity in the public Internet, for accessing in a secure

fashion private resources from anywhere. Internet Protocol Security (IPsec) is a standardized VPN technol-

ogy for serving multiple connectivity scenarios. Implementation of cryptography is widely considered as a

performance bottleneck and a target for optimization.

We present a set of system conﬁguration optimizations for the Linux 2.6 kernel network stack implementation,

supported by extensive measurements. These optimizations achieve signiﬁcant throughput gains. Our work

demonstrates that comparable performance between plain IP and IPsec connections is possible without altering

the implementation of the cryptographic algorithms.

1 INTRODUCTION

Internet Protocol (IP) is the standard protocol for

moving packets between a source and a destina-

tion host in the Internet. There are no inherent se-

curity mechanisms deﬁned in IP. Thus, it is easy

to manipulate IP packets e.g., alter their contents,

change source or destination addresses, and inject

fake ones (Bellovin, 2004). Internet Protocol Secu-

rity (IPsec) is a set of open protocols deﬁned by IETF

in RFC 4301–4309. IPsec aims to offer security in the

IP layer, for both IPv4 and IPv6.

IPsec deﬁnes three fundamental security proto-

cols: Authentication Header (AH), Encapsulating Se-

curity Payload (ESP), and Internet Key Exchange

(IKE). AH provides sender authentication, data in-

tegrity, and anti-replay protection. ESP additionally

provides data conﬁdentiality. IKE is the key manage-

ment protocol and provides the mechanisms to initiate

and to periodically refresh cryptographic parameters

of the AH and ESP.

IPsec operates in either transport or tunnel mode.

In transport mode IPsec protects only the IP payload

i.e., the upper layer information contained in an IP

datagram. In tunnel mode IPsec protects the whole IP

datagram i.e., the IP header and the IP payload.

IPsec in tunnel mode is commonly used to realize

a Virtual Private Network (VPN) over the public In-

ternet. VPN connectivity is attracting signiﬁcant in-

terest lately due to the increasing need of accessing

private resources in a secure manner from any point

of the public Internet. The IPsec endpoints can be

routers (site-to-site VPN), end-systems (host-to-host

VPN), or mixed (host-to-network VPN).

We focus on a site-to-site VPN, like between

the head and branch ofﬁces of a company. In this

case, IPsec operates in tunnel mode and both end-

points hold the same pre-shared keys. Thus, the gate-

ways need not engage in key management operations

through the IKE protocol. As a control scenario,

we also considered transport mode. Our focus is on

attainable throughput for saturating IPsec-protected

G. Iatrou M., G. Voyiatzis A. and N. Serpanos D. (2009).

NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC PERFORMANCE ON LINUX.

In Proceedings of the International Conference on Security and Cryptography, pages 83-91

DOI: 10.5220/0002225600830091

 SciTePress

links of 10, 100, and 1000 Mbps .

In this paper we provide insight on the perfor-

mance of IPsec as implemented in Linux kernel ver-

sion 2.6. We present experimental data and propose a

set of system conﬁguration optimizations. These opti-

mizations lead to signiﬁcant throughput gains without

intrusive modiﬁcations of the implementation. Our

work demonstrates that careful system engineering

should be applied ﬁrst, before trying to optimize (an

already optimized) implementation of cryptography.

The paper is organized as follows. Section 2

presents IPsec with emphasis on implementation on

Linux kernel and performance issues.

Section 3 describes the testbed environment used

for performance analysis and Section 4 the measure-

ments and discusses our ﬁndings. Section 5 presents

the set of system optimizations we propose and dis-

cusses their beneﬁts. Finally, Section 6 concludes our

ﬁndings and proposes future work in the area.

2 IPSEC BACKGROUND

2.1 Protocols

IPsec is a suite of protocols for securing IP commu-

nications by authenticating and encrypting each IP

packet of a data stream.

A set of extra headers for the IP datagram that ac-

tually provide the VPN services is deﬁned by two new

protocols: ESP and AH. The AH protocol offers data

source authentication, data integrity, anti-replay pro-

tection, but does not offer conﬁdentiality. The ESP

protocol provides all the features offered by AH pro-

tocol and in addition data privacy. The cryptographic

transformations that implement the security services

require a set of keys that are available between the

send peers. These can be either preshared keys or can

be negotiated using IKE protocol.

AH is used to provide connectionless integrity and

data origin authentication for IP datagrams using an

Integrity Check Value (ICV), as well as protection

against replays utilizing a monotonically increasing

number for each packet, which can be optionally ver-

iﬁed by the receiver. AH provides authentication for

as many ﬁelds of IP header as possible, as well as for

next level protocol data. However, some IP header

ﬁelds may change in transit and values of such ﬁelds

cannot be protected by AH. Thus, the protection pro-

vided to the IP header by AH is piecemeal. The size

of the AH header is 12 bytes plus the variable length

ICV. For point-to-point communication, suitable in-

tegrity algorithms for the ICV computation include

keyed Message Authentication Codes (MACs) based

on symmetric encryption algorithms (e.g., AES) or on

one-way hash functions (e.g., MD5, SHA1, etc.) Usu-

ally the output of the algorithm is truncated to 96 bit

for its use in ICV.

ESP can provide the same services as AH and

furthermore data conﬁdentiality using cryptographic

transformations. Whenever ESP only is used, both

conﬁdentiality and integrity services are recom-

mended, in order to avoid some attacks (Bellovin,

1996), (Degabriele and Paterson, 2007). The primary

difference in the integrity provided by ESP and AH

is the extent of the coverage. Speciﬁcally, ESP does

not protect any IP header ﬁelds unless those ﬁelds are

encapsulated by ESP e.g., via use of tunnel mode.

IKE is a component of IPsec used for performing

mutual authentication and establishing and maintain-

ing security associations (SAs). Among other things,

these deﬁne the speciﬁc services provided to the data-

gram, which cryptographic algorithms will be used to

provide the services, and the keys used as input to the

cryptographic algorithms. Although establishing this

shared state in a manual fashion is possible, it does

not scale well, and thus the use of IKE is required.

2.2 Transport and Tunnel Mode

VPN tunnels are usually implemented through packet

encapsulation. The full packet is wrapped in a new

header at the layer VPN operates, in order to provide

transparent peer-to-peer connectivity. IPsec supports

two modes of operation instead: transport mode and

tunnel mode.

In transport mode, encryption and authentication

services are provided only for the payload of the orig-

inal IP datagram. Transport mode is used for host-

to-host communication and it is not compatible with

gateway services.

In tunnel mode the original IP datagram is fully

encapsulated, providing security services for both the

IP header and the payload. Since a new IP header

is added, tunnel mode is appropriate for gateway-to-

gateway service setups. ESP and AH protocols are

available in both modes.

2.3 Cryptography

Implementation experience with IPsec in both man-

ual keying mode and in IKE protocol mode has shown

that there are so many choices for system administra-

tors to make, that it is difﬁcult to achieve interoper-

ability without careful pre-agreement (Eastlake 3rd,

2005). Thus, the IPsec Working Group agreed that

there should be a small number of named suites that

cover typical security policies (Hoffman, 2005).

SECRYPT 2009 - International Conference on Security and Cryptography

These suites are optional and do not prevent im-

plementers from allowing individual selection of each

security algorithms. The proposed cryptographic al-

gorithms are 3DES and AES128 in CBC mode for

encryption operations and SHA1 and AES-XCBC for

integrity operations.

2.4 Linux Implementation

FreeS/WAN project provided the ﬁrst implementa-

tion of IPsec for Linux. The implementation con-

sisted of a kernel IPsec stack (KLIPS) and user-space

key management daemon named

pluto

. The com-

munication between the two parts is realized over

the IPsec-standardized

PF KEY

interface (McDonald

et al., 1998).

Starting from Linux kernel 2.6 series, a “na-

tive” IPsec implementation was opted. Being

standards-compliant, the kernel component imple-

ments the

PF KEY

interface and can be used in

conjunction with any standards-compatible user-

space key management component. Examples of

such components include

pluto

(now part of the

OpenSwan/StrongSwan project),

isakmpd

from the

OpenBSD project,

racoon

from the KAME project,

and manual keying (no IKE component).

The in-kernel IPsec component interacts with

the network processing stack through the standard-

ized

XFRM

in-kernel framework. The Linux ker-

nel provides as generic, in-kernel modules heavily-

optimized implementations of various cryptographic

algorithms. IPsec reuses through XFRM these mod-

ules for its cryptographic needs.

2.5 Performance Issues

IPsec introduces extra processing overhead to the net-

work stack, in terms of performing mutual authenti-

cation and establishing and maintaining security as-

sociations (SAs), as well as cryptographic data trans-

formations. Cryptographic operations for encryption,

decryption and hashing introduce overhead unrelated

to characteristics of the trafﬁc imposed by protocols

beyond the network layer. Instead, SAs manipulation

has an impact to the performance that is highly cor-

related to session-like properties of the trafﬁc (Shue

et al., 2005).

In the past, various performance compensating

methodologies have been proposed, such as crypto

ofﬂoading to specialized hardware (Bellows et al.,

2003), key caching (Shue et al., 2007), and usage

of speciﬁc cryptographic algorithms (Elkeelany et al.,

2002).

To the best of our knowledge, the impact of IPsec

on generic bulk data transfers remains unaddressed,

as well as the opportunity to utilize related protocol

characteristics and implementation speciﬁcs to further

compensate the performance overhead of IPsec, for

both network throughput and CPU usage. We seek to

address these issues in the following.

3 METHODOLOGY

For the performance analysis of IPsec we built a

testbed that provides both an easy to deploy and re-

produce environment and versatility in a variety of

measurements scenarios. Our primal focus is on the

performance impacts of IPsec for bulk data transfers.

3.1 Testbed

The testbed environment consists of personal com-

puters running Slackware 10.1 Linux distribution,

with custom, vendor independent kernel builds and

optimized-for-throughput TCP/IP stack. In prelimi-

nary tests, various kernel versions were used, namely

2.6.0, 2.6.4, 2.6.7, and 2.6.11. All reported results in

this paper are taken with kernel version 2.6.11.7.

To ensure the reproducibility of the tests and the

accuracy of CPU usage instrumentation, we conﬁg-

ure the systems without any unneeded services run-

ning, except OpenSSH, which is used to control them

and doesn’t essentially affect measurements. Fur-

thermore, network interfaces were connected directly

(back-to-back) with shielded twisted-paired cables,

certiﬁed for speeds up to 1 Gbps.

To measure network throughput, we use the

netperf

tool (Jones, 2009). Netperf is capable of

measuring network throughput, generating multiple

packet sizes, and utilizing different socket sizes, us-

ing a single stream. We measure the amount of data

sent using a packet stream in a predeﬁned time pe-

riod. In our tests, we use packet sizes of 64, 128,

256, 512, 1024, 2048, 4096, 8192, 16384, and 32768

bytes. Each experiment runs for a constant time of

ten seconds. This time is sufﬁcient for TCP through-

put stabilization. Netperf is conﬁgured for up to 12

iterations (minimum 4) to a reach conﬁdence level

of 95 with a 3% width of conﬁdence interval. We

use the stock kernel socket sizes. We will show later

that careful socket size changes contribute to achiev-

ing higher throughput.

The built-in features of netperf can be used for

CPU usage estimation. In order to to achieve more

ﬁne-grained estimations, we opted to collect the ap-

propriate statistics directly from the kernel, using the

NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC PERFORMANCE ON LINUX

Table 1: Testbed components.

ID CPU core Clock/FSB (MHz) RAM (MB) NIC (Mbit/s)

P2 Intel Pentium 2 541 / 75 384 RTL8139 10/100

A AMD Athlon XP 1667 / 266 512 RTL8139 10/100

P41, P42 Intel Pentium 4 1800 / 100 256 Intel 82546 10/100/1000

P4M Intel Pentium 4M 2200 / 266 512 BCM 4401 10/100

X1, X2 Intel Xeon 2800 / 533 512 Intel 82546EB 10/100/1000

64 128 256 512 1024 2048 4096 8192 16384 32768

Throughput (Mbit/sec)

Packet size (bytes)

Network throughput

Plain IP

64bit DES

64bit DES-160bit SHA1

64bit DES-128bit MD5

256bit AES

256bit AES-160bit SHA1

256bit AES-128bit MD5

192bit AES

192bit AES-160bit SHA1

192bit AES-128bit MD5

192bit 3DES

192bit 3DES-160bit SHA1

192bit 3DES-128bit MD5

160bit SHA1

128bit MD5

128bit AES

128bit AES-160bit SHA1

128bit AES-128bit MD5

Figure 1: IP and IPsec comparison - 10 Mbps link.

/proc

interface and

/proc/stat

in particular. A cus-

tom application controls remotely the conﬁguration

for each system and collects the results from netperf

and the samples from

/proc/stat

Finally, for an in-depth analysis of performance

bottlenecks, we use

oprofile

, a transparent, low-

overhead, system-wide proﬁler (Levon, 2008). In to-

tal, we use nine personal computers with varying ca-

pabilities, as shown in Table 1.

3.2 Experiments

We set up IPsec in both transport and tunnel mode

with preshared keys, and measure the performance

of plain IP and IPsec with different algorithms for

authentication (namely MD5 and SHA1) and for en-

cryption (namely DES, 3DES, and AES), for a single

unidirectional TCP stream trafﬁc. We collect mea-

surements for link speeds of 10, 100, and 1000 Mbps.

We use two pairs of systems for testing the 10

Mbps link: (P2,P4M) and (X1,X2). Systems P2 and

X1 act as senders and system P4M and X2 as the re-

ceivers of the TCP stream. In all experiments, the

MTU was set to the maximum allowed of 1500 bytes.

We use three pairs of systems, covering the full

range of available hardware capabilities, for testing

the 100 Mbps link: (P2,P4M), (A,P4M), and (X1,X2).

This variety allows us to compare the scalability of the

IPsec implementation on different hardware.

We use the high-end systems for testing the 1

Gbps link: (P41,P42) and (X1,X2). For these sys-

tems, we further experiment with customized TCP/IP

options, interrupt coalescence capabilities, MTU size,

and different network cards. We use the same IPsec

setup for all experiments.

4 RESULTS

We group the results of the experiments according to

the link speed, since it is the deﬁnitive constraint in

terms of network throughput. Furthermore, it pro-

vides a metric for the performance characterization of

each hardware platform.

4.1 Link of 10 Mbps

IPsec has negligible impact for both network through-

put and CPU utilization overhead, even for the low-

end systems, as shown in Figure 1. Compared to plain

IP, encryption and authentication modes of IPsec can

saturate the link with small increase in CPU utiliza-

tion. There is only a small difference in maximum

throughput achieved. This difference is the result of

the increased packet sizes due to IPsec encapsulation.

SECRYPT 2009 - International Conference on Security and Cryptography

100

64 128 256 512 1024 2048 4096 8192 16384 32768

Throughput (Mbit/sec)

Packet size (bytes)

Network throughput

Plain IP

64bit DES

64bit DES-160bit SHA1

64bit DES-128bit MD5

256bit AES

256bit AES-160bit SHA1

256bit AES-128bit MD5

192bit AES

192bit AES-160bit SHA1

192bit AES-128bit MD5

192bit 3DES

192bit 3DES-160bit SHA1

192bit 3DES-128bit MD5

160bit SHA1

128bit MD5

128bit AES

128bit AES-160bit SHA1

128bit AES-128bit MD5

Figure 2: (P2,P4M) IP and IPsec comparison - 100 Mbps link.

100

64 128 256 512 1024 2048 4096 8192 16384 32768

Throughput (Mbit/sec)

Packet size (bytes)

Network throughput

Plain IP

64bit DES

64bit DES-160bit SHA1

64bit DES-128bit MD5

256bit AES

256bit AES-160bit SHA1

256bit AES-128bit MD5

192bit AES

192bit AES-160bit SHA1

192bit AES-128bit MD5

192bit 3DES

192bit 3DES-160bit SHA1

192bit 3DES-128bit MD5

160bit SHA1

128bit MD5

128bit AES

128bit AES-160bit SHA1

128bit AES-128bit MD5

Figure 3: (X1,X2) IP and IPsec comparison - 100 Mbps link.

Tunnel mode exhibits a 3% throughput loss com-

pared to transport mode. The primal reason for this

is that tunnel mode encapsulates the full IP datagram

and thus, the packet size in the wire is increased.

4.2 Link of 100 Mbps

We distinguish three system setups here: the low-

end (P2,P4M), the medium (A,P4M), and the high-

end (X1,X2). In the low-end (P2,P4M), the impact

of cryptographic operations is signiﬁcant and propor-

tional to their computational complexity, as Figure 2

depicts. Also, the number of packets to process per

time unit strongly affects the overall throughput. The

throughput grows in a nearly logarithmic rate with the

packet size in all but four cases: two of low computa-

tional complexity (MD5 and SHA1) and two of high

one (3DES+MD5, 3DES+SHA1).

The pair (A,P4M) has enough processing power

to handle all algorithm and key size combinations,

with virtually no throughput loss for the whole spec-

trum of packet sizes. The only exceptions are the

setup of DES+SHA1 and the ones of 3DES. The setup

of 3DES suffers a 40-50 Mbps penalty on through-

put and exhibits a 5 Mbps average variation with the

packet size.

The high-end setup (X1,X2) doesn’t bridge the

performance gap of 3DES identiﬁed in (A,P4M).

However, the gap is now more narrow: 35-40 Mbps,

as Figure 3 depicts.

In all setups, the observed difference between

transport and tunnel mode is bound to 3-5%.

NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC PERFORMANCE ON LINUX

200

400

600

800

1000

64 128 256 512 1024 2048 4096 8192 16384 32768

Throughput (Mbit/sec)

Packet size (bytes)

Network throughput

Plain IP

64bit DES

64bit DES-160bit SHA1

64bit DES-128bit MD5

256bit AES

256bit AES-160bit SHA1

256bit AES-128bit MD5

192bit AES

192bit AES-160bit SHA1

192bit AES-128bit MD5

192bit 3DES

192bit 3DES-160bit SHA1

192bit 3DES-128bit MD5

160bit SHA1

128bit MD5

128bit AES

128bit AES-160bit SHA1

128bit AES-128bit MD5

Figure 4: (X1,X2) IP and IPsec comparison - 1 Gbps link.

4.3 Link of 1 Gbps

The setup (P41,P42) does not provide enough pro-

cessing power to saturate the link, even for plain IP.

Furthermore, IPsec suffers from a throughput penalty

of more than 50%. The total throughput results in re-

spect to packet size and cryptographicalgorithm com-

plexity are similar to those of (P2,P4M) for the 100

Mbps link.

The setup (X1,X2) of Xeon processors saturates

the link in the case of plain IP for packet sizes larger

than 256 bytes, as Figure 4 depicts. However, it is

able to provide conﬁdentiality and integrity protec-

tion only for up to 300 Mbps, as show in Figure 4. In

all setups, the difference between transport and tunnel

mode is negligible. It is interesting to note the case of

AES128, combined or not with some integrity algo-

rithm. In all cases, the achieved throughput is almost

doubled moving from packet size of 64 bytes to 8192

bytes.

Further examination of the collected traces reveals

that the CPU usage is 100%, as Figure 5 depict. The

vast majority of the time is spent in the softirq state.

The cryptographic processing of each packet takes

place in this state. In the case of (X1,X2) the second

more-time consuming state is IRQ. In this state the

processor handles the interrupt received from the net-

work card. These interrupts occur whenever a packet

event takes place, such as on sending and receiving a

packet.

5 OPTIMIZATIONS

The results of Section 4 indicate that commodity sys-

tems of medium capabilities can be utilized to imple-

ment Linux IPsec gateways for links up to 100 Mbps.

However, there is some area for improvement for 1

Gbps links. The results from the 1 Gbps link on high-

end systems provide an interesting insight: serving

the interrupts caused by the packet events seems to

have a considerable impact on cryptographic algo-

rithm execution of protocol processing.

The implementation of IPsec in a system can be

considered as a latency component: each and every

packet must pass through the IPsec implementation

for one or more of the following operations: encryp-

tion, decryption, hash generation, hash veriﬁcation.

Since the function calls required to implement these

accumulate and form a longer execution path, it is

preferable to process as many bytes as possible on

each path traversal. In the following, we explore pos-

sible optimizations in all layers of TCP/IP that can

affect packet processing time.

TCP/IP is not a static and monolithic set of proto-

cols. Protocol parameters can be conﬁgured for spe-

ciﬁc end-to-end link characteristics. We saw that in-

terrupt processing has a critical role on performance;

interrupt coalescence is an excellent candidate for our

purpose. Also, MTU size can have an immediate

impact, since it can affect maximum allowed packet

size. This can reduce the number of packets needed

SECRYPT 2009 - International Conference on Security and Cryptography

100

128bit AES

128bit AES-128bit MD5

128bit AES-160bit SHA1

128bit MD5

160bit SHA1

192bit 3DES

192bit 3DES-128bit MD5

192bit 3DES-160bit SHA1

192bit AES

192bit AES-128bit MD5

192bit AES-160bit SHA1

256bit AES

256bit AES-128bit MD5

256bit AES-160bit SHA1

64bit DES

64bit DES-128bit MD5

64bit DES-160bit SHA1

Plain IP

CPU time (%)

IPsec setup

CPU usage

softirq

irq

system

user

Figure 5: (X1,X2) CPU utilization - 1 Gbps link.

to transmit a speciﬁc volume of information and thus,

reduce overall packet processing time and total num-

ber of interrupts.

The IEEE 802.3 standard dictates a backwards-

compatible MTU size of 1500 bytes. For a saturated

Gigabit link, the kernel must be able to cope with

more than 80,000 packets per second. The so-called

“Jumbo frames” have been proposed as a means to

reduce packet rate for a given transmission rate. The

size of jumbo frames is not standardized but there are

currently available products that support MTU sizes

of 4096, 8192, 9000, and up to 16110 bytes.

There is no formal agreement on the maximum

MTU size for devices supporting jumbo frames. Fur-

thermore, usage of the “path MTU discovery” pro-

tocol is not widely adopted (Mogul and Deering,

1990), (Mathis and Heffner, 2007). These facts can

lead to connection problems when jumbo frames are

enabled along a network path with network devices

of different vendors. For controlled environments,

where an a priori agreement between interested par-

ties can be achieved, jumbo frames are a desirable

feature, since it leads to higher performance, in the

means of less CPU and bus utilization which can pos-

sibly induce higher throughput.

We explore in the following the throughput im-

provement gained by each of the above parameters.

5.1 TCP/IP Stack Optimizations

Performance improvement of TCP over large

bandwidth-delay products paths is accomplished with

the addition of standardized extensions (Jacobson

et al., 1992), (Mathis et al., 1996). The Linux

kernel supports some of these extensions for high

performance networking in a customizable, online

conﬁgurable way. From the available arsenal, we

choose to enable the options for timestamps, window

scaling and SACK. Furthermore, we experiment with

the TCP window size and the default values for TCP

send and receive buffers. Speciﬁcally, we got the

optimum results by setting the min, default and max

values for both sending and receiving socket buffers

to 87380, 4194304 and 4194304 bytes accordingly.

The kernel selects the appropriate value depending

on the available memory.

These customizations lead to a gain of 20 Mbps

for plain IP and IPsec in AH mode.The larger the

size of the packets, the bigger the gain. Notably,

the TCP/IP stack optimizations are more beneﬁcial

to IPsec in ESP mode (AES, DES, and 3DES). They

contribute up to 150 Mbps more throughput. In gen-

eral, TCP optimizations not only provided a through-

put boost, but also exhibited more stable behavior.

5.2 Interrupt Coalescence

Whenever a packet is received by the network inter-

face card (NIC), it raises an interrupt. This interrupt

NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC PERFORMANCE ON LINUX

200

400

600

800

1000

2000 4000 6000 8000 10000 12000 14000 16000

Throughput (Mbit/sec)

MTU size (bytes)

Network throughput

Plain IP

128bit MD5

160bit SHA1

128bit AES

256bit AES-160bit SHA1

192bit 3DES

Figure 6: MTU contribution.

must be served by the appropriate interrupt handler of

the kernel. The handler processes the event in IRQ

context, where all interrupts are disabled. Thus, it is

necessary to minimize the processing time, and dele-

gate the rest of the network processing to a “softirq”

task. This task can be scheduled for later execution.

The kernel must process thousands of packets per sec-

ond in a saturated link. While processing the pack-

ets, the kernel must continuously suspend and resume

other processes. This interrupt “storm” has an impor-

tant impact not only on the network but also on the

overall system performance, even for modern, high-

speed processors.

The Linux kernel implements NAPI, a heuristics-

based workaround to cope with this storm (Salim

et al., 2001). NAPI is a hardware agnostic, hybrid in-

terrupts/polling mechanism. If the interrupts for the

NIC reach a certain rate, the kernel disables these

interrupts and processes the packets using a polling

mechanism. When the rate drops below the threshold,

the kernel switches back to interrupt-handling mode.

This approach provides interrupt overload reduction,

removes packet re-ordering issues in SMP architec-

tures, and handles (early drop) system overloading

due to network trafﬁc better.

We run a set of experiments using the NAPI-

enabled Intel e1000 network drivers in system pair

(X1,X2). For packet sizes less than 1024 bytes there is

a small throughput decrease of about 10 Mbps. How-

ever, for larger packets, there is a throughput increase

of 50 Mbps.

The differences between transport and tunnel

mode are negligible for MTU size of 1500 bytes.

We further experimented with the so-called “jumbo”

frames for an MTU of 9000 bytes. Our results lead to

three observations:

• Six times larger MTU has an amplifying effect

upon the previous results: up to 40 Mbps less for

small packets and 40-120 Mbps more for larger

ones.

• The increased MTU yields to more unstable re-

sults, with considerable standard deviation. Until

now, the standard deviation was near zero.

• There is a signiﬁcant throughput increase of 20-

100 Mbps for small and up to 450 Mbps (AH,

MD5) for large packets in tunnel mode. This is

the only case that we observed a differentiation

between transport and tunnel mode.

5.3 MTU

We extensively tested the effects of MTU size in the

system pair (X1,X2), after enabling the TCP opti-

mizations described above and the NAPI. We tested

effects in plain IP, in IPsec in tunnel mode using AH

(MD5 and SHA1), ESP (AES128, 3DES), and com-

bined ESP and AH (AES256 and SHA1).

The performance gains are rather strong, as Fig-

ure 6 depicts: MD5 peaks at 985 Mbps, the same

as plain IP and the combined ESP and AH operation

mode achieves 100 Mbps more throughput.

SECRYPT 2009 - International Conference on Security and Cryptography

6 CONCLUSIONS AND FUTURE

WORK

In this paper we analyzed the performance of the

Linux native IPsec implementation, for both transport

and tunnel mode. The analysis indicates that even

with commodity systems, we can easily saturate links

up to 100 Mbps, without any signiﬁcant penalty for

throughput. IPsec falls short of expectations in sat-

urating Gigabit links. The implementation of cryp-

tographic algorithms can be an attractive target for

optimization. However, detailed system analysis re-

vealed that the problem is not processing power per

se. Rather, it is the combined effect of the IRQ storm

and the softirq kernel state due to IPsec processing,

even with increased MTU sizes. Once the real cause

is identiﬁed, careful system engineering can lead to

signiﬁcantly increased IPsec throughput.

Future work in this area includes extensive testing

of advances in Linux kernel network stack, and use of

hardware-based cryptographic processors for ofﬂoad-

ing security operations. Another direction is the com-

parison with the BSD IPsec stack variants and valida-

tion of our ﬁndings in higher link speeds; 10 Gbps is

a good candidate for this. Finally, it would be inter-

esting to compare our results in scenarios with user-

space based VPN solutions.

REFERENCES

Bellovin, S. (2004). A look back at “security problems in

the TCP/IP protocol suite”. In ACSAC ’04: Proceed-

ings of the 20th Annual Computer Security Applica-

tions Conference, pages 229–249, Washington, DC,

USA. IEEE Computer Society.

Bellovin, S. M. (1996). Problem areas for the IP security

protocols. In Proceedings of the Sixth USENIX Secu-

rity Symposium, pages 205–214.

Bellows, P., Flidr, J., Gharai, L., Perkins, C., Chodowiec,

P., and Gaj, K. (2003). IPsec-protected transport of

HDTV over IP.

Degabriele, J. P. and Paterson, K. G. (2007). Attacking

the IPsec standards in encryption-only conﬁgurations.

Cryptology ePrint Archive, Report 2007/125.

Eastlake 3rd, D. (2005). Cryptographic Algorithm Imple-

mentation Requirements for Encapsulating Security

Payload (ESP) and Authentication Header (AH). RFC

4305 (Proposed Standard). Obsoleted by RFC 4835.

Elkeelany, O., Matalgah, M., Sheikh, K., Thaker, M.,

Chaudhry, Medhi, G., and Qaddour, J. D. (2002). Per-

formance analysis of IPSec protocol: encryption and

authentication.

Hoffman, P. (2005). Cryptographic Suites for IPsec. RFC

4308 (Proposed Standard).

Jacobson, V., Braden, R., and Borman, D. (1992). TCP Ex-

tensions for High Performance. RFC 1323 (Proposed

Standard).

Jones, R. (2009). Netperf. Retrieved April 27, 2009 from

http://www.netperf.org.

Levon, J. (2008). OProﬁle - A System Proﬁler

for Linux. Retrieved April 27, 2009 from

http://oproﬁle.sourceforge.net/.

Mathis, M. and Heffner, J. (2007). Packetization Layer Path

MTU Discovery. RFC 4821 (Proposed Standard).

Mathis, M., Mahdavi, J., Floyd, S., and Romanow, A.

(1996). TCP Selective Acknowledgment Options.

RFC 2018 (Proposed Standard).

McDonald, D., Metz, C., and Phan, B. (1998). PF

KEY

Key Management API, Version 2. RFC 2367 (Infor-

mational).

Mogul, J. and Deering, S. (1990). Path MTU discovery.

RFC 1191 (Draft Standard).

Postel, J. (1981). Transmission Control Protocol. RFC 793

(Standard). Updated by RFC 3168.

Salim, J. H., Olsson, R., and Kuznetsov, A. (2001). Beyond

softnet. In ALS ’01: Proceedings of the 5th annual

Linux Showcase & Conference, pages 18–18, Berke-

ley, CA, USA. USENIX Association.

Shue, C., Shin, Y., Gupta, M., and Choi, J. Y. (2005). Anal-

ysis of IPSec overheads for VPN servers. In IEEE

ICNPs NPSec Workshop.

Shue, C. A., Gupta, M., and Myers, S. A. (2007). IPSec:

Performance Analysis and Enhancements. In IEEE

Conference on Communications (ICC).

NETWORK STACK OPTIMIZATION FOR IMPROVED IPSEC PERFORMANCE ON LINUX