SECURITY SENSOR PROVIDING ANALYSIS OF ENCRYPTED

NETWORK DATA

Daniel Hamburg

Integrated Information Systems Group, Ruhr University

Bochum, Germany

York T

uchelmann

Integrated Information Systems Group, Ruhr University

Bochum, Germany

Keywords:

Transport Layer Security Sensor, Data Collection, Local Application Proxy, Encrypted Attacks, Intrusion

Detection, Intrusion Prevention, Secure Socket Layer.

Abstract:

Common Intrusion Detection Systems are susceptible to encrypted attacks, i.e. attacks that employ security

protocols to conceal malign data. In this work, we introduce a software sensor, called Transport Layer Security

Sensor (TLSS), providing detection engines access to network data encrypted at Transport Layer. Transport

Layer Encryption, such as SSL, is typically implemented by a local application and not the OS. TLSS resides

on the monitored host and executes cryptographic functions on behalf of local applications. TLSS decrypts

incoming encrypted network packets and passes the data to the application, e.g., a Web server software. In

addition, cleartext data is also passed to a detection engine for analysis. We present an implementation of

TLSS designed for Web servers providing SSL-secured HTTP access and evaluate sensor’s performance.

1 INTRODUCTION

Till this day, the transaction volume of online shop-

ping on the Internet grows. One major ﬁnding for

this growth is that online merchants provided apro-

priate mechanism to protect transactions (boede +

partners integrated marketing, 2004). For instance,

a favored technique is the use of Secure Socket

Layer (SSL), which emerged to the de-facto stan-

dard in protecting web applications. SSL estab-

lishes a conﬁdential and integer communication be-

tween client and server making wiretapping infeasi-

ble. Vice versa, adversaries can deploy these cryp-

tographic secured channels to conceal attacks inside

encrypted data. Intrusion Detection Systems (IDS)

depend upon the availability of unencrypted data. In

particular, Network based Intrusion Detection Sys-

tems (NIDS) are blind to attacks encapsulated in en-

crypted network trafﬁc. Thus, related works pro-

pose the use of Host based Intrusion Detection Sys-

tems (HIDS) to monitor servers accessible via en-

crypted tunnels. Though hooking into the operating

system (OS), HIDS are blind to attacks inside pro-

tocols which encrypt the Transport Layer payload,

such as SSL. For instance, web servers and browsers

perform cryptographic functions by themselves. The

OS passes incoming encrypted SSL data to the corre-

sponding application, which decrypts and processes

it. Though, data captured by a HIDS sensor is en-

crypted.

A standard solution to provide analysis of data en-

capsulated in encrypted connections is to end the en-

crypted tunnel at a web proxy, which does not guaran-

tee the non-repudiation assurance. This is crucial in

particular for ﬁnancial transactions. Non repudiation

ensures that neither of two business partners is able to

dispute his commitment to the transaction afterwards.

To bypass this drawback, another common approach

decrypts trafﬁc without truncating the tunnel (Breach

Security Inc., 2004). For doing so, an intermediary

hardware device is equipped with the private key of

every original end point. However, export of private

keys from the host is a perilous process and should be

omitted whenever possible; both, attackers and legit-

imated insiders (e.g., administrators) which have ac-

cess to private keys can impersonate the correspond-

ing hosts.

In this work we introduce the Transport Layer Se-

curity Sensor (TLSS), which is a special data captur-

ing tool, as part of an IDS. The goal of TLSS is to fea-

ture analysis of network trafﬁc encrypted at the Trans-

port Layer, as with SSL. TLSS is a two part software

sensor installed on the host to be monitored. Using

the sensor it is feasible to capture and block incoming

172

Hamburg D. and Tüchelmann Y. (2006).

SECURITY SENSOR PROVIDING ANALYSIS OF ENCRYPTED NETWORK DATA.

In Proceedings of WEBIST 2006 - Second International Conference on Web Information Systems and Technologies - Internet Technology / Web

Interface and Applications, pages 172-177

DOI: 10.5220/0001254401720177

 SciTePress

network data after it has been decrypted, and outgoing

trafﬁc before it will be encrypted. TLSS is capable to

forward gathered data to available local (or remote)

detection engines. A main beneﬁt is that TLSS nei-

ther harms non-repudiation assurances nor does it re-

quire export of private keys from the monitored host.

Furthermore, we discuss the practicability of TLSS.

In a proof-of-concept implementation, we prove evi-

dence that a standard detection engine in combination

with TLSS is able to detect attacks encapsulated in

encrypted SSL data. In addition, we give a detailed

analysis of TLSS’s performance.

This paper is structured as follows: In Section 2, we

review related works and introduce, elaborate and de-

tail our Transport Layer Security Sensor in Section 3.

Afterwards, we describe in Section 4 the proof-of-

concept, and summarize and discuss the measured

performance results in Section 5. Finally, we con-

clude our work with a summary of our major results

and outlook on future work in Section 6.

2 RELATED WORKS

Ristic (Ristic, 2005) introduces an approach to defend

Web servers from malicious data inside encrypted

SSL connections. He proposes the integration of an

IDS inside the Web server software as loadable mod-

ule. The basic idea is to hook in the unencrypted

web server application’s internal information ﬂow. He

presents a module for the Apache httpd Web server

consisting of a proprietary intrusion detection and

prevention engine as proof-of-concept. The idea is a

straightforward solution for standalone Web servers

but prevents integration into existing IDS used to

monitor several servers or entire networks. Further,

the idea is limited to open source or extendable server

software.

Almgren and Lindqvist (Almgren and Lindqvist,

2001) introduce an application-integrated approach to

collect data aimed at a server. The authors propose

the use of corresponding APIs to hook in the server

application, and to take control of data ﬂows inside

the application. Gathered data is sent to a detection

engine. Further, data ﬂows can be suspended until the

detection engine evaluates data, though providing pre-

vention functionality. In addition, the authors provide

an implementation for a web server. The basic idea

is similar to the one proposed in (Ristic, 2005) but

provides cooperation with existing detection engines.

Still, the solution is limited to open source server soft-

ware or software that enables external modules to take

over process control. In addition, there is a high de-

velopment effort, as every monitored application re-

quires a special sensor.

3 TRANSPORT LAYER

SECURITY SENSOR (TLSS)

The efﬁciency of Intrusion Detection Systems highly

depends upon the data they analyze, hence encrypted

data poses a serious threat. We propose a new sensor,

providing detection engines access to data encapsu-

lated in encrypted tunnels. In this context, sensor in-

dicates a data capturing software tool installed on the

monitored host, gathering network packets’ headers

and payload.

3.1 Rationale

Transport Layer Security Protocols, (e.g. SSL, Se-

cure Shell (SSH)), are a favored technique to secure

transported data on public networks. Those protocols

encrypt Application Layer headers and payload, but

do not change headers of protocols on the Internet-

or Transport Layer, (e.g., IP and TCP). Cryptographic

functions for Transport Layer Security Protocols are

generally implemented by an application and not the

OS. For instance, common web servers and browsers

include cryptographic functions for SSL connections

and do not rely on the OS to offer these capabili-

ties. The OS passes incoming encrypted data to the

corresponding application which decrypts and pro-

cesses it. Hence, capturing local data ﬂows between

OS and application, e.g. with a HIDS sensor, results

in encrypted Application Layer headers and appli-

cation payload, valueless for common detection en-

gines. Hooking up applications and analyzing unen-

crypted internal data ﬂows is costly and not feasible

for closed source software. Therefore, we propose to

outsource the cryptographic functions from the appli-

cation to a software security module, residing on the

same host as the application, and to capture the un-

encrypted local trafﬁc between this module and the

application. Transport Layer Security Sensor (TLSS)

integrates both a module providing security functions

and a data capturing sensor.

3.2 Design of TLSS

TLSS consists of two parts, the Local Application

Proxy (LAP) performing cryptographic functions and

a software sensor to capture data. The basic idea of

TLSS is to outsource cryptographic functions from

the application to LAP, thus, enabling the software

sensor to capture unencrypted data exchanged be-

tween LAP and application. Incoming encrypted net-

work packets aimed at the application ﬁrst pass the

network stack implemented by the OS. Subsequently,

the OS hands the packets to LAP which decrypts and

forwards them to the application. On the other hand,

the application passes outgoing packets to the LAP

SECURITY SENSOR PROVIDING ANALYSIS OF ENCRYPTED NETWORK DATA

173

which applies cryptographic mechanisms and hands

on secured data to the OS. Local communication be-

tween LAP and application is not encrypted, thus we

use the software sensor to capture Application Layer

headers and payload. In addition, the software sensor

captures Internet- and Transport Layer headers from

the OS to detect low-level attacks, e.g. illegal combi-

nations of TCP-ﬂags (Daniels and Spafford, 1999).

3.3 Discussion

In contrast to existing solutions, TLSS does not re-

quire export of secret keys from the monitored host.

In the last resort, private keys are locally migrated

from the application to LAP. Typically, keys are

stored on the host’s hard disc and can be accessed

by the local TLSS, rendering migration unnecessary.

Use of special cryptographic hardware to store pri-

vate keys, (e.g. Smart cards), or to accelerate cryp-

tographic computations does not foreclose the use of

TLSS. LAP has to be adapted to interact with the

hardware using corresponding drivers.

Compared to a web proxy, (also known as SSL ter-

minator), TLSS does not affect non-repudiation as-

surances as it does not truncate secure connections.

For instance, SSL allows a client to establish a se-

cured connection to a server, i.e. between the SSL

Layers of the two hosts

. A web proxy, which is

a dedicated host, implements the SSL layer, hence

ends the secured connection. The non-repudiation as-

surance holds between client host and web proxy but

not between client and server host. As TLSS resides

on the server host, the secure connection terminates

at the server, thus ensuring non-repudiation between

client and server host.

One limit of TLSS is its boundary to applications

that allow outsourcing of cryptographic functions.

This does not necessitate open source software, but

the possibility to shutdown or bypass the application’s

internal cryptographic functions. We argue that this

assumption holds for most server software used on

the Web, as web services usually implement security

as optional add on. With TLSS, the server applica-

tion offers the unsecured service and LAP appends

security functions. To do so, the server’s OS passes

incoming encrypted data, (e.g. according to the TCP-

/UDP-port number it is aimed at), to LAP. LAP com-

municates with the server application either via API

function calls or via a virtual local network interface.

Using a virtual network interface, TLSS is indepen-

dent of the server application’s API, which is a con-

trast to application-based IDS, which rely upon the

availability of the API. Another advantage is that one

TLSS can serve multiple applications. For instance,

In the TCP/IP network model, the SSL Layer is located

between the Transport Layer and the Application Layer

one LAP can offer SSL-functions to several applica-

tions on the same host.

4 PROOF-OF-CONCEPT

We implemented the Transport Layer Security Sensor

for a Web server, offering HTTP over SSL access to

resources. Local Application Proxy acts as SSL proxy

for the server, performing all SSL related functions,

(e.g., authentication, key exchange, encryption).

4.1 Implementation

LAP uses the Stunnel 4.11 application (Trojnara,

2004) in addition with the OpenSSL software li-

braries (OpenSSL Development Team, 2005) to serve

as local SSL proxy for an Apache httpd Web server

2.0.52-3.1 (Apache Software Foundation, 2005a) on

a Fedora Core 3 Gnu/Linux system.

Figure 1 illustrates the communication between a

client and a server equipped with TLSS. In the ﬁrst

phase of communication, client and server establish

a SSL connection. Subsequently, the client initiates

SSL encrypted HTTP requests via his web browser.

The server receives packets, containing requests, on

the eth0 network interface card (NIC). Packets tra-

verse the network stack and are passed to Stunnel,

which decrypts SSL data and sends the encapsulated

HTTP request to httpd, using the loopback device lo.

lo is a virtual network interface used for local inter-

application communication on Linux systems. Out-

going HTTP responses from httpd to the client pass

the same way reversely.

Figure 1: Communication ﬂow between a server equipped

with TLSS and a client host.

The software sensor captures both the communica-

tion on the eth0 and the lo interface. It captures only

IP- and TCP-headers from eth0 and TCP-payload on

WEBIST 2006 - INTERNET TECHNOLOGY

174

lo, i.e. the decrypted HTTP-headers and -payload.

The sensor hands captured packet data to a local de-

tection engine which analyzes packets captured on

eth0 to detect attacks on the Internet and Transport

Layer and packets from lo to detect attacks inside

the encrypted SSL payload, e.g., attacks using hostile

HTTP-header and -payload data.

Packets captured on the lo interface do not include

the correct attacker’s IP address and port number, but

instead always the IP address 127.0.0.1 in combina-

tion with apparent random port numbers. LAP uses

those alternative addresses to communicate with the

local httpd server software. We adapted Stunnel to

generate a list of mappings:

(Stunnel’s port number on loopback interface) →

(client’s IP address, port number)

The list enables an administrator to link the alterna-

tive addresses to the attackers original ones. We plan

to adopt LAP to automatically change IP address and

port number to the correct values, before sending data

to a detection engine.

4.2 Function Test

The focal point of implementation is to demonstrate

the functionality of our idea, i.e. making possible

the detection of attacks encrypted at Transport Layer.

Test cases consist of a client establishing a SSL con-

nection to the server and consequently sending pack-

ets with malicious HTTP-requests through the en-

crypted tunnel. A local Snort-Inline IPS (Syngress

Author Team, 2004) detection engine on the server

host analyzes captured data.

In the ﬁrst test case, the server is equipped with a

TLSS, forwarding trafﬁc captured on the lo device to

the detection engine. Captured trafﬁc on lo is unen-

crypted, therefore the detection engine traces attacks

and generates alarms. TLSS achieves the same re-

sults, as well with malicious HTTP-headers, as with

malicious HTTP-payload.

The second test case consists of a system equipped

with a local sensor but no LAP. The sensor captures

packets including TCP payload directly from the NIC,

alike HIDS sensors, and passes them to the detection

engine. The packet’s payload contains malicious data,

but as it is nested inside encrypted SSL packets, the

detection engine does not observe the attacks.

Results indicate, that TLSS enables IDS to detect

attacks inside network trafﬁc encrypted at Transport

Layer.

5 PERFORMANCE

TLSS’s performance is an important criterion to eval-

uate the applicability of our idea. Therefore we mea-

sured the performance of a Web server equipped with

a TLSS. In particular, we want to evaluate LAP’s in-

ﬂuence on the server’s performance, as it executes all

SSL functions in lieu of the Web server application.

Evaluating software sensor’s performance is secon-

darily, therefore we deactivate data capturing. Per-

formance of data analysis is ignored, as it is not part

of TLSS.

As a benchmark, we compare the results to the

ones of a reference system which employs the mod

ssl

2.0.52-3.1 SSL-module.

5.1 Metric and Experimental Setup

The metric average latency and packet loss ratio in-

dicates system’s performance. In this context, latency

is deﬁned as the time delay between the moment the

client initiates a HTTPS-request and the moment he

receives the proper HTTPS-response including all re-

quested objects. Latency is a standard criterion for

evaluation of web server performance (Iyengar et al.,

1997), while the packet loss ratio indicates server’s

congestion.

The testbed consists of two HTTPS-server, i.e.,

the reference system and one equipped with TLSS,

three clients

and one management station, all inter-

connected over a switch. Clients, management sta-

tion and HTTPS-servers have AMD Athlon 64 3000+

CPUs, 512 MByte RAM and Broadcom NetXtreme

Ethernet adapters applicable on 100 MBit/s and 1

GBit/s networks. The operating system of all hosts

is Fedora Linux Core 3.

Both servers host an identical HTML-page of size

836 Byte, including references to four jpg-pictures

Clients are equipped with the Apache Web testing tool

JMeter (Apache Software Foundation, 2005b) to sim-

ulate Web users’ accessing the server, and to obtain

values for latency and packet loss ratio. A JMeter

instance on the management station coordinates the

clients.

In a ﬁrst test case we vary the encryption algorithm

and the duration of HTTP Keep Alive. Encryption al-

gorithm and corresponding key length have direct in-

ﬂuence on the performance of SSL sessions (Apos-

tolopoulos et al., 1999), while HTTP Keep Alive in-

ﬂuences the HTTP transfer encapsulated in a SSL

session, therefore also SSL latency. In addition, we

compare results for a 100 MBit/s and 1 GBit/s net-

work. We perform each measurement with 150, 300,

600 and 900 simultaneous user sessions respectively

to simulate increasing workload.

In this context, client indicates a host sending HTTPS-

requests to the server.

The picture’s size is 91.3, 23.5, 14.5 and 3.96 kByte

respectively

SECURITY SENSOR PROVIDING ANALYSIS OF ENCRYPTED NETWORK DATA

175

To do a detailed analysis of system’s behavior close

to satiation, we employ a second test case, with a 100

MBit/s network connection, no HTTP Keep Alive and

3-DES with 168 Bit key length. The number of paral-

lel user sessions is raised with high granularity (15,

30, 75, 150, 300, 450, 600 parallel sessions). We

measure the performance of TLSS with an activated

software sensor to evaluate our assumption, that LAP

and not the software sensor dominates TLSS’s perfor-

mance. The sensor captures data but does not save it

on hard disk.

5.2 Results

Figure 2 represents average latencies of LAP and the

reference system using the RC4 encryption algorithm,

on both 100 MBit/s and 1 GBit/s network. Laten-

cies for more than 600 user sessions on a 1 GBit/s

network are not printed because LAP gets overloaded

and starts dropping requests.

1000

2000

3000

4000

5000

6000

150 300 600 900

Parallel User sessions

Latency / ms

LAP,100 MBit/s

reference system, 100 MBit/s

LAP, 1 GBit/s

reference system,1 GBit/s

Figure 2: Average Latency for RC4 128 Bit, no HTTP Keep

Alive (Test case 1).

It is remarkable, that 100 MBit/s network offers

better performance, i.e. minor latency, than 1 GBit/s,

both for LAP as for the reference system. One pos-

sible reason is that higher number of parallel requests

reaches the server on a 1 GBit/s network than on 100

MBit/s. This leads to an increased number of par-

allel processes initiated by httpd

. As mentioned

in (Iyengar et al., 1997), we assume that, provoked

by Web server’s internal procedures, the CPU wastes

time switching between httpd processes which leads

to an increased latency .

On a 1 GBit/s network, LAP outperforms the refer-

ence system for 600 simultaneous user sessions. This

is due to limitations of the Web server software in

conjunction with standard settings, allowing a max-

imum of 256 parallel client sessions. The increased

The Web server is operated in prefork mode, i.e. using

one process per client request

latency of the reference system is caused by requests

waiting to be processed by httpd. Increasing the num-

ber of maximum client sessions results in minor la-

tency.

Figure 3 indicates latencies for AES in combina-

tion with varying HTTP Keep Alive Time. Using

HTTP Keep Alive decreases the latency for the ref-

erence system but not for LAP. We’ll discuss this in

the next section.

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

150 300 600 900

Parallel User sessions

Latency / ms

LAP, no HTTP KeepAlive

reference system, no HTTP KeepAlive

LAP, HTTP KeepAlive=15s

reference system, HTTP KeepAlive=15s

Figure 3: Average Latency for AES 128 Bit, 100 MBit/s

(Test case 1).

Figure 4 presents the average latencies of the refer-

ence system and TLSS, with activated data capturing.

2000

4000

6000

8000

10000

12000

14000

15 30 75 150 300 450 600

Parallel User sessions

Latency / ms

reference system

TLSS

Figure 4: Average Latency for 3DES 168 Bit, no HTTP

Keep Alive, 100 MBit/s (Test case 2).

5.3 Analysis of Results

The results indicate that the reference system per-

forms slightly better than LAP, i.e. the reference

system has a minor latency. The reference system’s

performance advantages are due to the fast internal

communication between the SSL module, perform-

WEBIST 2006 - INTERNET TECHNOLOGY

176

ing cryptographic functions, and the Web server soft-

ware. While the reference system employs API func-

tion calls, the communication between LAP and Web

server software utilizes the virtual lo device. For a

small number of parallel sessions, the internal com-

munication takes up most part of the server’s com-

puting time, resulting in a high relative performance

advantage, (i.e. performance advantage / total la-

tency), for the reference system. For an increasing

sessions’ quantity, execution of cryptographic algo-

rithms consumes the major part of CPU time. Cryp-

tographic computation’s performance

is similar for

both systems, therefore an increasing number of par-

allel user sessions diminishes the reference system’s

relative performance advantage.

As mentioned in the previous section, Figure 3 in-

dicates that activating HTTP Keep Alive does not lead

to a decreased latency for LAP. In contrast to LAP,

the reference system’s SSL module is integrated in

the Web server software. Hence, if HTTP Keep Alive

is activated on the reference system, the SSL module

automatically caches the corresponding SSL sessions

which leads to decreased latency. We plan to integrate

this feature in an improved version of LAP.

Results presented in Figure 4 indicate that activa-

tion of packet capturing has no negative impact on

sensor’s performance. The maximum capacity han-

dled by LAP is 600 simultaneous user sessions, while

reference system handles up to 750 sessions.

Overall results indicate that TLSS slightly de-

creases the performance of the monitored server and

can lead to a decreased availability in case of high

workload. This is due to the low performance of the

loopback interface used for local communication be-

tween LAP and server application, because data tra-

verses the network stack of the virtual device. We

argue that our implementation of TLSS is suitable for

Web servers that do not operate in the range of maxi-

mum workload.

6 CONCLUSION AND FUTURE

WORKS

In this paper we introduced Transport Layer Secu-

rity Sensor (TLSS) as part of an Intrusion Detection

System. TLSS facilitates detection engines to ana-

lyze network trafﬁc encrypted at the Transport Layer,

such as SSL. We designed TLSS to cooperate with

third-party detection engines, though enabling the in-

tegration into existing IDS. In contrast to existing so-

lutions, TLSS neither denies the non-repudiation as-

surance, nor exports fundamental secrets (e.g. private

Both Stunnel and mod ssl use the OpenSSL (OpenSSL

Development Team, 2005) libraries

keys) to other hosts.

Our implementation of TLSS for a SSL-enabled

Web server indicates the viability of our idea, but re-

quires further improvements in terms of performance.

The focal point is to increase the maximum number

of parallel user sessions handled by TLSS.

In addition, we plan to extend TLSS to provide pre-

analysis and anonymization of data before sending it

to a detection engine. While the ﬁrst will decrease de-

tection engine’s workload, latter will prevent that an

attacker, wiretapping the communication between en-

gine and sensor, gathers sensitive informations from

the transmitted data stream.

REFERENCES

Almgren, M. and Lindqvist, U. (2001). Application-

integrated data collection for security monitoring. In

Recent Advances in Intrusion Detection (RAID 2001)

Proceedings, Davis.

Apache Software Foundation, A. (2005a). Apache http

server project. http://httpd.apache.org/.

Apache Software Foundation, A. (2005b). Apache jmeter

project.

Apostolopoulos, Peris, and Saha (1999). Transport layer se-

curity: How much does it really cost? In INFOCOM:

The Conference on Computer Communications, joint

conference of the IEEE Computer and Communica-

tions Societies.

boede + partners integrated marketing (2004). Research -

consumer awareness and concerns.

Breach Security Inc., B. (2004). Breach viewl ssl. White

Paper.

Daniels, T. and Spafford, E. (1999). Identiﬁcation of host

audit data to detect attacks on low-level ip vulnerabil-

ities. Journal of Computer Security, 7(1):3–35.

Iyengar, A., MacNair, E., and Nguyen, T. (1997). An anal-

ysis of web server performance. In Proceedings of

GLOBECOM ’97.

OpenSSL Development Team, O. (2005). Openssl project.

http://www.openssl.org/.

Ristic, I. (2005). Apache Security. O’Reilly Media, Inc.

Syngress Author Team, S. (2004). Snort 2.1 Intrusion De-

tection. Syngress Publishing, Rockland.

Trojnara, M. (2004). Stunnel - ssl encryption wrapper.

http://stunnel.mirt.net/.

SECURITY SENSOR PROVIDING ANALYSIS OF ENCRYPTED NETWORK DATA

177