TOWARDS RUN-TIME PROTOCOL ANOMALY DETECTION AND

VERIFICATION

InSeon Yoo and Ulrich Ultes-Nitsche

Department of Computer Science, University of Fribourg,

Chemin du Musee 3, Fribourg, CH1700, Switzerland.

Keywords:

Run-time Protocol Veriﬁcation, Protocol Anomaly Detection, SDL, EFSM/CEFSM

Abstract:

‘How to verify incoming packets whether they follow standards or not?’ and ‘How to detect protocol anomalies

in real-time?’, we seek to answer these questions. In order to solve these questions, we have designed a

packet veriﬁer with packet inspection and sanity check. In this work, we specify TCP transaction behaviours

declaratively in a high-level language called Speciﬁcation and Description Language (SDL). This speciﬁcation

will be then compiled into an inspection engine program for oberving packets. In addition, the SanityChecker

covers protocol header anomalies.

1 INTRODUCTION

Protocols are created with speciﬁcations, known as

RFCs, to dictate proper use and communication. An

anomaly is deﬁned as something different, abnormal,

peculiar, or not easily classiﬁed. Protocol anomaly

refers to all exceptions related to protocol format and

protocol behaviour with respect to common practice

on the Internet and standard speciﬁcations. This in-

cludes network and transport layer protocol anoma-

lies in layer 3 and layer 4 and application layer proto-

col anomalies in layer 6 and layer 7.

Protocol anomaly detection is essential for under-

standing new attacks. Without names and prior docu-

mentation, new attacks can only be defended against

by understanding their intrusion methods and their ef-

fects. However, important to note is that not all threats

or attacks exhibit themselves as protocol anomalies.

Some types of application logic attacks, denial of ser-

vice (DoS) attacks, viruses, and reconnaissance meth-

ods all appear as perfectly legitimate network trafﬁc.

In this paper, we present our packet veriﬁer model

based on a speciﬁcation of security protocols given

in a high-level language, called Speciﬁcation and De-

scription Language (SDL). Then, we address how to

verify TCP protocol transition.

2 EXAMPLE CASES OF

PROTOCOL ANOMALIES

2.1 Ping of Death

Attackers sends a fragmented PING request that ex-

ceeds the maximum IP packet size(64KB), causing

vulnerable systems to crash. The idea behind the Ping

of Death and similar attacks is that the user sends a

packet that is malformed in such a way that the target

system will not know how to handle the packet. The

Ping of Death attack (Fyodor, 1996) sent IP packets

of a size greater than 65,535 bytes to the target com-

puter. IP packets of this size are abnormal, but appli-

cations can be built that are capable of creating them.

Carefully programmed operating systems could de-

tect and safely handle abnormal IP packets, but some

failed to do this. ICMP ping utilities often included

large-packet capability and became the namesake of

the problem, although UDP and other IP-based proto-

cols also could transport Ping of Death.

2.2 Land Attack

The land attacks (CISCO, 1997) are also known as IP

DOS (Denial of Service) (Fyodor, 1997). The land at-

tack involves the perpetrator sending a stream of TCP

SYN packets that have the source IP address and TCP

port number set to the same value as the destination

299

Yoo I. and Ultes-nitsche U. (2004).

TOWARDS RUN-TIME PROTOCOL ANOMALY DETECTION AND VERIFICATION.

In Proceedings of the First International Conference on E-Business and Telecommunication Networks, pages 299-304

DOI: 10.5220/0001395802990304

 SciTePress

address and port number, i.e., that of the attacked host.

Some implementations of TCP/IP cannot handle this

theoretically impossible condition, causing the oper-

ating system to go into a loop as it tries to resolve

repeated connections to itself.

2.3 SYN ﬂood attack

Figure 1: SYN Flooding

The client system begins by sending a SYN mes-

sage to the server (CERT/CA-1996-21, 2000). The

server then acknowledges the SYN message by send-

ing SYN-ACK message to the client like in Figure1.

The client then ﬁnishes establishing the connection by

responding with an ACK message. The connection

between the client and the server is then open, and the

service-speciﬁc data can be exchanged between the

client and the server.

The TCP SYN attack exploits this design by hav-

ing an attacking source host send SYN packets with

random source addresses to a victim host. The vic-

tim destination host sends a SYN ACK back to the

random source address and adds an entry to the con-

nection queue. Since the SYN ACK is destined for

an incorrect or nonexistent host, the last part of the

three-way handshake is never completed, and the en-

try remains in the connection queue until a timer ex-

pires, typically within about one minute. By gener-

ating phony SYN packets from random IP addresses

at a rapid rate, it’s possible to ﬁll up the connection

queue and deny TCP services to legitimate users.

2.4 Teardrop attack

Teardrop attack (Hoggan, 2000) targets a vulnerabil-

ity in the way fragmented IP packets are reassembled.

Fragmentation is necessary when IP datagrams are

larger than the maximum transmission unit (MTU) of

a network segment across which the datagrams must

traverse. In order to successfully reassemble packets

at the receiving end, the IP header for each fragment

includes an offset to identify the fragment’s position

in the original unfragmented packet. In a Teardrop

attack, packet fragments are deliberately fabricated

with overlapping offset ﬁelds causing the host to hang

or crash when it tries to reassemble them. Under nor-

mal conditions packet fragments will yield a positive

integer value as can be derived from the diagram be-

low.

Figure 2: IP TearDrop Attack - Correct reassemble

However, the teardrop attack sends a fragment that

deliberatley forces the calculated value for the end

pointer to be less than the value for the offset pointer.

This can be achieved by ensuring that the second frag-

ment speciﬁes a FRAGMENT OFFSET that resides

within the data portion of the ﬁrst fragment and has a

length such that the end of the data carried by the sec-

ond fragment is short enough to ﬁt within the length

speciﬁed by the ﬁrst fragment. Diagramatically this

can be shown as follows:

Figure 3: IP TearDrop Attack - Incorrect reassemble

When the IP module performing the reassembly

attempts a memory copy of the fragment data into

the buffer assigned to the complete datagram, the

calculated length of data to be copied (that is the

end pointer minus the offset pointer) yields a neg-

ative value. The memory copy function expects an

unsigned integer value and so the negative value is

viewed as a very large positive integer value. The re-

sults of such an action depends upon the IP implemen-

tation, but typically cause stack corruption, failure of

the IP module or a system hang.

3 PACKET VERIFIER MODEL

The purposes of the packet veriﬁer are validating

compliance to standards, and validating expected us-

age of protocols e.g. protocol anomaly detection. The

packet veriﬁer checks the protocol header of pack-

ets, veriﬁes packet size, checks TCP/UDP header

ICETE 2004 - SECURITY AND RELIABILITY IN INFORMATION SYSTEMS AND NETWORKS

300

Figure 4: Components of the Packet Veriﬁer

length, veriﬁes TCP ﬂags and all packet parame-

ters, does TCP protocol type veriﬁcation, and anal-

yses TCP Protocol header and TCP protocol ﬂags.

In addition, the packet veriﬁer contains an inspec-

tion engine like in Fig.4, including an observer and

an main object model, to validate expected usage of

protocols with SDL (the Speciﬁcation and Descrip-

tion Language) (ITU-T, 1992) speciﬁcations. SDL

speciﬁcation deﬁnes the possible behaviours of pro-

tocols. While investigating the encoded packets, the

observer/SDL-parser validates whether or not each

sequence of packets follows what is required by the

SDL speciﬁcations.

3.1 Protocol Speciﬁcation

In our work, we have abstracted from our state ma-

chine speciﬁcation (see Figure5) to capture only the

essential details of TCP protocols. Using a more ab-

stract speciﬁcation, where the state machines accept

a superset of what is permitted by the standards, and

is still sufﬁcient to deal with incomplete protocol runs

meeting the standards (such as in the case of the SYN

ﬂood attack). We present a speciﬁcation of the TCP

state machine in this section. In order to achieve the

goal of identifying TCP transition invariants for a set

of reachable states, our method ﬁrst searches through

all the reachable states from the initial state to ﬁnd the

invariants. It then reduces the number of the searched

states. As the number of the searched states reduced,

the number of invariants increases.

A TCP connection is always initiated with the

three-way handshake, which establishes and negoti-

ates the actual connection over which data will be

sent. The whole session begins with a SYN packet,

then a SYN/ACK packet and ﬁnally and an ACK

packet to acknowledge the whole session establish-

ment. Our TCP speciﬁcation is depicted pictorially in

Figure 5. A new session starts in the LISTEN state.

Data transfer takes place in the connection ESTAB-

LISHED state. If the TCP connection is initiated from

an external site, then the state machine goes through

SYN

RCVD and ACK WAIT states to reach the ES-

TABLISHED state. If the connection is initiated from

an internal machine, then the ESTABLISHED state

is reached through the SYN SENT state. In order to

tear down the connection, either side can send a TCP

segment with the FIN bit set. If the FIN packet is

sent by an internal host, the state machine waits for

an ACK of FIN to come in from the outside. Data

may continue to be received till this ACK to the FIN

is received. It is also possible that the external site

may initiate a closing of the TCP connection. In

this case we may receive a FIN, or a FIN + ACK

from the external site. This scenario is represented

by the states FIN WAIT 1, FIN WAIT 2, CLOSING,

TIME WAIT 1, and TIME WAIT 2 states. Our state

machine characterizes receive and send events al-

together to understand and to check properly. If

the connection termination is initiated by an external

host, note that the TCP RFCs do not have the states

WAIT, LAST ACK WAIT, and LAST ACK

since they deal with packets observed at one of the

ends of the connection. In that case, it is reasonable

to assume that no packets will be sent by a TCP stack

implementation after it receives a FIN from the other

end. In our case, we are observing trafﬁc at an inter-

mediate node, e.g. ﬁrewall, so the tear down process

is similar regardless of which end initiated the tear

down. To reduce clutter, the following classes of ab-

normal transitions are not shown: conditions where

an abnormal packet is discarded without a state tran-

sition, e.g., packets received without correct sequence

numbers after connection establishment and packets

with incorrect ﬂag settings. Beause these parts will

be checked by the SanityChecker.

3.2 Protocol SanityChecker

To cover other protocol aspects apart from TCP state

speciﬁcation, we are building a sanity checker. This

performs layer 3 and layer 4 sanity checks. These in-

clude verifying packet size, checking UDP and TCP

header lengths, dropping IP options and verifying the

TCP ﬂags to ensure that packets have not been manu-

ally crafted by a malicious user, and that all packet pa-

rameters are correct. In the IP protocol, according to

the Internet Protocol Standard (RFC791, 1981), an IP

header length should always be greater than or equal

to the minimal Internet header length (20 octets) and

a packet’s total length should always be greater than

its header length. IP address checks will also be im-

portant since land attacks use the same IP address for

source and destination. According to the TCP stan-

dard (RFC793, 1981), neither the source nor the des-

TOWARDS RUN-TIME PROTOCOL ANOMALY DETECTION AND VERIFICATION

301

Figure 5: TCP Protocol State Machine

tination TCP port number can be zero, and TCP ﬂags,

e.g. URG and PSH ﬂags, can be used only when a

packet carries data. Thus, for instance, combinations

of SYN and URG or SYN and PSH become invalid.

In addition, any combination of more than one of the

SYN, RST, and FIN ﬂags is also invalid.

SanityChecker examines every packet within a 10

second window, and at the end of each window it

will record any malicious activity it sees using sys-

log. SanityChecker currently detects some attacks; all

TCP scans, all UDP scans, SYN ﬂood attacks, Land

attacks, and ping of death attacks. SanityChecker as-

sumes any TCP packet other than a RST may be used

to scan for services. If packets of any type are re-

ceived by more than 7 different ports within the win-

dow, an event is logged. The same criteria are used for

UDP scans. If SanityChecker sees more than 8 SYN

packets to the same port with no ACK’s or FIN’s asso-

ciated with the SYN’s, a SYN ﬂood event is logged.

Any TCP SYN packets with source and destination

address and ports the same is a identiﬁed as a land at-

tack. If more than 5 ICMP ECHO REPLIES are seen

within the window, SanityChecker assumes it may be

a Smurf attack (CERT, 1998). Note that this is not a

certainty. SanityChecker also assumes that any frag-

mented ICMP packet is bad, so this catches attacks

such as the ping of death. To make the certatinty

higher, this SanityChecker cooperateswith the proto-

col inspection engine with SDL. Furthermore, when

the SanityChecker cooperates with the protocol in-

spection engine, this can work as a sequence veriﬁer

to matches the current TCP packet’s sequence num-

bers against a state kept for that TCP connection. For

example, in teardrop attack case, fragmented pack-

ets can be dealt with packet’s IP id, and sequence.

The inspection engine with SDL examines all packet

transitions and remember IP id, and sequences, so the

SanityChecker can detect reassemble problems with

this engine.

4 GENERATING THE STATE

MACHINE SPECIFICATION

We made the speciﬁcation by hand, our next step is

applying SDL to accomplish this speciﬁcation. SDL

is an International Telecommunication Union (ITU)

standard, based on the concept of a system of Com-

municating Extended Finite State Machine (CEFSM)

ICETE 2004 - SECURITY AND RELIABILITY IN INFORMATION SYSTEMS AND NETWORKS

302

Model (E.Hopcroft and D.Ullman, 1979). To under-

stand how SDL can work based on the CEFSM, we

address the dynamic semantics of the ﬁnite state ma-

chine, and SDL underlying model in this section.

4.1 Speciﬁcation Development

We present how we specify TCP state transition with

CEFSM in this section. A CEFSM is deﬁned as a 6-

tuple < S, s

, E, f, V, X >, as we mentioned above.

Where, S is a set of states, s

is an initial state, E is

a set of events with their parameter lists, f is a state

transition relation. V is a set of local variables along

with their types and initial values, if any. X is a set

of signals. For a state, an input event, and a predicate

composed of a subset of V, the state transition rela-

tion f has a next state, a set of output events and their

parameters, and an action list describing how the lo-

cal variables are updated. The purpose of SDL in our

project to verify whether the TCP transition follows

the standards. To do this, we made very simple TCP

transition using SDL based on Figure 5. For TCP state

transition, our CEFSM is like following:

• S = { listen, syn rcvd, syn sent, ack wait, estab-

lished, ﬁn wait 1, ﬁn wait 2, closing, close wait 1,

close wait 2, time wait, last ack wait, last ack,

closed }

• s

= listen

• E = { send(ip id, ﬂags), recv(ip id, ﬂags)}

• f : { f(listen, recv(ip id, SYN), ip seq per id =

0) -> (syn rcvd, ip seq per id = ip seq per id +

ip seq, SYN, ACK), f(listen, send(ip id, SYN),

SYN, ip seq per id != 0) -> (syn sent, , ), ..... }

• V = ip seq per id, ip seq

• X = ACK, SYN, FIN

In this SDL speciﬁcation, Timeout, and other ﬂags

e.g., RST, PSH, URG are not included. Timeout and

RST, PSH, URG ﬂags can be dealt with low-level

implementation part. To detect packet fragmenta-

tion, the SDL speciﬁcation part can tell the packet se-

quence and proper ﬂag, and low-level implementation

part cooperate with this SDL, other ﬂag combination,

and timeout part. Figure 6. shows the StateTransition

process which we built in SDL. Like SDL-GR(Figure

6.), sdl-PR is also in [Table 1].

5 CONCLUSION

We have discussed protocol anomalies and address a

packet veriﬁer model. The purposes of the packet ver-

iﬁer are to validate compliance to standards, and to

validate expected usage of protocols, especially pro-

tocol anomaly detection. Considering performance in

Table 1: Part of SDL-PR Source in the process StateTransi-

tion

PROCESS StateTransition ;

NEWTYPE PacketInfo

STRUCT

ip Integer;

seq Integer;

ﬂag TCPFlags;

OPERATORS

Unexpected: Integer, TCPFlags -> PacketInfo;

SanityCheck: Integer, Integer -> PacketInfo;

ENDNEWTYPE;

DCL pkt PacketInfo;

DCL tcp id, tcp seq, tcp seq per id Integer := 0;

DCL tcp ﬂag TCPFlags;

DCL cur process PId; /* current process */

Timer t;

START;

NEXTSTATE idle ;

STATE syn sent ;

INPUT Packet(tcp id, tcp seq, tcp ﬂag) ;

TASK pkt := SanityCheck(tcp id, tcp seq) ;

DECISION tcp ﬂag ;

( ACKFIN ):NEXTSTATE close wait 2 ;

( ACKSYN ):NEXTSTATE established ;

( SYN ): NEXTSTATE syn rcvd ;

ELSE: TASK pkt := Unexpected(tcp id, tcp ﬂag) ;

NEXTSTATE - ;

ENDDECISION;

ENDSTATE;

STATE syn rcvd ;

INPUT NONE;

OUTPUT ROP(tcp id, ACKSYN) to SENDER ;

NEXTSTATE ack WAIT ;

ENDSTATE;

real-system, we are implementing a SanityChecker to

cover protocol header anomalies, and using SDL, we

specify TCP transition behaviours, the speciﬁcation

will then be compiled into an inspection engine pro-

gram for observing packets. At the moment, we spec-

iﬁed TCP protocol transition. Through this state ma-

chine, we will implement the inspection engine. This

will be our future work. We believe that this protocol

anomaly analysis and the packet veriﬁer model will be

useful to detect protocol anomalies and verify proper

usage of protocols.

TOWARDS RUN-TIME PROTOCOL ANOMALY DETECTION AND VERIFICATION

303

Figure 6: Process StateTransition of the TCP Protocol State Machine in SDL

REFERENCES

CERT (1998). Advisory ca-1998-01 smurf ip denial-of-

service attacks. In Online Publication.

CERT/CA-1996-21 (2000). Advisory ca-1996-21 tcp syn

ﬂooding and ip spooﬁng attacks. In Online publica-

tion.

CISCO (1997). Security advisory: Tcp loopback dos attack

(land.c) and cisco devices.

E.Hopcroft, J. and D.Ullman, J. (1979). Introduction to Au-

tomata Theory, languages, and computation. Addison

Wesley.

Fyodor (1996). Ping of death attack. In INSECURE.ORG.

Fyodor (1997). The land attack(ip dos). In ENSE-

CURE.ORG.

Hoggan, D. (1994-2000). Teardrop attack. In The Internet

Book: Introduction and Reference.

ITU-T, C. (1992). Recommendation Z.100: Speciﬁcation

and Description Language (SDL). General Secre-

tariat, Geneve, Switzerland.

RFC791 (1981). Internet protocol. In DARPA Internet Pro-

gram Protocol Speciﬁcation.

RFC793 (1981). Transmission control protocol. In DARPA

Internet Program Protocol Speciﬁcation.

ICETE 2004 - SECURITY AND RELIABILITY IN INFORMATION SYSTEMS AND NETWORKS

304