![](bg3.png)
Figure 4: Components of the Packet Verifier
length, verifies TCP flags and all packet parame-
ters, does TCP protocol type verification, and anal-
yses TCP Protocol header and TCP protocol flags.
In addition, the packet verifier contains an inspec-
tion engine like in Fig.4, including an observer and
an main object model, to validate expected usage of
protocols with SDL (the Specification and Descrip-
tion Language) (ITU-T, 1992) specifications. SDL
specification defines the possible behaviours of pro-
tocols. While investigating the encoded packets, the
observer/SDL-parser validates whether or not each
sequence of packets follows what is required by the
SDL specifications.
3.1 Protocol Specification
In our work, we have abstracted from our state ma-
chine specification (see Figure5) to capture only the
essential details of TCP protocols. Using a more ab-
stract specification, where the state machines accept
a superset of what is permitted by the standards, and
is still sufficient to deal with incomplete protocol runs
meeting the standards (such as in the case of the SYN
flood attack). We present a specification of the TCP
state machine in this section. In order to achieve the
goal of identifying TCP transition invariants for a set
of reachable states, our method first searches through
all the reachable states from the initial state to find the
invariants. It then reduces the number of the searched
states. As the number of the searched states reduced,
the number of invariants increases.
A TCP connection is always initiated with the
three-way handshake, which establishes and negoti-
ates the actual connection over which data will be
sent. The whole session begins with a SYN packet,
then a SYN/ACK packet and finally and an ACK
packet to acknowledge the whole session establish-
ment. Our TCP specification is depicted pictorially in
Figure 5. A new session starts in the LISTEN state.
Data transfer takes place in the connection ESTAB-
LISHED state. If the TCP connection is initiated from
an external site, then the state machine goes through
SYN
RCVD and ACK WAIT states to reach the ES-
TABLISHED state. If the connection is initiated from
an internal machine, then the ESTABLISHED state
is reached through the SYN SENT state. In order to
tear down the connection, either side can send a TCP
segment with the FIN bit set. If the FIN packet is
sent by an internal host, the state machine waits for
an ACK of FIN to come in from the outside. Data
may continue to be received till this ACK to the FIN
is received. It is also possible that the external site
may initiate a closing of the TCP connection. In
this case we may receive a FIN, or a FIN + ACK
from the external site. This scenario is represented
by the states FIN WAIT 1, FIN WAIT 2, CLOSING,
TIME WAIT 1, and TIME WAIT 2 states. Our state
machine characterizes receive and send events al-
together to understand and to check properly. If
the connection termination is initiated by an external
host, note that the TCP RFCs do not have the states
CLOSE
WAIT, LAST ACK WAIT, and LAST ACK
since they deal with packets observed at one of the
ends of the connection. In that case, it is reasonable
to assume that no packets will be sent by a TCP stack
implementation after it receives a FIN from the other
end. In our case, we are observing traffic at an inter-
mediate node, e.g. firewall, so the tear down process
is similar regardless of which end initiated the tear
down. To reduce clutter, the following classes of ab-
normal transitions are not shown: conditions where
an abnormal packet is discarded without a state tran-
sition, e.g., packets received without correct sequence
numbers after connection establishment and packets
with incorrect flag settings. Beause these parts will
be checked by the SanityChecker.
3.2 Protocol SanityChecker
To cover other protocol aspects apart from TCP state
specification, we are building a sanity checker. This
performs layer 3 and layer 4 sanity checks. These in-
clude verifying packet size, checking UDP and TCP
header lengths, dropping IP options and verifying the
TCP flags to ensure that packets have not been manu-
ally crafted by a malicious user, and that all packet pa-
rameters are correct. In the IP protocol, according to
the Internet Protocol Standard (RFC791, 1981), an IP
header length should always be greater than or equal
to the minimal Internet header length (20 octets) and
a packet’s total length should always be greater than
its header length. IP address checks will also be im-
portant since land attacks use the same IP address for
source and destination. According to the TCP stan-
dard (RFC793, 1981), neither the source nor the des-
TOWARDS RUN-TIME PROTOCOL ANOMALY DETECTION AND VERIFICATION
301