![](bg2.png)
Figure 1: Interposed library.
quences can be logged.
Kuperman et al. (Kuperman and Spafford, 1998)
applied Curry’s technique to IDS for detecting buffer
overflow. Figure 1 shows a sequence when a library
call is encountered. The IDS acquires audit data at
user level using an interposed library. Every library
function is thus intercepted. The interposed library
can acquire audit data at the user level without need-
ing to modify either application or system libraries.
The authors show the cost of audit on their approach
is low, and report that this approach is suitable for
buffer overflow detection.
However, in their system there are two problems.
The first problem is that malicious attackers can easily
bypass the interposed library, because only the library
function is checked. If the attacker calls any system-
call directly, the system-call will not be checked by
their system. The secondly, every library function is
checked every time. As a result, the interposed library
adds an overhead to every library function.
This paper discusses an intrusion detection system
with interposed system-calls and a user level access
control mechanism using a dynamic linker. This in-
trusion detection system resolves the first problem de-
scribed above. The system is based on the access con-
trol mechanism at the user level. It operates in con-
junction with the interposing of system-calls. As a re-
sult, our intrusion detection system can detect system-
calls that bypass the library interposition.
The proposed access control mechanism can re-
solve the second problem described above. The
mechanism monitors the execution of target programs
at appropriate timing using dynamic linker. Dynamic
linking is employed, such that a program and a partic-
ular library referenced by the program are not com-
bined together by the linker at link-time. In the pro-
posed mechanism, a dynamic linker resolves a refer-
ence to call the access control module. The module
runs at the user level and decides to grant or deny the
library function after the check of access rights. If the
module denies the access, the library function is not
called. In addition, the checkpoint of library functions
can be selected arbitrarily. Important library functions
that call system-calls, can be controlled by the mech-
anism with a little overhead. The reference of non-
important routines that do not call system-calls such
as math functions, are resolved to call the module. As
a result, the library functions are called directly with-
out overhead. Generally, library functions that cause
security problems invoke system-calls. Therefore, it
is believed that this approach is reasonable.
2 RELATED WORK
IDS are an example of a defense mechanism against
attacks that abuse software vulnerabilities (Wagner
and Dean, 2001) (Hofmeyr et al., 1998) (Wagner and
Soto, 2002). IDS observe program execution and
detect malicious behavior of the program. As men-
tioned previously, because an anomaly detection sys-
tem is based on a normal behavior database, the dis-
advantages is that the false positive ratio is significant.
Therefore, main issue of intrusion detection systems
is reduction of overhead and false positive and false
negative ratios.
Sekar et al. (Sekar et al., 2001) reported an over-
head due to the execution of learning and/or detection
code of between 3% and 4%. Sekar et al. also re-
ported that the overhead due to system-call interposi-
tion is between 100% and 250%. In addition, Ooyama
et al. (Oyama et al., 2003) reported that the run-time
of httpd increased 47%. These IDS are based on the
kernel. By contrast, the IDS proposed in this paper is
based on both the user and the kernel. Kuperman et
al. reports that the overhead is between 3.2%. Thus,
IDS based on user level possess overhead advantages.
However, Kuperman reports the overhead of 57.2% in
the worst case.
Jain et al. (Jain and Sekar, 2000) presented an ap-
proach to developing a user-level infrastructure for
system-call interception and extension. This approach
requires a process switch to intercept system-calls. As
a result, there is a high overhead. The mechanism
proposed here does not involve process switching to
intercept library functions and system-calls.
3 REQUIREMENT OF
INTRUSION DETECTION
SYSTEM
3.1 Problem
Most of the existing kernel based IDS detect intrusion
using history of system-call sequence. In the IDS, the
false positive ratio is a crucial problem. The reduction
in the false positive ratio facilitates their introduction
ICETE 2004 - SECURITY AND RELIABILITY IN INFORMATION SYSTEMS AND NETWORKS
264