Intrusion Detection System Test Framework for SCADA Systems

Henrik Waagsnes and Nils Ulltveit-Moe

Department of ICT, University of Agder, Jon Lilletunsvei 9, 4879 Grimstad, Norway

Keywords:

IDS, Test Framework, SCADA, IEC 60870-5-104, SIEM.

Abstract:

This paper presents a SCADA intrusion detection system test framework that simulates SCADA trafﬁc and

detects malicious network activity. The framework combines several existing components such as Kali Linux,

Conpot, QTester104 and OpenMUC in a virtual machine based framework to provide realistic SCADA trafﬁc.

It is agnostic to Intrusion Detection System (IDS) type, and is demonstrated in a case study comparing two

popular signature-based IDS engines: Suricata and Snort. The IDS engines include rule-sets for the IEC

60870-5-104 and other SCADA protocols. Detected events from IDS sensors are sent to a distributed Elastic

cluster which visualises them using Kibana dashboards. The experiments show that there is some difference in

behaviour between Suricata and Snort’s ability to detect malicious trafﬁc using the same SCADA ruleset, but

these issues are relatively easy to mitigate. The IDS test framework also measures the latency from detection

and until the IDS alerts are presented in the incident management system, which shows that Suricata has

slightly better performance than Snort.

1 INTRODUCTION

This paper describes the design and implementation

of a testbed for assessing IDS performance in SCADA

networks. It combines off-the-shelf components such

as the Elastic stack, Kali Linux, and published IDS

rule sets to generate simulated trafﬁc and analyse re-

sults. The paper’s experimental section evaluates two

popular open–source IDS engines against SCADA-

related trafﬁc produced by several public tools. Over-

all, this provides a useful end-to-end solution for

testing IDS tools against SCADA–oriented attacks,

which are an increasingly prevalent security issue

nowadays. The long-term objective is to extend it as

a control system platform for performing realistic ex-

ercises in a simulated operations centre as shown in

Figure 1. The aim is to improve the situational aware-

ness for security analysts and other stakeholders (per-

sonnel managing power lines etc) during emergency

situations. This would allow performing realistic ex-

ercises on cyber-attack events that naturally will occur

infrequently, thereby increasing the readiness towards

such attacks. It will especially allow performing exer-

cises that simulate advanced persistent threat scenar-

ios caused by government agencies or other organisa-

tions with signiﬁcant funding that aim at performing

targeted attacks on critical infrastructure as demon-

strated for example by the Ukrainian BlackEnergy at-

Figure 1: CIEM Test lab for evaluating situational aware-

ness.

tack

SCADA systems used in the energy sector encom-

passes the following: collecting of information via

Remote Terminal Units (RTUs), transferring it back

to the central site, carrying out necessary analysis and

control, and then displaying that information in a Hu-

man Machine Interface (HMI). A SCADA communi-

cation protocol is a standard for data representation

and data transfer over a communication channel on a

master/slave basis. IEC60870-5-104 and DNP3 are

Cyber-Attack Against Ukrainian Critical Infrastructure

https://ics-cert.us-cert.gov/alerts/IR-ALERT-H-16-056-01

Waagsnes, H. and Ulltveit-Moe, N.

Intrusion Detection System Test Framework for SCADA Systems.

DOI: 10.5220/0006588202750285

In Proceedings of the 4th International Conference on Information Systems Security and Privacy (ICISSP 2018), pages 275-285

ISBN: 978-989-758-282-0

275

two of the most frequently used SCADA communica-

tion protocols in the electrical energy industry. IEC

60870-5 is widely used in Europe, while DNP3 is

widely used in North America.

The main contributions of this paper are threefold:

ﬁrst, it presents a novel intrusion detection system test

framework for SCADA networks based on a set of

virtual machines that can be used to simulate differ-

ent attack scenarios. This framework builds upon and

integrates several other SCADA tools in a novel way,

and also allows for simulating the HMI of a power

plant. Secondly, this framework is demonstrated in a

comparative study between the signature-based IDS

Snort

and Suricata

to verify whether an IEC60870-

5-104 SCADA ruleset for Snort indeed is compatible

with Suricata or not. Third, it performs a comparative

study of the efﬁciency of Snort and Suricata when in-

tegrated in the Elastic stack

(formerly known as the

Elasticsearch, Logstash and Kibana (ELK) stack).

This paper is organised as follows: The next sec-

tion gives an introduction to SCADA protocols. Sec-

tion 3 discusses related works. Section 4 describes

the proposed framework architecture and developed

dashboards. Section 5 discusses experiments and re-

sults in context of validation and testing of the imple-

mented framework. Section 6 contains a discussion

of the proposed framework, evaluation of results, and

other thoughts about the conducted research. Section

7 contains a conclusion of the presented framework

and achieved results, and ﬁnally section 8 contains a

discussion around possible future work and sugges-

tions for improvements.

2 INTRODUCTION TO SCADA

PROTOCOLS

IEC 60870-5-104 is a standard SCADA communica-

tion protocol for telecontrol in electrical energy sys-

tems. The protocol describes data representation and

data transfer over communication channels on a mas-

ter/slave basis. The SCADA Master polls data from

the connected SCADA slaves (RTUs). A RTU is used

in the electrical power sector to control circuit break-

ers, transformer stations etc. The SCADA Master can

display the information in a Human-Mahine Interface

(HMI). Figure 2 shows a simpliﬁed SCADA archiec-

ture.

Snort: https://www.snort.org

Suricata https://suricata-ids.org

Elastic: https://www.elastic.co

Figure 2: Example SCADA system.

Start 68H

APDU Length

Control Field Octet 1

Control Field Octet 2

Control Field Octet 3

ASDU defined in IEC 101

and IEC 104

Control Field Octet 4

ASDU ASDU Field

Data Unit

Identifier

Data Unit

Type

Identification

(TI)

Variable

Structure

Qualifier

Cause of Transmission (CoT)

Common Adress

Information Object Address

Information Object

Time Tag

Information

Object

Length

APDU

APCI

ASDU

Figure 3: IEC 60870-5-104 frame format.

2.1 IEC 60870-5-104

IEC 60870 is a collection of open standards created by

the International Electrotechnical Commission (IEC)

for the transmission of SCADA telemetry control

and data (International Electrotechnical Commission,

2006). IEC 60870 normally refers to IEC 60870-5-

101 when discussed in context of SCADA in the elec-

trical energy sector. This is a standard for power sys-

tem monitoring, control and associated communica-

tions. When it was launched in 1995, the protocol

was designed for serial communication. IEC 60870-

5-104 is an extension of IEC 60870-5-101 that was

released in 2004, which allows the serial frames to

be transmitted over TCP/IP. The Application Protocol

Data Unit (APDU) shown in ﬁgure 3 consists of two

parts: the Application Protocol Control Information

(APCI) and Application Service Data Unit (ASDU).

The APCI is comprised of the ﬁrst 6 octets in the

APDU, and contains a start character, 68H, a length

ﬁeld (containing the length of the APDU) and a con-

trol ﬁeld. The APCI control ﬁeld can be of the follow-

ing types: Information, Supervisory or Unnumbered.

The last two bits indicate the type; 10 for supervisory,

11 for unnumbered and 00 for information.

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

276

The ASDU contains the data unit identiﬁer and the

data payload of one or more information objects. The

Type Identiﬁcation (TI) ﬁeld deﬁnes the data types by

referring to the 8-bit code types. The TI groups that

are currently deﬁned by IEC. TI number 9 for exam-

ple refers to the reference code M_ME_NA_1 indi-

cating “Measured value, Normalised value”.

2.2 Security Issues

The security in SCADA systems and the communi-

cation protocols used to exchange data is important,

to prevent cyber-attacks on critical infrastructures that

they control. The root cause of cyber security vulner-

abilities in industrial control systems have been found

to be (Graham et al., 2016):

• Poorly secured legacy systems;

• Lack of trained cyber security specialists;

• Delayed patching of software vulnerabilities;

• Lack of cyber-security situational awareness;

• Lack of communication security;

• Remote access for system engineers and vendors.

Many SCADA systems lack security incident event

detection and monitoring (SIEM) functionality to in-

crease the cyber-security awareness and detect cyber-

attacks on these systems. Without network moni-

toring, it is impossible to detect suspicious activity

and identify potential threats. Another problem is

that some systems are rarely or never updated. This

means the SCADA systems might contain vulnerabil-

ities in ﬁrmware or software that can be exploited by

attackers. Some vendors even allow SCADA devices

to communicate remotely over unencrypted commu-

nication links. Authentication solutions are often

conﬁgured with poor passwords or even with de-

fault passwords, and SCADA protocols typically lack

authentication and cryptographic message integrity

checks. The reason for this being that the Remote

Terminal Unit (RTU) devices originally did not have

the processing capability to do such operations. Many

SCADA protocols are therefore designed to be open,

robust (from a control perspective), reliable and easy

to operate, and not necessarily designed to provide se-

cure communication.

The IEC 60870-5-104 protocol which is used to

control RTUs in the electrical energy sector has the

following security properties:

• Lack of Conﬁdentiality: All IEC 60870-5-104

messages are transmitted in clear-text across the

network.

• Lack of Integrity: There are no cryptographic in-

tegrity checks built into the protocol.

• Lack of Authentication: There is no authentica-

tion of communicating entities in the protocol.

3 RELATED WORKS

This paper proposes an IDS test framework based

on virtual machines for SCADA systems. It uses a

combination of the Elastic stack as a big-data based

SIEM solution, the SCADA HoneyNet ConPot and

Kali Linux as the attacker. The solution is experimen-

tally veriﬁed in a comparative analysis of the perfor-

mance of Snort and Suricata for SCADA rules focus-

ing on the IEC 60870-5-104 telecontrol protocol for

the electrical energy sector. The solution and com-

parative analysis is, to the best of our knowledge, not

formely described in the scientiﬁc literature.

There are some examples of previous SCADA test

beds, for example the US Idaho National Lab SCADA

test bed (Wei et al., 2010). University of Arizona set

up a test bed called TASCA for analysing the security

of SCADA control systems (Davis et al., 2006). The

latter uses PowerWorld as a grid simulator to simu-

late the effect of cyber-attacks on the power grid. Our

framework currently supports grid simulation based

on the Open Substation HMI (OSHMI)

, and plans

more elaborate grid simulation support in the future.

Queen’s University Belfast has published a pa-

per based on research in the PRECYSE FP7 project

that presents a SCADA-speciﬁc cyber-security test-

bed used for investigating man-in-the-middle at-

tacks (Yang et al., 2012). They built a framework

with three Windows-based hosts to simulate real-time

SCADA master/slave communication that displays

the output on a HMI. The framework also includes

a Linux host to simulate an attacker. Due to conﬁ-

dentiality and security concerns, the speciﬁc software

used in this framework is withheld by the publish-

ers (Yang et al., 2012). This solution does not use a

virtualised infrastructure, which makes it less ﬂexible

for dynamically setting up and changing test scenar-

ios. It furthermore only covers the IDS host, which

runs a proprietary IDS (ITACA), and does not contain

a SIEM solution that can be used for the comparative

analysis as well as for doing research on situational

awareness. Our solution is more ﬂexible by running

as a virtualised infrastructure which uses different vir-

tual machines for different test components.

The same researchers published a paper, present-

ing a set of Snort IDS rules for IEC 60870-5-104

SCADA networks (Y. Yang et al., 2013). They used

a Deep Packet Inspection (DPI) method, which in-

OSHMI https://github.com/focusenergy/oshmi

Intrusion Detection System Test Framework for SCADA Systems

277

cludes signature-based and model-based approaches.

Our experiments use a set of Snort IDS signatures for

SCADA systems from various sources, amongst oth-

ers the model-based IDS ruleset for the IEC 60870-5-

104 protocol from (Y. Yang et al., 2013), as well as

the Modbus and DNP3 IDS rules that come with the

Snort VRT ruleset. These signatures can not only de-

tect several known malicious attacks and suspicious

threats, but also identify the sources of the attacks

which can aid security analysts in potentially prevent-

ing future intrusions from these sources. The Modbus

and DNP3 rules are not described in this paper, since

the results do not add signiﬁcant value compared to

the ﬁndings in the IEC 60870-5-104 rule set.

We reuse this IDS rule set to perform a compara-

tive analysis of how well Suricata and Snort perform

on these rules, something that is not done in the origi-

nal paper. This also acts as a proof of concept experi-

ment for verifying the whole SCADA IDS test frame-

work.

An ENIP (Ethernet/IP) SCADA rule set for Suri-

cata was developed in (Wong et al., 2017). This

research conﬁrms that Suricata is a good choice

for SCADA IDS monitoring on resource constrained

hardware, but does not perform a comparative analy-

sis of other SCADA protocols, including IEC 60870-

5-104.

A quantitative analysis of Snort and Suricata is

performed in (White et al., 2013). They used large

amounts of pcap data and custom scripts to compare

the two IDS engines on ﬂexible and scalable hard-

ware. To conduct a quantitative analysis they cap-

tured a variety of metrics, including packets per sec-

ond, memory usage and CPU utilization. The IDS

rulesets Emerging Threats (ET) Open/ET-Free, ET-

Pro and Snort VRT were used in their experiments.

They conclude that Suricata outperforms Snort, even

on a single core. Suricata also has an average lower

memory and CPU utilization than Snort. This paper

does not do a comparative analysis of SCADA spe-

ciﬁc rulesets to detect compatibility issues as well as

a latency comparison between the IDSs in the Elastic

stack.

There are also some other examples of com-

parative analyses of Snort and Suricata, for exam-

ple (White et al., 2013; Pihelgas, 2012; Brumen and

Legvart, 2016), but none of these consider SCADA

protocols.

Our framework simulates IEC60870-5-104 com-

munication, and can be conﬁgured to simulate Mod-

bus TCP and DNP3 as well. In contrast to other quan-

titative analyses of IDS engine performance, we have

compared the detection capabilities of Snort and Suri-

cata in context of SCADA rule-sets.

4 APPROACH

The SCADA IDS Test Framework provides a con-

trolled environment for penetration testing, real-time

simulation, intrusion detection and visualisation of at-

tacks targeting SCADA networks. It can simulate

real-time IEC 60870-5-104 trafﬁc between SCADA

devices and attacks can be performed by utilizing the

attacker machine in the framework. The framework

also includes a simulated Siemens S7-200 PLC to per-

form attacks against the Modbus protocol. Two sep-

arate intrusion detection systems (IDSs) are individu-

ally looking for malicious SCADA trafﬁc in the con-

trolled environment. An alert is triggered each time

one of the IDS implementations detect malicious traf-

ﬁc. The alert is then forwarded to a centralised se-

curity information and event management (SIEM) so-

lution, responsible for collecting, analysing, indexing

and visualising IDS events.

4.1 Framework Architecture

The framework’s architecture is categorised into four

elements, as illustrated in ﬁgure 4:

• Security information and event management

(SIEM)

• Intrusion detection systems (IDS)

• SCADA target side

• Attacker side

The attacker side is a Kali Linux machine

with over

600 penetration testing tools. The SCADA target

side consist of three machines running the client and

server side of the IEC 60870-5-104 communication,

and one machine simulating a Siemens S7-200 PLC.

The Attacker and SCADA machines are connected to

the same local network. The network trafﬁc travers-

ing the lab network is mirrored and analysed by two

separate IDS implementations: Suricata and Snort.

The SCADA rules for IEC 60870-5-104 and the Dig-

ital Bond Quickdraw SCADA rules for Modbus and

DNP3 are used in both IDS solutions.

Suricata is conﬁgured to log alerts in a structured

Extensible Event (EVE) JSON format. This output

format is not available in Snort. For the basis of

comparison between alerts triggered by Suricata and

Snort, it is advantageous if the output log format is

the same. To achieve this, Snort has been conﬁgured

to log alerts in Uniﬁed2 format. The Python library

py-idstools is then used to convert the Uniﬁed2 for-

mat to EVE JSON (Ish, 2017).

Kali Linux: http://www.kali.org

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

278

The Elastic stack

(formerly known as ELK stack)

is used as a SIEM solution in this framework (Lah-

madi and Beck, 2015). It includes the following ser-

vices:

Logstash is a service that collects, parses and

transforms logs. It is conﬁgured to listen for log data

on TCP port 5044 in the management network. File-

beat is log ﬁle transfer service that is installed on both

IDS implementations and conﬁgured to forward new

entries (IDS alerts) in the log ﬁle to Logstash on port

5044. Logstash then analyses the incoming data and

adds/removes speciﬁc ﬁelds and tags (e.g. GeoIP in-

formation) as will be discussed later. The reﬁned data

is then sent to the search engine (Elasticsearch based

on Apache Lucene) and stored in a cluster of nodes.

Kibana is used as a data analysis framework to ex-

plore the Elasticsearch data. It provides the possibil-

ity to investigate historical and real-time events by uti-

lizing a search engine and use custom dashboards to

visualise the data.

X-pack

is a commercial SIEM solution imple-

mented on top of the open source Elastic stack. This

framework is available with a basic license which

only includes a monitoring feature. It provides real-

time monitoring of the performance of the Elastic-

search cluster and the Kibana work load. Figure 4

gives an overview of the framework architecture.

To provide automatic start up at reboot, scheduled

log rotation and synchronous time settings, the sys-

tem was automated using system daemons, scripts for

automation as well as network segmentation with a

separate management and simulation network to re-

duce the risk that attackers are aware of the security

monitoring.

4.2 Dashboards

Kibana uses a dashboard based approach for data

visualisation and analysis. An advantage with this

framework, is that it allows security analysts to eas-

ily reconﬁgure the user interface according to their

operative needs. For our test lab, this means that dif-

ferent scenarios and visualisations can be tested out

to evaluate the usability, accessibility and situational

awareness of the operators.

In total six dashboards were developed for differ-

ent purposes. The ﬁrst dashboard is a visualisation of

the trafﬁc ﬂow data generated by Suricata. Two sepa-

rate dashboards were created to analyse the alert data

from Suricata and Snort, respectively. A dashboard

was also developed to compare alert data generated

from the two IDS implementations. An experimental

Elastic stack: https://www.elastic.co

X-pack: https://www.elastic.co/x-pack

dashboard was created to visualise relationships be-

tween data. The last dashboard was created to anal-

yse the additional latency added by Logstash process-

ing. The time frame for each dashboard can easily be

changed by adjusting a parameter at the top right. The

time frame can be anything from the past 15 minutes

to the past 5 years.

Network Trafﬁc Monitoring

The network trafﬁc dashboard shown in ﬁgure 5 anal-

yses the network trafﬁc in real-time. The left side uses

the geographical data added by Logstash. The dash-

board displays pie chart visualisations of the top 10

countries and cities generation trafﬁc, and plots the

geographical location into a map. The right side anal-

yses the actual trafﬁc and visualises the destination IP

addresses, source IP addresses, source port, destina-

tion port, underlying protocols and TCP states.

IDS Alert Monitoring

The IDS alert dashboard provides a histogram that

counts the unique events at any given time. The dash-

board also provides a measurement count of alerts

with various severity levels. If the count reaches a

custom set limit, the measurement changes color from

green to yellow or red. The geographical location of

any source address triggering an alert, is plotted in a

map. Pie charts are used to visualise relevant data,

and a table is used to count the unique signatures trig-

gered.

IDS Comparison

The IDS comparison dashboard in ﬁgure 6 is used

to compare Suricata’s and Snort’s ability to detect at-

tacks. Snort is monitored on the right side and Suri-

cata on the left side. The dashboard contains measure-

ment visualisations to count unique signatures and the

total number of alerts triggered by Suricata and Snort,

respectively. The size of the IDS name text in the mid-

dle is automatically adjusted by number of alerts trig-

gered. At the end of the dashboard is a table that lists

every triggered alert and counts every occurrence.

Data Relation Dashboard

The experimental dashboard implemented in this

framework, is a network graph that can be used to vi-

sualise the relations between the data.

Intrusion Detection System Test Framework for SCADA Systems

279

Security information and event management (Security information and event management (

Intrusion detection systems (IDSs)Intrusion detection systems (IDSs)

ELK Stack

Kali Linux

Lab network

Suricata IDS Snort IDS

Mirrored

ELKmonitor

Logstash

Elasticsearch

Node-1

Kibana

Beats (filebeat)

Suricata IDS

Snort IDS

eve.jsonSuricata.yamlRules Unified2 logsnort.confRules

Py-idstools:u2eve

eve.json

Management network

IEC Server

(IEC 60870-5-104

Server)

OSHMI

QTester104 (IEC 60870-5-104

protocol tester)

X-Pack

With Basic License

Architecture

Conpot version

0.5.1

Telnet server

PuTTY (Telnet Client)

Elasticsearch

Node-2

Elasticsearch

Node-3

Distributed

OpenMUC j60870

Figure 4: System Architecture.

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

280

Processing Latency Dashboard

The last dashboard implemented in this framework,

is used to visualise the additional latency added by

Logstash processing. The IDS adds a timestamp ﬁeld

to the eve.json ﬁle when it detects malicious trafﬁc.

Logstash also adds a timestamp ﬁeld when the data is

processed and shipped to Elasticsearch. These times-

tamp ﬁelds can be used to ﬁnd the latency by calculat-

ing the deviation between them. This is done by us-

ing a Kibana functionality called scripted ﬁelds which

performs calculations on the ﬂy by analysing the in-

coming Elasticsearch data. The implemented dash-

board compares latency for Suricata and Snort alerts.

It plots the average latency in a line chart every sec-

ond and calculates max, min, average, median and

standard deviation (upper/lower) on the ﬂy. The dash-

board also visualises the latency in a pie chart.

Figure 5: Network trafﬁc monitoring dashboard.

Figure 6: IDS comparison dashboard.

5 EXPERIMENTS AND RESULTS

This section discusses experiments and results in the

context of validation and testing of the implemented

framework. Five experiments were performed to mea-

sure the latency of the framework and verify that:

• Normal trafﬁc does not trigger any false positives;

• Signature-based rules do not trigger on trafﬁc

from an unauthorised client;

• Protocol-violation rules trigger on IEC 60870-5-

104 protocol violations;

• Trafﬁc pattern based rules do not trigger on illegal

trafﬁc patterns.

5.1 IEC 60870-5-104 client/server

Communication

The main goal of carrying out these experiments, is

to trigger alerts speciﬁed in the implemented IEC

60870-5-104 rule-set (Y. Yang et al., 2013). This is

done by generating realistic IEC 60870-5-104 back-

ground trafﬁc in the lab network and in addition per-

form attacks and Illegal actions. The alerts triggered

by Suricata and Snort are compared by log analysis

and real-time visualisation. The signatures included

in the rule set can be categorised into three categories;

signature-based rules, protocol based rules and trafﬁc-

pattern-based rules.

5.1.1 Normal Communication:

To simulate normal trafﬁc patterns between autho-

rised IEC 60870-5-104 clients and servers, exper-

iments were conducted using the IEC Server soft-

ware on a virtual machine with IP address 10.0.0.75,

listening on port number 2404 (Jaentsch, 2013).

QTester104 is used as an IEC client on a virtual ma-

chine with IP address 10.0.0.66 (Olsen, 2017). Wire-

shark

is used to inspect and validate the IEC 60870-

5-104 trafﬁc traversing the lab network. Since Suri-

cata is conﬁgured to log all data trafﬁc, Kibana can be

used to analyse normal trafﬁc as well. No alerts (false

positives) were triggered either by Suricata or Snort

during this experiment, as expected.

5.1.2 Signature-based Rules:

Several experiments were conducted to simulate ille-

gal SCADA operations to trigger the signature-based

rules, by using tools like QTester104, IEC Server,

OpenMUC j60870 and packet sender. Initially, Suri-

cata did not detect and trigger alerts for two of the

signatures as shown in ﬁgure 7. The compatibility is-

sue was that the regular expression matching in Suri-

cata with the /R (relative) option requires a preceeding

match in the same buffer, but there is no buffer keep-

ing track of previous matches. This was the case for

two of the signatures in this rule set, sid: 6666601 and

sid: 6666602. This issue was mitigated by removing

the “R” from the regular expression.

Wireshark: https://www.wireshark.org

Intrusion Detection System Test Framework for SCADA Systems

281

SCADA protocol IEC 60870-5-104

IDS Engine Suricata Snort

Description

Signature ID

6666601

X (modification) X

Non-IEC/104 Communication on an IEC/104 Port

6666602 X (modification) X

Spontaneous Messages Storm

6666603 X X

Unauthorized Read Command to an IEC/104 Server

6666604

X X

Unauthorized Interrogation Command to an IEC/104

Server

6666605

X X

Unauthorized Counter Interrogation Command to an

IEC/104 Server

6666606

X X

Remote Control or Remote Adjustment Command

from Unauthorized 104 Client

6666607

X X

Set Point Command from an Unauthorized IEC/104

Client

6666608

X X

Reset Process Command from Unauthorized Client

6666609

X X

Broadcast Request from Unauthorized Client

6666610

X X

Potential Butter Overflow

Figure 7: Signature-based rules.

5.1.3 Protocol Violation and Trafﬁc-pattern

based Rules:

To trigger the protocol violation rules, an experiment

was conducted using using the Ettercap plugin de-

veloped by PMaynard for performing a man-in-the-

middle attack (Maynard, 2017). The attack moni-

tors the IEC 60870-5-104 communication between a

client and a server, and injects illegal protocol packets

into the network.

Both Suricata and Snort detected all malicious

trafﬁc injected in the experiment, and triggered alerts

based on the protocol-based rules (SIDs 6666611 -

6666621). All the trafﬁc-pattern based rules (SIDs

6666622 - 6666624) were also triggered for both IDSs

using generated trafﬁc.

5.2 Latency

The two different IDS implementations in the test

framework detect malicious trafﬁc, and use Filebeat

to transmit the log entries to Logstash. Logstash anal-

yses, modiﬁes and transfers the data to a Elasticsearch

cluster. The latency considered here is the time it

takes from malicious trafﬁc is detected by an IDS un-

til the event is indexed and stored in the cluster. This

experiment uses a custom latency dashboard imple-

mented in the framework to analyse the additional la-

tency added by Logstash and conversion-tools. Nor-

mal amounts of IEC 60870-5-104 trafﬁc are sent be-

tween client and server, using the QTester104 and IEC

Server tools (Olsen, 2017)(Jaentsch, 2013).

The ﬁrst experiment lasted ten minutes and trig-

gered approximately 200 alerts for both Suricata and

Snort during normal conditions. The boxplot in ﬁgure

8 is an ensemble of ten latency measurements to get a

representative picture of the variance of the test data.

The ﬁgure shows a comparison of Suricata and Snort

under normal conditions. The thin line describes the

highest and lowest latency measured for both IDS so-

lutions. Outlier values are excluded from the ﬁgure.

The box represents the upper and lower quartile of

the latency and the yellow line in the middle repre-

sent the median. From the ﬁgure it appears that Snort

generally has higher latency a Suricata. The median

value was approximately 2.5 seconds for Suricata and

5.1 seconds for Snort. The upper and lower quartile

of Snort latency is higher with a larger variance than

Suricata. Both solutions have a bit skewed distribu-

tion compared to a normal distribution.

Figure 8: Boxplot comparison of latency in Suricata and

Snort under normal trafﬁc.

6 DISCUSSION

The main advantage of using the Open Source Elastic

stack rather than a fully commercial alternative such

as Splunk for the IDS test framework, is that Elas-

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

282

tic has an affordable licencing policy, and does not

severely restrict the number of dashboards such as

Splunk does in its evaluation mode. These restric-

tions essentially made Elastic a better alternative for

our project. Comparing Splunk and Elastic as SIEM

solutions under similar conditions would be an inter-

esting extension in a larger follow-up study.

Another possibility would be to use Graylog2

which has similar features as Elastic, is Open Source,

and also is based on Elasticsearch for log data min-

ing. Graylog2 excels at postmortem debugging, secu-

rity and activity analysis. Our main reason for choos-

ing Elastic over Graylog2, is that it has more ﬂexible

graphing and dashboard handling. Such functional-

ity would be useful for extending the framework as a

more general situational awareness platform support-

ing both security monitoring and crisis management

in the future. Kibana is user friendly by allowing data

scientists to fairly easily add more functionality to the

system in the form of new dashboards. It is also pos-

sible to add support for more advanced time series

analytics by integrating Grafana

. A limitation with

Kibana is that it lacks advanced user management. It

is however possible to set up simple access control us-

ing HTTP basic authentication. More advanced solu-

tions will require purchasing X-Pack which provides

encryption, authentication and authorisation.

The solution proposed here is based on virtual

machines. This plug-in architecture for IDS engines

makes the framework easily extensible for other IDS

technologies. Other similar architectures have been

suggested based on Docker containers, which are

more lightweight than virtual machines and allows

for enabling independence between cloud applica-

tions and infrastructure

. Our solution will needs to

use at least some virtual machines to be able to test

real operating system instances. Docker could be con-

sidered as an option in the future.

Elasticsearch can be used for building complex

search requests, but the main challenge in future re-

search will be not only conveying the IDS alerts but

also performing more complex data analysis, alert

correlation and data mining to identify the relevant in-

formation for security operators and other stakehold-

ers and reduce the amount of false alarms. Clustering,

behaviour analysis and machine learning techniques

used in anomaly detection would be natural exten-

sions of this research to improve the overall attack

detection capabilities of the system in the future.

Possible missions for the target framework in-

cludes testing the performance in terms of accuracy

Graylog2: https://github.com/Graylog2

Grafana https://grafana.com/

Docker: https://elk-docker.readthedocs.io/

and speed of detection, validating new IDS solutions

or validating different rule sets on IDSs. Another pos-

sibility is to extend the framework as a comprehensive

hybrid SIEM/IDS solution that uses several different

tools such as log analysis, network and host-based in-

trusion detection systems.

Future challenges that can be investigated using

this framework includes research on how to model

new protocols and how to simulate really big data

scenarios where a large cluster of sensors as well as

Elastic shards need to collaborate on the data min-

ing. Autoconﬁguration of the framework is another

challenge which can use techniques such as Network

Function Virtualisation and Software Deﬁned Net-

working. Alert normalisation is to some extent han-

dled by Logstash. Future research and standardisa-

tion is however required to deﬁne a common ontology

that ensures semantic interoperability between differ-

ent types of alerts (Krauß and Thomalla, 2016). The

platform will also act as a research vehicle for visual-

ising and analysing results as to prove speciﬁed sim-

ulation scenarios as well as for improving situational

awareness during such scenarios.

The framework is extensible and scalable. The

IDS side can be extended with additional IDS solu-

tions by adding new virtual machines to the frame-

work that support a given IDS technology. The

Logstash conﬁguration can also be adapted to cate-

gorise different kinds of IDS solutions. Logstash is

horizontally scalable meaning that the performance

can be scaled up by adding more hardware nodes.

It can furthermore form groups of nodes running the

same information pipeline. Adaptive buffering ca-

pabilities in the Elastic stack provides smooth data

streaming even with variable throughput loads. If

Logstash becomes a bottleneck, then more nodes

(cloud instances/virtual machines) can be added.

Elasticsearch is also horizontally scalable by al-

lowing the performance to be scaled up using more

hardware nodes. The nodes in a cluster form a full

mesh topology, which means that each node main-

tains a connection to each of the other nodes. The

cluster has a single master node which is chosen au-

tomatically by the cluster and which can be replaced

if the current master node fails. An index is a log-

ical namespace which points to primary and replica

shards, which are instances managed automatically

by Elasticsearch

. Each document is stored in a sin-

gle primary shard. When a document is being in-

dexed, it is indexed ﬁrst on the primary shard, then

on all replicas of the primary shard. A replica is a

copy of the primary shard, used to increase failover

and performance. The number of primary and replica

https://www.elastic.co

Intrusion Detection System Test Framework for SCADA Systems

283

shards can be manually conﬁgured to optimise imple-

mentation.

The case study used the implemented framework

to carry out several experiments. The experiments

demonstrate which attacks and types of communica-

tion that trigger different signatures in the rules for

IEC 60870-5-104. The experiments were performed

manually, which is an expensive and more error-prone

approach than automatic testing. Automating the test

setup using OpenMUC with automated testsuites for

example based on JUnit is therefore a natural next

step in improving the test framework.

The experiments aim at demonstrating IEC

60870-5-104 trafﬁc that triggers alerts using IDS

rulesets for this protocol. Normal trafﬁc did not

trigger any of the signatures speciﬁed in the IEC

60870-5-104 rule-set. Several methods and tools

were used to trigger the signature-based rules. The

protocol-based rules and trafﬁcpattern-based rules

were demonstrated using a man-in-the-middle packet

injection and demonstration unauthorised trafﬁc. This

case study detected some compatibility issues be-

tween the two IDSs that relatively easily could be mit-

igated.

The experiments also demonstrated a SYN ﬂood

denial-of-service attack and analysis of the addi-

tional latency applied by the Logstash processing and

conversion-tools. During some of the experiments,

Suricata triggered a larger amount of alerts than Snort.

This is because the default alert threshold settings are

different for Suricata and Snort, respectively. The

purpose of these is to avoid that the security oper-

ations centre becomes swamped by too many alerts

if the same rule triggers too many times. These val-

ues can be manually changed by editing the threshold

conﬁguration ﬁle for both solutions.

The platform should in the future be extended to

use the OpenMUC j60870 library to simulate all the

IEC 60870- 5-104 communication. This library al-

lows simulating all IEC message types on both the

client and server side. Both QTester104 and IEC

server have their limitations because they not are com-

plete implementations of the IEC 60870-5-104 proto-

col.

7 CONCLUSION

This paper proposes a SCADA Intrusion Detection

System Test Framework that can be used to conduct

research on security in SCADA communication and

validate existing IDS signatures before implementing

them in a production network. The framework con-

sists of four parts; attacker side, SCADA target side,

IDS side and SIEM side. It simulates real-time IEC

60870-5-104 communication between a client and a

server. Both Suricata and Snort IDS are included in

this framework to analyse the network trafﬁc and de-

tect malicious activity. The solutions are compared

in a comparison dashboard that was created for the

framework.

There is generally little difference between Suri-

cata and Snort’s ability to detect malicious trafﬁc.

Suricata is mostly compatible with signatures writ-

ten in Snort’s lightweight rules description language.

We discovered some compatibility issues that can be

mitigated by modifying the ruleset slightly. All IEC

60870-5-104 signatures were triggered for both IDSs

after these modiﬁcations.

The perceived latency is generally higher for Snort

events than for Suricata events. The reason for this is

probably the additional processing time applied by the

uniﬁed2 to eve.json conversion.

8 FUTURE WORK

As future work, we aim at expanding the SCADA In-

trusion Detection System Test Framework. The natu-

ral next step would be to implement other rule-sets,

create custom signatures, use anomaly based intru-

sion detection features and experience with other IDS

packages, like Bro or commercial IDS solutions.

The HMI software that is implemented in the

framework is currently only used for simulation pur-

poses. A possible extension of this implementation,

could be to interconnect the HMI software with the

IEC 60870-5-104 client. It would then be possible

to see and simulate the effect of an attack in the

HMI. Another possible extension is to implement host

based IDS solutions (HIDS) in the framework to anal-

yse machine level activity, detect abnormal login pat-

terns and ﬁle changes. New entries in the HIDS and

SCADA logs could be shipped to Logstash and be

stored in the cluster. This extension would provide the

network administrators with more information that

can be used to detect malicious activity in the SCADA

system.

Another idea is to integrate Apache Hadoop and

connect it to the cluster using the ES-Hadoop con-

nector. This would connect the massive data storage

and deep processing power of Hadoop with the real-

time search and analytics of Elasticsearch. It would

also be interesting to use the machine learning fea-

ture included in X-pack to predict attacks based on

past events. Another possibility is to implement the

IEC 62351 security enhancement of IEC 60870-5-104

protocol in the framework. Finally, future research in-

ICISSP 2018 - 4th International Conference on Information Systems Security and Privacy

284

cludes to perform usability testing and experiments to

determine the level of situational awareness for oper-

ators using the framework.

ACKNOWLEDGEMENTS

This research has been supported by NC Spectrum

AS and CIEM, the Centre for Integrated Emergency

Management at University of Agder, Norway. Thanks

to the anonymous reviewers for good suggestions on

how to improve the paper.

REFERENCES

Brumen, B. and Legvart, J. (2016). Performance analysis

of two open source intrusion detection systems. In

2016 39th International Convention on Information

and Communication Technology, Electronics and Mi-

croelectronics (MIPRO), pages 1387–1392.

Davis, C. M., Tate, J. E., Okhravi, H., Grier, C., Overbye,

T. J., and Nicol, D. (2006). SCADA Cyber Security

Testbed Development. In 2006 38th North American

Power Symposium, pages 483–488.

Graham, J., Hieb, J., and Naber, J. (2016). Improving cyber-

security for Industrial Control Systems. In 2016 IEEE

25th International Symposium on Industrial Electron-

ics (ISIE), pages 618–623.

International Electrotechnical Commission (2006). Interna-

tional Standard IEC 60870-5-104: Telecontrol Equip-

ment and Systems — Part 5-104: Transmission Pro-

tocols — Network Access for IEC 60870-5-101 using

Standard Transport Proﬁles. IEC, Geneva, Switzer-

land. Second Edition (Reference number IEC 60870-

5-104:2006(E)).

Ish, J. (accessed 2017). py-idstools: Snort and

Suricata Rule and Event Utilities in Python.

https://github.com/jasonish/py-idstools.

Jaentsch, K. (2013). IEC Server Manual. http://area-

x1.lima-city.de/.

Krauß, D. and Thomalla, C. (2016). Ontology-based detec-

tion of cyber-attacks to scada-systems in critical in-

frastructures. In 2016 Sixth International Conference

on Digital Information and Communication Technol-

ogy and its Applications (DICTAP), pages 70–73.

Lahmadi, A. and Beck, F. (2015). Powering Monitoring

Analytics with ELK stack. In 9th International Con-

ference on Autonomous Infrastructure, Management

and Security (AIMS 2015).

Maynard, P. (accessed 2017). Ettercap-104-mitm: Plugin

for IEC 60870-5-104.

Olsen, R. L. (accessed 2017). QTester104

iec 60870-5-104 protocol tester.

https://sourceforge.net/projects/qtester104/.

Pihelgas, M. (2012). A Comparative Analysis of Open-

Source Intrusion Detection Systems. PhD thesis,

Tallinn University of Technology & University of

Tartu.

Wei, D., Lu, Y., Jafari, M., Skare, P., and Rohde, K. (2010).

An integrated security system of protecting smart grid

against cyber attacks. In Innovative Smart Grid Tech-

nologies (ISGT), 2010, pages 1–7. IEEE.

White, J. S., Fitzsimmons, T., and Matthews, J. N. (2013).

Quantitative analysis of intrusion detection systems:

Snort and suricata. SPIE Defense, Security, and Sens-

ing, pages 875704–875704.

Wong, K., Dillabaugh, C., Seddigh, N., and Nandy, B.

(2017). Enhancing Suricata intrusion detection sys-

tem for cyber security in SCADA networks. In 2017

IEEE 30th Canadian Conference on Electrical and

Computer Engineering (CCECE), pages 1–5.

Y. Yang, K. McLaughlin, B. Pranggono, T. Littler, S. Sezer,

and H. F. Wang (2013). Intrusion Detection System

for IEC 60870-5-104 Based SCADA Networks. Tech-

nical report, Queen’s University Belfast and Brunel

University.

Yang, Y., McLaughlin, K., Littler, T., Sezer, S., Im,

E. G., Yao, Z. Q., Pranggono, B., and Wang, H. F.

(2012). Man-in-the-middle attack test-bed investigat-

ing cyber-security vulnerabilities in smart grid scada

systems. In International Conference on Sustainable

Power Generation and Supply (SUPERGEN 2012),

pages 1–8.

Intrusion Detection System Test Framework for SCADA Systems

285