An Approach to Train and Evaluate the Cybersecurity Skills of

Participants in Cyber Ranges based on Cyber-Risk Models

Gencer Erdogan

, Åsmund Hugo

, Antonio Álvarez Romero

Dario Varano

, Niccolò Zazzeri

and Anže Žitnik

Software and Service Innovation, SINTEF Digital, Oslo, Norway

Research & Innovation, Atos, Seville, Spain

Department of Information Engineering, University of Pisa, Pisa, Italy

Trust-IT Services, Pisa, Italy

XLAB, Ljubljana, Slovenia

n.zazzeri@trust-itservices.com, anze.zitnik@xlab.si

Keywords: Cyber Range, Cybersecurity, Cyber-risk Models, Training Scenario, Exercise, Evaluation, Real-time, White

Team, Green Team, Blue Team, Red Team.

Abstract: There is an urgent need for highly skilled cybersecurity professionals, and at the same time there is an

awareness gap and lack of integrated training modules on cybersecurity related aspects on all school levels.

In order to address this need and bridge the awareness gap, we propose a method to train and evaluate the

cybersecurity skills of participants in cyber ranges based on cyber-risk models. Our method consists of five

steps: create cyber-risk model, identify risk treatments, setup training scenario, run training scenario, and

evaluate the performance of participants. The target users of our method are the White Team and Green Team

who typically design and execute training scenarios in cyber ranges. The output of our method, however, is

an evaluation report for the Blue Team and Red Team participants being trained in the cyber range. We have

applied our method in three large scale pilots from academia, transport, and energy. Our initial results indicate

that the method is easy to use and comprehensible for training scenario developers (White/Green Team),

develops cyber-risk models that facilitate real-time evaluation of participants in training scenarios, and

produces useful feedback to the participants (Blue/Red Team) in terms of strengths and weaknesses regarding

cybersecurity skills.

1 INTRODUCTION

There is an urgent need for highly skilled, multi-

disciplined cybersecurity professionals, given the

increasingly aggressive cyber-landscape public and

private organizations are facing. As pointed out by the

European Cyber Security Organization (ECSO), at

education level, there is a big awareness gap and lack

of integrated training modules on cybersecurity

related aspects on all school levels, starting from low

awareness and skills of teachers themselves. The

same is true for professional training on university

level, including lack of cybersecurity modules in

higher education training programs for vital service

domains etc. In addition, there are only few existing

cybersecurity higher education programs in Europe.

Moreover, it is reported that at professional level,

there is a lack of accessible tools for continuous

awareness, training and skills development on

cybersecurity aspects (ECSO, 2016).

In order to address the need of continuous

cybersecurity awareness, training and skills

development, we have developed a method to train

and evaluate the cybersecurity skills of participants in

cyber ranges based on cyber-risk models. Our method

assumes that the technical capabilities to simulate

infrastructure on which exercises are executed

already exist and that necessary instructions and

training for the cyber range has already been given to

the participants.

Thus, the contribution of this paper is a method

for creating cyber-risk models that facilitate the

training and evaluation of cybersecurity skills of

participants in cyber-ranges. The method is described

using an example in the context of

CYBERWISER.eu, which is a web-based cyber

Erdogan, G., Hugo, Å., Romero, A., Varano, D., Zazzeri, N. and Žitnik, A.

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models.

DOI: 10.5220/0009892105090520

In Proceedings of the 15th International Conference on Software Technologies (ICSOFT 2020), pages 509-520

ISBN: 978-989-758-443-5

509

range platform we have developed as part of the

international EU project with the same name

(CYBERWISER.eu, 2020a).

Cyber ranges often have different groups of

people referred to as "teams" who have different

roles. In this paper, we consider the Green, White,

Red, and Blue teams (Damodaran & Smith, 2015).

The Green Team consists of individuals who operate

the range infrastructure and support tool systems. In

collaboration with the White Team, the Green Team

manages on-demand definition of training scenarios.

The White Team represents the instructor/s of the

training, whether course based or as an exercise. The

White Team collaborates with the Green Team to

deploy and configure training scenarios. The White

Team also evaluates the participants' progress. The

Red Team carries out cyberattacks against the

infrastructure simulated on the cyber range as part of

a training scenario. The Blue Team detects and

responds to the attacks performed by the Red Team

and/or automatically by the tools in the cyber range.

To address the abovementioned needs and

develop artefacts that appropriately meet these needs,

we define the following success criteria.

Success Criterion 1: The method must be easy to

use and comprehensible for cyber-range training

scenario (exercise) developers.

The main target user group of our method is people

who design and develop cyber-range training

scenarios/exercises. That is, the Green Team and the

White Team. The method must therefore be easy to use

and comprehensible for the intended target audience.

Success Criterion 2: The method must provide

necessary guidelines to create cyber-risk models that

facilitate real-time evaluation of participants.

Cyber range training scenarios are dynamic in

nature and the participants being trained need to make

decisions and take actions on-the-fly. Thus, to

correctly evaluate the cybersecurity skills of the

participants, we need to evaluate their decisions and

actions in real-time while the exercise is running.

Success Criterion 3: The method must produce

useful feedback to the participants in terms of

exercise evaluations.

For the participants to learn from their decisions

and actions taken in an exercise in the cyber range,

they need to receive feedback explaining to what

extent they have successfully carried out the exercise.

Thus, we need to provide useful feedback to the

participants evaluating their achievements.

In Section 2, we describe our research method. In

Section 3, we describe the architecture of our cyber

range platform, before we explain our method for

training and evaluation in Section 4. In Section 5, we

describe related work. In Section 6, we discuss the

extent to which we have fulfilled our success criteria

described above, before concluding in Section 7.

2 RESEARCH METHOD

Figure 1 illustrates the three steps of our research

method, which is in line with the design science

approach by Wieringa (2014). Although the steps are

illustrated sequentially, the method was carried out

iteratively where some of the steps were revisited

during the process.

Figure 1: Research method.

In Step 1, we identified three success criteria

which act as requirements for our method for training

and evaluation based on the background and needs as

explained in Section 1.

In Step 2, we developed our method for training

and evaluation. The method consists of five main

steps: create cyber-risk model, identify risk

treatments, setup training scenario, run training

scenario, and finally evaluate the performance of

participant. All steps are supported by tools that

collectively comprise our cyber range training

platform. The cyber range training platform will be

explained in more detail in Section 3.

In Step 3, we evaluated our method for training

and evaluation in real-world pilot studies to assess the

feasibility of our approach w.r.t. our success criteria.

3 CYBER RANGE

ARCHITECTURE

Before we describe the steps of our method for

training and evaluation, it is necessary to explain the

overall architecture of our cyber range platform in

which the training and evaluation method is applied.

As illustrated in Figure 2, the platform consists of

components that may be grouped into four parts:

simulated infrastructure, scenario environment, cyber

range, and user interface. The Simulated

Infrastructure represents the first layer of the

platform. This layer consists of virtual machines and

virtual networks that simulate an organization's ICT

system. Depending on the objective of a training

ICSOFT 2020 - 15th International Conference on Software Technologies

510

scenario, the participant being trained will either

carry out attacks on the simulated ICT system

automatically using the Attack Simulator, or mitigate

an ongoing attack using the Countermeasure

Simulator. Manual attacks/mitigations are also

possible. The participants in a Blue or Red Team may

also run scans to check for potential vulnerabilities

using the Vulnerability Assessment Tools.

The Monitoring Sensors are software programs

implemented in the simulated ICT system to monitor

host activity or network activity. The host activity

sensors detect potential threats, while the network

activity sensors analyse network traffic and detect and

prevent network intrusion. These sensors send events

to the Anomaly Detection Reasoner and the

Economic Risk Evaluator.

The second layer of the cyber range platform is

the Scenario Environment which is implemented as

an IaaS and contains the components necessary for

the training scenarios. At the centre of the Scenario

Environment, we find the Economic Risk Evaluator,

which uses Economic Risk Models to produce real-

time risk assessment in terms of monetary loss based

on the observed behaviour of the participants in the

training scenario. The real-time feature builds on

continuous observation of the dynamic behaviour of

the training scenarios with the help of the Monitoring

Sensors, Anomaly Detection Reasoner, and the

Vulnerability Assessment Tools which all produce

input to the Economic Risk Evaluator.

To create a realistic experience for the

participants, the Attack Simulator can be configured

to automatically launch pre-defined attacks towards a

specific target. The Performance Evaluator is notified

about the success of an automated attack which

indicates whether the participants, acting as

defenders, were able to prevent the attack. Similarly,

the Countermeasure Simulator can execute mitigation

measures that prevent further attacks automatically

after a certain event has occurred.

The Performance Evaluator component takes as

input the risk assessment produced by the Economic

Risk Evaluator, as well as actions taken by the

participants, and based on these inputs produces an

evaluation report. This evaluation report is forwarded

to the Centralized Logging Component where it is

archived and made available to the participant.

The third layer of the cyber range platform is the

Cyber Range, which includes the components Digital

Library, Training Manager, and Simulated

Infrastructure Manager, as well as the Scenario

Environment and the Simulated Infrastructure. The

Training Manager provides user interfaces for easy

design and configuration of training scenarios, their

creation, deployment, as well as un-deployment and

resource removal after a completed training session.

The Digital Library is a repository storing all the

virtual machine images and additional software,

required for building the training scenarios, as well as

the designed scenarios themselves. The Simulated

Infrastructure Manager acts as an interface to the

underlying IaaS to control the virtual machines and

networks of the Scenario Environments based on

instructions from the Training Manager.

Additionally, Simulated Infrastructure Manager

provides the participants access to the virtual

Figure 2: Cyber range architecture.

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models

511

machines in the Scenario Environment by exposing a

VNC interface, simply available through the user’s

web browser.

The fourth and final layer of the cyber range

platform is the User Interface consisting of a Web

Portal and the Cross-learning Facilities. The

CYBERWISER.eu Web Portal is the single entry-

point to the platform and its services for all end-users

(white, green, blue, red teams, etc.). The

authentication process is provided by the Cross-

Learning Facilities component through a Single Sign-

On (SSO) service. The Cross-Learning Facilities

component provides, among others, training

materials in terms of literature, courses,

communication tools such a chat service, dashboards

for the users and a link to the Cyber-Range Service.

Scores, achieved by participants in the Cyber Range

training exercises, are transferred to the Cross-

Learning Facilities and can be viewed here as well.

4 METHOD FOR TRAINING AND

EVALUATION

Figure 3 illustrates our method for training and

evaluating cybersecurity skills of participants in

cyber ranges based on cyber-risk models. The

following sections explain each of the steps.

Figure 3: Method for training and evaluation.

4.1 Step 1: Create Cyber-Risk Model

As indicated in Figure 3, the first step of our method

expects as input a description of a training scenario,

which is basically a description of an exercise to be

carried out on the cyber range platform with the

purpose of training cybersecurity skills. This

description is expected to be provided by

cybersecurity experts. For example, someone who

has the role as Chief Security Information Officer and

who is interested in training their cybersecurity staff.

Examples of roles to be trained are Vulnerability

Analysts or Threat Analysts (CIISec, 2019). As part

of our cyber range platform, we do also provide

guidelines for how to describe training scenarios.

However, these guidelines are out of the scope of this

paper. The reader is referred to (CYBERWISER.eu,

2019b) for a detailed explanation on how to describe

training scenarios.

In the following, we consider a training scenario

example developed as part of applying the method in a

real-world case pilot as part of the CYBERWISER.eu

project. The training scenario we consider describes an

exercise to train technical security staff in mitigating an

SQL injection attack. That is, the exercise concerns

defensive training where the technical security staff

needs to mitigate an SQL injection attack their ICT

infrastructure, which is simulated on the cyber range

platform, is exposed to. From a cyber range training

perspective, the team trained in defensive exercises are

typically referred to as the Blue Team.

Based on the above training scenario description

we follow an approach described in earlier work

(Erdogan, Gonzalez, Refsdal, & Seehusen, 2017) to

first create a graphical cyber-risk model using

CORAS (Lund, Solhaug, & Stølen, 2011) in order to

capture the SQL injection attack pattern, and then

schematically develop a corresponding machine-

readable risk assessment algorithm with respect to the

graphical risk model. The CORAS risk model and the

risk assessment algorithm are the output of Step 1.

Figure 4 illustrates the CORAS risk model created

for the SQL injection training scenario example

described above. When creating risk models, we

make use of existing libraries and catalogues such as

CAPEC (CAPEC, 2020), OWASP (OWASP, 2020)

and CWE (CWE, 2020) in order to create risk models

that are in line with standard attack patterns.

The CORAS risk model in Figure 4 illustrates that

a threat Hacker initiates the treat scenario S1: Initiate

SQL Injection. Moreover, the hacker may exploit the

vulnerabilities CWE-89: Improper neutralization of

special elements used in an SQL command and CWE-

390: Detection of SQL-related error conditions

without action which leads to the unwanted incident

U1: SQL injection successful. Finally, we see that the

unwanted incident has an impact on the security

assets A1: Confidentiality and A2: Integrity, which

are the assets we want to protect.

In addition to threats, threat scenarios, vulnerabi-

lities, unwanted incidents, and security assets, we use

ICSOFT 2020 - 15th International Conference on Software Technologies

512

CORAS risk models to capture the risk assessment

values likelihood, conditional likelihood and

consequence. In a standard CORAS risk model, these

values are written directly in the model. However, in

our approach we parameterize these values in the

model to later develop the corresponding risk

assessment algorithms. Considering our example in

Figure 4, the likelihood of S1 is represented by l_S1,

the conditional likelihood going from S1 to U1 is

represented by cl_S1_to_U1, the likelihood of U1 is

represented by l_U1, the consequence of U1 on A1 is

represented by c_U1_A1, and the consequence of U1

on A2 is represented by c_U1_A2.

Finally, to capture the dynamic behaviour of the

SQL injection attack that the simulated infrastructure

is exposed to during the training scenario, we include

what we refer to as indicators in the risk models. By

indicator we mean a piece of information that is

relevant for assessing the risk level. The risk level in

our approach is represented as monetary loss. We

distinguish between the following four kinds of

indicators.

 Business configuration (IN-32, IN-C81C, and

IN-C81I): Indicator values are obtained by

asking business related questions. The

indicator values are thus based on the

knowledge of the participant.

 Test results (IN-37): Indicator values are

obtained by carrying out tests, such as

vulnerability scans or automated attacks. The

indicator values are thus based on test results.

Test results and business configuration

indicators are non-intrusive in the sense that

they do not require the implementation of

sensors in the simulated infrastructure.

 Network-layer monitoring: Indicator values are

obtained by monitoring the network layer. This

Figure 4: CORAS risk model with indicators.

type of indicator is intrusive in the sense that

sensors need to be deployed in the network

layer of the simulated infrastructure under

analysis.

 Application-layer monitoring (IN-44 and IN-

56): Indicator values are obtained by

monitoring the application layer. This type of

indicator is also intrusive; a sensor needs to be

installed in the machine under analysis.

Note that Figure 4 does not illustrate any network

monitoring indicator as they were not relevant for this

example.

Having created the graphical risk model, next we

schematically translate the model into an executable

risk assessment algorithm in terms of an R script (R-

project, 2020). Figure 5 illustrates an excerpt of the R

script created based on the risk model in Figure 4. The

excerpt illustrates that we represent the risk model as

a Bayesian Network in order to do calculations using

the likelihood and consequence values captured in the

risk model. The likelihood and consequence values

are in turn calculated with respect to their respective

indicators. Due to space restrictions, it is

Figure 5: Excerpt of R script.

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models

513

not possible to show the complete R script. However,

the logic for computing likelihood and consequence

values are as follows:

 The indicators used to assess likelihood values

(IN-32, IN-37, IN-44, and IN-56) are

formulated either as Yes/No questions or as

quantitative questions. Based on the answer,

we increase or decrease the likelihood value of

the risk illustrated in Figure 4. For example,

consider that the answer to indicators IN-32

and IN-44 are "Yes", and that the answer to

indicator IN-56 is "25 in the last 10 minutes",

then we would increase the likelihood l_S1 to

Very High (in a scale of {Very Low, Low,

Medium, High, Very High}).

 The indicators used to assess consequence

values (IN-C81C and IN-C81I) basically asks

the participant what the consequence of an

unwanted incident is, given that the unwanted

incident materializes.

For the complete R script, the reader is referred to

a technical report in which the risk model in Figure 4

as well as other risk models tried out in context of

real-world pilots are explained in detail

(CYBERWISER.eu, 2020b).

4.2 Step 2: Identify Risk Treatments

In Step 2, we base ourselves on the risk model created

in Step 1 to identify risk countermeasures. We

identify countermeasures using CORAS treatment

diagrams which represent strategies and action plans

the implementation of which reduces risks to an

acceptable level (Lund et al., 2011).

Figure 6 illustrates the same risk model as in

Figure 4. However, in Figure 6, we have removed all

indicators and identified several risk countermeasures

for the vulnerabilities CWE-89 and CWE-390. In

total, we identified 10 countermeasures for

vulnerability CWE-89, and 2 countermeasures for

vulnerability CWE-390, but due to space restrictions

we illustrate 5 of 12 countermeasures in the treatment

diagram in Figure 6. The main source from which we

identified the countermeasures were the webpages

documenting CWE-89 and CWE-390 (CWE, 2020).

Having created CORAS treatment diagrams, next

we describe each countermeasure in detail using a table

template including a unique ID of the countermeasure

(for example, M8 illustrated in Figure 6), the name of

the countermeasure (for example, Validate input), a

detailed description of the countermeasure, and source

of the countermeasures (for example, URL to specific

countermeasures in CWE). An excerpt of the detailed

description of countermeasure M8, according to CWE-

89 (CWE, 2020), is: "When performing input

validation, consider all potentially relevant properties,

including length, type of input, the full range of

acceptable values, missing or extra inputs, syntax,

consistency across related fields, and conformance to

business rules. As an example of business rule logic,

'boat' may be syntactically valid because it only

contains alphanumeric characters, but it is not valid if

the input is only expected to contain colours such as

'red' or 'blue'."

Thus, the output of Step 2 is a CORAS treatment

diagram with countermeasures and an accompanying

table, according to the template described above,

describing the countermeasures in detail. These out-

puts are used in the cyber training scenarios to provide

the participants a set of countermeasure options.

4.3 Step 3: Setup Training Scenario

In Step 3, we setup the training scenario in the cyber

range platform. All the components in the cyber range

platform (described in Section 3) play a role in setting

Figure 6: CORAS treatment diagram.

ICSOFT 2020 - 15th International Conference on Software Technologies

514

up and executing a training scenario. However, our

method considers mainly the components that are

relevant for training and evaluating based on risk

models. When describing the setup and execution of

training scenarios, we therefore mainly discuss the

components Economic Risk Evaluator, Counter-

measure Simulator, and Performance Evaluator, and

assume that all other components are setup and work

properly.

As illustrated in Figure 7, setting up the training

scenario depends on the risk models. As illustrated in

the top-left part of Figure 7, we first develop risk

models with indicators and then schematically

translate a risk model with indicators to an R script

(that is, Step 1 explained in Section 4.1). Next, as

illustrated on bottom-left part of Figure 7, we develop

treatment diagrams (risk models with

countermeasures) and then describe all identified

countermeasures in detail using a table-based

template (that is, Step 2 explained in Section 4.2).

The R scripts are implemented in the Economic

Risk Evaluator, which has the responsibility of

executing the scripts. This includes retrieving values

from the components Monitoring Sensors, Anomaly

Detection Reasoner, and Vulnerability Assessment

Tools and assign the values to their corresponding

indicators in the risk model. Recall that we have

different types of indicators as described in Section

4.1. The component Vulnerability Assessment Tools

provide values to test result indicators, the component

Monitoring Sensors and Anomaly Detection

Reasoner collectively provide values to network layer

monitoring indicators and application layer

monitoring indicators.

Each countermeasure identified and described as

part of Step 2 is implemented in the Countermeasure

Simulator. The Countermeasure Simulator is a

component that has a register of a set of possible

countermeasures for each risk model considered in a

training scenario. Basically, the Countermeasure

Simulator is responsible of making a set of relevant

countermeasures available to the participants during

the training scenario.

4.4 Step 4: Run Training Scenario

In Step 4, we execute the training scenario with

respect to the training scenario configuration carried

out in Step 3.

The execution of a training scenario depends on

the use case of the training scenario. There are four

kinds of use cases: (A) Blue Team vs. Red Team

where the purpose is for the Blue Team to protect the

simulated ICT system from cyber-risk attacks

performed by the Red Team, (B) Red Team vs. Blue

Team where the purpose is for the Red Team to attack

the simulated ICT system protected by the Blue

Team, (C) Blue Team vs. Cyber Range Platform

where the purpose is for the Blue Team to protect the

simulated ICT system from cyber-risk attacks

automatically carried out by the platform, and finally

(D) Red Team vs. Cyber Range Platform where the

purpose is for the Red Team to attack the simulated

ICT system protected by the platform.

Figure 7: Setting up training scenario in the cyber range.

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models

515

Considering our example described in Section

4.1, the purpose is for the Blue Team to protect the

simulated ICT system from SQL injections. In this

example, we do not consider a team versus another

team. Thus, we are considering use case (C) Blue

Team vs. Cyber Range Platform. The automatic SQL

injections are initiated by the person having the role

as tutor/teacher using the component Attack

Simulator in the platform. The person having the

teacher/tutor role is typically referred to as a member

of the White Team in context of cyber ranges.

Assuming members of the Blue Team and White

Team are logged in the cyber range platform, the

training scenario will commence as follows. First, the

White Team initiates SQL injections using the

component Attack Simulator. The Blue Team

monitors the status of the simulated ICT system via

the graphical user interface of the Economic Risk

Evaluator. Then, if the Blue Team detects that the risk

level for SQL injection is going up, they need to

investigate how and where the attack is carried out

(on the simulated ICT system) and select appropriate

countermeasures from the Countermeasure Simulator

and apply them. Having applied the countermeasures,

the Blue Team continuously checks the risk level to

see if the level goes down or up. The important task

here is to understand what kind of cyber-risk attack

the system is exposed to and based on that select

appropriate countermeasures. After a predefined time

(running time of the exercise), the White Team stops

the exercise. All actions taken by the Blue Team is

logged. After the exercise has been stopped, the risk

assessment and the countermeasures selected by the

Blue Team are fed into the Performance Evaluator as

illustrated in Figure 7.

4.5 Step 5: Evaluate the Performance

of Participants

Based on the training scenario results obtained by

carrying out the exercise in Step 4, the Performance

Evaluator calculates automatically in Step 5 how well

the participants have carried out the training scenario.

The Performance Evaluator considers the current

status of the exercise, but also its history, that is, how

the exercise has evolved.

The evaluation is carried out with respect to

predefined assessment algorithms implemented in the

Performance Evaluator, which vary depending on the

exercise. Each performance evaluation algorithm

defined and implemented in the Performance Evalua-

tor expects a series of inputs that must be obtained from

the training scenario exercise. The values of those

inputs are assigned by analysing and processing the

raw logs the exercise generates (output of Step 4).

In the following the different types of information

leveraged to produce the performance evaluation

reporting for the participants are listed, along with

some examples:

 Flags. Flags mark relevant moments in the

exercise, mainly related to achievements of the

participant. For example, the execution of a

Denial of Service attack by a participant who is

part of the Red Team or the blocking of the

TCP for a suspicious IP address by a participant

who is part of the Blue Team. The Attack

Simulator and the Countermeasure Simulator

are the main components producing this

information (flags).

 Elaborated logs coming from monitoring tools

deployed in the monitored infrastructure. Such

logs produce information about, for example,

an attacker performing a dictionary attack to

achieve brute-force login into a certain

machine.

 Results of vulnerability scans executed against

a certain infrastructure element. The

vulnerability scans (part of the Vulnerability

Assessment Tools) are useful for the Red Team

to find weaknesses that can be exploited. It can

also be used by a Blue Team to identify

vulnerabilities that need to be mitigated. An

example of a vulnerability may be the improper

neutralization of special characters in an SQL

query which may eventually be exploited with

an SQL injection attack.

 Cyber risk exposure evolution: the amount of

money being exposed and its evolution over

time is relevant for the evaluation. Pairs (cyber

risk, timestamp) are fed to the Performance

Evaluator from the Economic Risk Evaluator to

describe the corresponding trajectory. Using

configured thresholds, the Performance

Evaluator will analyse this trajectory to

evaluate the celerity and effectiveness the

participant showed to react to the attack

(selected countermeasures by the participant).

 Questionnaire: In addition to the above

information, the Performance Evaluator takes

input coming from a questionnaire filled out by

the White Team evaluating the participants.

The predefined evaluation algorithm in the

Performance Evaluator is configured to give

different scorings depending on the answer

given by the White Team.

In the case of our SQL injection exercise example,

the participant playing the role of defender is

presented with different alternatives in terms of

countermeasures, thanks to the Countermeasure

ICSOFT 2020 - 15th International Conference on Software Technologies

516

Simulator. The countermeasures are presented to the

participant in terms of preconfigured scripts.

However, some of the countermeasures are more

useful than others (expressed as ratings in each

countermeasure), and each countermeasure comes

with a cost expressed in monetary value.

To make the exercise represent a real-world

situation, a limited budget is allocated for the

participant to select ("buy") countermeasures and

apply them on the underlying cyber risk attack.

When choosing a countermeasure, the

corresponding script is executed, and this event is

registered with a timestamp and sent to the

Performance Evaluator. The Economic Risk Evaluator

sends the evolution of the cyber risk exposure which is

also used as input of the performance evaluation

algorithm. Recall that, depending on the selected

countermeasure, the risk level may vary.

Based on the information produced by the

participant during a training scenario as pointed out

above, the Performance Evaluator will be able to

evaluate whether:

 The Blue Team correctly identified the cyber-

attack that the simulated ICT system is exposed

to.

 The Blue Team selected correct and

appropriate countermeasures.

 The selected countermeasures were the most

cost-efficient.

 The Blue Team mitigated the cyber-risk attack

within the expected time of the training

scenario.

In addition to the above, the White Team has the

possibility to provide their own assessment based on

their expert knowledge by filling out the

abovementioned questionnaire. The questionnaire

provides the assessment carried out by the White

Team in a structured manner to the Performance

Evaluator.

As indicated in Figure 3, the main output of the

Performance Evaluator is a report which provides a

grading and evaluates the performance of the

participants. To provide a richer feedback, this grade

is broken into chapters which provide the participants

a more detailed assessment explaining aspects the

participants showed more strength and where the

weak points are.

5 RELATED WORK

Cyber ranges have traditionally been developed and

used by military institutions for cybersecurity training

in the context homeland defence strategy

(Damodaran & Smith, 2015; Davis & Magrath, 2013;

Ferguson, Tall, & Olsen, 2014). However, over the

years, there have been proposed various cyber range

solutions to bring cybersecurity training to both

public and private organizations (Yamin, Katt, &

Gkioulos, 2020).

Secure Eggs (Essentials and Global Guidance for

Security) by NRI Secure (NRISecure, 2020), enPiT-

Security (SecCap) (EnpitSecurity, 2020), and CYber

Defense Exercise with Recurrence (CYDER) are

approaches and security training programs focusing

on basic cybersecurity hands on and awareness

training (Beuran, Chinen, Tan, & Shinoda, 2016).

There are various approaches focusing on

cybersecurity skills training within specific domains

such as smart grid (Ashok, Krishnaswamy, &

Govindarasu, 2016) and cybersecurity assurance

(Somarakis, Smyrlis, Fysarakis, & Spanoudakis,

2019). In contrast to domain specific approaches, our

approach is generic and may be applied for training

cybersecurity skills in any domain. This is also

demonstrated by the fact that we have applied our

approach in the context of three different large scale

real-world pilots: academic pilot, transport pilot, and

energy pilot (CYBERWISER.eu, 2020a).

Several approaches focus mainly on the cyber

range architecture and improving the efficiency and

performance of cyber ranges. Pham, Tang, Chinen,

and Beuran (2016) suggest a cyber range framework

named CyRIS/CyTrONE focusing on improving the

accuracy of the training setup, decreasing the setup

time and cost, and making training possible

repeatedly and for a large number of participants. The

authors also report on an evaluation of the

performance of their approach (Beuran et al., 2018).

Russo, Costa, and Armando (2018) argue that the

design, validation, and deployment of scenarios are

costly and error-prone activities that may require

specialized personnel for weeks or even months, and

that misconfiguration in the resulting scenario can

spoil the entire cyber exercise. To address these

challenges brought by architectural shortcomings, the

authors propose a framework for automating the

design, model validation, generation and testing of

cyber training scenarios. As part of developing our

cyber range platform described in Section 3, we

considered requirements related to scalability and

efficiency of deploying and running training

scenarios. However, as explained in Section 1, the

contribution of this paper is our method for creating

cyber-risk models that facilitate the training and

evaluation of cybersecurity skills of participants in

cyber-ranges. The reader is referred to a technical

report for further information on considerations

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models

517

related to scalability and efficiency of our cyber range

platform (CYBERWISER.eu, 2019a).

The approaches provided by Russo et al. (2018)

and Braghin et al. (2019) are similar to our approach

in the sense that they use some form of attack models

as a foundation to design and execute training

scenarios. Russo et al. (2018) introduce a Scenario

Definition Language (SDL) based on the OASIS

Topology and Orchestration Specification for Cloud

Applications (TOSCA). Braghin et al. (2019) provide

a domain specific language for scenario construction

in which it is possible to capture configuration

problems as well as structural vulnerabilities. We use

CORAS risk modes which are acyclic directed graphs

as described in Section 4. Thus, from a modelling

perspective, our approach complements existing

approaches. To the best of our knowledge, our

approach is unique compared to existing approach in

the sense that we develop machine readable risk

assessment algorithms based on the cyber-risk

models in order to facilitate real-time risk assessment

as well as real-time evaluation of the participants'

cybersecurity skills as explained in Section 4.

In their systematic literature review, Yamin et al.

(2020) report that current cyber range approaches that

apply some form of cyber-attack modelling are not

validating the models against real world scenarios and

use mostly artificial educational scenarios. In this

respect, our approach requires that the risk models are

developed with respect to training scenario

descriptions requested by stakeholders. Based on

experience so far in applying our method in real-

world academic pilot, transport pilot, and energy

pilot, it is reasonable to argue that our approach

validates the risk models against real world scenarios,

thus supporting its feasibility.

6 EVALUATION

In this section, we discuss the extent to which we have

fulfilled our success criteria described in Section 1.

6.1 Fulfilment of Success Criterion 1

The first criterion states: The method must be easy to

use and comprehensible for cyber-range training

scenario (exercise) developers.

The steps of our method (Section 4) are well in line

with activities typically carried out in cyber ranges

collaboratively by the White Team and the Green

Team (Yamin et al., 2020). Step 1 and Step 2 fall under

the training scenario design activities typically carried

out by the White Team. Step 3 and Step 4 are part of

environment configuration and the management of

training scenario execution typically carried out by the

Green Team. Finally, Step 5 is carried out as part of

learning activities including tutoring, scoring and

analysis of scoring carried out by the White Team.

As explained in Section 5, there are several

approaches that use some form of attack modelling

for the purpose of designing and executing training

scenarios. In our approach, we use the CORAS risk

modelling language. CORAS has been empirically

shown to be intuitively simple for stakeholders with

different backgrounds (Solhaug & Stølen, 2013).

CORAS is also based on international standards like

ISO 27005 and ISO 31000, which means that the

language supports well known and widely used

cybersecurity concepts (see Section 4). In the context

of cyber ranges, it is expected that the White Team

and Green Team are familiar with concepts such as

threat scenario, vulnerability, unwanted incident, etc.

With respect to the technical aspects of our

method considering the development of R scripts

based on the cyber-risk models, this is an activity

expected to be carried out by the Green Team as part

of environment configuration. However, the White

Team on the other hand can support the Green Team

by explaining the logic of the expected cyber-risk

assessment algorithms. These activities are thus in

line with the expected roles of White Team and Green

Team members (Yamin et al., 2020).

A threat to validity in terms of the generality of

our method is that the method has been applied only

on our cyber range architecture described in Section

3. However, according to the taxonomy provided by

Yamin et al. (2020), we see that the architecture of

our cyber range is in line with general cyber range

architectures covering components within scenario

development, environment setup and configuration,

monitoring, teaming (red, blue, white, green),

learning (training material), and management of the

cyber range. It is therefore a straightforward task to

map our method to existing cyber range architectures,

in order to apply our method in different cyber ranges.

At the time of writing, our method has been tried

out in the context of the CYBERWISER.eu project,

within three large scale pilots (cases) from three

different domains: academia, transport, and energy.

Although the stakeholders in these pilots come from

different domains, they all are cybersecurity experts

who took the role as White Team, while the technical

team of the project took the role as Green Team. All

steps of our method were carried out and tried out in

order to train students (in the case of the academic

pilot) and security staff (in the case of the transport

and energy pilot). More empirical studies of our

ICSOFT 2020 - 15th International Conference on Software Technologies

518

method are planned (CYBERWISER.eu, 2019b).

However, the fact that all steps of our method were

successfully carried out collaboratively in different

domains with people from various background,

supports the feasibility of our method in real world

cyber range applications.

Thus, from a methodological point of view, it is

reasonable to argue that our method is easy to use and

comprehensible by scenario developers.

6.2 Fulfilment of Success Criterion 2

The second criterion states: The method must provide

necessary guidelines to create cyber-risk models that

facilitate real-time evaluation of participants.

The first step of our method provides detailed

explanation on how to develop cyber-risk models

based on a training scenario description, how to

identify indicators to capture the dynamic behaviour of

the training scenario, and finally translate the cyber risk

models with indicators to machine readable risk

assessment algorithms. These risk assessment algo-

rithms are used to assess the risk level of an ongoing

cyber-attack in a training scenario in real-time. From a

team perspective, these algorithms are used to assess

how well the Red Team is carrying out an attack in

real-time (the higher the risk level the better).

The second step of our method explains how to

identify risk countermeasures by creating CORAS

treatment diagrams. These countermeasures are

implemented in the cyber range and made available

to the Blue Team during a training scenario as risk-

countermeasure options. Based on the selected

countermeasures in a training scenario, the risk level

may go down. This factor is used to assess how well

the Blue Team is performing in real-time protecting

the simulated infrastructure from exposed cyber-

attacks. That is, the lower the risk level (due to

selected countermeasures) the better the Blue Team is

performing.

To this end, our method provides necessary

guidelines to create cyber-risk models that facilitate

real-time evaluation of participants.

6.3 Fulfilment of Success Criterion 3

The third criterion states: The method must produce

useful feedback to the participants in terms of

exercise evaluations.

As explained in Section 4.5, our method supports

the evaluation of participants carrying out training

scenario exercises on cyber ranges. The novel

contribution of our approach relies on the automation

of the evaluation process, using information collected

during the exercise to provide automated evaluation

in real-time.

The evaluation process in our approach takes as

input the cyber risk exposure of the cyber-attack

considered in the training scenario, flags logging the

achievements of the participants, logs from monitoring

tools, and results from vulnerability scans as explained

in Section 4.5. In order to capture evaluation carried

out manually by the White Team, complementing the

automatic evaluation, our approach provides a

questionnaire to be filled out by the White Team based

on their expert knowledge and observations.

According to Yamin et al. (2020), current cyber

range approaches mainly use scoreboards in which

progress of participants is presented based upon the

task they completed. Our approach also provides

scoring in terms of grading. However, compared to

existing approaches, our approach contributes with

additional evaluations in the sense that the

Performance Evaluator provides a report in which

detailed information about strengths and weaknesses

of the participant's cybersecurity skills are presented,

including indications on how the participant may

improve weaknesses. Our initial experience is that the

participants find this evaluation very useful in order to

plan further customized cybersecurity learning

activities.

7 CONCLUSIONS

In general, there is an urgent need for highly skilled,

multi-disciplined cybersecurity professionals, and at

the same time there is an awareness gap and lack of

integrated training modules on cybersecurity related

aspects on all school levels. In order to address this

need and bridge the awareness gap, we have

developed a method to train and evaluate the

cybersecurity skills of participants in cyber ranges

based on cyber-risk models.

The target users of our method are the White and

Green Teams typically considered in cyber ranges.

The method uses cyber-risk models to support the

design and execution of training scenarios. The

output of our method is a performance report for the

participants being trained, that is, participants in the

Red or Blue Teams. The report provides detailed

information based on the executed training scenario

exercises reporting strengths and weaknesses of the

participant's cybersecurity skills, as well as directions

for improving the weaknesses.

We have applied our method in three pilot cases

from academia, transport, and energy. Our initial

results indicate that the method is easy to use and

An Approach to Train and Evaluate the Cybersecurity Skills of Participants in Cyber Ranges based on Cyber-Risk Models

519

comprehensible for training-scenario developers

(White and Green Team), develops cyber-risk models

that facilitate real-time evaluation of participants in

training scenarios, and produces useful feedback to

the participants (Blue and Red Team) in terms of

grading and detailed evaluation of strengths and

weaknesses regarding cybersecurity skills.

As next steps, we will carry out empirical

evaluations focusing on user experience in the

abovementioned large-scale pilots and based on our

findings continue improving our method.

ACKNOWLEDGEMENTS

This work has been conducted as part of the

CYBERWISER.eu project (786668) funded by the

European Commission within the Horizon 2020

research and innovation programme.

REFERENCES

Ashok, A., Krishnaswamy, S., & Govindarasu, M. (2016).

PowerCyber: A remotely accessible testbed for Cyber

Physical security of the Smart Grid. Paper presented at

the 2016 IEEE Power & Energy Society Innovative

Smart Grid Technologies Conference (ISGT).

Beuran, R., Chinen, K.-i., Tan, Y., & Shinoda, Y. (2016).

Towards effective cybersecurity education and training

(IS-RR-2016-003). Retrieved from https://dspace.

jaist.ac.jp/dspace/handle/10119/13769

Beuran, R., Tang, D., Pham, C., Chinen, K.-i., Tan, Y., &

Shinoda, Y. (2018). Integrated framework for hands-on

cybersecurity training: CyTrONE. Computers &

Security, 78, 43-59.

Braghin, C., Cimato, S., Damiani, E., Frati, F., Mauri, L., &

Riccobene, E. (2019). A Model Driven Approach for

Cyber Security Scenarios Deployment. In Computer

Security (pp. 107-122): Springer.

CAPEC. (2020). Common Attack Pattern Enumeration and

Classification. Retrieved from https://capec.mitre.org/

index.html

CIISec. (2019). CIISec Roles Framework, Version 0.3.

Retrieved from https://www.ciisec.org/ CIISEC/

Resources/Roles_Framework.aspx

CWE. (2020). Common Weakness Enumeration. Retrieved

from https://cwe.mitre.org/

CYBERWISER.eu. (2019a). Deliverable 2.5, Platform

Design, Final Version. Retrieved from https://www.

cyberwiser.eu/

CYBERWISER.eu. (2019b). Deliverable 5.1, General

Requirements and Guidelines. Retrieved from

https://www.cyberwiser.eu/

CYBERWISER.eu. (2020a). CYBERWISER.eu - Cyber

Range & Capacity Building in Cybersecurity. Retrieved

from https://www.cyberwiser.eu/

CYBERWISER.eu. (2020b). Deliverable 2.6, Risk Model

Templates, Initial Version. Retrieved from

https://www.cyberwiser.eu/

Damodaran, S. K., & Smith, K. (2015). CRIS Cyber Range

Lexicon, Version 1.0. Retrieved from https://apps.dtic.

mil/dtic/tr/fulltext/u2/a627477.pdf

Davis, J., & Magrath, S. (2013). A survey of cyber ranges

and testbeds. Retrieved from https://apps.dtic.mil/

dtic/tr/fulltext/u2/a594524.pdf

ECSO. (2016). European Cybersecurity Strategic Research

and Innovation Agenda (SRIA) for a contractual

Public-Private-Partnership (cPPP). Retrieved from

https://www.ecs-org.eu/documents/ecs-cppp-sria.pdf

EnpitSecurity. (2020). SecCap. Retrieved from

https://www.seccap.jp/

Erdogan, G., Gonzalez, A., Refsdal, A., & Seehusen, F.

(2017). A method for developing algorithms for

assessing cyber-risk cost. Paper presented at the 2017

IEEE International Conference on Software Quality,

Reliability and Security (QRS).

Ferguson, B., Tall, A., & Olsen, D. (2014). National cyber

range overview. Paper presented at the 2014 IEEE

Military Communications Conference.

Lund, M. S., Solhaug, B., & Stølen, K. (2011). Model-

Driven Risk Analysis - The CORAS Approach:

Springer-Verlag Berlin Heidelberg.

NRISecure. (2020). Secure Eggs (Essentials and Global

Guidance for Security). Retrieved from https://www.

nri-secure.co.jp/service/learning/secureeggs

OWASP. (2020). Open Web Application Security Project.

Retrieved from https://owasp.org/

Pham, C., Tang, D., Chinen, K.-i., & Beuran, R. (2016).

Cyris: A cyber range instantiation system for

facilitating security training. Paper presented at the

Seventh Symposium on Information and

Communication Technology.

R-project. (2020). The R Project for Statistical Computing.

Retrieved from https://www.r-project.org/

Russo, E., Costa, G., & Armando, A. (2018). Scenario

design and validation for next generation cyber ranges.

Paper presented at the 2018 IEEE 17th International

Symposium on Network Computing and Applications.

Solhaug, B., & Stølen, K. (2013). The CORAS Language-

Why it is designed the way it is. Paper presented at the

11th International Conference on Structural Safety and

Reliability (ICOSSAR’13).

Somarakis, I., Smyrlis, M., Fysarakis, K., & Spanoudakis,

G. (2019). Model-Driven Cyber Range Training: A

Cyber Security Assurance Perspective. In Computer

Security (pp. 172-184): Springer.

Wieringa, R. J. (2014). Design science methodology for

information systems and software engineering:

Springer.

Yamin, M. M., Katt, B., & Gkioulos, V. (2020). Cyber

Ranges and Security Testbeds: Scenarios, Functions,

Tools and Architecture. Computers & Security, 88,

101636.

ICSOFT 2020 - 15th International Conference on Software Technologies

520