Cyber Exercises in Computer Science Education

Melisa Gaﬁc

1 a

, Simon Tjoa

1 b

, Peter Kieseberg

1 c

, Otto Hellwig

2 d

and Gerald Quirchmayr

Institute of IT Security Research, St. P

olten University of Applied Sciences, 3100 St. P

olten, Austria

University of Vienna, 1010 Vienna, Austria

Keywords:

Cyber Exercises, Cyber Security, Cyber Resilience, HEI.

Abstract:

Due to the strong dependence of companies on their ICT and the high relevance of stable services to remain

competitive in the global market, cyber security and resilience play an increasingly important role. However,

information security is not only an important issue in the corporate context but also in the societal context. For

this reason, nearly all computer science programs at higher education institutions (HEI) incorporate this topic.

In this paper, we introduce a table-top cyber security exercise lecture format and the experiences gathered

over the last years. The approach is currently used to teach computer science students as well as information

security students at two higher education institutions in Austria. Additionally, we brieﬂy highlight how the

approach was adapted in order to satisfy the compelling need to teach the course remotely due to Corona

restrictions.

1 INTRODUCTION

During COVID 19-crisis the security and resilience of

critical information system have been more important

than ever before. Breaches and cyber security inci-

dents impressively highlighted the importance of cy-

ber security and especially incident response (ENISA,

2020). In order to ensure resilience of systems, and

to prepare for a such unpredictable cyber threats, it

is necessary to continuously train people how to pur-

posefully react on these threats and to communicate

within the team under difﬁcult circumstances (Wil-

hemson and Svensson, 2014). Therefore, exercises

especially cyber exercises, play a central role in es-

tablishing a resilient society.

Cyber exercises have gained a lot of attention

throughout recent years, especially in the cyber se-

curity sector, as an important tool for security train-

ing, awareness-building and testing incident response.

The EU emphasises the importance of this ﬁeld in

its strategy for the digital decade (European Com-

mission, 2020). Large exercises, that make it to the

news, such as CyberStorm (Cybersecurity & Infras-

tructure Security Agency (CISA), ), Locked Shields

(The NATO Cooperative Cyber Defence Centre of

https://orcid.org/0000-0002-0402-4283

https://orcid.org/0000-0003-2280-9604

https://orcid.org/0000-0002-2847-2152

https://orcid.org/0000-0002-9172-0681

Excellence, 2021) or Cyber Europe (ENISA, 2021),

represent only a small fraction of the exercises car-

ried out. Conducting cyber exercises also has an ed-

ucational value. Through interactive activities, such

as simulations and scenarios, exercise participants

apply knowledge in practical situations using tech-

niques and tools they are familiar with, thereby deep-

ening their understanding of a particular type of inci-

dent (Dewar, 2018).

Although cyber security has gained a lot of atten-

tion and also found its way to most modern computer

science curricula, the development of skills in the area

of cyber exercises still did not get a lot of attention.

A major discriminator to more traditional forms of

training, especially lectures with test, is the simula-

tion of real-life environments, realistic work environ-

ments and especially stress. To change the situation,

we started to research how the topic can be taught to

students in order to empower them to gain the neces-

sary skill set to develop, run and evaluate cyber exer-

cises in their future.

The major contribution of this paper is the pre-

sentation of a didactic concept highlighting how the

increasingly important topic of cyber exercises can be

taught in higher education institutions (HEI). It fur-

ther shares our experiences of nearly ten years and

outlines the challenges, pitfalls and beneﬁts of cyber

exercises in computer science curricula. Using cyber

exercises as a learning method allows students at HEI

to simulate cyber security incidents for hypothetical

404

Gaﬁc, M., Tjoa, S., Kieseberg, P., Hellwig, O. and Quirchmayr, G.

Cyber Exercises in Computer Science Education.

DOI: 10.5220/0010845800003120

In Proceedings of the 8th International Conference on Information Systems Security and Privacy (ICISSP 2022), pages 404-411

ISBN: 978-989-758-553-1; ISSN: 2184-4356

situations and practice their required decision-making

expertise and capabilities.

The remainder of this paper is structured as fol-

lows: In Section 2 we provide an overview about ex-

isting publications on cyber security exercises as well

as on educational planning games. In Section 3 we

outline our teaching concept, which was elaborated in

the past decade. In Section 5, we highlight the feed-

back of the students of the corresponding courses. We

conclude our work in Section 6.

2 RELATED WORK

In this section, we outline the relevant research in the

context of cyber exercises in higher education. Be-

sides providing an overview on selected approaches

in teaching, we highlight essential literature, which

have been inﬂuenced the design and conception of the

herein presented approach.

The effectiveness and importance of plan-

ning/simulation games as a learning and teaching tool

has been outlined in the meta-analysis by Vogel (Jen-

nifer J. Vogel et al., 2006). It was found that interac-

tive activities such as games and simulations increase

motivation and learning outcomes compared to the

traditional teaching methods.

Also Prensky discusses in his paper (Prensky,

2002) how challenging is to keep students motivated

through the entire learning process. In contrast to the

traditional environment in HEI, playing interactive

games can be engaging and achieving some scores or

prizes can be very relaxing and motivating.

Steinkuehler emphasizes in (Steinkuehler, 2010)

that through games, students can acquire various

skills and be more enthusiastic about learning. How-

ever, beside acquired knowledge, students, as a com-

plete individuals, have to develop different skills and

gain experiences that may help them to think or react

rationally in new situations (Blazenka Divjak, 2011).

2.1 Cyber Exercise Guidelines

As deﬁned in ISO 22398:2013 exercises are ”a pro-

cess to train for, assess, practice, and improve perfor-

mance in an organization” (ISO, 2013). Derived from

this deﬁnition, cyber exercises can be deﬁned as an

event, in which organizations simulate a cyber secu-

rity incident in order to develop and test skills in the

prevention, detection, mitigation and recovery of op-

erations from cyber attack or security incident.

In order to facilitate a process of planning and or-

ganizing such events, many studies have been con-

ducted, which identiﬁed key components of a cyber

exercise. In 2015, the Spanish National Cybersecu-

rity Institute Incibe published a taxonomic classiﬁca-

tion scheme that provides a comprehensive survey on

existing cyber exercises (Incibe, 2015). Based on col-

lected information about existing cyber exercises, the

authors deﬁned a set of metrics and indicators for cy-

ber exercise proﬁling and developed a taxonomy pro-

posal to better plan and improve future cyber exer-

cises.

In Cybersecurity and Cyberdefense Exercises Re-

port (Dewar, 2018), the authors identiﬁed goals,

types, actors and resources as a core elements of cy-

ber defense exercise and also give insights into the ex-

periences and lessons learned based on various After

Action Reports (AAR).

The ENISA Good Practice Guide, which is widely

used in the EU, describes the general organizational

process from preparation to implementation of lo-

cal and national cyber exercise (ENISA, 2009). The

guide systematically explains the key steps in the life-

cycle for exercises (identifying, planning, conducting,

and evaluating). According to ENISA, it is essential

to incorporate experiences from previous exercises

into the planning and setting of framework conditions

in order to achieve the best possible outcome. There-

fore, in addition to this guideline, the Latest Report on

National and International Cyber Security Exercises

(ENISA, 2015) was published. The report analyses

the consisting data set of over 200 exercises and dis-

cusses the outcomes of previous exercises.

Similar to ENISA Good Practice Guide, the

Department of Homeland Security published The

Homeland Security Exercise and Evaluation Program

(HSEEP) (FEMA, 2020). HSEEP’s life cycle (Pro-

gram Management, Exercise Development and De-

sign, Exercise Conduct, Exercise Evaluation, Im-

provement Planning) is very ﬂexible and can be

adapted to different types of exercises.

The Swedish Defence University (FHS) has pub-

lished a Handbook for planning, running, and evalu-

ating information technology and cyber security exer-

cises (Wilhemson and Svensson, 2014). Besides the

detailed description of exercise planning steps, this

handbook contains practical experiences from previ-

ous exercises and a list of criteria for the technical

exercise environment (e.g. Communication prepara-

tions, Exercise network, Equipment etc.).

In Cyber Exercise Playbook (Kick, 2015), MITRE

describes practical guidance on cyber exercises pro-

cess and gives an overview of essential activities of

every phase. This playbook also provides tips and

common pitfalls of previous exercises as well as sam-

ple documents and templates to assist planners of ex-

ercises.

Cyber Exercises in Computer Science Education

405

2.2 Cyber Exercises

To get an overall overview of cyber exercise we an-

alyzed after action reports (AAR) of previous cyber

exercises and their results of the execution (goals,

objectives, scenario, participants etc.). Based on

the number of participants, three largest and popu-

lar cyber-exercises are Cyber Europe (ENISA, 2021),

Cyber Storm (Cybersecurity & Infrastructure Security

Agency (CISA), ) and Locked Shields (The NATO

Cooperative Cyber Defence Centre of Excellence,

2021).

Cyber Europe is organised by the European

Union Agency for Network and Information Security

(ENISA) and takes place every two years. These exer-

cises include both public and private institutions and

participants from all 28 EU Member States and two

European Free Trade Association (EFTA). The usual

high-level strategic goals of this exercises are testing

EU-level cooperation processes and training EU- and

national-level capabilities (ENISA, 2018).

The largest national cyber exercise conducted in

the US is organised by the US Department of Home-

land Security (DHS). In Cyber Storm VI (US Depart-

ment of Homeland Security, 2020) more than 1200

experts from both the private and the public sector

from the US and abroad participated in order to evalu-

ate and improve Nation’s cyber security response ca-

pabilities and to strengthen relationships between the

Federal Government and its partners (US Department

of Homeland Security, 2020).

Unlike the two previous exercises, Locked Shields

(The NATO Cooperative Cyber Defence Centre of

Excellence, 2021) is an annual full-scale exercise,

which is organised by NATO’s Cooperative Cyber

Defence Centre of Excellence (CCDCOE). A differ-

ence to other exercises is, that the exercise did not

only focus on national IT systems and critical in-

frastructure, but also included military systems from

nearly 30 nations. In Locked Shields exercises all as-

pects of cyber attacks are in real-time simulated (i.e

decision-making, legal, communication) (The NATO

Cooperative Cyber Defence Centre of Excellence,

2021).

Panoptes (Dimitris Gritzalis, Spyros Papageor-

giou, 2016) is a Greek National Cyber Defence Exer-

cise organized annually since 2010 by the Cyber De-

fence Directorate of the HNDGS. This exercise is a

mostly ofﬂine Cyber Defense Exercise (CDX) (Dim-

itris Gritzalis, Spyros Papageorgiou, 2016), so that cy-

ber attacks are not simulated in a real-time. More than

200 participants from different sectors gain an oppor-

tunity to evaluate their own capabilities on various

scenarios (i.e. Incident Handling, Digital Forensics,

Information Sharing, Following policies and proce-

dures etc.).

Cyber Atlantic (ENISA, 2011) is the joint EU-

US table top Cyber execise organized by ENISA and

DHS in 2011. An overall objective was to explore

and identify potential improvements in communica-

tion and collaboration between EU Member states

and US during cyber crisis management activities

(ENISA, 2011).

3 PERFORMING CYBER

EXERCISE IN HEI

In this section, we introduce our approach to teach

students planning, conducting and evaluating tabletop

cyber exercises in university courses. Related litera-

ture presented in Section 2, especially the guidelines

(ENISA, 2009), (FEMA, 2020), (Kick, 2015) have in-

ﬂuenced the design and conception of the herein pre-

sented approach.

In order to set the scope and deﬁne the learning

objectives we made use of the Bloom’s taxonomy

(Bloom, 1956). In the following, the main learning

objectives are outlines:

• The students understand current threats and are

able to model threats.

• The students are able to derive exercise objectives

from an exercise task and create suitable high-

level scenario.

• The students decide on roles and responsibilities

within the project according to the strengths and

weaknesses of the planning team members.

• The students are able to reﬁne the high-level sce-

nario into a master scenario event list (MSEL) and

the according injects.

• The students are able to manage and run a cyber

exercise.

• The students can select evaluation criteria and ap-

ply them to come up with recommendations and

improvements.

In order to achieve the main objectives, the lec-

ture follows a ﬁve-step process (i.e. Knowledge trans-

fer, Group formation, Planning, Conducting, Evalua-

tion)(Hellwig, 2016) depicted in Figure 1.

3.1 Knowledge Transfer

The ﬁrst part of the lecture is dedicated to the knowl-

edge transfer of cyber exercise fundamentals. In this

ICISSP 2022 - 8th International Conference on Information Systems Security and Privacy

406

Figure 1: Cyber exercises procedure for courses in HEI.

part the theory of cyber exercises in general and ta-

ble top exercises in particular are presented. The stu-

dents learn about current exercise frameworks and

best practices in the area, the anatomy of cyber at-

tacks ATT&CK (MITRE, 2021), Cyber Kill Chain

(LogSign, 2020), Diamond Model of intrusion (Calta-

girone et al., 2013), current threat landscape (ENISA,

2020) and techniques for developing scenarios like

Twitter account BadThingsDaily (Hafner, ), CWE

(Mitre, 2021) and others.

As the level of knowledge in this area often varies

amongst the students, inverted classroom settings

showed good results. Providing the students with ap-

propriated sources and guiding questions enables stu-

dents to learn on their own pace and to dive into the

topic.

3.2 Group Formation

Within the next section of the lecture student teams

are formed. As planning and running a suitable exer-

cise is a challenging task, it is important the team size

is not too small (i.e. 8-12 members).

After the teams are established, they have to come

up with a strategy how to assign roles and responsi-

bilities. Past courses demonstrated that an effective

way to allocate roles and responsibilities often starts

by capturing the strengths of each team member, fol-

lowed by reﬂecting which roles (i.e. team leader,

facilitator, observer/evaluator, counter-player) require

certain strengths in order to have a good exercise team

structure.

3.3 Planning

The planning phase is the most critical part of the

lecture. Planning errors can lead to severe problems

while conducting the exercise. Therefore, students

have to be closely coached during this phase. This

planning phase consists of two parts.

In a ﬁrst part students are asked to set the objec-

tives for the exercises and to derive the skills they

want to train and rehearse during the exercise. To

deﬁne goals, objectives and capabilities, students of-

ten use the capability target method from HSEEP

Framework (FEMA, 2020) and the SMART Method-

ology of goal setting (Speciﬁc, Measurable, Achiev-

able, Relevant, Time-Bound). After deﬁnition of the

objectives, the students get the assignment to identify

a suitable and realistic high-level scenario to achieve

the objectives.

A high-level scenario, as a combination of daily

business and cyber attacks, provides stakeholders

with an initial idea and intention of the exercise. At

the end of the ﬁrst part, students have developed rough

plan that contains the general set-up of the cyber ex-

ercise, its goals and objectives, potential participants,

roles and responsibilities of the planning team and the

high-level scenario.

Succeeding in the second part of the planning

phase, a high-level storyboard is developed to set the

frame for the exercise. To come up with a suitable

storyboard, it has to be deﬁned what happens in each

phase of the scenario. The transition from one phase

to the next is characterized by time jump (e.g. phase

1: worming up events to learn the roles, phase 2: in-

cident identiﬁcation and containment, phase 3: erad-

ication and recovery). The concept of phases plays

Cyber Exercises in Computer Science Education

407

Figure 2: Setting of Cyber Exercise.

an important role in tabletop exercises as they help to

synchronize the scenario even if participants are re-

acting in a different way than expected.

As soon as the high level scenario is outlined

within the storyboard, the detailed planning can take

place. The goal is to put the participants in a realistic

situation in order to subsequently simulate a logical

sequence of events. Participants act individually in a

situation and can thus inﬂuence others with their be-

havior. For planning purposes, this means that there

must be at least two different approaches to reacting

to each situation in the exercise. However, it is nec-

essary to make a balance between the path the plan-

ning group envisions and the ﬂexibility the exercise

requires.

The scenario must be managed throughout the im-

plementation and adapted to the actions of the partic-

ipants. For this purpose, injects are created to con-

trol the exercise and may be the decisive factor in

whether an execution of exercise is successful. Typi-

cally, students create a template for injects and break

them down into three categories: scenario relevant in-

jects (i.e attacks, cyber incidents), media (i.e. press

statements), legal injects (i.e. law, regulations like

GDPR). A key document in the planning stage is the

Master Scenario Event List (MSEL), that provides an

overview (Inject ID, delivery method, target, title, de-

scription, assumptions and expected actions) of all in-

jects and ensures timely and organized execution of

injects.

In the next step of the planning phase it is nec-

essary to deﬁne communications rules with and be-

tween participants during the exercise (i.e. what com-

munications channels are allowed, and how does the

organisational (communication) structure looks like).

Moreover, game-rules are deﬁned (e.g. allowed soft-

ware and hardware, team resources, penalties for any

violations during an exercise). Additionally, metrics

are deﬁned to make the success of a (cyber) exercise

measurable. Usually, metrics correlate closely with

the goals of the exercise, as the effectiveness of the

exercise is critical for the achievement of the objec-

tives.

The last step in the planning phase is the deﬁnition

of the evaluation method. An evaluation method is to

be designed by each of the group in order to ensure

that all players of the exercise learn from the exer-

cise. Moreover, before conducting the exercise, stu-

dents perform some ﬁnal preparations in order to test

the injects and ensure readiness to start the exercise.

3.4 Conduct

After a successful design of cyber table top exercise,

the preparation of the environment takes place. Typ-

ically this involves setting up the lecture rooms in

an adequate manner. For clariﬁcation, such a set up

could consist of the following steps:

• Assign rooms for the different teams (e.g. recov-

ery team, crisis management team)

ICISSP 2022 - 8th International Conference on Information Systems Security and Privacy

408

• Setup the rooms (e.g. table arrangement, ﬂip

charts)

• Test of technical equipment used during the exer-

cise (e.g. email addresses, projectors, phones)

After the setup is complete, a ﬁnal test is carried

out by each group. This is also the last chance to

coach the students managing the exercise without in-

terfering the exercise. As soon as all ﬁnal checks are

ﬁnished and the exercise environment is ready, the

students simulating the exercise participants are wel-

comed in a brieﬁng by the students managing the ex-

ercise.

The aim of the brieﬁng is to explain the exercise

rules (e.g. allowed communication media), organi-

zational issues (e.g. assignment of roles, course of

the exercise, explanation on delivery and answering

of injects, exercise time, exercise end) and the starting

point of the scenario. In the course of the exercise, the

injects are delivered by the exercise team. Dynamic

injects support the exercise management team to ad-

just the difﬁculty depending on the performance of

the player. In this phase, the lecturers take the role of

an observer in order to capture the performance and

provide feedback to both teams following the exer-

cise. The main setting of conducting cyber exercise is

shown in Figure 2.

The end of exercise must be clearly announced by

the exercise leader and is usually achieved, if one of

the following states is reached:

• exercise goals have been achieved: the ideal state

is that all essential goals are reached by players.

• certain tasks have been completed: another option

to determine the end is when all relevant activities

have been performed by the exercise participants.

• if it is not certain how to proceed with the exer-

cise: although it has turned out that this case is

unlikely to occur, every exercise team leader must

be prepared to end the exercise earlier than ex-

pected due to unforeseen events or reactions by

the players.

3.5 Evaluation

After the exercise is declared ﬁnished by the exercise

planning team, the evaluation begins. The evaluation

consists of the two parts hot wash up and after action

report.

Immediate captured feedback is collected within a

so called hot wash up. The goal using this method is

to get the impressions, experiences and opinions right

after the exercise. Topics covered by the hot wash up

are amongst others: strength and weaknesses of the

exercise, performance of the players, achievement of

the exercise goals & objectives, lessons learned of all

stakeholders.

The lecture ends for the students by handing in an

after action report, which contains the goal and ob-

jectives of the performed exercise, information about

the scenario and participants results and observation

during the exercise as well as feedback of the partici-

pants.

4 VARIATION DURING CORONA

As the cyber exercise has been planned as physical

table top exercise, the concept had to be strongly

adapted to work in a remote setting during the Corona

restrictions. All physical components, which have

been normally used, had to be replaced by virtual

means. Figure 3 provides an overview on the tech-

nologies used in the various phases of the exercise

process.

As Microsoft Teams was already well known by

the students, we decided to use this platform as foun-

dation for the course (i.e. for communication and col-

laboration). Every student group was assigned to an

individual team of up to 12 students (e.g. Team A,

Team B). Furthermore, teams have been created for

conducting the exercises (Exercise A-B). Instead of

different rooms, which have been used in physical ta-

ble top exercises during the exercise in the past, it was

decided to use private channels.

Since the scenario deﬁnition and planning process

mostly is creative process, we decided to use Mural

to facilitate collaboration. Mural enables visual col-

laboration similar to the work on a whiteboard or ﬂip

chart and therefore was a good technological replace-

ment for mind mapping, brainstorming or working

with sticky notes in the classroom.

To provide a central master scenario event list and

store details about the various injects, we created a

Sharepoint list. This represented the Master Scenario

Event List and supported the students to coordinate

the delivery of injects as well as capturing the reac-

tion at a central place. For the evaluation, we decided

to perform a hot wash in Microsoft Teams and we pre-

pared a form on a website to capture the experience of

the students.

Instead of Microsoft Teams, at the University of

Vienna students used an open source conferencing

system BigBlueButton for planning and the Discord

Server for conducting exercise. Based on our experi-

ences, these tools are also a good alternative and have

appeared to be suitable for performing virtual cyber

exercises.

Although the COVID edition of the exercise was

Cyber Exercises in Computer Science Education

409

Figure 3: Technologies used in the virtual edition.

challenging at the beginning, we could also identify

advantages in the collaboration. Students have not

been bound by time or place. A major challenge dur-

ing the exercise in the virtual edition was the coor-

dination of the exercise. For the improvement of the

overall coordination, we are currently developing an

application based on the captured requirements dur-

ing the exercise.

5 EVALUATION OF THE

PROPOSED APPROACH

In the last 9 years, cyber exercises have been used at

St. P

olten University of Applied Sciences and Univer-

sity of Vienna for computer science and cyber secu-

rity students in various settings (i.e. weekly teaching,

block teaching). The results and experienced gained

during this period are promising. In the following, we

brieﬂy provide the lessons learned and results of the

evaluations.

The interactive way to elaborate a cyber security

scenario and the development of a realistic cyber ex-

ercise motivates the students. The feedback of the

students also indicated that this way of game based

learning encourages students to further deepen their

cyber security knowledge.

During the pandemics the well established con-

cept had to be adapted to a remote teaching solution

twice. While in the ﬁrst year there had been minor

issues (e.g. challenges in the technical transformation

of the exercises) when rapidly transforming the lec-

ture, in the second year the remote lecture was a big

success.

6 CONCLUSION AND FUTURE

WORK

The COVID-19 crisis and the various cyber incidents

in recent months have made us aware of the impor-

tance of functioning and secure IT systems. Cyber

security and resilience gains importance as business

processes and services increasingly depend on func-

tioning ICT systems. Thus, it is no big surprise that

security found its way into most modern computer

science curricula. Cyber exercises are an effective

way to create awareness of cyber security incidents

and empowers students to experience various scenar-

ios.

In this paper we introduced an approach for or-

ganising table-top cyber exercise in computer sci-

ence education. The approach consists of ﬁve steps:

Knowledge Transfer, Group Formation, Plan, Con-

duct, Evaluate. This approach can be used as tem-

plate for institutions or companies that want to orga-

nize table-top exercises for educational purposes.

The virtual planning and performance of cyber ex-

ercises for teaching purposes is still in its infancy.

First promising results have been achieved in the last

ICISSP 2022 - 8th International Conference on Information Systems Security and Privacy

410

two years. However, the usage of a variety of tools

and media is still a problem to solve. We therefore

are working on a prototype, which combines the re-

quired capabilities in order to enhance the educational

experience.

REFERENCES

Blazenka Divjak, D. T. (2011). The impact of game-based

learning on the achievement of learning goals and mo-

tivation for learning mathematics - literature review.

Bloom, B. (1956). Blooms taxonomy. https://www.

bloomstaxonomy.net/. accessed Oktober 2021.

Caltagirone, S., Pendergast, A. D., and Betz, C. (2013). The

diamond model of intrusion analysis a summary by

sergio caltagirone.

Cybersecurity & Infrastructure Security Agency (CISA).

Cyber storm: Securing cyber space. https://www.cisa.

gov/cyber-storm-securing-cyber-space. accessed July

2021.

Dewar, R. S. (2018). Cybersecurity and Cyberdefense Ex-

ercises. Center for Security Studies - ETH Z

urich.

Dimitris Gritzalis, Spyros Papageorgiou (2016).

Panoptes: The greek national cyber de-

fence exercise. https://www.infosec.aueb.gr/

Publications/CEER-ENISA-2016\%20Gritzalis\

%20Papageorgiou.pdf. accessed July 2021.

ENISA (2009). Good Practice Guide on National Exer-

cises. European Union Agency for Cybersecurity.

ENISA (2011). Cyber atlantic. http://www.bic-trust.eu/

ﬁles/2011/12/slides15.pdf. accessed July 2021.

ENISA (2015). The 2015 report on national and interna-

tional cyber security exercises - survey, analysis and

recommendations.

ENISA (2018). Cyber europe 2018 - after action

report. hhttps://www.enisa.europa.eu/publications/

cyber-europe-2018-after-action-report. accessed July

2021.

ENISA (2020). The year in a review: ENISA Threat Land-

scape 2020. ENISA.

ENISA (2021). Cyber exercises - cyber europe programme.

European Commission (2020). The eu’s cy-

bersecurity strategy for the digital decade.

https://eur-lex.europa.eu/legal-content/EN/TXT/

HTML/?uri=CELEX:52020JC0018&from=EN.

FEMA (2020). Homeland Security Exercise and Evaluation

Program. Federal Emergency Management Agency.

Hafner, R. Bad things daily - tabletop scenarios. https:

//badthings.tedivm.com/. accessed Oktober 2021.

Hellwig, O. (2016). Die gestaltung von cyber-

ubungen

als planspiel. Planspiele-Vernetzung gestalten:

Forschungsergebnisse und Praxisbeispiele f

ur mor-

gen, 8:187.

Incibe (2015). Cyber exercises taxonomy. Spanish National

Cybersecurity Institute.

ISO (2013). ISO 22398:2013 Societal security – Guidelines

for exercises. International Organization for Standard-

ization.

Jennifer J. Vogel et al. (2006). Computer gaming and in-

teractive simulations for learning: A meta-analysis.

pages 229–243.

Kick, J. (2015). Cyber Exercise Playbook. The MITRE

Corporation.

LogSign (2020). 7 steps of cyber kill chain. https://www.

logsign.com/blog/7-steps-of-cyber-kill-chain/. ac-

cessed September 2021.

Mitre (2021). Common weakness enumeration. https://cwe.

mitre.org/. accessed Oktober 2021.

MITRE (2021). MITRE ATT&CK adversary. https://attack.

mitre.org/. accessed Oktober 2021.

Prensky, M. (2002). The motivation of gameplay.

Steinkuehler, C. (2010). Video games and digital literacies.

The NATO Cooperative Cyber Defence Centre of Excel-

lence (2021). Locked shields.

US Department of Homeland Security (2020). Cyber storm

vi: National cyber exercise. https://www.cisa.gov/

cyber-storm-vi. accessed July 2021.

Wilhemson, N. and Svensson, T. (2014). Handbook for

planning, running and evaluating information tech-

nology and cyber security exercises. Swedish National

Defence College.

Cyber Exercises in Computer Science Education

411