Data Collection in Cyber Exercises Through Monitoring Points: Observing, Steering, and Scoring

Tobias Pfaller, Florian Skopik, Lenhard Reuter, Maria Leitner

2025

Abstract

Cyber security exercises are an essential means to train people and increase their skill levels in IT operations, cyber incident response, and forensic investigations. Unfortunately, carrying out high-quality exercises requires tremendous human effort in planning, deploying, executing and evaluating well-planned cyber exercise scenarios. While planning a scenario is often only a one time effort, and deployment can be highly automatized today, their repeated execution and evaluation is a resource-intensive task. Usually human experts manually observe the participants to recognize any difficulties in carrying out the exercise and to keep track of the participants’ progress. This is an essential prerequisite to not only support participants during the exercise, but also to drive the scenario further through timely injects, and provide feedback after the exercise. All this manual effort makes exercises a costly activity, reduces scalability and hinders their wide adoption. We argue that with automating observations, recognizing participant progress with only little to no human effort, and even steering the delivery of customized injects, cyber exercises could be carried out much more cost-effective. In this paper, we therefore introduce the concept of monitoring points which enable the scenario-dependent collection of technical data and the calculation of behavior and progress metrics to rate participants in exercises. This is the foundational basis for steering an exercise on the one side, and evaluation on the other side. We showcase our concept and implementation in course of a demonstrator consisting of a cyber exercise comprising 14 participants and discuss its applicability.

Paper Citation

in Harvard Style

Pfaller T., Skopik F., Reuter L. and Leitner M. (2025). Data Collection in Cyber Exercises Through Monitoring Points: Observing, Steering, and Scoring. In Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP; ISBN 978-989-758-735-1, SciTePress, pages 355-366. DOI: 10.5220/0013309900003899

in Bibtex Style

@conference{icissp25,
author={Tobias Pfaller and Florian Skopik and Lenhard Reuter and Maria Leitner},
title={Data Collection in Cyber Exercises Through Monitoring Points: Observing, Steering, and Scoring},
booktitle={Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP},
year={2025},
pages={355-366},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013309900003899},
isbn={978-989-758-735-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 1: ICISSP
TI - Data Collection in Cyber Exercises Through Monitoring Points: Observing, Steering, and Scoring
SN - 978-989-758-735-1
AU - Pfaller T.
AU - Skopik F.
AU - Reuter L.
AU - Leitner M.
PY - 2025
SP - 355
EP - 366
DO - 10.5220/0013309900003899
PB - SciTePress