The Dual-Edged Sword: The Impact of Large Language Models in Network Infrastructure Security

David Debono, Anastasia Sare

2025

Abstract

Large Language Models (LLMs) have become essential tools for network infrastructure and security engineers, assisting in a wide range of daily administrative tasks. However, the widespread use of these models without adequate cybersecurity expertise could potentially compromise network security. This study examines the compliance of various LLMs, including GPT-3.5, GPT-4, Microsoft Copilot, and Gemini, with CIS benchmarks. We evaluate the capabilities and limitations of these models in adhering to MySQL and MongoDB CIS benchmarks on a Linux system using both qualitative and quantitative metrics. Four distinct test cases were developed to assess the performance of GPT-3.5 and GPT-4. The first test evaluated the models' compliance and knowledge of security standards without explicitly mentioning the standards. The second test assessed the models' zero-shot knowledge when CIS benchmarks were explicitly referenced, while the third test examined the effectiveness of follow-up prompts based on the results of the second test. In the fourth test, GPT-4 was provided with the actual standard in PDF format. Additionally, the zero-shot capabilities of Gemini and Microsoft Copilot were also evaluated. Among the models tested, GPT-4 demonstrated the highest compliance with CIS benchmarks, particularly in zero-shot learning and assisted scenarios. However, challenges were noted with certain configurations, and the use of prompt engineering techniques proved crucial in maximizing compliance. With a maximum score of 76.3% compliance, the findings suggest that while LLMs can assist in providing secure configurations aligned with international standards, expert knowledge and supervision remain essential to mitigate potential vulnerabilities.

Paper Citation

in Harvard Style

Debono D. and Sare A. (2025). The Dual-Edged Sword: The Impact of Large Language Models in Network Infrastructure Security. In Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 2: ICISSP; ISBN 978-989-758-735-1, SciTePress, pages 125-132. DOI: 10.5220/0013090700003899

in Bibtex Style

@conference{icissp25,
author={David Debono and Anastasia Sare},
title={The Dual-Edged Sword: The Impact of Large Language Models in Network Infrastructure Security},
booktitle={Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 2: ICISSP},
year={2025},
pages={125-132},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013090700003899},
isbn={978-989-758-735-1},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Information Systems Security and Privacy - Volume 2: ICISSP
TI - The Dual-Edged Sword: The Impact of Large Language Models in Network Infrastructure Security
SN - 978-989-758-735-1
AU - Debono D.
AU - Sare A.
PY - 2025
SP - 125
EP - 132
DO - 10.5220/0013090700003899
PB - SciTePress