GOLLUM: Guiding cOnfiguration of firewaLL Through aUgmented Large Language Models

Roberto Lorusso, Antonio Maci, Antonio Coscia

2025

Abstract

Artificial intelligence (AI) tools offer significant potential in network security, particularly for addressing issues like firewall misconfiguration, which can lead to security flaws. Configuration support services can help prevent errors by providing clear general-purpose language instructions, thus minimizing the need for manual references. Large language models (LLMs) are AI-based agents that use deep neural networks to understand and generate human language. However, LLMs are generalists by construction and may lack the knowledge needed in specific fields, thereby requiring links to external sources to perform highly specialized tasks. To meet these needs, this paper proposes GOLLUM, a conversational agent designed to guide firewall configurations using augmented LLMs. GOLLUM integrates the pfSense firewall documentation via a retrieval augmented generation approach, providing an example of actual use. The generative models used in GOLLUM were selected based on their performance on the state-of-the-art NetConfEval and CyberMetric datasets. Additionally, to assess the effectiveness of the proposed application, an automated evaluation pipeline, involving RAGAS as test dataset generator and a panel of LLMs for judgment, was implemented. The experimental results indicate that GOLLUM, powered by LLama3-8B, provides accurate and faithful support in three out of four cases, while achieving > 80% of answer correctness in configuration-related queries.

Paper Citation

in Harvard Style

Lorusso R., Maci A. and Coscia A. (2025). GOLLUM: Guiding cOnfiguration of firewaLL Through aUgmented Large Language Models. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 489-496. DOI: 10.5220/0013221900003890

in Bibtex Style

@conference{icaart25,
author={Roberto Lorusso and Antonio Maci and Antonio Coscia},
title={GOLLUM: Guiding cOnfiguration of firewaLL Through aUgmented Large Language Models},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART},
year={2025},
pages={489-496},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013221900003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART
TI - GOLLUM: Guiding cOnfiguration of firewaLL Through aUgmented Large Language Models
SN - 978-989-758-737-5
AU - Lorusso R.
AU - Maci A.
AU - Coscia A.
PY - 2025
SP - 489
EP - 496
DO - 10.5220/0013221900003890
PB - SciTePress