A Study on Vulnerability Explanation Using Large Language Models

Lucas B. Germano, Julio Cesar Duarte

2025

Abstract

In the quickly advancing field of software development, addressing vulnerabilities with robust security measures is essential. While much research has focused on vulnerability detection using Large Language Models (LLMs), limited attention has been given to generating actionable explanations. This study explores the capability of LLMs to explain vulnerabilities in Java code, structuring outputs into four dimensions: why the vulnerability exists, its dangers, how it can be exploited, and mitigation recommendations. In this context, smaller LLMs struggled to produce outputs in the required JSON format, with CodeGeeX4 showing high semantic similarity to GPT-4o but generating many incorrect formats. CodeLlama 34B emerged as the best overall performer, balancing output quality and formatting consistency. Despite these findings, comparisons with the GPT-4o baseline revealed no significant differences to rank the models effectively. Human evaluation further revealed that all models, including GPT-4o, struggled to adequately explain complex vulnerabilities, underscoring the challenges in achieving comprehensive explanations.

Paper Citation

in Harvard Style

Germano L. and Duarte J. (2025). A Study on Vulnerability Explanation Using Large Language Models. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 1404-1411. DOI: 10.5220/0013379200003890

in Bibtex Style

@conference{icaart25,
author={Lucas Germano and Julio Duarte},
title={A Study on Vulnerability Explanation Using Large Language Models},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={1404-1411},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013379200003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - A Study on Vulnerability Explanation Using Large Language Models
SN - 978-989-758-737-5
AU - Germano L.
AU - Duarte J.
PY - 2025
SP - 1404
EP - 1411
DO - 10.5220/0013379200003890
PB - SciTePress