Comparative Study of Large Language Models Applied to the Classification of Accountability Documents

Pedro Vinnícius Bernhard, João Dallyson Sousa de Almeida, Anselmo Cardoso de Paiva, Geraldo Braz Junior, Renan Coelho de Oliveira, Lúis Jorge Enrique Rivero Cabrejos, Darlan Quintanilha

2025

Abstract

Public account oversight is crucial, facilitated by electronic accountability systems. Through those systems, audited entities submit electronic documents related to government and management accounts, categorized according to regulatory guidelines. Accurate document classification is vital for adhering to court standards. Advanced technologies, including Large Language Models (LLMs), offer promise in optimizing this process. This study examines the use of LLMs to classify documents pertaining to annual accounts received by regulatory bodies. Three LLM models were examined: mBERT, XLM-RoBERTa and mT5. These LLMs were applied to a dataset of extracted texts specifically compiled for the research, based on documents provided by the Tribunal de Contas do Estado do Maranhao (TCE/MA), and evaluated based on the F1-score. The results ˜ strongly suggested that the XLM-RoBERTa model achieved an F1-score of 98.99% ±0.12%, while mBERT achieved 98.65% ± 0.29% and mT5 showed 98.71% ± 0.75%. These results highlight the effectiveness of LLMs in classifying accountability documents and contributing to advances in natural language processing. These approaches can potentially be exploited to improve automation and accuracy in document classifications.

Paper Citation

in Harvard Style

Bernhard P., Sousa de Almeida J., Cardoso de Paiva A., Braz Junior G., Coelho de Oliveira R., Cabrejos L. and Quintanilha D. (2025). Comparative Study of Large Language Models Applied to the Classification of Accountability Documents. In Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 1: ICEIS; ISBN 978-989-758-749-8, SciTePress, pages 944-951. DOI: 10.5220/0013439800003929

in Bibtex Style

@conference{iceis25,
author={Pedro Bernhard and João Sousa de Almeida and Anselmo Cardoso de Paiva and Geraldo Braz Junior and Renan Coelho de Oliveira and Lúis Cabrejos and Darlan Quintanilha},
title={Comparative Study of Large Language Models Applied to the Classification of Accountability Documents},
booktitle={Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 1: ICEIS},
year={2025},
pages={944-951},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013439800003929},
isbn={978-989-758-749-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 27th International Conference on Enterprise Information Systems - Volume 1: ICEIS
TI - Comparative Study of Large Language Models Applied to the Classification of Accountability Documents
SN - 978-989-758-749-8
AU - Bernhard P.
AU - Sousa de Almeida J.
AU - Cardoso de Paiva A.
AU - Braz Junior G.
AU - Coelho de Oliveira R.
AU - Cabrejos L.
AU - Quintanilha D.
PY - 2025
SP - 944
EP - 951
DO - 10.5220/0013439800003929
PB - SciTePress