Document Analysis with LLMs: Assessing Performance, Bias, and Nondeterminism in Decision Making

Stephen Price, Danielle L. Cote

2025

Abstract

In recent years, large language models (LLMs) have demonstrated their ability to perform complex tasks such as data summarization, translation, document analysis, and content generation. However, their reliability and efficacy in real-world scenarios must be studied. This work presents an experimental evaluation of an LLM for document analysis and candidate recommendation using a set of resumes. Llama3.1, a state-of-the-art open-source model, was tested with 30 questions using data from five resumes. On tasks with a direct answer, Llama3.1 achieved an accuracy of 99.56%. However, on more open-ended and ambiguous questions, performance, and reliability decreased, revealing limitations such as bias toward particular experience, primacy bias, nondeterminism, and sensitivity to question phrasing.

Paper Citation

in Harvard Style

Price S. and Cote D. (2025). Document Analysis with LLMs: Assessing Performance, Bias, and Nondeterminism in Decision Making. In Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-730-6, SciTePress, pages 207-214. DOI: 10.5220/0013094300003905

in Bibtex Style

@conference{icpram25,
author={Stephen Price and Danielle Cote},
title={Document Analysis with LLMs: Assessing Performance, Bias, and Nondeterminism in Decision Making},
booktitle={Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2025},
pages={207-214},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013094300003905},
isbn={978-989-758-730-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Document Analysis with LLMs: Assessing Performance, Bias, and Nondeterminism in Decision Making
SN - 978-989-758-730-6
AU - Price S.
AU - Cote D.
PY - 2025
SP - 207
EP - 214
DO - 10.5220/0013094300003905
PB - SciTePress