MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification

Laura Fieback, Laura Fieback, Jakob Spiegelberg, Hanno Gottschalk

2025

Abstract

Large Vision Language Models (LVLMs) have shown remarkable capabilities in multimodal tasks like visual question answering or image captioning. However, inconsistencies between the visual information and the generated text, a phenomenon referred to as hallucinations, remain an unsolved problem with regard to the trustworthiness of LVLMs. To address this problem, recent works proposed to incorporate computationally costly Large (Vision) Language Models in order to detect hallucinations on a sentence- or subsentence-level. In this work, we introduce MetaToken, a lightweight binary classifier to detect hallucinations on token-level at negligible cost. Based on a statistical analysis, we reveal key factors of hallucinations in LVLMs. MetaToken can be applied to any open-source LVLM without any knowledge about ground truth data providing a calibrated detection of hallucinations. We evaluate our method on four state-of-the-art LVLMs outperforming baseline methods by up to 46.50pp in terms of area under precision recall curve values.

Paper Citation

in Harvard Style

Fieback L., Spiegelberg J. and Gottschalk H. (2025). MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification. In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-728-3, SciTePress, pages 126-137. DOI: 10.5220/0013165700003912

in Bibtex Style

@conference{visapp25,
author={Laura Fieback and Jakob Spiegelberg and Hanno Gottschalk},
title={MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification},
booktitle={Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2025},
pages={126-137},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013165700003912},
isbn={978-989-758-728-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification
SN - 978-989-758-728-3
AU - Fieback L.
AU - Spiegelberg J.
AU - Gottschalk H.
PY - 2025
SP - 126
EP - 137
DO - 10.5220/0013165700003912
PB - SciTePress