A Framework for Identifying Underspecification in Image Classification Pipeline Using Post-Hoc Analyzer

Prabhat Parajuli, Teeradaj Racharak

2025

Abstract

Underspecification — a failure mode where multiple models perform well during development but fail to generalize to new, unseen data due to reliance on insufficient or spurious features — remains critical challenge in machine learning (ML) and deep learning (DL). In this paper, we focus on a specific aspect of underspecification: the inconsistency in feature learning. We hypothesize that models with similar performance should exhibit consistent behavior in terms of feature reliance. However, in practice — especially in deep learning —this consistency is often lacking due to various factors influencing the learning process. To uncover where this inconsistency occurs, we propose a framework leveraging XAI techniques (specifically LIME and SHAP) to identify underspecification by analyzing inconsistencies across ML pipeline components, including feature extractors, optimizers, and weight initialization. Experiments on MNIST, Imagenette, and Cats vs Dogs reveal significant variability in feature reliance, particularly due to the choice of feature extractor. This variability highlights how different factors contribute to the learning of varied features, ultimately leading to potential underspecification. While this study focuses on the impact of specific pipeline components, our framework can be extended to analyze other factors contributing to underspecification in machine learning systems.

Paper Citation

in Harvard Style

Parajuli P. and Racharak T. (2025). A Framework for Identifying Underspecification in Image Classification Pipeline Using Post-Hoc Analyzer. In Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-730-6, SciTePress, pages 426-436. DOI: 10.5220/0013374200003905

in Bibtex Style

@conference{icpram25,
author={Prabhat Parajuli and Teeradaj Racharak},
title={A Framework for Identifying Underspecification in Image Classification Pipeline Using Post-Hoc Analyzer},
booktitle={Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2025},
pages={426-436},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013374200003905},
isbn={978-989-758-730-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - A Framework for Identifying Underspecification in Image Classification Pipeline Using Post-Hoc Analyzer
SN - 978-989-758-730-6
AU - Parajuli P.
AU - Racharak T.
PY - 2025
SP - 426
EP - 436
DO - 10.5220/0013374200003905
PB - SciTePress