Improving Image Classification Tasks Using Fused Embeddings and Multimodal Models

Artur Oliveira, Mateus Espadoto, Roberto Hirata Jr., Roberto M. Cesar Jr.

2025

Abstract

In this paper, we address the challenge of flexible and scalable image classification by leveraging CLIP embeddings, a pre-trained multimodal model. Our novel strategy uses tailored textual prompts (e.g., “This is digit 9”, “This is even/odd”) to generate and fuse embeddings from both images and prompts, followed by clustering for classification. We present a prompt-guided embedding strategy that dynamically aligns multimodal representations to task-specific or grouped semantics, enhancing the utility of models like CLIP in clustering and constrained classification workflows. Additionally, we evaluate the embedding structures through clustering, classification, and t-SNE visualization, demonstrating the impact of prompts on embedding space separability and alignment. Our findings underscore CLIP’s potential for flexible and scalable image classification, supporting zero-shot scenarios without the need for retraining.

Paper Citation

in Harvard Style

Oliveira A., Espadoto M., Hirata Jr. R. and Cesar Jr. R. (2025). Improving Image Classification Tasks Using Fused Embeddings and Multimodal Models. In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-728-3, SciTePress, pages 232-241. DOI: 10.5220/0013365600003912

in Bibtex Style

@conference{visapp25,
author={Artur Oliveira and Mateus Espadoto and Roberto Hirata Jr. and Roberto Cesar Jr.},
title={Improving Image Classification Tasks Using Fused Embeddings and Multimodal Models},
booktitle={Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2025},
pages={232-241},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013365600003912},
isbn={978-989-758-728-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - Improving Image Classification Tasks Using Fused Embeddings and Multimodal Models
SN - 978-989-758-728-3
AU - Oliveira A.
AU - Espadoto M.
AU - Hirata Jr. R.
AU - Cesar Jr. R.
PY - 2025
SP - 232
EP - 241
DO - 10.5220/0013365600003912
PB - SciTePress