Exploring the Impact of Knowledge Graphs on Zero-Shot Visual Object State Classification

Filippos Gouidis, Filippos Gouidis, Konstantinos Papoutsakis, Theodore Patkos, Antonis Argyros, Antonis Argyros, Dimitris Plexousakis, Dimitris Plexousakis

2024

Abstract

In this work, we explore the potential of Knowledge Graphs (KGs) towards an effective Zero-Shot Learning (ZSL) approach for Object State Classification (OSC) in images. For this problem, the performance of traditional supervised learning methods is hindered mainly by data scarcity, as they attempt to encode the highly varying visual features of a multitude of combinations of object state and object type classes (e.g. open bottle, folded newspaper). The ZSL paradigm does indicate a promising alternative to enable the classification of object state classes by leveraging structured semantic descriptions acquired by external commonsense knowledge sources. We formulate an effective ZS-OSC scheme by employing a Transformer-based Graph Neural Network model and a pre-trained CNN classifier. We also investigate best practices for both the construction and integration of visually-grounded common-sense information based on KGs. An extensive experimental evaluation is reported using 4 related image datasets, 5 different knowledge repositories and 30 KGs that are constructed semi-automatically via querying known object state classes to retrieve contextual information at different node depths. The performance of vision-language models for ZS-OSC is also assessed. Overall, the obtained results suggest performance improvement for ZS-OSC models on all datasets, while both the size of a KG and the sources utilized for their construction are important for task performance.

Paper Citation

in Harvard Style

Gouidis F., Papoutsakis K., Patkos T., Argyros A. and Plexousakis D. (2024). Exploring the Impact of Knowledge Graphs on Zero-Shot Visual Object State Classification. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 738-749. DOI: 10.5220/0012434800003660

in Bibtex Style

@conference{visapp24,
author={Filippos Gouidis and Konstantinos Papoutsakis and Theodore Patkos and Antonis Argyros and Dimitris Plexousakis},
title={Exploring the Impact of Knowledge Graphs on Zero-Shot Visual Object State Classification},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP},
year={2024},
pages={738-749},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012434800003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2: VISAPP
TI - Exploring the Impact of Knowledge Graphs on Zero-Shot Visual Object State Classification
SN - 978-989-758-679-8
AU - Gouidis F.
AU - Papoutsakis K.
AU - Patkos T.
AU - Argyros A.
AU - Plexousakis D.
PY - 2024
SP - 738
EP - 749
DO - 10.5220/0012434800003660
PB - SciTePress