Few-shot Linguistic Grounding of Visual Attributes and Relations using Gaussian Kernels

Daniel Koudouna, Kasim Terzić

2021

Abstract

Understanding complex visual scenes is one of fundamental problems in computer vision, but learning in this domain is challenging due to the inherent richness of the visual world and the vast number of possible scene configurations. Current state of the art approaches to scene understanding often employ deep networks which require large and densely annotated datasets. This goes against the seemingly intuitive learning abilities of humans and our ability to generalise from few examples to unseen situations. In this paper, we propose a unified framework for learning visual representation of words denoting attributes such as “blue” and relations such as “left of” based on Gaussian models operating in a simple, unified feature space. The strength of our model is that it only requires a small number of weak annotations and is able to generalize easily to unseen situations such as recognizing object relations in unusual configurations. We demonstrate the effectiveness of our model on the predicate detection task. Our model is able to outperform the state of the art on this task in both the normal and zero-shot scenarios, while training on a dataset an order of magnitude smaller.

Paper Citation

in Harvard Style

Koudouna D. and Terzić K. (2021). Few-shot Linguistic Grounding of Visual Attributes and Relations using Gaussian Kernels. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 146-156. DOI: 10.5220/0010261301460156

in Bibtex Style

@conference{visapp21,
author={Daniel Koudouna and Kasim Terzić},
title={Few-shot Linguistic Grounding of Visual Attributes and Relations using Gaussian Kernels},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP},
year={2021},
pages={146-156},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010261301460156},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP
TI - Few-shot Linguistic Grounding of Visual Attributes and Relations using Gaussian Kernels
SN - 978-989-758-488-6
AU - Koudouna D.
AU - Terzić K.
PY - 2021
SP - 146
EP - 156
DO - 10.5220/0010261301460156
PB - SciTePress