Combining Text and Image Knowledge with GANs for Zero-Shot Action Recognition in Videos

Kaiqiang Huang, Luis Miralles-Pechuán, Susan Mckeever

2022

Abstract

The recognition of actions in videos is an active research area in machine learning, relevant to multiple domains such as health monitoring, security and social media analysis. Zero-Shot Action Recognition (ZSAR) is a challenging problem in which models are trained to identify action classes that have not been seen during the training process. According to the literature, the most promising ZSAR approaches make use of Generative Adversarial Networks (GANs). GANs can synthesise visual embeddings for unseen classes conditioned on either textual information or images related to the class labels. In this paper, we propose a Dual-GAN approach based on the VAEGAN model to prove that the fusion of visual and textual-based knowledge sources is an effective way to improve ZSAR performance. We conduct empirical ZSAR experiments of our approach on the UCF101 dataset. We apply the following embedding fusion methods for combining text-driven and image-driven information: averaging, summation, maximum, and minimum. Our best result from Dual-GAN model is achieved with the maximum embedding fusion approach that results in an average accuracy of 46.37%, which is improved by 5.37% at least compared to the leading approaches.

Paper Citation

in Harvard Style

Huang K., Miralles-Pechuán L. and Mckeever S. (2022). Combining Text and Image Knowledge with GANs for Zero-Shot Action Recognition in Videos. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP; ISBN 978-989-758-555-5, SciTePress, pages 623-631. DOI: 10.5220/0010903100003124

in Bibtex Style

@conference{visapp22,
author={Kaiqiang Huang and Luis Miralles-Pechuán and Susan Mckeever},
title={Combining Text and Image Knowledge with GANs for Zero-Shot Action Recognition in Videos},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP},
year={2022},
pages={623-631},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010903100003124},
isbn={978-989-758-555-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP
TI - Combining Text and Image Knowledge with GANs for Zero-Shot Action Recognition in Videos
SN - 978-989-758-555-5
AU - Huang K.
AU - Miralles-Pechuán L.
AU - Mckeever S.
PY - 2022
SP - 623
EP - 631
DO - 10.5220/0010903100003124
PB - SciTePress