Zero-Shot Action Recognition with Knowledge Enhanced Generative Adversarial Networks

Kaiqiang Huang, Luis Miralles-Pechuán, Susan Mckeever

2021

Abstract

Zero-Shot Action Recognition (ZSAR) aims to recognise action classes in videos that have never been seen during model training. In some approaches, ZSAR has been achieved by generating visual features for unseen classes based on the semantic information of the unseen class labels using generative adversarial networks (GANs). Therefore, the problem is converted to standard supervised learning since the unseen visual features are accessible. This approach alleviates the lack of labelled samples of unseen classes. In addition, objects appearing in the action instances could be used to create enriched semantics of action classes and therefore, increase the accuracy of ZSAR. In this paper, we consider using, in addition to the label, objects related to that action label. For example, the objects ‘horse’ and ‘saddle’ are highly related to the action ‘Horse Riding’ and these objects can bring additional semantic meaning. In this work, we aim to improve the GAN-based framework by incorporating object-based semantic information related to the class label with three approaches: replacing the class labels with objects, appending objects to the class, and averaging objects with the class. Then, we evaluate the performance using a subset of the popular dataset UCF101. Our experimental results demonstrate that our approach is valid since when including appropriate objects into the action classes, the baseline is improved by 4.93%.

Paper Citation

in Harvard Style

Huang K., Miralles-Pechuán L. and Mckeever S. (2021). Zero-Shot Action Recognition with Knowledge Enhanced Generative Adversarial Networks. In Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA; ISBN 978-989-758-534-0, SciTePress, pages 254-264. DOI: 10.5220/0010717000003063

in Bibtex Style

@conference{ncta21,
author={Kaiqiang Huang and Luis Miralles-Pechuán and Susan Mckeever},
title={Zero-Shot Action Recognition with Knowledge Enhanced Generative Adversarial Networks},
booktitle={Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA},
year={2021},
pages={254-264},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010717000003063},
isbn={978-989-758-534-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA
TI - Zero-Shot Action Recognition with Knowledge Enhanced Generative Adversarial Networks
SN - 978-989-758-534-0
AU - Huang K.
AU - Miralles-Pechuán L.
AU - Mckeever S.
PY - 2021
SP - 254
EP - 264
DO - 10.5220/0010717000003063
PB - SciTePress