Temporal Bilinear Encoding Network of Audio-visual Features at Low Sampling Rates

Feiyan Hu, Eva Mohedano, Noel O’Connor, Kevin Mcguinness

2021

Abstract

Current deep learning based video classification architectures are typically trained end-to-end on large volumes of data and require extensive computational resources. This paper aims to exploit audio-visual information in video classification with a 1 frame per second sampling rate. We propose Temporal Bilinear Encoding Networks (TBEN) for encoding both audio and visual long range temporal information using bilinear pooling and demonstrate bilinear pooling is better than average pooling on the temporal dimension for videos with low sampling rate. We also embed the label hierarchy in TBEN to further improve the robustness of the classifier. Experiments on the FGA240 fine-grained classification dataset using TBEN achieve a new state-of-the-art (hit@1=47.95%). We also exploit the possibility of incorporating TBEN with multiple decoupled modalities like visual semantic and motion features: experiments on UCF101 sampled at 1 FPS achieve close to state-of-the-art accuracy (hit@1=91.03%) while requiring significantly less computational resources than competing approaches for both training and prediction.

Paper Citation

in Harvard Style

Hu F., Mohedano E., O’Connor N. and Mcguinness K. (2021). Temporal Bilinear Encoding Network of Audio-visual Features at Low Sampling Rates. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 637-644. DOI: 10.5220/0010337306370644

in Bibtex Style

@conference{visapp21,
author={Feiyan Hu and Eva Mohedano and Noel O’Connor and Kevin Mcguinness},
title={Temporal Bilinear Encoding Network of Audio-visual Features at Low Sampling Rates},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP},
year={2021},
pages={637-644},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010337306370644},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 5: VISAPP
TI - Temporal Bilinear Encoding Network of Audio-visual Features at Low Sampling Rates
SN - 978-989-758-488-6
AU - Hu F.
AU - Mohedano E.
AU - O’Connor N.
AU - Mcguinness K.
PY - 2021
SP - 637
EP - 644
DO - 10.5220/0010337306370644
PB - SciTePress