The VVAD-LRS3 Dataset for Visual Voice Activity Detection

Adrian Lubitz, Matias Valdenegro-Toro, Frank Kirchner, Frank Kirchner

2023

Abstract

Robots are becoming everyday devices, increasing their interaction with humans. To make human-machine interaction more natural, cognitive features like Visual Voice Activity Detection (VVAD), which can detect whether a person is speaking or not, given visual input of a camera, need to be implemented. Neural networks are state of the art for tasks in Image Processing, Time Series Prediction, Natural Language Processing and other domains. Those Networks require large quantities of labeled data. Currently there are not many datasets for the task of VVAD. In this work we created a large scale dataset called the VVAD-LRS3 dataset, derived by automatic annotations from the LRS3 dataset. The VVAD-LRS3 dataset contains over 44K samples, over three times the next competitive dataset (WildVVAD). We evaluate different baselines on four kinds of features: facial and lip images, and facial and lip landmark features. With a Convolutional Neural Network Long Short Term Memory (CNN LSTM) on facial images an accuracy of 92% was reached on the test set. A study with humans showed that they reach an accuracy of 87.93% on the test set.

Paper Citation

in Harvard Style

Lubitz A., Valdenegro-Toro M. and Kirchner F. (2023). The VVAD-LRS3 Dataset for Visual Voice Activity Detection. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 2: HUCAPP; ISBN 978-989-758-634-7, SciTePress, pages 39-46. DOI: 10.5220/0011612900003417

in Bibtex Style

@conference{hucapp23,
author={Adrian Lubitz and Matias Valdenegro-Toro and Frank Kirchner},
title={The VVAD-LRS3 Dataset for Visual Voice Activity Detection},
booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 2: HUCAPP},
year={2023},
pages={39-46},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011612900003417},
isbn={978-989-758-634-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 2: HUCAPP
TI - The VVAD-LRS3 Dataset for Visual Voice Activity Detection
SN - 978-989-758-634-7
AU - Lubitz A.
AU - Valdenegro-Toro M.
AU - Kirchner F.
PY - 2023
SP - 39
EP - 46
DO - 10.5220/0011612900003417
PB - SciTePress