Gesture Recognition on a New Multi-Modal Hand Gesture Dataset

Monika Schak, Alexander Gepperth

2022

Abstract

We present a new large-scale multi-modal dataset for free-hand gesture recognition. The freely available dataset consists of 79,881 sequences, grouped into six classes representing typical hand gestures in human-machine interaction. Each sample contains four independent modalities (arriving at different frequencies) recorded from two independent sensors: a fixed 3D camera for video, audio and 3D, and a wearable acceleration sensor attached to the wrist. The gesture classes are specifically chosen with investigations on multi-modal fusion in mind. For example, two gesture classes can be distinguished mainly by audio, while the four others are not exhibiting audio signals – besides white noise. An important point concerning this dataset is that it is recorded from a single person. While this reduces variability somewhat, it virtually eliminates the risk of incorrectly performed gestures, thus enhancing the quality of the data. By implementing a simple LSTM-based gesture classifier in a live system, we can demonstrate that generalization to other persons is nevertheless high. In addition, we show the validity and internal consistency of the data by training LSTM and DNN classifiers relying on a single modality to high precision.

Paper Citation

in Harvard Style

Schak M. and Gepperth A. (2022). Gesture Recognition on a New Multi-Modal Hand Gesture Dataset. In Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-549-4, pages 122-131. DOI: 10.5220/0010982200003122

in Bibtex Style

@conference{icpram22,
author={Monika Schak and Alexander Gepperth},
title={Gesture Recognition on a New Multi-Modal Hand Gesture Dataset},
booktitle={Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2022},
pages={122-131},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010982200003122},
isbn={978-989-758-549-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Gesture Recognition on a New Multi-Modal Hand Gesture Dataset
SN - 978-989-758-549-4
AU - Schak M.
AU - Gepperth A.
PY - 2022
SP - 122
EP - 131
DO - 10.5220/0010982200003122