Pervasive Hand Gesture Recognition for Smartphones using Non-audible Sound and Deep Learning

Ahmed Ibrahim, Ayman El-Refai, Sara Ahmed, Mariam Aboul-Ela, Hesham Eraqi, Mohamed Moustafa

2021

Abstract

Due to the mass advancement in ubiquitous technologies nowadays, new pervasive methods have come into the practice to provide new innovative features and stimulate the research on new human-computer interactions. This paper presents a hand gesture recognition method that utilizes the smartphone’s built-in speakers and microphones. The proposed system emits an ultrasonic sonar-based signal (inaudible sound) from the smartphone’s stereo speakers, which is then received by the smartphone’s microphone and processed via a Convolutional Neural Network (CNN) for Hand Gesture Recognition. Data augmentation techniques are proposed to improve the detection accuracy and three dual-channel input fusion methods are compared. The first method merges the dual-channel audio as a single input spectrogram image. The second method adopts early fusion by concatenating the dual-channel spectrograms. The third method adopts late fusion by having two convectional input branches processing each of the dual-channel spectrograms and then the outputs are merged by the last layers. Our experimental results demonstrate a promising detection accuracy for the six gestures presented in our publicly available dataset with an accuracy of 93.58% as a baseline.

Paper Citation

in Harvard Style

Ibrahim A., El-Refai A., Ahmed S., Aboul-Ela M., Eraqi H. and Moustafa M. (2021). Pervasive Hand Gesture Recognition for Smartphones using Non-audible Sound and Deep Learning. In Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA; ISBN 978-989-758-534-0, SciTePress, pages 310-317. DOI: 10.5220/0010656200003063

in Bibtex Style

@conference{ncta21,
author={Ahmed Ibrahim and Ayman El-Refai and Sara Ahmed and Mariam Aboul-Ela and Hesham Eraqi and Mohamed Moustafa},
title={Pervasive Hand Gesture Recognition for Smartphones using Non-audible Sound and Deep Learning},
booktitle={Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA},
year={2021},
pages={310-317},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010656200003063},
isbn={978-989-758-534-0},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Joint Conference on Computational Intelligence (IJCCI 2021) - Volume 1: NCTA
TI - Pervasive Hand Gesture Recognition for Smartphones using Non-audible Sound and Deep Learning
SN - 978-989-758-534-0
AU - Ibrahim A.
AU - El-Refai A.
AU - Ahmed S.
AU - Aboul-Ela M.
AU - Eraqi H.
AU - Moustafa M.
PY - 2021
SP - 310
EP - 317
DO - 10.5220/0010656200003063
PB - SciTePress