Classification and Direction Detection of Ambient Sounds on Microsoft HoloLens to Support Hearing-impaired People

Beauclair Dongmo Ngnintedem, Eric Mense, Johannes Rückert, Christoph M. Friedrich, Christoph M. Friedrich

2022

Abstract

Hearing-impaired people are exposed to greater dangers in everyday life, due to the fact that they are not able to perceive danger and warning signals. This paper addresses this problem by developing an application, that could help by classifying and detecting the direction of ambient sounds using Microsoft HoloLens 2 devices. The developed application implements a client-server architecture. The server-side REST-API supports not only the classification of sounds from audio files via deep-learning methods, but also allows the results of the sound source localization to be saved and read. The sound source localization is performed by a Maix Bit microcontroller with a 6-channel microphone array. For the user integration and interaction with the application, a 3D scene has been designed using Unity and the Mixed Reality Toolkit (MRTK). The implemented application showcases how classification and direction detection of ambient sounds could be done on the Microsoft HoloLens to support hearing-impaired people.

Paper Citation

in Harvard Style

Ngnintedem B., Mense E., Rückert J. and Friedrich C. (2022). Classification and Direction Detection of Ambient Sounds on Microsoft HoloLens to Support Hearing-impaired People. In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: WHC, ISBN 978-989-758-552-4, pages 857-863. DOI: 10.5220/0010973500003123

in Bibtex Style

@conference{whc22,
author={Beauclair Ngnintedem and Eric Mense and Johannes Rückert and Christoph Friedrich},
title={Classification and Direction Detection of Ambient Sounds on Microsoft HoloLens to Support Hearing-impaired People},
booktitle={Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: WHC,},
year={2022},
pages={857-863},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010973500003123},
isbn={978-989-758-552-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 4: WHC,
TI - Classification and Direction Detection of Ambient Sounds on Microsoft HoloLens to Support Hearing-impaired People
SN - 978-989-758-552-4
AU - Ngnintedem B.
AU - Mense E.
AU - Rückert J.
AU - Friedrich C.
PY - 2022
SP - 857
EP - 863
DO - 10.5220/0010973500003123