Object Detection Featuring 3D Audio Localization for Microsoft HoloLens - A Deep Learning based Sensor Substitution Approach for the Blind

Martin Eckert, Matthias Blex, Christoph M. Friedrich

2018

Abstract

Finding basic objects on a daily basis is a difficult but common task for blind people. This paper demonstrates the implementation of a wearable, deep learning backed, object detection approach in the context of visual impairment or blindness. The prototype aims to substitute the impaired eye of the user and replace it with technical sensors. By scanning its surroundings, the prototype provides a situational overview of objects around the device. Object detection has been implemented using a near real-time, deep learning model named YOLOv2. The model supports the detection of 9000 objects. The prototype can display and read out the name of augmented objects which can be selected by voice commands and used as directional guides for the user, using 3D audio feedback. A distance announcement of a selected object is derived from the HoloLens’s spatial model. The wearable solution offers the opportunity to efficiently locate objects to support orientation without extensive training of the user. Preliminary evaluation covered the detection rate of speech recognition and the response times of the server.

Paper Citation

in Harvard Style

Eckert M., Blex M. and Friedrich C. (2018). Object Detection Featuring 3D Audio Localization for Microsoft HoloLens - A Deep Learning based Sensor Substitution Approach for the Blind. In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 5: HEALTHINF; ISBN 978-989-758-281-3, SciTePress, pages 555-561. DOI: 10.5220/0006655605550561

in Bibtex Style

@conference{healthinf18,
author={Martin Eckert and Matthias Blex and Christoph M. Friedrich},
title={Object Detection Featuring 3D Audio Localization for Microsoft HoloLens - A Deep Learning based Sensor Substitution Approach for the Blind},
booktitle={Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 5: HEALTHINF},
year={2018},
pages={555-561},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006655605550561},
isbn={978-989-758-281-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 5: HEALTHINF
TI - Object Detection Featuring 3D Audio Localization for Microsoft HoloLens - A Deep Learning based Sensor Substitution Approach for the Blind
SN - 978-989-758-281-3
AU - Eckert M.
AU - Blex M.
AU - Friedrich C.
PY - 2018
SP - 555
EP - 561
DO - 10.5220/0006655605550561
PB - SciTePress