An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings

Jin Chen, Satesh Ramnath, Tyron Samaroo, Fani Maksakuli, Arber Ruci, E’edresha Sturdivant, Zhigang Zhu, Zhigang Zhu

2023

Abstract

This paper presents a mobile-based solution that integrates 3D vision and voice interaction to assist people who are blind or have low vision to explore and interact with their surroundings. The key components of the system are the two 3D vision modules: the 3D object detection module integrates a deep-learning based 2D object detector with ARKit-based point cloud generation, and an interest direction recognition module integrates hand/finger recognition and ARKit-based 3D direction estimation. The integrated system consists of a voice interface, a task scheduler, and an instruction generator. The voice interface contains a customized user request mapping module that maps the user’s input voice into one of the four primary system operation modes (exploration, search, navigation, and settings adjustment). The task scheduler coordinates with two web services that host the two vision modules to allocate resources for computation based on the user request and network connectivity strength. Finally, the instruction generator computes the corresponding instructions based on the user request and results from the two vision modules. The system is capable of running in real time on mobile devices. We have shown preliminary experimental results on the performance of the voice to user request mapping module and the two vision modules.

Paper Citation

in Harvard Style

Chen J., Ramnath S., Samaroo T., Maksakuli F., Ruci A., Sturdivant E. and Zhu Z. (2023). An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings. In Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE, ISBN 978-989-758-642-2, SciTePress, pages 180-187. DOI: 10.5220/0011984400003497

in Bibtex Style

@conference{improve23,
author={Jin Chen and Satesh Ramnath and Tyron Samaroo and Fani Maksakuli and Arber Ruci and E’edresha Sturdivant and Zhigang Zhu},
title={An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings},
booktitle={Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE,},
year={2023},
pages={180-187},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011984400003497},
isbn={978-989-758-642-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE,
TI - An Integrated Mobile Vision System for Enhancing the Interaction of Blind and Low Vision Users with Their Surroundings
SN - 978-989-758-642-2
AU - Chen J.
AU - Ramnath S.
AU - Samaroo T.
AU - Maksakuli F.
AU - Ruci A.
AU - Sturdivant E.
AU - Zhu Z.
PY - 2023
SP - 180
EP - 187
DO - 10.5220/0011984400003497
PB - SciTePress