Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens

Alessandro Farasin, Alessandro Farasin, Francesco Peciarolo, Marco Grangetto, Elena Gianaria, Paolo Garza

2020

Abstract

This paper presents a mixed reality system that, using the sensors mounted on the Microsoft Hololens headset and a cloud service, acquires and processes in real-time data to detect and track different kinds of objects and finally superimposes geographically coherent holographic texts on the detected objects. Such a goal has been achieved dealing with the intrinsic headset hardware limitations, by performing part of the overall computation in a edge/cloud environment. In particular, the heavier object detection algorithms, based on Deep Neural Networks (DNNs), are executed in the cloud. At the same time we compensate for cloud transmission and computation latencies by running light scene detection and object tracking on board the headset. The proposed pipeline allows meeting the real-time constraint by exploiting at the same time the power of state of art DNNs and the potential of Microsoft Hololens. This paper presents the design choices and describes the original algorithmic steps we devised to achieve real time tracking in mixed reality. Finally, the proposed system is experimentally validated.

Paper Citation

in Harvard Style

Farasin A., Peciarolo F., Grangetto M., Gianaria E. and Garza P. (2020). Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP; ISBN 978-989-758-402-2, SciTePress, pages 165-172. DOI: 10.5220/0008877901650172

in Bibtex Style

@conference{visapp20,
author={Alessandro Farasin and Francesco Peciarolo and Marco Grangetto and Elena Gianaria and Paolo Garza},
title={Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens},
booktitle={Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP},
year={2020},
pages={165-172},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008877901650172},
isbn={978-989-758-402-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP
TI - Real-time Object Detection and Tracking in Mixed Reality using Microsoft HoloLens
SN - 978-989-758-402-2
AU - Farasin A.
AU - Peciarolo F.
AU - Grangetto M.
AU - Gianaria E.
AU - Garza P.
PY - 2020
SP - 165
EP - 172
DO - 10.5220/0008877901650172
PB - SciTePress