AI-Informed Interactive Task Guidance in Augmented Reality

Viacheslav Tekaev, Raffaele de Amicis

2025

Abstract

This paper presents a proof of concept for an augmented reality (AR) and artificial intelligence (AI)-powered task guidance system, demonstrated through the task of opening a door handle. The system integrates an AR frontend, deployed on an Oculus Quest Pro, with an AI backend that combines computer vision for real-time object detection and tracking, and natural language processing (NLP) for dynamic user interaction. Objects such as door handles are identified using YOLOv8-seg, and their 3D positions are calculated to align with the user’s environment, ensuring accurate task guidance. The AI backend supports local and cloud processing, maintaining performance even without internet connectivity. The system provides adaptive feedback, adjusting guidance based on user actions, such as correcting improper rotation of a knob. Real-time communication between components is achieved via WebSocket, minimizing latency. Technical challenges like tracking accuracy, latency, and synchronization are addressed through calibration and stress testing under vary-ing conditions. The study emphasizes the system’s adaptability to complex scenarios, offering error-handling mechanisms and smooth interaction through AR overlays. This proof of concept highlights the potential of AR-AI integration for task guidance in diverse applications.

Paper Citation

in Harvard Style

Tekaev V. and de Amicis R. (2025). AI-Informed Interactive Task Guidance in Augmented Reality. In Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP; ISBN 978-989-758-728-3, SciTePress, pages 101-112. DOI: 10.5220/0013187800003912

in Bibtex Style

@conference{grapp25,
author={Viacheslav Tekaev and Raffaele de Amicis},
title={AI-Informed Interactive Task Guidance in Augmented Reality},
booktitle={Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP},
year={2025},
pages={101-112},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013187800003912},
isbn={978-989-758-728-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 20th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP
TI - AI-Informed Interactive Task Guidance in Augmented Reality
SN - 978-989-758-728-3
AU - Tekaev V.
AU - de Amicis R.
PY - 2025
SP - 101
EP - 112
DO - 10.5220/0013187800003912
PB - SciTePress