Augmenting Human-Robot Collaboration Task by Human Hand Position Forecasting

Shyngyskhan Abilkassov, Michael Gentner, Michael Gentner, Mirela Popa

2024

Abstract

Human-Robot collaboration (HRC) plays a critical role in enhancing productivity and safety across various industries. While reactive motion re-planning strategies have proven useful, there is a pressing need for proactive control involving computing human intentions to enable efficient collaboration. This work addresses this challenge by proposing a deep learning-based approach for forecasting human hand trajectories and a heuristic optimization algorithm for proactive robotic task sequencing problem optimization. This work presents a human hand trajectory forecasting deep learning model that achieves state-of-the-art performance on the Ego4D Future Hand Prediction benchmark in all evaluation metrics. In addition, this work presents a problem formulation and a Dynamic Variable Neighborhood Search (DynamicVNS) heuristic optimization algorithm enabling robot to pre-plan their task sequence to avoid human hands. The proposed algorithm exhibits significant computational improvements over the generalized VNS approach. The final framework efficiently incorporates predictions made by the deep learning model into the task sequencer, which is evaluated in an experimental setup for the HRC use-case of the UR10e robot in a visual inspection task. The results indicate the effectiveness and practicality of the proposed approach, showcasing its potential to improve human-robot collaboration in various industrial settings.

Paper Citation

in Harvard Style

Abilkassov S., Gentner M. and Popa M. (2024). Augmenting Human-Robot Collaboration Task by Human Hand Position Forecasting. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 262-269. DOI: 10.5220/0012306300003660

in Bibtex Style

@conference{visapp24,
author={Shyngyskhan Abilkassov and Michael Gentner and Mirela Popa},
title={Augmenting Human-Robot Collaboration Task by Human Hand Position Forecasting},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP},
year={2024},
pages={262-269},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012306300003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP
TI - Augmenting Human-Robot Collaboration Task by Human Hand Position Forecasting
SN - 978-989-758-679-8
AU - Abilkassov S.
AU - Gentner M.
AU - Popa M.
PY - 2024
SP - 262
EP - 269
DO - 10.5220/0012306300003660
PB - SciTePress