Towards Multi-View Hand Pose Recognition Using a Fusion of Image Embeddings and Leap 2 Landmarks

Sergio Esteban-Romero, Romeo Lanzino, Marco Raoul Marini, Manuel Gil-Martín

2025

Abstract

This paper presents a novel approach for multi-view hand pose recognition through image embeddings and hand landmarks. The method integrates raw image data with structural hand landmarks derived from the Leap Motion Controller 2. A Vision Transformer (ViT) pretrained model was used to extract visual features from dual-view grayscale images, which were fused with the corresponding Leap 2 hand landmarks, creating a multimodal representation that encapsulates both visual and landmark data for each sample. These fused embeddings were then classified using a multi-layer perceptron to distinguish among 17 distinct hand poses from the Multi-view Leap2 Hand Pose Dataset, which includes data from 21 subjects. Using a Leave-OneSubject-Out Cross-Validation (LOSO-CV) strategy, we demonstrate that this fusion approach offers a robust recognition performance (F1 Score of 79.33 ± 0.09 %), particularly in scenarios where hand occlusions or challenging angles may limit the utility of single-modality data.

Paper Citation

in Harvard Style

Esteban-Romero S., Lanzino R., Marini M. and Gil-Martín M. (2025). Towards Multi-View Hand Pose Recognition Using a Fusion of Image Embeddings and Leap 2 Landmarks. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 918-925. DOI: 10.5220/0013234300003890

in Bibtex Style

@conference{icaart25,
author={Sergio Esteban-Romero and Romeo Lanzino and Marco Marini and Manuel Gil-Martín},
title={Towards Multi-View Hand Pose Recognition Using a Fusion of Image Embeddings and Leap 2 Landmarks},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={918-925},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013234300003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Towards Multi-View Hand Pose Recognition Using a Fusion of Image Embeddings and Leap 2 Landmarks
SN - 978-989-758-737-5
AU - Esteban-Romero S.
AU - Lanzino R.
AU - Marini M.
AU - Gil-Martín M.
PY - 2025
SP - 918
EP - 925
DO - 10.5220/0013234300003890
PB - SciTePress