Hand Gesture Recognition Using MediaPipe Landmarks and Deep Learning Networks

Manuel Gil-Martín, Marco Raoul Marini, Iván Martín-Fernández, Sergio Esteban-Romero, Luigi Cinque

2025

Abstract

.Advanced Human Computer Interaction techniques are commonly used in multiple application areas, from entertainment to rehabilitation. In this context, this paper proposes a framework to recognize hand gestures using a limited number of landmarks from the video images. This hand gesture recognition system comprises an image processing module that extracts and processes the coordinates of 21 hand points called landmarks, and a deep neural network module that models and classifies the hand gestures. These landmarks are extracted automatically through MediaPipe software. The experiments were carried out over the IPN Hand dataset in an independent-user scenario using a Subject-Wise Cross Validation. They cover the use of different landmark-based formats, normalizations, lengths of the gesture representations, and number of landmarks used as inputs. The system obtains significantly better accuracy when using the raw coordinates of the 21 landmarks through 125 timesteps and a light Recurrent Neural Network architecture (80.56 ± 1.19 %) or the hand anthropometric measures (82.20 ± 1.15 %) compared to using the speed of the hand landmarks through the gesture (72.93 ± 1.34 %). The proposed framework studied the effect of different landmark-based normalizations over the raw coordinates, obtaining an accuracy of 83.67 ± 1.12 % when using as reference the wrist landmark from each frame, and an accuracy of 84.66 ± 1.09 % when using as reference the wrist landmark from the first video frame of the current gesture. In addition, the proposed solution provided high recognition performance even when only using the coordinates from 6 (82.15 ± 1.16 %) or 4 (81.46 ± 1.17 %) specific hand landmarks using as reference the wrist landmark from the first video frame of the current gesture.

Paper Citation

in Harvard Style

Gil-Martín M., Marini M., Martín-Fernández I., Esteban-Romero S. and Cinque L. (2025). Hand Gesture Recognition Using MediaPipe Landmarks and Deep Learning Networks. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART; ISBN 978-989-758-737-5, SciTePress, pages 24-30. DOI: 10.5220/0013053500003890

in Bibtex Style

@conference{icaart25,
author={Manuel Gil-Martín and Marco Marini and Iván Martín-Fernández and Sergio Esteban-Romero and Luigi Cinque},
title={Hand Gesture Recognition Using MediaPipe Landmarks and Deep Learning Networks},
booktitle={Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART},
year={2025},
pages={24-30},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013053500003890},
isbn={978-989-758-737-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART
TI - Hand Gesture Recognition Using MediaPipe Landmarks and Deep Learning Networks
SN - 978-989-758-737-5
AU - Gil-Martín M.
AU - Marini M.
AU - Martín-Fernández I.
AU - Esteban-Romero S.
AU - Cinque L.
PY - 2025
SP - 24
EP - 30
DO - 10.5220/0013053500003890
PB - SciTePress