Self-Supervised-Based Multimodal Fusion for Active Biometric Verification on Mobile Devices

Youcef Ouadjer, Chiara Galdi, Sid-Ahmed Berrani, Sid-Ahmed Berrani, Mourad Adnane, Jean-Luc Dugelay

2024

Abstract

This paper focuses on the fusion of multimodal data for an effective active biometric verification on mobile devices. Our proposed Multimodal Fusion (MMFusion) framework combines hand movement data and touch screen interactions. Unlike conventional approaches that rely on annotated unimodal data for deep neural network training, our method makes use of contrastive self-supervised learning in order to extract powerful feature representations and to deal with the lack of labeled training data. The fusion is performed at the feature level, by combining information from hand movement data (collected using background sensors like accelerometer, gyroscope and magnetometer) and touch screen logs. Following the self- supervised learning protocol, MMFusion is pre-trained to capture similarities between hand movement sensor data and touch screen logs, effectively attracting similar pairs and repelling dissimilar ones. Extensive evaluations demonstrate its high performance on user verification across diverse tasks compared to unimodal alternatives trained using the SimCLR framework. Moreover, experiments in semi-supervised scenarios reveal the superiority of MMFusion with the best trade-off between sensitivity and specificity.

Paper Citation

in Harvard Style

Ouadjer Y., Galdi C., Berrani S., Adnane M. and Dugelay J. (2024). Self-Supervised-Based Multimodal Fusion for Active Biometric Verification on Mobile Devices. In Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM; ISBN 978-989-758-684-2, SciTePress, pages 739-744. DOI: 10.5220/0012359000003654

in Bibtex Style

@conference{icpram24,
author={Youcef Ouadjer and Chiara Galdi and Sid-Ahmed Berrani and Mourad Adnane and Jean-Luc Dugelay},
title={Self-Supervised-Based Multimodal Fusion for Active Biometric Verification on Mobile Devices},
booktitle={Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM},
year={2024},
pages={739-744},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012359000003654},
isbn={978-989-758-684-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM
TI - Self-Supervised-Based Multimodal Fusion for Active Biometric Verification on Mobile Devices
SN - 978-989-758-684-2
AU - Ouadjer Y.
AU - Galdi C.
AU - Berrani S.
AU - Adnane M.
AU - Dugelay J.
PY - 2024
SP - 739
EP - 744
DO - 10.5220/0012359000003654
PB - SciTePress