Near-infrared Lipreading System for Driver-Car Interaction

Samar Daou, Ahmed Rekik, Ahmed Rekik, Achraf Ben-Hamadou, Achraf Ben-Hamadou, Abdelaziz Kallel, Abdelaziz Kallel

2023

Abstract

In this paper, we propose a new lipreading approach for driver-car interaction in a cockpit monitoring environment. Furthermore, we introduce and release the first lipreading dataset dedicated to intuitive driver-car interaction using near-infrared driver monitoring cameras. In this paper, we propose a two-stream deep learning architecture that combines both geometric and global visual features extracted from the mouth region to improve the performance of lipreading based only on visual cues. Geometric features are extracted by a graph convolutional network applied to a series of 2D facial landmarks, while a 2D-3D convolutional network is used to extract the global visual features from the near-infrared frame sequence. These features are then decoded based on a multi-scale temporal convolutional network to generate the output word sequence classification. Our proposed model achieved high accuracy for both training scenarios overlapped speaker and unseen speaker with 98.5% and 92.2% respectively.

Paper Citation

in Harvard Style

Daou S., Rekik A., Ben-Hamadou A. and Kallel A. (2023). Near-infrared Lipreading System for Driver-Car Interaction. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP; ISBN 978-989-758-634-7, SciTePress, pages 631-638. DOI: 10.5220/0011692300003417

in Bibtex Style

@conference{visapp23,
author={Samar Daou and Ahmed Rekik and Achraf Ben-Hamadou and Abdelaziz Kallel},
title={Near-infrared Lipreading System for Driver-Car Interaction},
booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP},
year={2023},
pages={631-638},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011692300003417},
isbn={978-989-758-634-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP
TI - Near-infrared Lipreading System for Driver-Car Interaction
SN - 978-989-758-634-7
AU - Daou S.
AU - Rekik A.
AU - Ben-Hamadou A.
AU - Kallel A.
PY - 2023
SP - 631
EP - 638
DO - 10.5220/0011692300003417
PB - SciTePress