Continuous Sign-Language Recognition using Transformers and Augmented Pose Estimation

Reemt Hinrichs, Angelo Sitcheu, Jörn Ostermann

2023

Abstract

Sign language is used by deaf to communicate with other humans. It consists of not only hand signs or gestures but encompasses also facial expressions and further body movements. To make machine-human interaction accessible for deaf, automatic sign language recognition has to be implemented which allows a machine to understand the signs and gestures of deaf. For this purpose, continous sign-language recognition, which is the mapping of a (visual) sequence of signs forming a (sign) sentence to a sequence of (text) words, has to be developed. In this work, continuous sign-language recognition using transformers is proposed. Using additional pose estimation, body markers are extracted and augmented through data imputation and velocity-like features, and then used together with a transformer network for continuous sign-language recognition. Using the proposed method, better than state-of-the-art results were obtained on the RWTH-PHOENIX-Weather 2014 dataset, achieving 19.2%/19.5% dev/test word error rate (WER) on the signer-independent subset and 16.9%/17.4% dev/test WER on the simpler multi-signer subset. The feature augmentation was found to improve the baseline word error rate by about 2.7 %/ 2.9 % dev/test.

Paper Citation

in Harvard Style

Hinrichs R., Sitcheu A. and Ostermann J. (2023). Continuous Sign-Language Recognition using Transformers and Augmented Pose Estimation. In Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-626-2, pages 672-678. DOI: 10.5220/0011709100003411

in Bibtex Style

@conference{icpram23,
author={Reemt Hinrichs and Angelo Sitcheu and Jörn Ostermann},
title={Continuous Sign-Language Recognition using Transformers and Augmented Pose Estimation},
booktitle={Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2023},
pages={672-678},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011709100003411},
isbn={978-989-758-626-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Continuous Sign-Language Recognition using Transformers and Augmented Pose Estimation
SN - 978-989-758-626-2
AU - Hinrichs R.
AU - Sitcheu A.
AU - Ostermann J.
PY - 2023
SP - 672
EP - 678
DO - 10.5220/0011709100003411