Speech Recognition using Deep Canonical Correlation Analysis in Noisy Environments

Shinnosuke Isobe, Satoshi Tamura, Satoru Hayamizu

2021

Abstract

In this paper, we propose a method to improve the accuracy of speech recognition in noisy environments by utilizing Deep Canonical Correlation Analysis (DCCA). DCCA generates projections from two modalities into one common space, so that the correlation of projected vectors could be maximized. Our idea is to employ DCCA techniques with audio and visual modalities to enhance the robustness of Automatic Speech Recognition (ASR); A) noisy audio features can be recovered by clean visual features, and B) an ASR model can be trained using audio and visual features, as data augmentation. We evaluated our method using an audiovisual corpus CENSREC-1-AV and a noise database DEMAND. Compared to conventional ASR and feature- fusion-based audio-visual speech recognition, our DCCA-based recognizers achieved better performance. In addition, experimental results shows that utilizing DCCA enables us to get better results in various noisy environments, thanks to the visual modality. Furthermore, it is found that DCCA can be used as a data augmentation scheme if only a few training data are available, by incorporating visual DCCA features to build an audio-only ASR model, in addition to audio DCCA features.

Paper Citation

in Harvard Style

Isobe S., Tamura S. and Hayamizu S. (2021). Speech Recognition using Deep Canonical Correlation Analysis in Noisy Environments.In Proceedings of the 10th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM, ISBN 978-989-758-486-2, pages 63-70. DOI: 10.5220/0010268200630070

in Bibtex Style

@conference{icpram21,
author={Shinnosuke Isobe and Satoshi Tamura and Satoru Hayamizu},
title={Speech Recognition using Deep Canonical Correlation Analysis in Noisy Environments},
booktitle={Proceedings of the 10th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,},
year={2021},
pages={63-70},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010268200630070},
isbn={978-989-758-486-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM,
TI - Speech Recognition using Deep Canonical Correlation Analysis in Noisy Environments
SN - 978-989-758-486-2
AU - Isobe S.
AU - Tamura S.
AU - Hayamizu S.
PY - 2021
SP - 63
EP - 70
DO - 10.5220/0010268200630070