An Audio-Visual based Feature Level Fusion Approach Applied to Deception Detection

Safa Chebbi, Sofia Ben Jebara

2020

Abstract

Due to the increasing requirement of security and antiterrorism issues, research activities in the field of deception detection have been receiving a big attention. For this reason, many studies dealing with deception detection have been developed varying in terms of approaches, modalities, features and learning algorithms. Despite the wide range of proposed approaches in this task, there is no universal and effective system until today capable of identifying deception with a high recognition rate. In this paper, a feature level fusion approach, combining audio and video modalities, has been proposed to build an automated system that can help in decision making of honesty or lie. Thus a high feature vector size, combining verbal features (72 pitch-based ones) and nonverbal ones related to facial expressions and body gestures, is extracted. Then, a feature level fusion is applied in order to select the most relevant ones. A special interest is given to mutual information-based criteria that are well adapted to continuous and binary features combination. Simulation results on a realistic database of suspicious persons interrogation achieved 97% as deception/truth classification accuracy using 19 audio/video mixed features, which outperforms the state-of-the-art results.

Paper Citation

in Harvard Style

Chebbi S. and Ben Jebara S. (2020). An Audio-Visual based Feature Level Fusion Approach Applied to Deception Detection. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP; ISBN 978-989-758-402-2, SciTePress, pages 197-205. DOI: 10.5220/0008896201970205

in Bibtex Style

@conference{visapp20,
author={Safa Chebbi and Sofia Ben Jebara},
title={An Audio-Visual based Feature Level Fusion Approach Applied to Deception Detection},
booktitle={Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP},
year={2020},
pages={197-205},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008896201970205},
isbn={978-989-758-402-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP
TI - An Audio-Visual based Feature Level Fusion Approach Applied to Deception Detection
SN - 978-989-758-402-2
AU - Chebbi S.
AU - Ben Jebara S.
PY - 2020
SP - 197
EP - 205
DO - 10.5220/0008896201970205
PB - SciTePress