Interpretation of Human Behavior from Multi-modal Brain MRI Images based on Graph Deep Neural Networks and Attention Mechanism

Refka Hanachi, Akrem Sellami, Imed Riadh Farah, Imed Riadh Farah

2021

Abstract

Interpretation of human behavior by exploiting the complementarity of the information offered by multimodal functional magnetic resonance imaging (fMRI) data is a challenging task. In this paper, we propose to fuse task-fMRI for brain activation and rest-fMRI for functional connectivity with the incorporation of structural MRI (sMRI) as an adjacency matrix to maintain the rich spatial structure between voxels of the brain. We consider then the structural-functional brain connections (3D mesh) as a graph. The aim is to quantify each subject’s performance in voice recognition and identification. More specifically, we propose an advanced multi-view graph auto-encoder based on the attention mechanism called MGATE, which seeks at learning better representation from both modalities task- and rest-fMRI using the Brain Adjacency Graph (BAG), which is constructed based on sMRI. It yields a multi-view representation learned at all vertices of the brain, which be used as input to our trace regression model in order to predict the behavioral score of each subject. Experimental results show that the proposed model achieves better prediction rates, and reaches competitive high performances compared to various existing graph representation learning models in the stateof-the-art.

Paper Citation

in Harvard Style

Hanachi R., Sellami A. and Farah I. (2021). Interpretation of Human Behavior from Multi-modal Brain MRI Images based on Graph Deep Neural Networks and Attention Mechanism. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 56-66. DOI: 10.5220/0010214400560066

in Bibtex Style

@conference{visapp21,
author={Refka Hanachi and Akrem Sellami and Imed Riadh Farah},
title={Interpretation of Human Behavior from Multi-modal Brain MRI Images based on Graph Deep Neural Networks and Attention Mechanism},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP},
year={2021},
pages={56-66},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010214400560066},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP
TI - Interpretation of Human Behavior from Multi-modal Brain MRI Images based on Graph Deep Neural Networks and Attention Mechanism
SN - 978-989-758-488-6
AU - Hanachi R.
AU - Sellami A.
AU - Farah I.
PY - 2021
SP - 56
EP - 66
DO - 10.5220/0010214400560066
PB - SciTePress