Multimodal Fusion Strategies for Outcome Prediction in Stroke

Esra Zihni, Esra Zihni, Vince Madai, Ahmed Khalil, Ivana Galinovic, Jochen Fiebach, John D. Kelleher, Dietmar Frey, Michelle Livne

2020

Abstract

Data driven methods are increasingly being adopted in the medical domain for clinical predictive modeling. Prediction of stroke outcome using machine learning could provide a decision support system for physicians to assist them in patient-oriented diagnosis and treatment. While patient-specific clinical parameters play an important role in outcome prediction, a multimodal fusion approach that integrates neuroimaging with clinical data has the potential to improve accuracy. This paper addresses two research questions: (a) does multimodal fusion aid in the prediction of stroke outcome, and (b) what fusion strategy is more suitable for the task at hand. The baselines for our experimental work are two unimodal neural architectures: a 3D Convolutional Neural Network for processing neuroimaging data, and a Multilayer Perceptron for processing clinical data. Using these unimodal architectures as building blocks we propose two feature-level multimodal fusion strategies: 1) extracted features, where the unimodal architectures are trained separately and then fused, and 2) end-to-end, where the unimodal architectures are trained together. We show that integration of neuroimaging information with clinical metadata can potentially improve stroke outcome prediction. Additionally, experimental results indicate that the end-to-end fusion approach proves to be more robust.

Paper Citation

in Harvard Style

Zihni E., Madai V., Khalil A., Galinovic I., Fiebach J., Kelleher J., Frey D. and Livne M. (2020). Multimodal Fusion Strategies for Outcome Prediction in Stroke. In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF; ISBN 978-989-758-398-8, SciTePress, pages 421-428. DOI: 10.5220/0008957304210428

in Bibtex Style

@conference{healthinf20,
author={Esra Zihni and Vince Madai and Ahmed Khalil and Ivana Galinovic and Jochen Fiebach and John D. Kelleher and Dietmar Frey and Michelle Livne},
title={Multimodal Fusion Strategies for Outcome Prediction in Stroke},
booktitle={Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF},
year={2020},
pages={421-428},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008957304210428},
isbn={978-989-758-398-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF
TI - Multimodal Fusion Strategies for Outcome Prediction in Stroke
SN - 978-989-758-398-8
AU - Zihni E.
AU - Madai V.
AU - Khalil A.
AU - Galinovic I.
AU - Fiebach J.
AU - Kelleher J.
AU - Frey D.
AU - Livne M.
PY - 2020
SP - 421
EP - 428
DO - 10.5220/0008957304210428
PB - SciTePress