Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation

Yifei Zhang, Olivier Morel, Marc Blanchon, Ralph Seulin, Mojdeh Rastgoo, Désiré Sidibé

2019

Abstract

Deep neural networks have been frequently used for semantic scene understanding in recent years. Effective and robust segmentation in outdoor scene is prerequisite for safe autonomous navigation of autonomous vehicles. In this paper, our aim is to find the best exploitation of different imaging modalities for road scene segmentation, as opposed to using a single RGB modality. We explore deep learning-based early and later fusion pattern for semantic segmentation, and propose a new multi-level feature fusion network. Given a pair of aligned multimodal images, the network can achieve faster convergence and incorporate more contextual information. In particular, we introduce the first-of-its-kind dataset, which contains aligned raw RGB images and polarimetric images, followed by manually labeled ground truth. The use of polarization cameras is a sensory augmentation that can significantly enhance the capabilities of image understanding, for the detection of highly reflective areas such as glasses and water. Experimental results suggest that our proposed multimodal fusion network outperforms unimodal networks and two typical fusion architectures.

Paper Citation

in Harvard Style

Zhang Y., Morel O., Blanchon M., Seulin R., Rastgoo M. and Sidibé D. (2019). Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP; ISBN 978-989-758-354-4, SciTePress, pages 336-343. DOI: 10.5220/0007360403360343

in Bibtex Style

@conference{visapp19,
author={Yifei Zhang and Olivier Morel and Marc Blanchon and Ralph Seulin and Mojdeh Rastgoo and Désiré Sidibé},
title={Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation},
booktitle={Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP},
year={2019},
pages={336-343},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007360403360343},
isbn={978-989-758-354-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP
TI - Exploration of Deep Learning-based Multimodal Fusion for Semantic Road Scene Segmentation
SN - 978-989-758-354-4
AU - Zhang Y.
AU - Morel O.
AU - Blanchon M.
AU - Seulin R.
AU - Rastgoo M.
AU - Sidibé D.
PY - 2019
SP - 336
EP - 343
DO - 10.5220/0007360403360343
PB - SciTePress