Multi-stream CNN based Video Semantic Segmentation for Automated Driving

Ganesh Sistu, Sumanth Chennupati, Senthil Yogamani

2019

Abstract

Majority of semantic segmentation algorithms operate on a single frame even in the case of videos. In this work, the goal is to exploit temporal information within the algorithm model for leveraging motion cues and temporal consistency. We propose two simple high-level architectures based on Recurrent FCN (RFCN) and Multi-Stream FCN (MSFCN) networks. In case of RFCN, a recurrent network namely LSTM is inserted between the encoder and decoder. MSFCN combines the encoders of different frames into a fused encoder via 1x1 channel-wise convolution. We use a ResNet50 network as the baseline encoder and construct three networks namely MSFCN of order 2 & 3 and RFCN of order 2. MSFCN-3 produces the best results with an accuracy improvement of 9% and 15% for Highway and New York-like city scenarios in the SYNTHIACVPR’ 16 dataset using mean IoU metric. MSFCN-3 also produced 11% and 6% for SegTrack V2 and DAVIS datasets over the baseline FCN network. We also designed an efficient version of MSFCN-2 and RFCN-2 using weight sharing among the two encoders. The efficient MSFCN-2 provided an improvement of 11% and 5% for KITTI and SYNTHIA with negligible increase in computational complexity compared to the baseline version.

Paper Citation

in Harvard Style

Sistu G., Chennupati S. and Yogamani S. (2019). Multi-stream CNN based Video Semantic Segmentation for Automated Driving. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP; ISBN 978-989-758-354-4, SciTePress, pages 173-180. DOI: 10.5220/0007248401730180

in Bibtex Style

@conference{visapp19,
author={Ganesh Sistu and Sumanth Chennupati and Senthil Yogamani},
title={Multi-stream CNN based Video Semantic Segmentation for Automated Driving},
booktitle={Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP},
year={2019},
pages={173-180},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007248401730180},
isbn={978-989-758-354-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019) - Volume 5: VISAPP
TI - Multi-stream CNN based Video Semantic Segmentation for Automated Driving
SN - 978-989-758-354-4
AU - Sistu G.
AU - Chennupati S.
AU - Yogamani S.
PY - 2019
SP - 173
EP - 180
DO - 10.5220/0007248401730180
PB - SciTePress