Different Modal Stereo: Simultaneous Estimation of Stereo Image Disparity and Modality Translation

Ryota Tanaka, Fumihiko Sakaue, Jun Sato

2020

Abstract

We propose a stereo matching method from the different modal image pairs. In this method, input images are taken in different viewpoints by different modal cameras, e.g., an RGB camera and an IR camera. Our proposed method estimates the disparity of the two images and translates the modality of the input images to different modality simultaneously. To achieve this simultaneous estimation, we utilize two networks, i.e., a disparity estimation method from a single image and modality translation method. Both methods are based on the neural networks, and then we train the network simultaneously. In this training, we focus on several consistencies between the different modal images. By these consistencies, two kinds of networks are effectively trained. Furthermore, we utilize image synthesis optimization on conditional GAN, and the optimization provides quite good results. Several experimental results by open databases show that the proposed method can estimate disparity and translate the modality even if the modalities of the input image pair are different.

Paper Citation

in Harvard Style

Tanaka R., Sakaue F. and Sato J. (2020). Different Modal Stereo: Simultaneous Estimation of Stereo Image Disparity and Modality Translation. In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP; ISBN 978-989-758-402-2, SciTePress, pages 554-560. DOI: 10.5220/0009180205540560

in Bibtex Style

@conference{visapp20,
author={Ryota Tanaka and Fumihiko Sakaue and Jun Sato},
title={Different Modal Stereo: Simultaneous Estimation of Stereo Image Disparity and Modality Translation},
booktitle={Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP},
year={2020},
pages={554-560},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009180205540560},
isbn={978-989-758-402-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 4: VISAPP
TI - Different Modal Stereo: Simultaneous Estimation of Stereo Image Disparity and Modality Translation
SN - 978-989-758-402-2
AU - Tanaka R.
AU - Sakaue F.
AU - Sato J.
PY - 2020
SP - 554
EP - 560
DO - 10.5220/0009180205540560
PB - SciTePress