3D Semantic Scene Reconstruction from a Single Viewport

Maximilian Denninger, Maximilian Denninger, Rudolph Triebel, Rudolph Triebel

2023

Abstract

We introduce a novel method for semantic volumetric reconstructions from a single RGB image. To overcome the problem of semantically reconstructing regions in 3D that are occluded in the 2D image, we propose to combine both in an implicit encoding. By relying on a headless autoencoder, we are able to encode semantic categories and implicit TSDF values into a compressed latent representation. A second network then uses these as a reconstruction target and learns to convert color images into these latent representations, which get decoded after inference. Additionally, we introduce a novel loss-shaping technique for this implicit representation. In our experiments on the realistic benchmark Replica dataset, we achieve a full reconstruction of a scene, which is visually and in terms of quantitative measures better than current methods while only using synthetic data during training. On top of that, we evaluate our approach on color images recorded in the wild.

Paper Citation

in Harvard Style

Denninger M. and Triebel R. (2023). 3D Semantic Scene Reconstruction from a Single Viewport. In Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE, ISBN 978-989-758-642-2, SciTePress, pages 15-26. DOI: 10.5220/0011747700003497

in Bibtex Style

@conference{improve23,
author={Maximilian Denninger and Rudolph Triebel},
title={3D Semantic Scene Reconstruction from a Single Viewport},
booktitle={Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE,},
year={2023},
pages={15-26},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011747700003497},
isbn={978-989-758-642-2},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Conference on Image Processing and Vision Engineering - Volume 1: IMPROVE,
TI - 3D Semantic Scene Reconstruction from a Single Viewport
SN - 978-989-758-642-2
AU - Denninger M.
AU - Triebel R.
PY - 2023
SP - 15
EP - 26
DO - 10.5220/0011747700003497
PB - SciTePress