PIU-Net: Generation of Virtual Panoramic Views from Colored Point Clouds

Michael Georg Adam, Eckehard Steinbach

2021

Abstract

As VR-systems become more and more widespread, the interest in high-quality content increases drastically. One way of generating such data is by digitizing real world environments using SLAM-based mapping devices, which capture both the geometry of the environment (typically as point clouds) and its appearance (typically as a small set of RGB panoramas). However, when creating such digital representations of real-world spaces, artifacts and missing data cannot be fully avoided. Furthermore, free movement is often restricted. In this paper, we introduce a technique, which allows for the generation of high quality panoramic views at any position within a captured real world scenario. Our method consists of two steps. First, we render virtual panoramas from the projected point cloud data. Those views exhibit imperfections in comparison to the real panoramas, which can be corrected in the second step by an inpainting neural network. The network itself is trained using a small set of panoramic images captured during the mapping process. In order to take full advantage of the panoramic information, we use a U-Net-like structure with circular convolutions. Further, a custom perceptual panoramic loss is applied. The resulting virtual panoramas show high quality and spatial consistency. Furthermore, the network learns to correct erroneous point cloud data. We evaluate the proposed approach by generating virtual panoramas along novel trajectories where the panorama positions deviate from the originally selected capturing points and observe that the proposed approach is able to generate smooth and temporally consistent walkthrough sequences.

Paper Citation

in Harvard Style

Adam M. and Steinbach E. (2021). PIU-Net: Generation of Virtual Panoramic Views from Colored Point Clouds. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP; ISBN 978-989-758-488-6, SciTePress, pages 269-276. DOI: 10.5220/0010198302690276

in Bibtex Style

@conference{visapp21,
author={Michael Georg Adam and Eckehard Steinbach},
title={PIU-Net: Generation of Virtual Panoramic Views from Colored Point Clouds},
booktitle={Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP},
year={2021},
pages={269-276},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010198302690276},
isbn={978-989-758-488-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP
TI - PIU-Net: Generation of Virtual Panoramic Views from Colored Point Clouds
SN - 978-989-758-488-6
AU - Adam M.
AU - Steinbach E.
PY - 2021
SP - 269
EP - 276
DO - 10.5220/0010198302690276
PB - SciTePress