Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes

Florian Fervers, Timo Breuer, Gregor Stachowiak, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens

2022

Abstract

Models for semantic segmentation require a large amount of hand-labeled training data which is costly and time-consuming to produce. For this purpose, we present a label fusion framework that is capable of improving semantic pixel labels of video sequences in an unsupervised manner. We make use of a 3D mesh representation of the environment and fuse the predictions of different frames into a consistent representation using semantic mesh textures. Rendering the semantic mesh using the original intrinsic and extrinsic camera parameters yields a set of improved semantic segmentation images. Due to our optimized CUDA implementation, we are able to exploit the entire c-dimensional probability distribution of annotations over c classes in an uncertainty-aware manner. We evaluate our method on the Scannet dataset where we improve annotations produced by the state-of-the-art segmentation network ESANet from 52.05% to 58.25% pixel accuracy. We publish the source code of our framework online to foster future research in this area (https://github.com/fferflo/semantic-meshes). To the best of our knowledge, this is the first publicly available label fusion framework for semantic image segmentation based on meshes with semantic textures.

Paper Citation

in Harvard Style

Fervers F., Breuer T., Stachowiak G., Bullinger S., Bodensteiner C. and Arens M. (2022). Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes. In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP; ISBN 978-989-758-555-5, SciTePress, pages 509-516. DOI: 10.5220/0010841800003124

in Bibtex Style

@conference{visapp22,
author={Florian Fervers and Timo Breuer and Gregor Stachowiak and Sebastian Bullinger and Christoph Bodensteiner and Michael Arens},
title={Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes},
booktitle={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP},
year={2022},
pages={509-516},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010841800003124},
isbn={978-989-758-555-5},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 5: VISAPP
TI - Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes
SN - 978-989-758-555-5
AU - Fervers F.
AU - Breuer T.
AU - Stachowiak G.
AU - Bullinger S.
AU - Bodensteiner C.
AU - Arens M.
PY - 2022
SP - 509
EP - 516
DO - 10.5220/0010841800003124
PB - SciTePress