Authors:
Weiwen Hu
1
;
Niccolò Parodi
1
;
2
;
Marcus Zepp
1
;
Ingo Feldmann
1
;
Oliver Schreer
1
and
Peter Eisert
1
;
3
Affiliations:
1
Fraunhofer Heinrich Hertz Institute, Berlin, Germany
;
2
Technische Universität Berlin, Germany
;
3
Humboldt-Universität zu Berlin, Germany
Keyword(s):
Computer Vision, Neural Radiance Field, Semantic Segmentation, Point Cloud, 3D.
Abstract:
Open-vocabulary segmentation, powered by large visual-language models like CLIP, has expanded 2D segmentation capabilities beyond fixed classes predefined by the dataset, enabling zero-shot understanding across diverse scenes. Extending these capabilities to 3D segmentation introduces challenges, as CLIP’s image-based embeddings often lack the geometric detail necessary for 3D scene segmentation. Recent methods tend to address this by introducing additional segmentation models or replacing CLIP with variations trained on segmentation data, which lead to redundancy or loss on CLIP’s general language capabilities. To overcome this limitation, we introduce SPNeRF, a NeRF based zero-shot 3D segmentation approach that leverages geometric priors. We integrate geometric primitives derived from the 3D scene into NeRF training to produce primitive-wise CLIP features, avoiding the ambiguity of point-wise features. Additionally, we propose a primitive-based merging mechanism enhanced with affin
ity scores. Without relying on additional segmentation models, our method further explores CLIP’s capability for 3D segmentation and achieves notable improvements over orig-inal LERF.
(More)