OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views

Large visual-language models (VLMs), like CLIP, enable open-set image segmentation to segment arbitrary concepts from an image in a zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from a pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-set 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representations do not align well with the image-based nature of the visual-language models. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well to the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges, we propose Open<PRE_TAG>NeRF</POST_TAG> which naturally operates on posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in an overall less complex architecture without the need for additional DINO regularization. Our Open<PRE_TAG>NeRF</POST_TAG> further leverages NeRF's ability to render novel views and extract open-set <PRE_TAG>VLM features</POST_TAG> from areas that are not well observed in the initial posed images. For 3D point cloud segmentation on the Replica dataset, Open<PRE_TAG>NeRF</POST_TAG> outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.