3D open-vocabulary scene understanding, which accurately perceives complex semantic properties of objects in space, has gained significant attention in recent years. In this paper, we propose GAGS, a framework that distills 2D CLIP features into 3D Gaussian splatting, enabling open-vocabulary queries for renderings on arbitrary viewpoints. The main challenge of distilling 2D features for 3D fields lies in the multiview inconsistency of extracted 2D features, which provides unstable supervision for the 3D feature field. GAGS addresses this challenge with two novel strategies. First, GAGS associates the prompt point density of SAM with the camera distances, which significantly improves the multiview consistency of segmentation results. Second, GAGS further decodes a granularity factor to guide the distillation process and this granularity factor can be learned in a unsupervised manner to only select the multiview consistent 2D features in the distillation process. Experimental results on two datasets demonstrate significant performance and stability improvements of GAGS in visual grounding and semantic segmentation, with an inference speed 2× faster than baseline methods.
三维开放词汇场景理解（3D Open-Vocabulary Scene Understanding）近年来受到广泛关注，其目标是准确感知空间中对象的复杂语义属性。在本文中，我们提出了 GAGS，一种将二维 CLIP 特征蒸馏到三维高斯点云（3D Gaussian Splatting）中的框架，从而支持在任意视角下进行开放词汇查询和渲染。 二维特征蒸馏到三维场景的主要挑战在于提取的二维特征在多视角间的一致性不足，这为三维特征场的监督带来了不稳定性。GAGS 通过两种创新策略解决了这一问题。首先，GAGS 将 SAM（Segment Anything Model）中的提示点密度与相机距离相关联，从而显著提升了分割结果的多视角一致性。其次，GAGS 解码了一个粒度因子来引导蒸馏过程，该粒度因子通过无监督方式学习，仅选择多视角一致的二维特征参与蒸馏。 在两个数据集上的实验结果表明，GAGS 在视觉定位和语义分割任务中取得了显著的性能和稳定性提升，同时推理速度比基线方法快 2 倍，展现了其在三维开放词汇场景理解中的卓越表现。