SpectralGaussians: Semantic, spectral 3D Gaussian splatting for multi-spectral scene representation, visualization and analysis
We propose a novel cross-spectral rendering framework based on 3D Gaussian Splatting (3DGS) that generates realistic and semantically meaningful splats from registered multi-view spectrum and segmentation maps. This extension enhances the representation of scenes with multiple spectra, providing insights into the underlying materials and segmentation. We introduce an improved physically-based rendering approach for Gaussian splats, estimating reflectance and lights per spectra, thereby enhancing accuracy and realism. In a comprehensive quantitative and qualitative evaluation, we demonstrate the superior performance of our approach with respect to other recent learning-based spectral scene representation approaches (i.e., XNeRF and SpectralNeRF) as well as other non-spectral state-of-the-art learning-based approaches. Our work also demonstrates the potential of spectral scene understanding for precise scene editing techniques like style transfer, inpainting, and removal. Thereby, our contributions address challenges in multi-spectral scene representation, rendering, and editing, offering new possibilities for diverse applications.
我们提出了一种基于3D高斯点绘（3DGS）的新型跨光谱渲染框架，该框架能够从配准的多视角光谱和分割图中生成逼真且语义丰富的点绘。这一扩展增强了对多光谱场景的表示，提供了关于底层材料和分割的深入见解。我们引入了一种改进的基于物理的高斯点绘渲染方法，通过估计每个光谱的反射率和光照，提升了渲染的准确性和真实感。在全面的定量和定性评估中，我们展示了我们的方法在性能上优于其他最新的基于学习的光谱场景表示方法（如XNeRF和SpectralNeRF），以及其他非光谱的最先进的基于学习的方法。我们的研究还展示了光谱场景理解在精确场景编辑技术（如风格迁移、修复和移除）中的潜力。因此，我们的贡献解决了多光谱场景表示、渲染和编辑中的挑战，为多种应用提供了新的可能性。