Thermography is especially valuable for the military and other users of surveillance cameras. Some recent methods based on Neural Radiance Fields (NeRF) are proposed to reconstruct the thermal scenes in 3D from a set of thermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS) prevails due to its rapid training and real-time rendering. In this work, we propose ThermalGaussian, the first thermal 3DGS approach capable of rendering high-quality images in RGB and thermal modalities. We first calibrate the RGB camera and the thermal camera to ensure that both modalities are accurately aligned. Subsequently, we use the registered images to learn the multimodal 3D Gaussians. To prevent the overfitting of any single modality, we introduce several multimodal regularization constraints. We also develop smoothing constraints tailored to the physical characteristics of the thermal modality. Besides, we contribute a real-world dataset named RGBT-Scenes, captured by a hand-hold thermal-infrared camera, facilitating future research on thermal scene reconstruction. We conduct comprehensive experiments to show that ThermalGaussian achieves photorealistic rendering of thermal images and improves the rendering quality of RGB images. With the proposed multimodal regularization constraints, we also reduced the model's storage cost by 90%.
热成像在军事及其他监控摄像机用户中具有重要价值。最近，一些基于神经辐射场（NeRF）的方法被提出用于从一组热成像和RGB图像中重建3D热场景。然而，与NeRF不同，3D Gaussian Splatting (3DGS) 因其快速训练和实时渲染的优势而更具优势。在这项工作中，我们提出了ThermalGaussian，这是首个能够渲染高质量RGB和热成像图像的3DGS方法。我们首先校准RGB相机和热成像相机，以确保这两种模态准确对齐。随后，我们使用配准后的图像来学习多模态3D高斯。为防止单一模态的过拟合，我们引入了多模态正则化约束。此外，我们还开发了针对热成像物理特性定制的平滑约束。此外，我们贡献了一个名为RGBT-Scenes的真实世界数据集，该数据集由手持热红外相机采集，旨在推动未来热场景重建的研究。我们进行了全面的实验，表明ThermalGaussian在热成像图像的渲染上实现了逼真的效果，并提高了RGB图像的渲染质量。通过所提出的多模态正则化约束，我们还将模型的存储成本减少了90%。