In the literature, it has been shown that the evolution of the known explicit 3D surface to the target one can be learned from 2D images using the instantaneous flow field, where the known and target 3D surfaces may largely differ in topology. We are interested in capturing 4D shapes whose topology changes largely over time. We encounter that the straightforward extension of the existing 3D-based method to the desired 4D case performs poorly. In this work, we address the challenges in extending 3D neural evolution to 4D under large topological changes by proposing two novel modifications. More precisely, we introduce (i) a new architecture to discretize and encode the deformation and learn the SDF and (ii) a technique to impose the temporal consistency. (iii) Also, we propose a rendering scheme for color prediction based on Gaussian splatting. Furthermore, to facilitate learning directly from 2D images, we propose a learning framework that can disentangle the geometry and appearance from RGB images. This method of disentanglement, while also useful for the 4D evolution problem that we are concentrating on, is also novel and valid for static scenes. Our extensive experiments on various data provide awesome results and, most importantly, open a new approach toward reconstructing challenging scenes with significant topological changes and deformations.
在文献中已有研究表明，可以通过瞬时流场从二维图像中学习已知显式三维曲面向目标曲面的演变过程，其中已知曲面和目标曲面在拓扑结构上可能存在较大差异。我们感兴趣的是捕捉拓扑结构随时间发生显著变化的四维形状。然而，现有三维方法的直接扩展在处理这种四维情况时表现较差。 在本研究中，我们针对在拓扑结构发生显著变化的情况下，将三维神经演化扩展到四维所面临的挑战，提出了两项新的改进。具体来说，我们引入了以下关键创新：(i) 一种新架构，用于离散化和编码形变，同时学习符号距离函数（SDF）；(ii) 一种用于实现时间一致性的技术；以及 (iii) 一种基于高斯点云的颜色预测渲染方案。此外，为了直接从二维图像中学习，我们提出了一种框架，可以从 RGB 图像中解耦几何信息和外观信息。这种解耦方法不仅对我们关注的四维演化问题有用，而且对于静态场景也是新颖且有效的。 通过在多种数据集上的广泛实验，我们的研究获得了卓越的结果，更重要的是，为重建具有显著拓扑变化和形变的复杂场景开辟了一种全新的方法。