In this work, we tackle the task of learning generalizable 3D human Gaussians from a single image. The main challenge for this task is to recover detailed geometry and appearance, especially for the unobserved regions. To this end, we propose single-view generalizable Human Gaussian model (HGM), a diffusion-guided framework for 3D human modeling from a single image. We design a diffusion-based coarse-to-fine pipeline, where the diffusion model is adapted to refine novel-view images rendered from a coarse human Gaussian model. The refined images are then used together with the input image to learn a refined human Gaussian model. Although effective in hallucinating the unobserved views, the approach may generate unrealistic human pose and shapes due to the lack of supervision. We circumvent this problem by further encoding the geometric priors from SMPL model. Specifically, we propagate geometric features from SMPL volume to the predicted Gaussians via sparse convolution and attention mechanism. We validate our approach on publicly available datasets and demonstrate that it significantly surpasses state-of-the-art methods in terms of PSNR and SSIM. Additionally, our method exhibits strong generalization for in-the-wild images.
在这项工作中，我们致力于从单张图片学习可泛化的3D人体高斯模型。这项任务的主要挑战是恢复详细的几何形状和外观，特别是对于未观察到的区域。为此，我们提出了单视图可泛化的人体高斯模型（HGM），这是一个由扩散引导的3D人体建模框架。我们设计了一个基于扩散的从粗到细的管道，其中扩散模型被适应用来细化从粗糙人体高斯模型渲染的新视角图像。然后将细化后的图像与输入图像一起使用，以学习一个细化的人体高斯模型。虽然这种方法在幻想未观察视角时有效，但由于缺乏监督，可能会生成不现实的人体姿态和形状。我们通过进一步编码来自SMPL模型的几何先验来规避这个问题。具体来说，我们通过稀疏卷积和注意力机制将几何特征从SMPL体积传播到预测的高斯模型中。我们在公开可用的数据集上验证了我们的方法，并展示了它在PSNR和SSIM方面显著超越了现有的最先进方法。此外，我们的方法对于野外图像也展现了强大的泛化能力。