This is the official Benchmark PyTorch implementation of the paper "DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering".
Abstract: Realistic human-centric rendering plays a key role in both computer vision and computer graphics. Rapid progress has been made in the algorithm aspect over the years, yet existing human-centric rendering datasets and benchmarks are rather impoverished in terms of diversity (e.g., outfit's fabric/material, body's interaction with objects, and motion sequences), which are crucial for rendering effect. Researchers are usually constrained to explore and evaluate a small set of rendering problems on current datasets, while real-world applications require methods to be robust across different scenarios. In this work, we present DNA-Rendering, a large-scale, high-fidelity repository of human performance data for neural actor rendering. DNA-Rendering presents several alluring attributes. First, our dataset contains over 1500 human subjects, 5000 motion sequences, and 67.5M frames' data volume. Upon the massive collections, we provide human subjects with grand categories of pose actions, body shapes, clothing, accessories, hairdos, and object intersection, which ranges the geometry and appearance variances from everyday life to professional occasions. Second, we provide rich assets for each subject -- 2D/3Dhuman body keypoints, foreground masks, smplx models, cloth/accessory materials, multi-view images and videos. These assets boost the current method's accuracy on downstream rendering tasks. Third, we construct a professional multi-view system to capture data, which contains 60 synchronous cameras with max 4096 x 3000 resolution, 15fps speed, and stern camera calibration steps, ensuring high-quality resources for task training and evaluation. Along with the dataset, we provide a large-scale and quantitative benchmark in full-scale, with multiple tasks to evaluate the existing progress of novel view synthesis, novel pose animation synthesis, and novel identity rendering methods. In this manuscript, we describe our DNA-Rendering effort as a revealing of new observations, challenges, and future directions to human-centric rendering. The dataset and code for data processing and benchmarking are publicly available at https://dna-rendering.github.io/ .
- 2024.08.08: 🍮🍮🍮Data Part6🍮🍮🍮 is released!
- 2024.06.25: 🍹🍹🍹A-pose data of shared motion sequences🍹🍹🍹 is released!
- 2024.05.24: 🍮🍮🍮Data Part5🍮🍮🍮 is released!
- 2024.05.13: 🏗️ We support directly using DNA-Rendering data in 3DGS now. Please refer to (DNA-Rendering#12) for the instruction!
- 2024.05.11: 🍮🍮🍮Data Part4🍮🍮🍮 is released!
- 2024.05.10: 🍹🍹🍹Data metioned in 4K4D 🍹🍹🍹 is fully released!
- 2024.04.29: 🍮🍮🍮Data Part3🍮🍮🍮 is released!
- 2024.03.08: 🍰🍰🍰Data metioned in 4K4D 🍰🍰🍰 is released!
It is NOT part of the official DNA-Rendering dataset, we released for fair comparison in the research community.
- 2023.10.28: 🔥🔥🔥Data Part2 is fully released!🔥🔥🔥
- 2023.09.28: 🍮🍮🍮Data Part2🍮🍮🍮 is released!
- 2023.08.31: 🔥🔥🔥Data Part1 is fully released!🔥🔥🔥 Except the raw RGB data and annotations, we also provide additional depth data!
- 2023.08.01: 🍮🍮🍮Data Part1🍮🍮🍮 is released!
- 2023.07.20: 🔥🔥🔥The technical report is released!🔥🔥🔥
- 2023.07.14: Our paper has been accepted by ICCV 2023!
- 2023.07.01: Technical report, data and code will be released soon. Please stay tuned!
- 2023.07.01: The demo video is uploaded. Check it out for an overview of this project!
- 2023.07.01: The project page is created.
- Scales: To our knowledge, our dataset far surpasses similar ones in terms of the number of actors, costumes, actions, clarity, and overall data volume.
scales.mp4
- Diversity: Our dataset covers a diverse range of scenarios, including everyday and special performances. It includes large variation of clothing and action types, with sufficient difficulty levels in clothing textures and motion complexity. This diversity and difficulty make it suitable for a variety of downstream research tasks.
diversity.mp4
- High-quality Annotations: Our dataset comes with off-the-shelf high-precision annotation, including 2D/3D human body keypoints, foreground masks, and SMPL-X models. We have specifically optimized our annotations for 3D human body scenarios, resulting in high-quality annotations.
annotations.mp4
- Benchmarks: We have provided the results of various state-of-the-art methods of rendering and animation on our dataset.
The dataset will be released soon.
Coming soon! We provide for each benchmark the pretrained model, code for training & evaluation reimplementation, and dataset for training.
| Benchmark | Aspect | Pretrained Model | Reimplementation | Dataset |
|---|---|---|---|---|
| instant-ngp | NovelView | |||
| NeuS | NovelView | |||
| Neural Volumes | NovelView/NovelPose | |||
| A-NeRF | NovelView/NovelPose | |||
| Neural Body | NovelView/NovelPose | |||
| Animatable Nerf | NovelView/NovelPose | |||
| HumanNeRF | NovelView/NovelPose | |||
| IBRNet | NovelID/CrossData | |||
| Pixel | NovelID/CrossData | |||
| Vision | NovelID/CrossData | |||
| Neural Human Performance | NovelID | |||
| KeyPointNerf | NovelID |
The code will be released soon!
- Release Code and pretrained model
- Release Dataset
- Technical Report
- Project page
@article{2023dnarendering,
title={DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering},
author={Wei Cheng and Ruixiang Chen and Wanqi Yin and Siming Fan and Keyu Chen and Honglin He and Huiwen Luo and Zhongang Cai and Jingbo Wang and Yang Gao and Zhengming Yu and Zhengyu Lin and Daxuan Ren and Lei Yang and Ziwei Liu and Chen Change Loy and Chen Qian and Wayne Wu and Dahua Lin and Bo Dai and Kwan-Yee Lin},
journal = {arXiv preprint},
volume = {arXiv:2307.10173},
year = {2023}
}