From 67909dc8ea3f1afee97603537d69102593528fde Mon Sep 17 00:00:00 2001 From: Guangkai Date: Fri, 25 Oct 2024 16:50:37 +0800 Subject: [PATCH] readme --- .gitignore | 3 -- README.md | 98 ++++++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 81 insertions(+), 20 deletions(-) diff --git a/.gitignore b/.gitignore index 8896a15..8de38fb 100755 --- a/.gitignore +++ b/.gitignore @@ -9,9 +9,6 @@ /venv /cache /datasets* -/prs-eth -/script/batch_train -/script/train_debug /pretrained_weights /weights diff --git a/README.md b/README.md index 1f6fc8e..1a9c102 100755 --- a/README.md +++ b/README.md @@ -37,6 +37,23 @@ Zhejiang University - 2024.3.10: Release [arXiv v1 paper](https://arxiv.org/abs/2403.06090v1). +## 📚 Download Resource Summary + + - Space-Huggingface demo: https://huggingface.co/spaces/guangkaixu/GenPercept. + - Models-all (including ablation study): https://huggingface.co/guangkaixu/genpercept-exps. + - Models-main-paper: https://huggingface.co/guangkaixu/genpercept-models. + - Models-depth: https://huggingface.co/guangkaixu/genpercept-depth. + - Models-normal: https://huggingface.co/guangkaixu/genpercept-normal. + - Models-dis: https://huggingface.co/guangkaixu/genpercept-dis. + - Models-matting: https://huggingface.co/guangkaixu/genpercept-matting. + - Models-seg: https://huggingface.co/guangkaixu/genpercept-seg. + - Models-disparity: https://huggingface.co/guangkaixu/genpercept-disparity. + - Models-disparity-dpt-head: https://huggingface.co/guangkaixu/genpercept-disparity-dpt-head. + - Datasets-input demo: https://huggingface.co/datasets/guangkaixu/genpercept-input-demo. + - Datasets-evaluation data: https://huggingface.co/datasets/guangkaixu/genpercept_datasets_eval. + - Datasets-evaluation results: https://huggingface.co/datasets/guangkaixu/genpercept-exps-eval. + + ## 🖥️ Dependencies ```bash @@ -53,22 +70,33 @@ Download the [stable-diffusion-2-1](https://huggingface.co/stabilityai/stable-di Then, place images in the ```./input/``` dictionary. We offer demo images in [Huggingface](guangkaixu/genpercept-input-demo), and you can also download with the script ```script/download_sample_data.sh```. Then, run inference with scripts as below. ```bash -# depth -bash script/infer/main_paper/inference_genpercept_depth.sh -# normal -bash script/infer/main_paper/inference_genpercept_normal.sh -# dis -bash script/infer/main_paper/inference_genpercept_dis.sh -# matting -bash script/infer/main_paper/inference_genpercept_matting.sh -# seg -bash script/infer/main_paper/inference_genpercept_seg.sh -# disparity -bash script/infer/main_paper/inference_genpercept_disparity.sh -# disparity_dpt_head -bash script/infer/main_paper/inference_genpercept_disparity_dpt_head.sh +# Depth +source script/infer/main_paper/inference_genpercept_depth.sh +# Normal +source script/infer/main_paper/inference_genpercept_normal.sh +# Dis +source script/infer/main_paper/inference_genpercept_dis.sh +# Matting +source script/infer/main_paper/inference_genpercept_matting.sh +# Seg +source script/infer/main_paper/inference_genpercept_seg.sh +# Disparity +source script/infer/main_paper/inference_genpercept_disparity.sh +# Disparity_dpt_head +source script/infer/main_paper/inference_genpercept_disparity_dpt_head.sh ``` +If you would like to change the input folder path, unet path, and output path, input these parameters like: +```bash +# Assign a values +input_rgb_dir=... +unet=... +output_dir=... +# Take depth as example +source script/infer/main_paper/inference_genpercept_depth.sh $input_rgb_dir $unet $output_dir +``` +For a general inference script, please see ```script/infer/inference_general.sh``` in detail. + ***Thanks to our one-step perception paradigm, the inference process runs much faster. (Around 0.4s for each image on an A800 GPU card.)*** @@ -135,9 +163,45 @@ with torch.inference_mode(): cv2.imwrite("output_segmentation_map.png", segmentation) ``` --> -## 🔧 Train +## 🔥 Train + +NOTE: We implement the training with the [accelerate](https://github.com/huggingface/accelerate) library, but find a worse training accuracy with multi gpus compared to one gpu, with the same training ```effective_batch_size``` and ```max_iter```. Your assistance in resolving this issue would be greatly appreciated. Thank you very much! + +### Preparation + +Datasets: TODO + +Place training datasets unser ```datasets/``` + +Download the [stable-diffusion-2-1](https://huggingface.co/stabilityai/stable-diffusion-2-1) from HuggingFace and put the checkpoints under ```./pretrained_weights/```. You can also download with the script ```script/download_sd21.sh```. + + +### Start Training + +The reproduction training scripts in [arxiv v3 paper](https://arxiv.org/abs/2403.06090v3) is released in ```script/```, whose configs are stored in ```config/```. Models with ```max_train_batch_size > 2``` are trained on an H100 and ```max_train_batch_size <= 2``` on an RTX 4090. Run the train script: + +```bash +# Take depth training of main paper as an example +source script/train_sd21_main_paper/sd21_train_accelerate_genpercept_1card_ensure_depth_bs8_per_accu_pixel_mse_ssi_grad_loss.sh +``` + +## 🎖️ Eval + +### Preparation + +1. Download [evaluation datasets](https://huggingface.co/datasets/guangkaixu/genpercept_eval/tree/main) and place them in ```datasets_eval```. +2. Download [our trained models](https://huggingface.co/guangkaixu/genpercept-exps) of main paper and ablation study in Section 3 of [arxiv v3 paper](https://arxiv.org/abs/2403.06090v3), and place them in ```weights/genpercept-exps```. + +### Start Evaluation + +The evaluation scripts are stored in ```script/eval_sd21```. + +```bash +# Take "ensemble1 + step1" as an example +source script/eval_sd21/eval_ensemble1_step1/0_infer_eval_all.sh +``` + -TODO ## 📖 Recommanded Works @@ -146,7 +210,7 @@ TODO - FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models. [arXiv](https://arxiv.org/abs/2308.05733), [GitHub](https://github.com/aim-uofa/FrozenRecon). -## 🏅 Results in Paper +## 👍 Results in Paper ### Depth and Surface Normal