more commits

able2608 · Aug 1, 2024 · 89e3bad · 89e3bad
commit 89e3bad
Show file tree

Hide file tree

Showing 119 changed files with 17,106 additions and 0 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,18 @@
+*.pkl
+*.pt
+*.mov
+*.pth
+*.mov
+*.npz
+*.npy
+*.boj
+*.onnx
+*.tar
+*.bin
+cache*
+.DS_Store
+*DS_Store
+outputs/
+**/__pycache__
+***/__pycache__
+*/__pycache__
diff --git a/README.md b/README.md
@@ -0,0 +1,109 @@
+<!-- main documents -->
+
+
+<div align="center">
+
+
+#This is my ComfyUi-windows implementation for the image animation project -> UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation
+
+[🎨 Source Project Page](https://unianimate.github.io/)
+
+</div>
+
+
+## Getting Started
+
+The ComfyUI nodes created are "Align & Generate poses for UniAnimate" & "Animate image with UniAnimate"
+
+I used a ComfyUI_windows_portable to test the nodes in a Windows 10 OS with 16GB RAM & 12GB VRAM Nvidia Graphics Card
+
+Download or clone this repository and place it in ComfyUI_windows_portable\ComfyUI\custom_nodes\
+
+You will need python>=3.9 in your ComfyUI Environment.
+I tested the project with the following pytorch versions which you can install as follows
+
+```
+conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
+
+``` 
+
+Or
+
+```
+conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
+
+```
+
+If not installed, then:
+
+
+```
+pip install opencv-python
+pip install pytorch_lightning
+pip install lightning_utilities #if not installed
+pip install lightning_fabric #if not installed
+pip install torchmetrics
+pip install xFormers = 0.0.20 or copy torch 2.0.1 and supporting libraries and xFormer from A1111 and place in your Environment\Lib\site-packages (or)
+pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118 (for pytorch==2.3.1)
+pip install oss2
+pip install einops
+pip install args
+pip install modelscope
+
+```
+
+
+Download the required models (Around 14GB) after installing modelscope :
+
+```
+python modeldownloader.py
+
+```
+
+After downloading all the models, move them manually from 'checkpoints/iic/unianimate/' to the 'checkpoints' directory
+Or move them via your command line interface:
+
+```
+python mv ./checkpoints/iic/unianimate/* ./checkpoints/
+
+```
+
+All the models should be in the  '\Path-to-UniAnimate\checkpoints' folder as follows:
+
+
+```
+./checkpoints/
+|---- dw-ll_ucoco_384.onnx
+|---- open_clip_pytorch_model.bin
+|---- unianimate_16f_32f_non_ema_223000.pth 
+|---- v2-1_512-ema-pruned.ckpt
+└---- yolox_l.onnx
+
+```
+
+You can now upload the workflow in your '\Path-to-UniAnimate\' folder which is titled 'basicUniAnimateWorkflow.json', install missing custom nodes with the ComfyUI Manager if necessary, upload a picture & video (You can use those in the 'assets' folder), and run!
+
+
+**<font color=red>&#10004; Some tips</font>**:
+
+- > In the 'Align & Generate poses for UniAnimate' node, the first frame in the target pose sequence is used to calculate the scale coefficient of the alignment. Therefore, if the first frame in the target pose sequence contains the entire face and pose (hand and foot), it can help obtain more accurate estimation and better video generation results.
+
+- > Generating 32 frames of video with a resolution of [512, 768] usually takes about 7 minutes.
+
+
+
+- > To run the "Animate image with UniAnimate" node, **~12G** GPU memory will be used. If your GPU is smaller than this, you can change the  `max_frames: 32` to other values, e.g., 24, 16, and 8.
+
+You can also generate a video first, and then upload the last frame of the video as a pic to generate the next frames.
+
+
+@article{
+      project={ComfyUi-windows implementation for the image animation project -> UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation},
+      developer={Isimemen Omoifo Jnr},
+      year={2024}
+}
+
+
+## Disclaimer
+
+I explicitly disclaim any responsibility for user-generated content. Users are solely liable for their actions while using these nodes and the generative model. I and the source project contributors have no legal affiliation with, nor accountability for, users' behaviors. It is imperative to use these nodes and the generative model responsibly, adhering to both ethical and legal standards.
diff --git a/__init__.py b/__init__.py
@@ -0,0 +1,3 @@
+from .uniAnimate_Inference import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
+
+__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
diff --git a/assets/1.gif b/assets/1.gif
diff --git a/assets/2.gif b/assets/2.gif
diff --git a/assets/3.gif b/assets/3.gif
diff --git a/assets/4.gif b/assets/4.gif
diff --git a/assets/5.gif b/assets/5.gif
diff --git a/assets/6.gif b/assets/6.gif
diff --git a/assets/7.gif b/assets/7.gif
diff --git a/assets/8.gif b/assets/8.gif
diff --git a/assets/images/IMG_20240514_104337.jpg b/assets/images/IMG_20240514_104337.jpg
diff --git a/assets/images/WOMEN-Blouses_Shirts-id_00004955-01_4_full.jpg b/assets/images/WOMEN-Blouses_Shirts-id_00004955-01_4_full.jpg
diff --git a/assets/images/WOMEN-Blouses_Shirts-id_00005125-03_4_full.jpg b/assets/images/WOMEN-Blouses_Shirts-id_00005125-03_4_full.jpg
diff --git a/assets/images/musk.jpg b/assets/images/musk.jpg
diff --git a/assets/network.png b/assets/network.png
diff --git a/assets/teaser.png b/assets/teaser.png
diff --git a/assets/videos/0001-0032.mp4 b/assets/videos/0001-0032.mp4
diff --git a/assets/videos/dance.mp4 b/assets/videos/dance.mp4
diff --git a/assets/videos/source_video.mp4 b/assets/videos/source_video.mp4
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		from .uniAnimate_Inference import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS

		__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']