MangaNinja: Line Art Colorization with Precise Reference Following

This repository represents the official implementation of the paper titled "MangaNinja: Line Art Colorization with Precise Reference Following".

Zhiheng Liu* · Ka Leong Cheng* · Xi Chen · Jie Xiao · Hao Ouyang · Kai Zhu · Yu Liu · Yujun Shen · Qifeng Chen · Ping Luo

We propose MangaNinja, a reference-based line art colorization method. MangaNinja automatically aligns the reference with the line art for colorization, demonstrating remarkable consistency. Additionally, users can achieve more complex tasks using point control. We hope that MangaNinja can accelerate the colorization process in the anime industry.

📢 News

• 2025-01-15: Inference code and paper are released.
• 2025-01-16: MangaNinja is available on windows, 6G VRAM need Auto install and Download Model. Thanks @sdbds ! You can found it here. 🔥
• 🏃: We will open an issue area to investigate user needs and adjust the model accordingly. This includes more memory-efficient structures, data formats for line art (such as binary line art), and considering retraining MangaNinjia on a better foundation model (sd3,flux).

🛠️ Setup

📦 Repository

Clone the repository (requires git):

git clone https://github.com/ali-vilab/MangaNinjia.git
cd MangaNinjia

💻 Dependencies

Install with conda:

conda env create -f environment.yaml
conda activate MangaNinjia

⚙️ Weights

• You could download them from HuggingFace: StableDiffusion, clip-vit-large-patch14, control_v11p_sd15_lineart and Annotators
• You could download our MangaNinjia model from HuggingFace
• The downloaded checkpoint directory has the following structure:

-- checkpoints
    |-- StableDiffusion
    |-- models
        |-- clip-vit-large-patch14
        |-- control_v11p_sd15_lineart
        |-- Annotators
            |--sk_model.pth
    |-- MangaNinjia
        |-- denoising_unet.pth
        |-- reference_unet.pth
        |-- point_net.pth
        |-- controlnet.pth

🎮 Inference

cd scripts
bash infer.sh

You can find all results in output/. Enjoy!

📍 Inference settings

The default settings are optimized for the best result. However, the behavior of the code can be customized:

• --denoise_steps: Number of denoising steps of each inference pass. For the original (DDIM) version, it's recommended to use 20-50 steps.
• --is_lineart: If the user provides an image and the task is to color the line art within that image, this parameter is not needed. However, if the input is already a line art and no additional extraction is necessary, then this parameter should be included.
• --guidance_scale_ref: Increasing makes the model more inclined to accept the guidance of the reference image.
• --guidance_scale_point: Increasing makes the model more inclined to input point guidance to achieve more customized colorization.
• --point_ref_paths and --point_lineart_paths (optional): Two 512x512 matrices are used to represent the matching points between the corresponding reference and line art with continuously increasing integers. That is, the coordinates of the matching points in both matrices will have the same values: 1, 2, 3, etc., while the values in other positions will be 0 (you can refer to the provided samples). Of course, we recommend using Gradio for point guidance.

🌱 Gradio

First, modify ./configs/inference.yaml to set the path of model weight. Afterwards, run the script:

python run_gradio.py

The gradio demo would look like the UI shown below.

A biref tutorial:

1. Upload the reference image and target image.

   Note that for the target image, there are two modes: you can upload an RGB image, and the model will automatically extract the line art; or you can directly upload the line art by checking the 'input is lineart' option.

   The line art images are single-channel grayscale images, where the input consists of floating-point values with the background set to 0 and the line art close to 1. Additionally, we would like to further communicate with our users: if the line art you commonly use is binarized, please let us know. We will fine-tune the model and release an updated version to better suit your needs. 😆

2. Click 'Process Images' to resize the images to 512*512 resolution.

3. (Optional) Starting from the reference image, alternately click on the reference and target images in sequence to define matching points. Use 'Undo' to revert the last action.

4. Click 'Generate' to produce the result.

🌺 Acknowledgements

This project is developped on the codebase of MagicAnimate. We appreciate this great work!

🎓 Citation

Please cite our paper:

@article{liu2024manganinja,
  author  = {Zhiheng Liu and Ka Leong Cheng and Xi Chen and Jie Xiao and Hao Ouyang and Kai Zhu and Yu Liu and Yujun Shen
             and Qifeng Chen and Ping Luo},
  title   = {MangaNinja: Line Art Colorization with Precise Reference Following},
  journal = {CoRR},
  volume  = {abs/xxxx.xxxxx},
  year    = {2024}
}