Official PyTorch implementation of EXE-GAN, publised on Neurocomputing.
[Homepage] [paper] [demo_youtube] [demo_bilibili]
Our paper has been published in a Neurocomputing Journal!!! (28 Nov. 2024). This paper has been going through a three-year review process!!
Our paper was first released on Sun, 13 Feb 2022. We are thankful for the community's recognition and attention to our project. We also recognized that there have been some great papers published after ours, and we encourage you to check out their projects as well:
- Paint by Example, codes (released at Wed, 23 Nov 2022, CVPR 2023)
- Reference-Guided Face Inpainting, codes (released at Mon, 13 Mar 2023, TCSVT 2023)
- PATMAT, codes (released at Wed, 12 Apr 2023, ICCV 2023)
cd EXE-GAN project
pip install -r requirements.txt
- Note that other versions of PyTorch (e.g., higher than 1.7) also work well, but you have to install the corresponding CUDA version.
- Training and testing codes
- Pre-trained models
- Prepare your dataset (download FFHQ, and CelebA-HQ)
- The folder structure of training and testing data is shown below:
root/
test/
xxx.png
...
xxz.png
train/
xxx.png
...
xxz.png
-
Prepare pre-trained checkpoints: Arcface.pth and psp_ffhq_encode.pt (put models in ./pre-train)
-
Training for 256X256 images
python train.py --path /root/train --test_path /root/test --size 256 --embedding_weight 0.1 --id_loss_weight 0.1 --percept_loss_weight 0.5 --arcface_path ./pre-train/Arcface.pth --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt
- Training for 512X512 images using larger masks
python train_largeMask.py --path /root/train --test_path /root/test --size 256 --embedding_weight 0.1 --id_loss_weight 0.1 --percept_loss_weight 0.5 --arcface_path ./pre-train/Arcface.pth --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt
- We found that the previous mask configuration for training 256X256 images is not appropriate for training 512X512 images.
- So we use co-mod gan mask for training, which is plugged in the train_largeMask.py
-
For editing images from the web, photos should be aligned by face landmarks and cropped to 256x256 by align_face.
-
Irregular masks (optional, if you would like to test on irregular masks, download Testing Set masks)
-
(use our FFHQ_60k pre-trained model EXE_GAN_model_256.pt and EXE_GAN_model_512.pt or trained *pt file by yourself.)
python test.py --path /root/test --size 256 --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt --ckpt ./checkpoint/EXE_GAN_model.pt --mask_root ./dataset/mask/testing_mask_dataset --mask_file_root ./dataset/mask --mask_type test_6.txt
- mask_root Irregular masks root
- mask_file_root file name list file folder
- mask_type could be ["center", "test_2.txt", "test_3.txt", "test_4.txt", "test_5.txt", "test_6.txt", "all"]
- size 256 for 256X256 images and 512 for 512X512 images
- If you don't have irregular masks, just using center masks is also fine.
python test.py --path /root/test --size 256 --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt --ckpt ./checkpoint/EXE_GAN_model.pt --mask_type center
- For editing images from the web, photos should be aligned by face landmarks and cropped to 256x256 by align_face.
(use our FFHQ_60k pre-trained model EXE_GAN_model.pt or trained *pt file by yourself.)
python guided_recovery.py --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt --ckpt ./checkpoint/EXE_GAN_model.pt --masked_dir ./imgs/exe_guided_recovery/mask --gt_dir ./imgs/exe_guided_recovery/target --exemplar_dir ./imgs/exe_guided_recovery/exemplar --sample_times 10 --eval_dir ./recover_out
- masked_dir: mask input folder
- gt_dir: the input gt_dir, used for editing
- exemplar_dir: exemplar_dir, the exemplar dir, for guiding the editing
- eval_dir: output dir
Ground-truth | Mask | Exemplar | Inpainted |
- Inherent diversity, set
--sample_times 10
higher to get more diverse results.
diversity 1 | diversity 2 | diversity 3 | diversity 4 |
- For editing images from the web, photos should be aligned by face landmarks and cropped to 256x256 by align_face.
(use our FFHQ_60k pre-trained model EXE_GAN_model.pt or trained *pt file by yourself.)
python exemplar_style_mixing.py --psp_checkpoint_path ./pre-train/psp_ffhq_encode.pt --ckpt ./checkpoint/EXE_GAN_model.pt --masked_dir ./imgs/exe_guided_recovery/mask --gt_dir ./imgs/exe_guided_recovery/target --exemplar_dir ./imgs/exe_guided_recovery/exemplar --sample_times 2 --eval_dir mixing_out
- masked_dir: mask input folder
- gt_dir: the input gt_dir, used for editing
- exemplar_dir: exemplar_dir, the exemplar dir, for guiding the editing
- eval_dir: output dir
- size 256 for 256X256 images and 512 for 512X512 images
- Inputs are shown below:
Ground-truth | Mask | Exemplar 1 | Exemplar 2 |
- Style mixing results
We also uploaded the mask editing tool. You can try this tool to generate your masks for editing.
python mask_gui.py
- If you find our code useful, please cite our paper:
@article{LU2025128996, title = {Do inpainting yourself: Generative facial inpainting guided by exemplars}, journal = {Neurocomputing}, volume = {617}, pages = {128996}, year = {2025}, issn = {0925-2312}, doi = {https://doi.org/10.1016/j.neucom.2024.128996}, url = {https://www.sciencedirect.com/science/article/pii/S0925231224017673}, author = {Wanglong Lu and Hanli Zhao and Xianta Jiang and Xiaogang Jin and Yong-Liang Yang and Kaijie Shi}, keywords = {Generative adversarial networks, Image generation, Image inpainting, Facial image inpainting}, } @ARTICLE{FACEMUG, author={Lu, Wanglong and Wang, Jikai and Jin, Xiaogang and Jiang, Xianta and Zhao, Hanli}, journal={IEEE Transactions on Visualization and Computer Graphics}, title={FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing}, year={2024}, volume={}, number={}, pages={1-15}, keywords={Facial features;Semantics;Codes;Generators;Image synthesis;Faces;Image color analysis;Generative adversarial networks;image-toimage translation;multimodal fusion;image editing;facial editing}, doi={10.1109/TVCG.2024.3434386}}
Model details and custom CUDA kernel codes are from official repositories: https://github.com/NVlabs/stylegan2
Codes for Learned Perceptual Image Patch Similarity, LPIPS came from https://github.com/richzhang/PerceptualSimilarity
To match FID scores more closely to tensorflow official implementations, I have used FID Inception V3 implementations in https://github.com/mseitzer/pytorch-fid