NTU Final Year Project (FYP): Facial Expression Transfer using StyleGAN

This project supports image and video-based facial expression transfer.
Accompanying video here

Prerequisite

1. Environment

• python 3.8.8

• torch 1.9.0+cu111

• torchvision 0.10.0+cu111

• cuda 11.7

• pillow 9.3.0

• numpy 1.10.1

• ninja 1.10.2.3

• requests 2.25.1

• pathlib2 2.3.5

• click 7.1.2

• tqdm 4.35.0

• Visual Studio 2019 (for windows to compile models)

2. Resources to be downloaded

1. data_processing/resources: download "shape_predictor_68_face_landmarks.dat" from here
2. pSp/pretrained: download "CurricularFace_Backbone.pth" from here
3. pSp/pretrained: download "psp_ffhq_encode.pt" from here
4. styleGAN2/pretrained: download "metfaces.pkl" from here
5. styleGAN2/pretrained: download "stylegan2-ffhq-config-f.pkl" from here
6. styleGAN2/resources: download "ffhq-dataset-v2.json" from here

Code usage

1. Data Processing

Change directory to "data_processing"

a) Video processing

Mode 1: frames to video
Mode 2: video to frames

NOTE: rmb to change file address in the script

python video_frames.py

b) Face alignment

Crop and align the face. Input could be an image or a folder of images.
Enable_smooth is only applicable to processing a folder of images. It assumes the person's head to stay in the same position, so the coordinates/angle of the face is only computed using the first image and applied to all the rest of images.

python face_alignment.py input_path output_path --enable_smooth

c) Latent code analysis

latent_code_analyzing.ipynb file: calculate and plot the magnitude of 18 layers of code_exp - code_neu

2. Generate images using StyleGAN2

Change directory to "styleGAN2"

a) generate random images using StyleGAN2

python generate.py --outdir=out --seeds=200-205 --network=pretrained/stylegan2-ffhq-config-f.pkl

b) Project an image to the latent space of StyleGAN2 (W space)

Obtain the latent code of an arbitrary image (in w space) using optimization method (default 1000 steps).
Results include: 1) projected image; 2) w latent code; 3) optimization progress video (set save-video to False to disable)

python projector.py --outdir=out --target=../data/frame18.jpg --network=pretrained/stylegan2-ffhq-config-f.pkl (--save-video False --num-steps 1000)

3. Generate images using pSp

Change directory to pSp

a) Embed arbitrary images to the latent space of StyleGAN2 (W+ space) using pSp encoder

parameters:
data_path: a folder containing image(s) to be embedded
exp_dir: result saving path

NOTE: if want to save inferenced latent code, go to models/stylegan2/model.py & uncomment line 489-503 & change w_plus_folder accordingly (recommend: exp_dir/w_plus)

python scripts/inference.py --data_path data/images --checkpoint_path pretrained/psp_ffhq_encode.pt --test_batch_size=1 --exp_dir data/results

Alternatively, just type.\commands.bat in terminal

b) Manipulate w+ latent code

parameters:
--target: latent code file (.npy) of target face
--source_exp: latent code file (.npy) of target expression
--source_neu: latent code file (.npy) of neutral expression
--alpha: alpha value that controlling transfer intensity
--save_dir: saving directory

python explore_w_plus.py --target target_code_file --source_exp source_exp_code_file --source_neu source_neu_code_exp ^
       --alpha alpha_value  --save_dir saving_directory

c) Generate image from w+ latent code

parameters:
input_path: a file or a folder of files containing w+ latent code (.npy)
output_path: could be an image filename (jpg/png) or a folder(default jpg)

NOTE: go to models/stylegan2/model.py & comment out line 489-503

python image_from_w_plus.py input_path output_path

d) Transfer expression for videos

This script takes target face (npy file) and driving video (a folder of npy files for all frames) as input, and outputs the transferred frames of the target video. Before running this scripts, users should have prepared all the npy files by running inference.py. Users may use data/processing/video_frames.py to combine all the frames to get the transferred video.

parameters:
target: the w+ latent code (.npy) for the target face
driving_source: a folder of latent code files where each file is for one frame of the driving video
source_neutral: a manually selected neutral expression (.npy) from the driving video
--seq_file: the sequence file generated during inference, which is used for combining transferred frames in correct sequence
--save_dir: result saving directory
--alpha: a float value controlling transfer intensity
--save_transferred_code: whether save the intermediate transferred code, which is used to generate transferred frames

NOTE: go to models/stylegan2/model.py & comment out line 489-503

python driving_video_pSp_byFrames.py target driving_source source_neutral --seq_file --save_dir --alpha --save_transferred_code

Acknowledgement

Some codes are adapted from the following repository:
https://github.com/NVlabs/stylegan2-ada-pytorch
https://github.com/eladrich/pixel2style2pixel