A python implementation of portrait lighting transfer using a mass transport approach.
3DMM model in original implementation is replaced by PRNet.
input img | reference img | relighting with color | with light
- python3.6
- dependencies
- download 256_256_resfcn256_weight.data-00000-of-00001 from GoogleDrive and put into folder data/net-data/
- run
sh env.sh
- Compile c++ files to .so for python use (ignore if you use numpy version)
cd python_portrait_relight/cython/ python setup.py build_ext -i
# python demo.py
# fast = False(rendering with numpy version)
portrait_s1.jpg: 800x800x3
portrait_r1.jpg: 873x799x3
render_texture: 16.000
render_texture: 13.967
relight time with color: 36.16
render_texture: 13.871
render_texture: 13.617
relight time with light: 30.65
...
# python demo.py
# fast = True(rendering with c++ version)
portrait_s1.jpg: 800x800x3
portrait_r1.jpg: 873x799x3
render_texture: 0.026
render_texture: 0.024
relight time with color: 6.27
render_texture: 0.025
render_texture: 0.024
relight time with light: 3.20
...Let input image be I, reference image be R and output image be O.
Let posI, posR be frontal 3d face position map of img I, R, with shape=[n, 3].\
Let colorI, colorR be rgb colors of the reconstructed vertices of img I, R, with shape=[n, 3].
Let normalI, normalR be normal vectors of the vertices of img I, R, with shape=[n, 3].
We obtain features fI=[colorI, posI[:,:,:2], nomralI], fR=[colorR, posR[:,:,:2], normalR] of img I, R, with shape=[n, 8].
Then we determine pdf transfer function t, so that f{t(fI)}=f{fR}, where f{x} is the probability density function of array x.
t(colorI) is the relighted image of I with R for reference.
Finally, we apply regrain algorithm for postprocessing.
- 3D Face Reconstruction and Dense Alignment. YadiraF/PRNet
- Face detection. biubug6/Pytorch_Retinaface
- pdf-transfer and regrain process. pengbo-learn/python-color-transfer
portrait lighting transfer using a mass transport approach
by Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris and Dimitris Samaras.
Author's matlab implementation