mariawang96/mfirrn

This repository contains the source codes for the paper "MULTI-GRANULARITY FEATURE INTERACTION AND RELATION REASONING FOR 3D DENSE ALIGNMENT AND FACE RECONSTRUCTION (ICASSP 2021)"

MFIRRN

Dense aligment Results

3D Face Reconstruction Result

->->->->->->

Citing this work

If you find our code or paper useful, please consider citing

@inproceedings{li2021multi,
  title={Multi-Granularity Feature Interaction and Relation Reasoning for 3D Dense Alignment and Face Reconstruction},
  author={Li, Lei and Li, Xiangzheng and Wu, Kangbo and Lin, Kui and Wu, Suping},
  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={4265--4269},
  year={2021},
  organization={IEEE}
}

Installation

First you have to make sure that you have all dependencies in place.

You can create an anaconda environment called mfirrn using

conda env create -n mfirrn python=3.6 ## recommended python=3.6+
conda activate mfirrn
sudo pip3 install torch torchvision 
sudo pip3 install numpy scipy matplotlib
sudo pip3 install dlib
sudo pip3 install opencv-python
sudo pip3 install cython

Then, download the baseline code.

• download the 3DDFA

Next, compile the extension modules.

cd utils/cython
python3 setup.py build_ext -i

Final, adopt our model in baseline code.

Copy our model 'model/Mfirrn' to  baseline
Replace "train.py" in the baseline with our "train.py"
Replace "benchmark.py" in the baseline with our "benchmark.py"

Generation

To generate results using a trained model, use

python3 main.py -f samples/test.jpg 

Note that we suggest you choose normal image due to dlib restrictions on face capture

• download our pre-trained model Mfirrn via Google

Due to the randomness of multi-granularity segmentation, the evaluation result will fluctuate in the range of 3.650 to 3.690.

Evaluation

To eval our MFIRRN , use

python benchmark.py

Training

To train our MFIRRN with wpdc Loss, use

cd training
bash train_wqdc.sh

Quantitative Results

NME2D	AFLW2000-3D Dataset (68 pts)	AFLW Dataset (21 pts)
Method	[0,30],[30,60],[60,90], Mean, Std	[0,30],[30,60],[60,90], Mean, Std
CDM	-, -, -, -, -	8.150, 13.020, 16.170, 12.440, 4.040
RCPR	4.260, 5.960, 13.180, 7.800, 4.740	5.430, 6.580, 11.530, 7.850, 3.240
ESR	4.600, 6.700, 12.670, 7.990, 4.190	5.660, 7.120, 11.940, 8.240, 3.290
SDM	3.670, 4.940, 9.760, 6.120, 3.210	4.750, 5.550, 9.340, 6.550, 2.450
DEFA	4.500, 5.560, 7.330, 5.803, 1.169	-, -, -, -, -
3DDFA(CVPR2016)	3.780, 4.540, 7.930, 5.420, 2.210	5.000, 5.060, 6.740, 5.600, 0.990
Nonlinear(CVPR2018)	-, -, -, 4.700, -	-, -, -, -, -
DAMDNet(ICCVW19)	2.907, 3.830, 4.953, 3.897, 0.837	4.359, 5.209, 6.028, 5.199, 0.682
MFIRRN	2.841, 3.572, 4.561, 3.658, 0.705	4.321, 5.051, 5.958, 5.110, 0.670

Qualitative Results of Dense Aligment

Qualitative Results of 3D Reconstruction

Futher Information

If you have any problems with the code, please list the problems you encountered in the issue area, and I will reply you soon. Thanks for baseline work 3DDFA.