yeongjoonju/cfr-gan

Complete Face Recovery GAN: Unsupervised Joint Face Rotation and De-Occlusion from a Single-View Image (WACV 2022)

Yeong-Joon Ju

😄 Profile

🎓 Affiliation: Pattern Recognition and Machine Learning Lab in Korea University (2020~)

I'm researching Neural Network Dissection interpreting hidden units in neural networks and Language Modeling adapted to specific domains or languages.

Research direction: face generation (vision) $\rightarrow$ dialogue generation (NLP) $\rightarrow$ neural network debugging (XAI) $\rightarrow$ 🌞 language modeling for multimodalities (NLP)

📃 Publications

• Y-J. Ju, J-H. Park, and S-W. Lee, "Neuron-based Debugging Framework for Neural Networks Using Class-conditional Visualizations", Arxiv, 2023 | code

  History -> NeurIPS 2023: rejected - 3 borderline accept and 1 reject

• Y-J. Ju, G-H. Lee, J-H. Hong, and S-W. Lee, "Complete Face Recovery GAN: Unsupervised Joint Face Rotation and De-Occlusion from a Single-View Image", WACV, 2022 | code

• J-H. Park, Y-J. Ju, and S-W. Lee, "Explaining Generative Diffusion Models via Visual Analysis for Interpretable Decision-Making Process", Expert Systems with Applications, 2024 | code

• T-J. Woo, W-J. Nam, Y-J. Ju, and S-W. Lee, "Compensatory Debiasing for Gender Imbalances in Language Models", ICASSP, 2023 (oral) | code

  Oral speaker: Yeong-Joon Ju

💻 Research Projects (national or industry-academic projects)

• Research on AI-based large-scale language modeling techniques for Military-related topic (hidden), Agency for Defense Development (ADD, 국방과학연구소), Korea (Aug. 2023 ~ Nov. 2023) Team leader (practitioner)

• Development of an AI assistant for elementary education teacher based on a large language model (LLM), i-Scream Edu (아이스크림에듀), Korea (May 2023 ~ Jan. 2024) Team leader (practitioner)

  Research of LLM for NLG

• Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation, Institute of Information & communications Technology Planning & Evaluation (IITP), Korea (Jun. 2022 ~ Dec. 2023) Team leader (practitioner)

  Research of plug-and-play debugging for the decision-making process of neural networks

• Development of facial skin change analysis and visualization techniques, Samsung Electronics (삼성전자), Korea (Jun. 2020 ~ Jun. 2021)

  Research of facial dense alignment technique based on optical flow

👍 Awards, etc.

• Reviewer at IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

• Grand Prize in The 6th SW·AI Hackerton, Sejong University, Korea (Jul. 2019)

  Project: Android application that manages seat status for discussion lounge / Development of seat status detection model and API based on Flask (Jul. 2019 ~ Jul. 2019)

• Grand Prize in The 7th Creative Design Competition, Sejong University, Korea (Jun. 2019)

  Project: Web application that automatically blurs out license plates and faces / Development of web application based on React.js, detection models, and API. (Mar. 2019 ~ Jun. 2019)

cfr-gan's Issues

error when processing folder with many images

!python inference.py --img_path /content/inp4 --save_path /content/out4/ --generator_path "/content/CFRNet_G_ep55_vgg.pth" --estimator_path "/content/trained_weights_occ_3d.pth" --aligner retinaface

Loading pretrained model from Pytorch_Retinaface/weights/Resnet50_Final.pth
remove prefix 'module.'
Missing keys:0
Unused checkpoint keys:0
Used keys:456
Finished loading detection model!
Load /content/trained_weights_occ_3d.pth
  5% 84/1743 [00:26<08:40,  3.18it/s]
Traceback (most recent call last):
  File "inference.py", line 101, in <module>
    input_img = estimator3d.align_convert2tensor(img_list[k:until], aligned=(args.aligner is None))
  File "/content/CFR-GAN/generate_pairs.py", line 228, in align_convert2tensor
    lm = self.estimate_five_landmarks(img)
  File "/content/CFR-GAN/generate_pairs.py", line 205, in estimate_five_landmarks
    landm = np.array(landms[0])
IndexError: index 0 is out of bounds for axis 0 with size 0

The code in this article cannot find the problem with the path

Download our trained weights.

Download BFM_model_80.mat and 3D face estimator weights here.
The next step
python inference.py --img_path test_imgs/input --save_path test_imgs/output --generator_path [saved_path] --estimator_path [saved_path]
generator_path ？
estimator_path？

About Paper

Thank you for your remarkable work!
I am interested in your work, but I can't find your paper.
How can I download this paper?
Thanks!

Instructions for download link and storage path of checkpoint files

The README is a complete mess, and the download links of the checkpoints are disorienting. It seems like the authors don't want you to smoothly run their code. You have to guess and experiment to have any chance of success. When you ask questions in the issues section, they even sarcastically blame you for not understanding the documentation.

Here are the download links that I have verified along with their relative paths.
(1)./model/CFRNet_G_ep55_vgg.pth:
https://koreaoffice-my.sharepoint.com/personal/yj_ju_korea_ac_kr/_layouts/15/download.aspx?SourceUrl=%2Fpersonal%2Fyj%5Fju%5Fkorea%5Fac%5Fkr%2FDocuments%2FCFR%2DGAN%2FCFRNet%5FG%5Fep55%5Fvgg%2Epth
(2)./mmRegressor/BFM/BFM_model_80.mat:
https://koreaoffice-my.sharepoint.com/personal/yj_ju_korea_ac_kr/_layouts/15/download.aspx?SourceUrl=%2Fpersonal%2Fyj%5Fju%5Fkorea%5Fac%5Fkr%2FDocuments%2FCFR%2DGAN%2FBFM%5Fmodel%5F80%2Emat
(3)./mmRegressor/BFM/trained_weights_occ_3d.pth:
https://koreaoffice-my.sharepoint.com/personal/yj_ju_korea_ac_kr/_layouts/15/download.aspx?SourceUrl=%2Fpersonal%2Fyj%5Fju%5Fkorea%5Fac%5Fkr%2FDocuments%2FCFR%2DGAN%2Ftrained%5Fweights%5Focc%5F3d%2Epth
(4)./mmRegressor/BFM/similarity_Lm3D_all.mat:
https://github.com/microsoft/Deep3DFaceReconstruction/tree/master/BFM

different rotation angles

Can we only restore the front face, but not with different rotation angles?

similarity_Lm3D_all.mat

where can I get similarity_Lm3D_all.mat ? Thank you for your reply !

can't install a requierment!

hi!
the package pytorch3d can't be install through pip or directly from repository!

where can I get 3D face estimator

Link to weights + Bfm doesnt exists

Hi,

I tried to download your network weights + Bfm but the links doesnt seems to work. Could you update them pls ?
Thanks in advance

Second question, thanks to the author

First of all, thank you very much for your answer
If you have the time,

Could you give me an example

I thank you very much

I have read https://github.com/yeongjoonJu/CFR-GAN/blob/main/Readme.md

and

https://github.com/yeongjoonJu/Occlusion-Robust-3D-Face-CFR-GAN/blob/main/readme.md

But I have not been able to reproduce your code CFR-GAN

I downloaded the git clone https://github.com/biubug6/Pytorch_Retinaface.git

BFM_model_80.mat

CFRNet_G_ep55_vgg.pth

Download weights of face parsing networks to faceParsing folder.

Download weights of baseline 3D networks to mmRegressor/network folder.

Download weights of face recognition networks to saved_models folder.

But I still don't know what to do next,

Thank you very much

Thank you, thank you

It would be even better if you could give me a video demonstration

Thank you very much

help!!!!!

hey even I am doing a project on face de occlusion does the given code contain the dataset and can you provide the project report of this code

Occlusion Detection?

How can I pass in an image of a face and determine whether or not it is occluded?

Thanks!

About the quality of generated pictures

I put my pics and run the inference.py，but the quality of the generated image is very poor. Not only the background, but also the face quality is very poor. How to improve it?

가중치 다운로드 불가

죄송합니다. 요청한 파일이 없습니다.

구글 드라이브로 trained weights를 다운 받을 수 없습니다.

혹시 해결 다운 받을 수 있는 방법이 있는지 궁금합니다.

감사합니다.

Trying to reproduce the paper results

Thank you for your big effort and for sharing the pre-trained model.

It seems that something is missing from my steps.

I'm trying to run the model on this image from the paper:

But here is the result I get:

Which is different from the output on the paper.

I did align the image using RetinaFace by cloning the repo and downloading the pre-trained weights and running the command:

!python /CFR-GAN/Code/inference.py --img_path /CFR-GAN/Input_images --save_path /CFR-GAN/Output_images --generator_path '/CFR-GAN/Saved_models/CFRNet_G_ep55_vgg.pth' --estimator_path '/CFR-GAN/Saved_models/trained_weights_occ_3d.pth' --aligner retinaface

What did I miss? Thanks in advance.