aejion/neuface

Official code for CVPR 2023 paper NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images.

neuface's Issues

Code converts SH to environment map

Hi, thanks so much for this amazing work!

I'm trying to visualize the environment map from the recovered lighting, and was not sure what is the right convention / coordinate system to use for correct visualization. So I was wondering if you could provide some info on how you converted SH to environment map and environment map to SH? Thanks so much!

The process of reconstructing a 3D model using multi-view images of my face

To reconstruct a 3D model using my facial images, I've seen that using ImFace is necessary for cropping the face mesh and rendering a mask for facial images.

Is it correct to use the "NeuFace/data_preprocess/cut_mesh.py" and "NeuFace/data_preprocess/render_mask.py" codes for the necessary steps in that process?

In cut_mesh.py code,
tu_mesh_path = '' # tu model path
high_mesh_path = '' # origin mesh path (high quality)
output_mesh_path = '' # output mesh path

What kind of file should I put in the tu model path?
Is it appropriate to put multi-view images on the original mesh path?

In render_mask.py code,
origin_path = r'' # origin image path
mask_path = r'./mask' # output mask path
mask_image_path = r'./image' # output image path
ply_path = r'' # cut mesh path (high quality)
tu_mesh_path = '' # tu mesh path

I would appreciate it if you could let me know what files go into each of the paths mentioned above.

Expression Transfer

Great work; thanks for this - how can we do expression transfer between 2 identities here?

Train NeuFace on captured multi-view images but do not use ImFace prior

Hi, thanks for your great work and code!

I try to run NeuFace on a multi-view face image dataset capture by myself. It contains multi-view images of a single identity.

By reading through the issues, I find that its not very easy to follow the "Train on FaceScape" pipeline. It seems that a lot of processing task should be done, including align the mesh to the ImFace coordinates, crop the face mesh, etc. So I want to follow the "Train on DTU" pipeline.

My question is that, have you tried to train NeuFace on the same identity from the FaceScape dataset, with and without the ImFace prior (in my understanding "Train on FaceScape" is the one with prior while "Train on DTU" is the one without prior)? If without the ImFace prior, can we obtain plausible relightable face reconstruction results? What about the performance drop compared to the one with prior?

Thanks.

How to export texture?

Thanks for your great work!

My goal: is to import highly detailed meshes and textures into 3D software like Blender, Unreal for rendering

How do export mesh and textures (diffuse, specular, normal)?

Like image

About dataset

Hi~  It's a really fascinating achievement！
I have some questions.After I preprocess facescape with python data_preprocess/preprocess.py in imface,do I get imface_image and image_model?
Should I use python run/train.py [--config file's name] to train imface in the next step to get imface_mask?
Because after I run train_pl.py he reports AttributeError: 'NoneType' object has no attribute 'shape'

How to use Neuface for facial reconstruction of self created photos

Thank you for your hard work. I didn't find out how to use self built data for facial reconstruction from Readme. Could you please tell me how to use self built data for facial reconstruction

Unable to train DTU dataset

Hello, your work is very excellent, but I have some problems when I want to use the DTU dataset for training. I hope you can help me answer them, if you have time.

about training data preparation

Hi~
I have some questions on the training data preparation.

According to README, the training data is from the Facescape dataset. Here → https://facescape.nju.edu.cn/Page_Data/ describes the Facescape dataset, and which part is useful (should be downloaded) for NeuFace? And after the data is acquired, which data preprocess script should I run to proprocess the data properly?

Looking forward to your reply, thank you!

Some missing input files

Hi,

It's a fascinating piece of work.

When I tried to run NeuFace on the Facescape dataset, I found some input files were missing. For example, "Rt_scale_dict.json".

How could I get these files or do I need some preprocessing on the dataset?

Thanks.

About the link to dropbox of DTU dataset

Thanks for your great work.
I am trying to evaluate the model on DTU dataset. However, the hyperlink in readme file of DTU dataset seems invalid.

How to reconstruct face model with my self multi-images?

I really appreciate your work.
Now i want to reconstruct myself face model with about 100 images, but i do not how to do with it. Can you tell me the process？

code release about relight the reconstructed scene

Hi, thanks for your great work and code.

I find there is only a novel view synthesis evaluation code. Do you have a plan to release the code to relight the reconstructed scene under a given environment map?