havenfeng/photometric_optimization

Photometric optimization code for creating the FLAME texture space and other applications

photometric_optimization's Issues

Releasing the code

Hey,
first of all thanks for your great work!

I was reading how you built the texture space from the FFHQ dataset in the Readme-file.
Is it possible that you release the code you used for building the texture space? I would be especially interested in seeing how you formulated the photometric loss.

How to customize landmark

hi, I need to replace the customized landmark for fitting. How to make the new ‘landmark_embedding.npy' file?

Custom image

Nice work!

I wonder how can I apply this for a custom image?

I guess I need both

• a landmark file ( landmarkpath = os.path.sep.join([input_folder, image_name + '.npy']) )
• image mask file ( image_mask_path = os.path.sep.join([image_mask_folder, image_name + '.npy']) )

for the custom image, right? Do I need more than this? How can I obtain these files for a custom image?

Thanks!

Question about texture_mean and texture_basis

Hi, I could not find texture_mean_image_completed.npy and texture_basis_image_completed.npy in FLAME project. Could you please share them or add them into FLAME project? Thx! @HavenFeng

Difference of flame and fwh?

Could you share your pretraind inpainting model?

Hi, I found it needs a inpainting model to complete the unvisible part of the face, could you share your pretrained inpainting model?

Is there an open source implementation of Pytorch3dRasterizer?

I was wondering if it is possible to see C++ implementation of Pytorch3dRasterizer.

Basel Face Model (BFM) to FLAME

How did you fit BFM model to FLAME? I have tried fitting a Flame mean shape to BFM mean shape, but couldn't find good shape parameters for FLAME model.

If you can give me some info on this it would be helpful.

projection

hi, many thanks for your great work! I wonder why there mul -1 here

Perspective projection

Are there any specific reasons for you using orthogonal projection instead of perspective? I have tried reconstruction of the head from the photo with your code, and it doesn't align well with the ground truth point cloud obtained through photogrammetry. I suspect the problem is with the orthogonal projection. So, my question is, can I expect it to work after replacing an ortho projection with the perspective one.
Thanks for the great work!

What is the difference between 3d lmks and 2d lmks?

Hi!
From the code, it is not clear what is the difference between 3d lmks and 2d lmks? Both have 68 points and both have an effective dimension of 3. As I can judge from the code, 3d lmks are only used for visualization. But what is the actual difference between them, and why 2dlmks have the prefix "2d" even if they are actually 3d?
Thanks

About the cam

thanks for your work!
I want to know why cam[:,0]=5?

About convergence rate

Thanks for your great work!
It can be seen from the log that Adam is very stable, but the convergence time is longer. I tried to torch.optim.lbfgs to speed up the convergence, but I don’t understand why nan always appears in training. Can you provide some solutions? Have been troubled for a long time, looking forward to your reply!

What are "Full Landmarks"?

Hi,

I looked through the code and tried to comprehend how the dynamic landmarks are calculated. Additionally to the indices and the barycentric coordinates of the dynamic and static landmarks I found full_lmk_faces_idx and full_lmk_bary_coords in the code.
What are these landmarks and why are they needed in addition to the dynamic and static landmarks?

Best,
Dominik

More detailed head reconstruction

Hello, I am a beginner, I want better head reconstruction results, how should I do it, can you give me some detailed suggestions？
Thank you.

The are a collisions in license

The Readme.md has the following text:
"The code is available for non-commercial scientific research purposes."

And the LICENSE file has a MIT license that allows commercial use.

Multiple file missing and wanting for an library list

When I am running the demo python photometric_fitting.py 00000 cuda
First problem:
FileNotFoundError: [Errno 2] No such file or directory: './data/generic_model.pkl'
I downloaded flame2023.pkl and rename to it. I am not sure this is the right thing to do.
Further another problem appeared.

/home/guest/anaconda3/envs/smart_class/lib/python3.10/site-packages/pytorch3d/io/obj_io.py:548: UserWarning: Mtl file does not exist: ./data/template.mtl
  warnings.warn(f"Mtl file does not exist: {f}")
making ./test_results//00000
----iter: 0, time: 2023-06-20-20:31:41
landmark: 0.12438790500164032, all_loss: 0.12438790500164032,
Traceback (most recent call last):
  File "/data/share/FLAME/Photometric_fitting/photometric_optimization/photometric_fitting.py", line 265, in <module>
    fitting.run(imagepath, landmarkpath)
  File "/data/share/FLAME/Photometric_fitting/photometric_optimization/photometric_fitting.py", line 218, in run
    single_params = self.optimize(images, landmarks, image_masks, savefolder)
  File "/data/share/FLAME/Photometric_fitting/photometric_optimization/photometric_fitting.py", line 91, in optimize
    util.tensor_vis_landmarks(images[visind], landmarks[visind]))
  File "/data/share/FLAME/Photometric_fitting/photometric_optimization/util.py", line 209, in tensor_vis_landmarks
    image_landmarks = plot_kpts(image, predicted_landmark, color)
  File "/data/share/FLAME/Photometric_fitting/photometric_optimization/util.py", line 250, in plot_kpts
    image = cv2.circle(image,(st[0], st[1]), 1, c, 2)
cv2.error: OpenCV(4.7.0) :-1: error: (-5:Bad argument) in function 'circle'
> Overload resolution failed:
>  - Can't parse 'center'. Sequence item with index 0 has a wrong type
>  - Can't parse 'center'. Sequence item with index 0 has a wrong type

In addition, I have to install different versions of libraries to fit the codes?
Can you please update an library list so we can install the correct versions?

Regarding a detail in the fitting code

Hi, thanks for sharing this great work!

I have a little question regarding the main fitting code in the following line:

photometric_optimization/photometric_fitting.py

Line 42 in 44fc6e0

cam = torch.zeros(bz, self.config.camera_params); cam[:, 0] = 5.

Why hard code cam[:, 0] to 5. ? Is this a default setting in FLAME so that it may benefit the convergence of the fitting process?

Thank you!

Head Pose Angles from Pose Parameters

Hi! How can I calculate the Head Pose in Euler angles from FLAME pose parameters? Thank you.

About dynamic_lmk_faces_idx

Hi
Could you tell me how to get the file "landmark_embedding.npy", especially, 'dynamic_lmk_faces_idx', THX

questions regarding more details and evaluation of the texture PCA model

Nice work! The textures look so good.
Do you have any paper for this work?
Or where can I get more details regarding the pipeline and some evaluation result of the final PCA texture model, like compactness, etc.?

Thanks a lot!

uv maps

Hi,
I am trying to create a UV map from a tracked face model using the world2uv function, but the values are in a weird range [-0.22; 0.13]. When I scale the values into [0,255], the picture looks like this:

Is this correct?