The training loss is always small and does not converge

[youngstu]

The training loss is always small and does not converge

[TimoBolkart]

You are right, the loss is always small as VOCA regresses vertex offsets. Rather than on the printed loss I recommend checking the Tensorboard graphs which show the progress better

[youngstu]

You are right, the loss is always small as VOCA regresses vertex offsets. Rather than on the printed loss I recommend checking the Tensorboard graphs which show the progress better

And I try to regress the blend shape coefficient directly, but it doesn't converge

[TimoBolkart]

What are you referring to as blend shape paramters? FLAME jaw pose and expression parameters?
How did you obtain these parameters?

[youngstu]

By the 3DMM method，only compute the expression parameters.

[TimoBolkart]

What is the 3DMM method? Can you please be a bit more specific?
If you use FLAME, the expression parameters are insufficient, as opening the mouth is modeled as jaw rotation. In this case you would need to regress the three parameters for jaw AND the expression parameters.

[youngstu]

https://cseweb.ucsd.edu/~ravir/6998/papers/p187-blanz.pdf

[TimoBolkart]

I am well aware of this paper, the model however for this is not publicly available.
So what model did you use and how did you compute the model parameters?

[youngstu]

The parameters of face identity and expression parameters are obtained by 3dmm face reconstruction, and then the expression parameters are used as ground truth value of voca to train.

[TimoBolkart]

So you fitted the model to images to get identity and expression parameters (supposedly of Basel Face model)? Is that what you did?
What data did you use to fit the model to?

[youngstu]

yes，i do. Some videos downloaded from the internet.

[youngstu]

These videos are mainly some speech videos of the host. And network removing the output_decoder module，regress the expression coeffs directly.

[TimoBolkart]

It is quite difficult to judge what is the problem as you are using 1) different data which are not comparable to the quality of the VOCA training data, 2) a different statistical model and 3) a different loss as you are regressing model parameters rather than vertex offsets. For this you must have changed the provided architecture.

If you want to experiment with regressing model parameters, I would recommend starting with parameters for the VOCA training data. Given that the data are provided in FLAME topology, it is easy to compute model parameters for these data. In this case you can at least rely on the amount and quality of the training data.

[youngstu]

Yes, I have made some changes to the code based on the voca structure, mainly the input and output parts.

Later, I will try a direct regression expression coefficient experiment based on the voca training set.

Thank you very much for your reply. Hope to communicate with you latter.

[TimoBolkart]

Having the right size and quality of the training data is quite essential for training. Good luck with your further experiments.

[youngstu]

I quite agree with you. Thanks.

[youngstu]

In the training set, the loss is small, but in the verification set, the loss is relatively large. Do you have any suggestions to solve the fitting problem?
I found a phenomenon that in the beginning of some iterations, the loss on the verification set is decreasing, and the more training iterations, the loss will even increase. Is it over fitting due to overtraining？

[QifengDai]

@youngstu Hi, I have the same problem as you, regressing bs coefficients but not converging, similarly, the verification loss is large, did you solve this problem?Thanks.