Problem in Reproducing the Depth score

[VIROBO-15]

Problem in Reproducing the Depth score

[Xiaoming-Zhao]

Can I know what is the batch size you used during training? E.g., what is the number of GPUs you used?

And which checkpoint you used for evaluation, e.g., how many iterations you trained?

[VIROBO-15]

I have used these hyperparameters
curriculum: {'res_dict': {256: {'batch_size': 8, 'num_steps': 32, 'img_size': 256, 'tex_size': 256, 'batch_split': 1, 'gen_lr': 0.002, 'disc_lr': 0.002}, 512: {'batch_size': 4, 'num_steps': 32, 'img_size': 512, 'tex_size': 512, 'batch_split': 1, 'gen_lr': 0.002, 'disc_lr': 0.002}, 1024: {'batch_size': 4, 'num_steps': 32, 'img_size': 1024, 'tex_size': 1024, 'batch_split': 2, 'gen_lr': 0.002, 'disc_lr': 0.002}}, 'res_dict_learnable_param': {256: {'batch_size': 4, 'num_steps': 32, 'img_size': 256, 'tex_size': 256, 'batch_split': 1, 'gen_lr': 0.002, 'disc_lr': 0.002}, 512: {'batch_size': 4, 'num_steps': 32, 'img_size': 512, 'tex_size': 512, 'batch_split': 2, 'gen_lr': 0.002, 'disc_lr': 0.002}, 1024: {'batch_size': 4, 'num_steps': 32, 'img_size': 1024, 'tex_size': 1024, 'batch_split': 2, 'gen_lr': 0.002, 'disc_lr': 0.002}}, 0: {'batch_size': 8, 'num_steps': 32, 'img_size': 256, 'tex_size': 256, 'batch_split': 1, 'gen_lr': 0.002, 'disc_lr': 0.002}, 200000: {}, 'dataset_path': '/proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/ffhq256x256.zip', 'pose_data_path': '/proj/cvl/users/x_fahkh/mn/ml-gmpi/runtime_dataset/ffhq256_deep3dface_coeffs', 'fov': 12.6, 'ray_start': 0.95, 'ray_end': 1.12, 'h_stddev': 0.289, 'v_stddev': 0.127, 'h_mean': 0.0, 'v_mean': 0.0, 'latent_dim': 512, 'stylegan2_w_dim': 512, 'generator_label_dim': 0, 'fade_steps': 10000, 'betas': (0, 0.9), 'unique_lr': False, 'weight_decay': 0, 'r1_lambda': 10.0, 'grad_clip': 10, 'dataset': 'FFHQ', 'z_dist': 'gaussian', 'raw_img_size': 256, 'eval_img_size': 256}

Used 8 GPUs for training...

Trained the model for the 17000 iterations

[Xiaoming-Zhao]

Thanks. Do you mind evaluating the checkpoint at 5k iterations, which is what we used for evaluation? I think the code should save all checkpoints during training so 5k-iteration checkpoint should be there.

[VIROBO-15]

When I used the 5k-iteration checkpoint the reported number comes to be for FFHQ256

depth:  0.5485293 0.36907703 
angle:  0.004393761830653509 0.006464534215649069 

[Xiaoming-Zhao]

Thanks a lot for checking this. I think the current number looks reasonable to me. Since the depth and angle scores are computed on renderings from randomly-sampled camera poses, I would say the difference of ~0.01 or 0.02 is within expectation.

Regarding the reason about why the evaluation from the checkpoint at 17k iteration differs a lot, my gut is that the discriminator may be too strong after that long training and the generator cannot beat it. The curve of loss_d and loss_g_fake in the tensorboard may not be stable at 17k iterations.

It is an interesting finding though. I think some techniques for stabilizing GAN training might be helpful here.