Multi-GPU VarNet training and batch size

[facebookresearch]

Multi-GPU VarNet training and batch size

[z-fabian]

VarNet only works with batch size 1, that is per GPU. This is because the slices have varying size. If you want to increase mini-batch size, that is number of training examples averaged per gradient update just increase the number of GPUs used for training.

[mmuckley]

This is a good point. I believe 32 GPUs were used.

[z-fabian]

Thank you for the info. Have you experimented with how to adjust the learning rate with different batch sizes, or how much impact it has on reconstruction quality? Intuitively, I would use a factor of N lower learning rate for a factor of N less GPUs (smaller batch size), but this would require some trial-and-error to figure out. I'm interested because I want to reproduce the results from the paper, but don't have access to 32 GPUs. Thanks again.

[mmuckley]

I don't have much experience myself. I've gotten up to about 0.914 SSIM with two GPUs and the same learning rate and a smaller model for testing the repository refactor. Your strategy for lowering the learning rate by the GPU factor seems like a reasonable one.

[adefazio]

Keeping the learning rate the same would be my recommendation, but you should consider reducing the number of layers, otherwise it's not going to be train in a reasonable amount of time.

[z-fabian]

Okay, thanks for the input. I am going to experiment with model size and if the number of GPUs have an effect on SSIM. I'm closing this issue now.

[zhan4817]

However, when I tried to train VarNet with batch_size = 2, it shows the error in the validation sanity check:

...
237, in get_low_frequency_lines
while mask[..., r, :]:
RuntimeError: bool value of Tensor with more than one value is ambiguous

Do you have any idea? Thanks!