How do you decide the denominator 512.0 for lr scaler?

[facebookresearch]

How do you decide the denominator 512.0 for lr scaler?

[TouvronHugo]

Hi @amirhfarzaneh ,
It is an arbitrary choice, this scaling strategy is commonly used to train convents.
Best,
Hugo

[amirhfarzaneh]

Hi @amirhfarzaneh ,
It is an arbitrary choice, this scaling strategy is commonly used to train convents.
Best,
Hugo

Great! thank you for the clarification.

[bartmch]

Hi, is the denominator really an arbitrary choice? Scaling learning rates are done to make sure the gradients are averaged correctly in DDP, right? So shouldn't args.batch_size * utils.get_world_size() > 1? In other words, the denominator cannot be arbitrary.
I'm not 100% sure hence the question marks.