Questions about the implementation of deepnorm

[jiaohuix]

Questions about the implementation of deepnorm

[jiaohuix]

@shumingma Ooooh! Sorry, I was careless to see mul as division, thank you for your correction!!! I understand deeper on deepnorm_init, and the corrected distribution is as follows:

[shumingma]

Hi @MiuGod0126

$\beta$ is a multiplier, so it should be:

linear2_weight = linear2.weight.detach().numpy().reshape((-1, )) * init_scale

instead of

linear2_weight = linear2.weight.detach().numpy().reshape((-1, )) / init_scale