Weight initialization different from the original paper

[dreamquark-ai]

Weight initialization different from the original paper

[Optimox]

Hi @yajiez,

Thanks for pointing this out.
I tried to remember why I used this initialization, I remember reading carefully one of the cited papers and tried to implement their scheme but I can't find it back at the moment.

From the tests I've done, initialization does not matter that much in terms of results but I agree that the current implementation differs from the original paper.

About the gain values, from the (unknown) paper, I remember that I was aiming for a standard deviation of sqrt(4/fan_in) for GLU layers and sqrt(1/fan_in) for other layers.

I will try to find back the paper where I got that, but you are right, we probably need to discuss about it and simply go with xavier_uniform with gain=1 to get closer to the orignial paper.

Thanks for your feeback!

[yajiez]

Hi @Optimox,

Thanks for your reply and explanation.

All make sense and it would be great if you could please post the references here later if you can recall it.

Thanks for sharing your work.

[Optimox]

Hey @yajiez,

I know it's been a while but I found the paper!
It's from Convolutional Sequence to Sequence Learning (cited in tabnet paper)
https://arxiv.org/pdf/1705.03122.pdf
If you go to the 3.5 Initialization section I think I tried to implement this, if you see that it does not match please let me know!

Thanks

[yajiez]

Aha, this was one of my favourite papers but I did not remember their initialization details. Seems a good time to read this paper again :)

Thank you @Optimox for pointing out the reference!