wagonhelm/nanmnist

MNIST Tutorial

nanmnist's Issues

Highly improper way of weight initialization

As far as I know, initializing weights as zeroes, even as a beginner is highly improper way of initialization. CS231n open lecture notes says,

Pitfall: all zero initialization. Lets start with what we should not do. Note that we do not know what the final value of every weight should be in the trained network, but with proper data normalization it is reasonable to assume that approximately half of the weights will be positive and half of them will be negative. A reasonable-sounding idea then might be to set all the initial weights to zero, which we expect to be the “best guess” in expectation. This turns out to be a mistake, because if every neuron in the network computes the same output, then they will also all compute the same gradients during backpropagation and undergo the exact same parameter updates. In other words, there is no source of asymmetry between neurons if their weights are initialized to be the same.

Thanks for your tutorial

Even though i'm not good at english, but I can clearly understand your article.
It's very helpful for me to know how the tutorial from Tensorflow work.
I want to implement your code to tutorial , but I got a lot of questions about dimensions.
Whatever, thanks your work 👍