Updates.py format

[lasagne]

Updates.py format

[benanne]

Momentum needs to keep track of the accumulated previous gradients. That is what the mparam_i variables are for. They need to be updated at each iteration as well, hence the addition of the tuple (mparam_i, v) to the updates list.

The mparam_i tensors are initialized to zero, but during training they will be updated with the current accumulated gradient.

You could indeed scale the second term in v with (1 - momentum). This would allow you to experiment with different values of momentum and different learning rates more easily, in fact. But in literature and in practice, most people currently do not do this, so we opted not to do it either.

[skaae]

Ok. I'm new in the theano graph world :)

mparam_i = theano.shared(np.zeros(param_i.get_value().shape, dtype=theano.config.floatX))

initialize a shared variable at zero which is later updated?

[benanne]

Exactly :)

[f0k]

You could indeed scale the second term in v with (1 - momentum). This would allow you to experiment with different values of momentum and different learning rates more easily, in fact. But in literature and in practice, most people currently do not do this, so we opted not to do it either.

It would be nice to have this as an option, though. Some people do use the less traditional formula. I'll make an issue for that so we don't forget.

[benanne]

We should also consider if it's worth complicating the API for this. We had a very similar discussion in #10 :)

[f0k]

We should also consider if it's worth complicating the API for this.

Hmm, you're right. It's enough if the user just scales the learning rate by (1-momentum), isn't it?

/edit: By the way, what is the weight decay doing in there? Shouldn't that just be part of the cost and then become part of the gradient automatically?

[benanne]

By the way, what is the weight decay doing in there? Shouldn't that just be part of the cost and then become part of the gradient automatically?

Yep. I copied over most of this from my galaxy challenge code, it's probably a remnant of that. Having it as part of the cost is much cleaner.

[benanne]

made a new issue for the weight decay thing, closing this one.