Closure

[vinbhaskara]

Closure

[kayuksel]

It also seems that I need to re-hypertune the learning rate for using AdamS, as the optimizer converges much slower when I experimented with that as a drop-in replacement for Adam.

[vinbhaskara]

Thank you for the question!

Unlike Conjugate Gradient/LGBFS, the AdamX (X=UCB/CB/S) optimizers do not re-evaluate the loss multiple times per update.

Only the current loss scalar value is necessary to estimate the running means and the variances. Therefore, closure is just used to pass the loss value for the current state of the network. So, it is one forward pass for one update.

Therefore, you can use either of the versions above.

We used the latter one as that'd keep loss and output tensors in the training function scope for further use.

[vinbhaskara]

You can also try keeping the same learning rate you used to have in the original Adam but tune on the variance hyperparameter eta for AdamS for best convergence by quickly running it for ~3 epochs and picking the best one.

[kayuksel]

Thank you for your response. I have tried the values in the paper for the eta value, will later try hypertuning it.

…

On Thu 6. Jun 2019 at 18:26, Vineeth S. Bhaskara ***@***.***> wrote: You can also try keeping the same learning rate you used to have in the original Adam but tune on the variance hyperparameter eta for AdamS for best convergence by quickly running it for ~3 epochs and picking the best one. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALF7SP3ZZGDS6MELGXLHBFLPZE3C5ANCNFSM4HVC736A> .