vinbhaskara/adams

Exploiting Uncertainty of Loss Landscape for Stochastic Optimization

adams's Issues

Closure

Hello,

Thank you for publishing this interesting work. I can't wait to evaluate it on my use-cases. I have investigated main.py in this repository; yet, the closure doesn't calculate the loss again in there; as opposed to the following example from PyTorch website.

for input, target in dataset:
    def closure():
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        return loss
    optimizer.step(closure)

My question is, if closure won't recalculate the loss, why it is necessary for AdamS? Which usage is proper for AdamS, the one from PyTorch website or the one in main.py? which is simply as follows:

        def closure(): return loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step(closure)