Retrained models really big

[facebookarchive]

Retrained models really big

[gcr]

This was discussed in #38, #24

@colesbury writes: Load the model, run clearState(), and save the model:

model = torch.load('model.t7')
model:clearState()
torch.save('model_small.t7', model)

Unfortunately, this doesn't save RAM while training, but it can save disk space.

[colesbury]

You can add model:clearState() here:

https://github.com/facebook/fb.resnet.torch/blob/master/checkpoints.lua#L29

However, this breaks the sharing gradInput, so you should not run with -shareGradInput. This is why I haven't added it to checkpointing code yet.

[colesbury]

So these numbers are for GPU with 12 GB of memory each (e.g. K40, Titan X) with batch size 256. You can use a smaller batch size, but then your on your own when it comes to the LR schedule:

ResNet-101 fits on 4 GPUs and requires ~10.5 GB per GPU
ResNet-152 fits on 8 GPUs and requires ~7 GB per GPU
ResNet-200 fits on 8 GPUS and requires ~9 GB per GPU

(You can also have a larger effective batch size by accumulating the gradients from multiple mini-batches before updating parameters)

[tunguz]

Thanks for your help. So far I have only been using the command-line level training. Where exactly do I have to insert those lines of code?

[tunguz]

Thanks. I guess at this point I care more about -shareGradInput than the size of model files, so I think I'll stick with the existing code after all.

Any idea how much GPU memory is required for retraining 101, 152 and 200 resnets?

[tunguz]

Yeah, as I don't see getting 4 - 8 12 GB any time soon, I think I'll stick with lower capacity models for now. :) Thanks for all your great help and insights!

[pedropgusmao]

Is it possible, during initialization, to create a dummy model that will only share with the original training model variables that are useful during testing and then save this dummy model instead?
For example, inside main.lua :

-- Create model
local model, criterion = models.setup(opt, checkpoint)

-- Create dummy model just for saving 
local dummymodel = model:clone()
dummymodel:share(model, 'bias')
dummymodel:share(model, 'weight')
dummymodel:share(model, 'running_mean') -- For BatchNorm
dummymodel:share(model, 'running_var')     -- For BatchNorm

--- Enter for cycle that trains for an epoch ...
... 
-- At the end of for loop epoch   
    checkpoints.save(epoch, dummymodel, trainer.optimState, bestModel)
end 

Could this help with the gradInput sharing?

[colesbury]

@pedropgusmao, yeah that might be the best option for now

[colesbury]

Fixed in f54d7c8