DGC warmup

[sands-lab]

DGC warmup

[hangxu0304]

No. However, you can change the sparsity dynamically in your training script by calling:

for i in range(epochs):
  grc.compressor.compress_ratio = 0.01*pow(2,i)

[yimjinkyu1]

Thank you,

I have also another question.

I think DGC algorithm deploy momentum correction algorithm.
Is it right your code has this algorithm in DGC code ?

The figures for improving the imagenet model speed of DGC original papers are higher than your paper indicators.

[hangxu0304]

Yes. The momentum correction is implemented in grace_dl/torch/memory/dgc.py .
I think the hyper-parameters used in our experiment are a bit different from the original author's, e.g. batch size, number of nodes, learning rate, whether apply compression only on high rank gradients, etc. Please refer to the original author's repo for an exact reproduction.