Quantization with 1D Convolutions

[RanyaJumah]

Quantization with 1D Convolutions

[vshampor]

Greetings, @RanyaJumah!

The Conv1d weights quantization should be already supported (NNCFConv1d module takes care of that). If this is not the case for you, please post the instructions on how to reproduce this.

However, the quantization of Conv1d outputs won't occur in the current NNCF develop branch. I created a PR #35 to fix this - check it out and see if it's working for you.

[vshampor]

Made another PR to quantize embedding weights:
#37

@RanyaJumah I'll be closing this if you don't mind because the problem described in the issue's title has been fixed; can continue any outstanding discussions here, though, if necessary.

[vshampor]

FYI, see also #39 for FP16 training support. Note, though, that due to FP16 computation errors we observe +- 1 quantum shifts between mathematically equivalent implementations on CPU and on CUDA, so use FP16 with quantization at your own risk.

[RanyaJumah]

Hi vshampor, and thanks for your quick reply :)
(1) It seems that the (NNCFConv1d) works as I followed the usage file and I tried to compress the encoder and decoder models of the AE separately, but I notice that embadding and GRU layers are not supported, are not? (Note: I use it for speech synthesis )

(2) Also, I tried to create different config files for the different compression algorithms, and there is only an error comes when I tried to implement the quantization algorithm as below, any idea about it?

return self.quantize(x)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/layers.py", line 330, in quantize
self.eps)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/dynamic_graph/wrappers.py", line 55, in wrapped
result = operator(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/quantize_functions.py", line 178, in asymmetric_quantize
return QuantizeAsymmetric.apply(input_, input_low_tuned, input_range_tuned, level_low, level_high, levels)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/quantize_functions.py", line 72, in forward
output = QuantizedFunctionsCUDA.Quantize_forward(input_, input_low, input_range, levels)
RuntimeError: "q_cuda_forward" not implemented for 'Half'

I would say it is really interesting library to be used for model optimization and hope that addational compression algorithms can be added especially the adversarial one :)

[vshampor]

(1) We haven't had the need to explicitly quantize embedding layers just yet - I think we will add this option some time soon.

The GRU layer is not implemented - we have a certain degree of LSTM support, but that may have to be updated to correspond to latest PyTorch status. @ljaljushkin might comment further on this.

(2) I suppose you are trying to use FP16 training. Our custom quantization kernels do not yet support FP16 input - something we should also probably do posthaste.

[RanyaJumah]

Thanks vshampor! Yes It is clear enough now :)
Just in case if someone have the same error for quantization, yes I used FP16 training.