Comments (6)
Greetings, @RanyaJumah!
The Conv1d weights quantization should be already supported (NNCFConv1d
module takes care of that). If this is not the case for you, please post the instructions on how to reproduce this.
However, the quantization of Conv1d outputs won't occur in the current NNCF develop branch. I created a PR #35 to fix this - check it out and see if it's working for you.
from nncf.
Made another PR to quantize embedding weights:
#37
@RanyaJumah I'll be closing this if you don't mind because the problem described in the issue's title has been fixed; can continue any outstanding discussions here, though, if necessary.
from nncf.
FYI, see also #39 for FP16 training support. Note, though, that due to FP16 computation errors we observe +- 1 quantum shifts between mathematically equivalent implementations on CPU and on CUDA, so use FP16 with quantization at your own risk.
from nncf.
Hi vshampor, and thanks for your quick reply :)
(1) It seems that the (NNCFConv1d) works as I followed the usage file and I tried to compress the encoder and decoder models of the AE separately, but I notice that embadding and GRU layers are not supported, are not? (Note: I use it for speech synthesis )
(2) Also, I tried to create different config files for the different compression algorithms, and there is only an error comes when I tried to implement the quantization algorithm as below, any idea about it?
return self.quantize(x)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/layers.py", line 330, in quantize
self.eps)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/dynamic_graph/wrappers.py", line 55, in wrapped
result = operator(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/quantize_functions.py", line 178, in asymmetric_quantize
return QuantizeAsymmetric.apply(input_, input_low_tuned, input_range_tuned, level_low, level_high, levels)
File "/usr/local/lib/python3.6/dist-packages/nncf-1.3.1-py3.6.egg/nncf/quantization/quantize_functions.py", line 72, in forward
output = QuantizedFunctionsCUDA.Quantize_forward(input_, input_low, input_range, levels)
RuntimeError: "q_cuda_forward" not implemented for 'Half'
I would say it is really interesting library to be used for model optimization and hope that addational compression algorithms can be added especially the adversarial one :)
from nncf.
(1) We haven't had the need to explicitly quantize embedding layers just yet - I think we will add this option some time soon.
The GRU layer is not implemented - we have a certain degree of LSTM support, but that may have to be updated to correspond to latest PyTorch status. @ljaljushkin might comment further on this.
(2) I suppose you are trying to use FP16 training. Our custom quantization kernels do not yet support FP16 input - something we should also probably do posthaste.
from nncf.
Thanks vshampor! Yes It is clear enough now :)
Just in case if someone have the same error for quantization, yes I used FP16 training.
from nncf.
Related Issues (20)
- [Good First Issue][NNCF]: Remove compress_to_fp16=False from examples HOT 3
- AttributeError: 'list' object has no attribute 'keys' when executing yolov8_quantize_with_accuracy_control example HOT 4
- The question about function create_compressed_model():RuntimeError: CUDA error: device-side assert triggered HOT 4
- Very inconsistent quantization results on Tensorflow HOT 10
- When will there be a c++ interface? HOT 2
- IndexError: list index out of range When I try to quantize llama models using OVQuantizer HOT 2
- [TorchFX] Torch FX/PyTorch 2 Export Quantization HOT 2
- ov.compile_model(quantized_model, device_name="GPU") is not working HOT 5
- [Good First Issue][NNCF][TorchFX]: Test model transformer HOT 10
- [Good First Issue][NNCF][TorchFX]: Test statistic aggregator HOT 5
- [TorchFX] TorchFX metatypes support HOT 2
- DeformConv2d Cannot be Quantized HOT 1
- [NNCF][TorchFX]: Test Fast Bias Correction algorithm HOT 2
- test_autoq_precision_init Fails due to connection timeout HOT 1
- Failed to export NNCF QAT model for Ultralytics YOLOv8 HOT 5
- [Good First Issue][NNCF]: [TorchFX] Test MinMax algorithm HOT 3
- [Good First Issue][NNCF]: [TorchFX] Test PTQ MinMax parameters HOT 4
- [Good First Issue][NNCF]: [TorchFX] Test PTQ quantizer config HOT 7
- Object Detection Quantization fails with OpenVINO 2024.3 HOT 1
- [TorchFX] Add edges shape to `test_models.py`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nncf.