Quantized

[declare-lab]

Quantized

[GarciaLnk]

Exporting to ONNX and quantizing worked fine for me using Optimum. You can simply use optimum-cli like this:

optimum-cli export onnx --model declare-lab/flan-alpaca-base flan-alpaca-base_onnx/
optimum-cli onnxruntime quantize --onnx_model flan-alpaca-base_onnx/ --avx2

But keep in mind that by default only MatMul and Add operators are quantized, you need to pass a custom config if you want all the operators to be quantized. You can find the base model exported and fully quantized in my HF repo: https://huggingface.co/GarciaLnk/onnx-models-quantized/tree/main/flan-alpaca-base