Pytorch quantization bias is not quantised on aarch64

[renato-arantes]

Pytorch quantization bias is not quantised on aarch64

[mgouicem]

By inspect here I can see that the bias is by default f32 but in the documentation here the quantized bias is s32.

@renato-arantes From oneDNN perspective, the datatype of the bias is user defined. Internally, it can be upconverted (first link).
Regarding the documentation you linked, this is just a tutorial to showcase how quantization workflow can be customized by oneDNN user (and that example uses signed int8 datatype for bias).

Now why would PyTorch not quantize the bias? This is a question to ask PyTorch maintainers, but in general, there is little reason to quantize the bias tensor as it is small when compared to layer weights and activations. Adding @milpuz01, @snadampal @malfet for more comments.

[shu1chen]

Hi @renato-arantes, do you mean quantized to s8 or s32? The accumulation datatype used during Op computation is governed by the accumulation_mode attribute of the primitive. By default, f32 is used for floating-point primitives (or f64 for f64 primitives) and s32 is used for integral primitives. You may change the default behavior by setting the dnnl::accumulation_mode to s32. More details are in the Data Types section of oneDNN developer guide.

The example cnn_inference_int8.cpp you mentioned shows the quantized bias is s8 because the data type of bias is set to s8 in line 114 auto conv_bias_md = memory::desc({conv_bias_tz}, dt::s8, tag::any);.
After modifying the example with auto conv_bias_md = memory::desc({conv_bias_tz}, dt::s32, tag::any);, It's then using s32 for bias, which can also be seen in the ONEDNN_VERBOSE log:

onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abcd::f0 dst_u8::blocked:acdb::f0,attr-scales:dst:0:f32 ,,8x256x13x13,10.259
onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abcd::f0 dst_s8::blocked:AcdB64a4b::f0,attr-scales:dst:0:f32 ,,384x256x3x3,1.24707
onednn_verbose,primitive,exec,cpu,convolution,brgconv:avx512_core_vnni,forward_training,src_u8:a:blocked:acdb::f0 wei_s8:a:blocked:AcdB64a4b::f0 bia_s32:a:blocked:a::f0 dst_u8:a:blocked:acdb::f0,attr-scales:src0:0:f32+dst:0:f32+wei:0:f32 attr-post-ops:eltwise_relu ,alg:convolution_direct,mb8_ic256oc384_ih13oh13kh3sh1dh0ph1_iw13ow13kw3sw1dw0pw1,0.651855
onednn_verbose,primitive,exec,cpu,reorder,jit:uni,undef,src_u8::blocked:acdb::f0 dst_f32::blocked:abcd::f0,attr-scales:src0:0:f32 ,,8x384x13x13,0.10498

[renato-arantes]

Hi @shu1chen,

Your answer is not related to my question that is about Pytorch, and not an example that you said I mentioned, but I did not. Maybe you are answering another question here by mistake?

Cheers,
Renato

[shu1chen]

By inspect here I can see that the bias is by default f32 but in the documentation here the quantized bias is s32.

Hi @renato-arantes, the second here is the same example in the source code as I referred.
I meant that perhaps you need to set accumulation_mode attribute of the primitive in Pytorch to change the default behavior.