Inconsistency of CMSIS-NN Quantization Method(Q-format) with ARM Documentation

[arm-software]

Inconsistency of CMSIS-NN Quantization Method(Q-format) with ARM Documentation

[mansnils]

Hi @LEE-SEON-WOO,
You are referring to the legacy API. Development on that stopped a long time ago. Long before CMSIS-NN v4.0.1. When CMSIS-NN got its own repository the files were removed. CMSIS-NN is intended to be used with TFLM as CMSIS-NN is a library and not a framework. The legacy API was not bit-exact to TFLM reference kernels. The current API does not use any floating point operations.. Hope that makes it clearer.

[mansnils]

Thanks for the link! Why can't NNOM use the new/existing CMSIS-NN API?
Please note it also use shift operations. The scales are converted to integer multipliers and shifts before inference, and this is done before calling CMSIS-NN (before inference). And also performance should be better than the legacy API since development stopped long time ago.

[LEE-SEON-WOO]

Hello @mansnils , thank you for your response. Unfortunately, it seems that my limited fluency in English may have caused some misunderstandings, for which I apologize for any confusion caused.

The primary reason I am raising this issue is because I believe the q-format method of quantization could be effective on MCUs as well, and I am curious about the reasons it is not supported. For example, I wonder if it is due to difficulties related to accuracy or efficiency that support is not provided.

I understand that the quantization method provided by ARM is in the form of Q(m,n), and TFLM provides it based on the formula q = S*r + Z (where q: Quant Value, S: Scale Factor, Z: Zero Point, r: real_value). From my experience using ARM's method, it seems to have several advantages. Firstly, it operates at a higher speed because it uses shift operations. Secondly, it allows for smaller additional computations and variable sizes.

As I only deal with models suitable for MCUs, I am not sure how well it works with larger models. Also, legacy APIs using the q-format are easily accessible in other open-source platforms like nnom.

Thank you.

[LEE-SEON-WOO]

Dear @mansnils,

Thank you for your response. Upon reviewing the CMSIS-NN documentation, I confirmed that the presence of _s indicates compatibility with TensorFlow Lite Micro. Additionally, when examining the algorithm, it is evident that functions such as MIN, MAX, and arm_nn_requantizelink are invoked. Compared to the previous _q7link algorithms, this appears to necessitate more computations. Furthermore, I came across a statement made by the author of NNOM a while back and am writing to verify its authenticity. Here is the link for reference: link.

Best regards.