gemm_api and Reorder in HuggingFace OPT model

[SriAlavandar]

gemm_api and Reorder in HuggingFace OPT model

[vpirogov]

@SriAlavandar, verbose logs specifies a function called by the application as the fourth parameter. In your case Tensorflow calls 3 different functions: inner product, matmul, and dnnl::sgemm. The fifth parameter indicates underlying implementation name (like brg_avx512_core) for specific function.

Adding @agramesh1 to cover Tensorflow implementation decisions.

[vpirogov]

+@orelyehuda

[SriAlavandar]

Hi @yehudaorel, @agramesh1 Did you get any chance to have a look into the above issue. what is the importance of reorder to tag::abcd to tag::acbd? is reorder done on Matrix B?

[yehudaorel]

Hi @SriAlavandar, I believe your assumption here is correct, the reorder is taken place before matmul operation to reorder weight matrix to optimal memory format dictated by tensorflow implementation. In this case, the verbose output shows reorder from NCHW (channels first) to NHWC (channels last).

[SriAlavandar]

Thanks @yehudaorel. I noticed only for few MatMul's reorder on weight matrix is being triggered.

Can you please let me know on what basis are we deciding here whether reorder needs to be triggered or not?

[yehudaorel]

Hi @SriAlavandar, there are many reasons for a reorder primitive to be executed, one of them being matrix transposition which seems to be the case here. Without full context to the model implementation I don't have a definitive answer but I would recommend to run tensorboard profiler and verify tensorflow-onednn operation mapping.

e.g