Why dnnl_gemm_bf16bf16f32 is much slower than dnnl_sgemm？

[wells-wei-wei]

Why dnnl_gemm_bf16bf16f32 is much slower than dnnl_sgemm？

[yehudaorel]

Hi @wells-wei-wei,

Regarding implementation of dnnl_gemm_bf16bf16f32, the recommended interface for this internal GEMM API call is via a MatMul primitive.

Please reference the solution provided here, as well as this example demonstrating MatMul as a replacement for sGEMM functions for additional details.

But when I use it, I find dnnl_gemm_bf16bf16f32 is much slower than dnnl_sgemm, why is that? I'm sure the host cpu support avx512_bf16, is this because I didn't add some options during build time?

In general without Intel AMX support, bf16 gemm based ops have little to no performance gain depending on problem shape/HW/etc., which might explain your results.

Could you run the workload with the following environment variable: ONEDNN_VERBOSE=all and share the output? this will tell you specific supported ISA's and primitive dispatch information.

Additional information such as problem size/shape, CPU, OS will be helpful!

[vpirogov]

@wells-wei-wei, in addition to what @yehudaorel shared there's an example that benchmarks and reports matmul primitive performance with various data types.