"invalid static_cast" on AVX512FP16 (e.g. Sapphire Rapids)

[Flamefire]

"invalid static_cast" on AVX512FP16 (e.g. Sapphire Rapids)

[Flamefire]

https://gitlab.com/libeigen/eigen/-/merge_requests/1639 was merged fixing that in Eigen upstream

[mihaimaruseac]

Thanks for reopening!

[tilakrayal]

@belitskiy fyi

[belitskiy]

I doubt this will address it, but building TF is not supported with GCC. For posterity, can you try with Clang?

https://www.tensorflow.org/install/source contains instructions on how to install Clang 17.
Alternatively, you can use a Docker container from https://hub.docker.com/r/tensorflow/build/tags. It comes with Clang pre-installed.

Also, for building from head, that guide contains the new target for building the wheel - //tensorflow/tools/pip_package:build_pip_package is no longer present at head.

[Flamefire]

I doubt this will address it, but building TF is not supported with GCC. For posterity, can you try with Clang?

https://www.tensorflow.org/install/source contains instructions on how to install Clang 17. Alternatively, you can use a Docker container from https://hub.docker.com/r/tensorflow/build/tags. It comes with Clang pre-installed.

I tried using the tensorflow:devel docker container and didn't find clang installed there and installing via apt-get required a few extra steps, i.e. not just "apt-get install clang-17". I'll do a final attempt but according to my analysis of the Eigen source code (see the issue I opened there) it is practically impossible it can succeed due to missing template specializations

[Flamefire]

I reproduced this on TensorFlow 2.15.0:

(cd /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/org_tensorflow && \
  exec env - \
    CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang \
    PATH=/root/.cache/bazelisk/downloads/bazelbuild/bazel-6.1.0-linux-x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python3 \
    PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
    TF2_BEHAVIOR=1 \
  /usr/lib/llvm-17/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/tensorflow/core/kernels/linalg/_objs/matrix_inverse_op/matrix_inverse_op.pic.d '-frandom-seed=bazel-out/k8-opt/bin/tensorflow/core/kernels/linalg/_objs/matrix_inverse_op/matrix_inverse_op.pic.o' -fPIC -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' -DHAVE_SYS_UIO_H -DTF_USE_SNAPPY '-DLLVM_ON_UNIX=1' '-DHAVE_BACKTRACE=1' '-DBACKTRACE_HEADER=<execinfo.h>' '-DLTDL_SHLIB_EXT=".so"' '-DLLVM_PLUGIN_EXT=".so"' '-DLLVM_ENABLE_THREADS=1' '-DHAVE_DEREGISTER_FRAME=1' '-DHAVE_LIBPTHREAD=1' '-DHAVE_PTHREAD_GETNAME_NP=1' '-DHAVE_PTHREAD_H=1' '-DHAVE_PTHREAD_SETNAME_NP=1' '-DHAVE_REGISTER_FRAME=1' '-DHAVE_SETENV_R=1' '-DHAVE_STRERROR_R=1' '-DHAVE_SYSEXITS_H=1' '-DHAVE_UNISTD_H=1' -D_GNU_SOURCE '-DHAVE_LINK_H=1' '-DHAVE_MALLINFO=1' '-DHAVE_SBRK=1' '-DHAVE_STRUCT_STAT_ST_MTIM_TV_NSEC=1' '-DLLVM_NATIVE_ARCH="X86"' '-DLLVM_NATIVE_ASMPARSER=LLVMInitializeX86AsmParser' '-DLLVM_NATIVE_ASMPRINTER=LLVMInitializeX86AsmPrinter' '-DLLVM_NATIVE_DISASSEMBLER=LLVMInitializeX86Disassembler' '-DLLVM_NATIVE_TARGET=LLVMInitializeX86Target' '-DLLVM_NATIVE_TARGETINFO=LLVMInitializeX86TargetInfo' '-DLLVM_NATIVE_TARGETMC=LLVMInitializeX86TargetMC' '-DLLVM_NATIVE_TARGETMCA=LLVMInitializeX86TargetMCA' '-DLLVM_HOST_TRIPLE="x86_64-unknown-linux-gnu"' '-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-linux-gnu"' '-DLLVM_VERSION_MAJOR=18' '-DLLVM_VERSION_MINOR=0' '-DLLVM_VERSION_PATCH=0' '-DLLVM_VERSION_STRING="18.0.0git"' -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS '-DBLAKE3_USE_NEON=0' -DBLAKE3_NO_AVX2 -DBLAKE3_NO_AVX512 -DBLAKE3_NO_SSE2 -DBLAKE3_NO_SSE41 '-DNO_LLVM_SUPPORT=0' -DCURL_STATICLIB -DTENSORFLOW_USE_CUSTOM_CONTRACTION_KERNEL -DTENSORFLOW_USE_MKLDNN_CONTRACTION_KERNEL '-DEIGEN_USE_AVX512_GEMM_KERNELS=0' '-DEIGEN_ALTIVEC_USE_CUSTOM_PACK=0' '-DEIGEN_NEON_GEBP_NR=4' '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/local_tsl -iquote bazel-out/k8-opt/bin/external/local_tsl -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/ml_dtypes -iquote bazel-out/k8-opt/bin/external/ml_dtypes -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/llvm-project -iquote bazel-out/k8-opt/bin/external/llvm-project -iquote external/local_xla -iquote bazel-out/k8-opt/bin/external/local_xla -iquote external/curl -iquote bazel-out/k8-opt/bin/external/curl -iquote external/boringssl -iquote bazel-out/k8-opt/bin/external/boringssl -iquote external/jsoncpp_git -iquote bazel-out/k8-opt/bin/external/jsoncpp_git -iquote external/onednn -iquote bazel-out/k8-opt/bin/external/onednn -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/float8 -Ibazel-out/k8-opt/bin/external/ml_dtypes/_virtual_includes/int4 -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/ArithCanonicalizationIncGen -Ibazel-out/k8-opt/bin/external/llvm-project/mlir/_virtual_includes/AsmParserTokenKinds -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem third_party/eigen3/mkl_include -isystem bazel-out/k8-opt/bin/third_party/eigen3/mkl_include -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes -isystem external/ml_dtypes/ml_dtypes -isystem bazel-out/k8-opt/bin/external/ml_dtypes/ml_dtypes -isystem external/llvm-project/mlir/include -isystem bazel-out/k8-opt/bin/external/llvm-project/mlir/include -isystem external/llvm-project/llvm/include -isystem bazel-out/k8-opt/bin/external/llvm-project/llvm/include -isystem external/curl/include -isystem bazel-out/k8-opt/bin/external/curl/include -isystem external/boringssl/src/include -isystem bazel-out/k8-opt/bin/external/boringssl/src/include -isystem external/jsoncpp_git/include -isystem bazel-out/k8-opt/bin/external/jsoncpp_git/include -isystem external/onednn/include -isystem bazel-out/k8-opt/bin/external/onednn/include -isystem external/onednn/src -isystem bazel-out/k8-opt/bin/external/onednn/src -isystem external/onednn/src/common -isystem bazel-out/k8-opt/bin/external/onednn/src/common -isystem external/onednn/src/common/ittnotify -isystem bazel-out/k8-opt/bin/external/onednn/src/common/ittnotify -isystem external/onednn/src/cpu -isystem bazel-out/k8-opt/bin/external/onednn/src/cpu -isystem external/onednn/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/onednn/src/cpu/gemm -isystem external/onednn/src/cpu/x64/xbyak -isystem bazel-out/k8-opt/bin/external/onednn/src/cpu/x64/xbyak -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS '-march=sapphirerapids' '-std=c++17' -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare '-ftemplate-depth=900' -fno-exceptions '-DTENSORFLOW_USE_XLA=1' -DINTEL_MKL -DENABLE_ONEDNN_V3 -DAMD_ZENDNN -msse3 -pthread '-DTENSORFLOW_USE_XLA=1' '-DINTEL_MKL=1' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c tensorflow/core/kernels/linalg/matrix_inverse_op.cc -o bazel-out/k8-opt/bin/tensorflow/core/kernels/linalg/_objs/matrix_inverse_op/matrix_inverse_op.pic.o)
...
In file included from tensorflow/core/kernels/linalg/matrix_inverse_op.cc:22:
In file included from external/eigen_archive/Eigen/Core:182:
external/eigen_archive/Eigen/src/Core/MathFunctions.h:429:12: error: no matching conversion for static_cast from 'const Eigen::internal::eigen_packet_wrapper<__attribute__((__vector_size__(4 * sizeof(long long)))) long long, 1>' to '__attribute__((__vector_size__(16 * sizeof(float)))) float' (vector of 16 'float' values)
  429 |     return static_cast<NewType>(x);

On TF master and 2.17-rc0 it does not fail due to using a different Eigen commit which had a workaround for that issue by removing the vectorization of such casts (degrading performance as they are possible)

[belitskiy]

tensorflow:devel is no longer supported. My link above points to a different set of images.

Given this works at head/2.17, what would be the action item here? Is it to update Eigen, to get rid of performance degradation that you mention?