Compilation of mlir:tf-opt fails with error "The repository '@llvm_zlib' could not be resolved"

[chudur-budur]

Compilation of mlir:tf-opt fails with error "The repository '@llvm_zlib' could not be resolved"

[vladbelit]

That method doesn't seem to work anymore, likely because the llvm-project source is patched when retrieved by Bazel, which doesn't happen when a local repo is used in accordance with those instructions.
However, with some minimal hacking, you can specify a particular commit/tag like so:
vladbelit@a17f4c1
(if you use a tagged archive, you might need to change strip_prefix, etc.)

In order to retrieve the SHA256 for a particular archive:

wget https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-18.1.7.tar.gz
shasum -a 256 llvmorg-18.1.7.tar.gz

Btw, even though the Docker image you're using works fine in your case, it's no longer supported.
You should use one from here: https://hub.docker.com/r/tensorflow/build/tags
Something like: docker pull tensorflow/build:latest-python3.12

It already contains Clang 17 (Clang 18 soonish).
Installing Python + setting up a Virtual env is no longer necessary, as Tensorflow's Python has been hermetic for some time now (except maybe in some cases??)

[chudur-budur]

Possible workaround. Add these dependecies in the top level WORKSPACE file:

http_archive(
    name = "llvm_zstd",
    build_file = "@llvm-raw//utils/bazel/third_party_build:zstd.BUILD",
    sha256 = "7c42d56fac126929a6a85dbc73ff1db2411d04f104fae9bdea51305663a83fd0",
    strip_prefix = "zstd-1.5.2",
    url = "https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz",
)

http_archive(
    name = "llvm_zlib",
    build_file = "@llvm-raw//utils/bazel/third_party_build:zlib-ng.BUILD",
    sha256 = "e36bb346c00472a1f9ff2a0a4643e590a254be6379da7cddd9daeb9a7f296731",
    strip_prefix = "zlib-ng-2.0.7",
    url = "https://github.com/zlib-ng/zlib-ng/archive/refs/tags/2.0.7.zip",
)

The compilation works but then it generates duplicate symbols from llvm_zstd and net_zstd and fails during the linking:

ERROR: /tensorflow_src/tensorflow/BUILD:1177:21: Linking tensorflow/libtensorflow_framework.so.2.18.0 [for tool] failed: (Exit 1): clang failed: error executing command (from target //tensorflow:libtensorflow_framework.so.2.18.0) /usr/lib/llvm-17/bin/clang @bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/libtensorflow_framework.so.2.18.0-2.params
ld.lld: error: duplicate symbol: FSE_versionNumber
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/net_zstd/_objs/zstdlib/entropy_common.pic.o:(FSE_versionNumber)
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/llvm_zstd/_objs/zstd/entropy_common.pic.o:(.text.FSE_versionNumber+0x0)

ld.lld: error: duplicate symbol: FSE_isError
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/net_zstd/_objs/zstdlib/entropy_common.pic.o:(FSE_isError)
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/llvm_zstd/_objs/zstd/entropy_common.pic.o:(.text.FSE_isError+0x0)

ld.lld: error: duplicate symbol: FSE_getErrorName
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/net_zstd/_objs/zstdlib/entropy_common.pic.o:(FSE_getErrorName)
>>> defined at entropy_common.c
>>>            bazel-out/k8-opt-exec-50AE0418/bin/external/llvm_zstd/_objs/zstd/entropy_common.pic.o:(.text.FSE_getErrorName+0x0)
...
...
ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[chudur-budur]

bump?

[Venkat6871]

Hi @chudur-budur ,

• Thank you for the reporting issue. There are other issues in the same line which are assigned to eng team, we will provide the resolution once it resolved.

[Venkat6871]

CC: @learning-to-play

[vladbelit]

For a more robust solution, handing this off to MLIR
@d0k