Comments (10)
This is due to both nvidia and zfs needing libtirpc and the zfs libtirpc is linked against musl while the nvidia one is linked against glibc. These two extensions are mutually incompatible with each other for now
from extensions.
@frezbo Will this fix be possible on 1.7 or will we have to wait for 1.8?
Sorry, probably only for 1.8, found other issues and fixing them, don't want to backport such major fixes
from extensions.
As heavy zfs users and soon-to-be nvidia extension users as well we would be glad providing assistance to help solving this one.
@frezbo what do you think would be the best way to fix this then ?
Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?
Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?
From the libtirpc source package, the autoconf snippet handling libc detection is located in in libtirpc-1.3.4/config.guess
:
<...>
case $UNAME_SYSTEM in
Linux|GNU|GNU/*)
LIBC=unknown
set_cc_for_build
cat <<-EOF > "$dummy.c"
#include <features.h>
#if defined(__UCLIBC__)
LIBC=uclibc
#elif defined(__dietlibc__)
LIBC=dietlibc
#elif defined(__GLIBC__)
LIBC=gnu
#else
#include <stdarg.h>
/* First heuristic to detect musl libc. */
#ifdef __DEFINED_va_list
LIBC=musl
#endif
#endif
EOF
cc_set_libc=`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g'`
eval "$cc_set_libc"
# Second heuristic to detect musl libc.
if [ "$LIBC" = unknown ] &&
command -v ldd >/dev/null &&
ldd --version 2>&1 | grep -q ^musl; then
LIBC=musl
fi
# If the system lacks a compiler, then just pick glibc.
# We could probably try harder.
if [ "$LIBC" = unknown ]; then
LIBC=gnu
fi
;;
esac
<...>
from extensions.
This has caused an issue for me as well
The libc.so seems to be an ASCII file when most are symbolic links to their partner .so.# files.
amd-ucode 20240410
nonfree-kmod-nvidia 535.129.03-v1.7.1
nvidia-container-toolkit 535.129.03-v1.14.6
zfs 2.2.3-v1.7.1
from extensions.
This is interesting though, zfs should be linked against the musl libc, so i guess the zfs extensions ship some extra libc, which it shouldn't
from extensions.
Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?
We do not want to do this, makes it a crazy hard to follow build steps
Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?
Nvidia has strict requirement for glibc and musl will not work at all
What we were discussing internally is to setup some LD_
options to let zfs use the right libtirpc
from extensions.
Build and use glibc in the zfs extension build as well (based on https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/glibc/pkg.yaml for instance) instead of musl ?
We do not want to do this, makes it a crazy hard to follow build steps
Looks like a lot of hassle Indeed...
Or somehow trick the build process of https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/libtirpc/pkg.yaml to use musl instead of the glibc ?
Nvidia has strict requirement for glibc and musl will not work at all
Yeah... i'm not so surprised about that.
What we were discussing internally is to setup some
LD_
options to let zfs use the right libtirpc
Like an alternate $LD_LIBRARY_PATH ?
What about using https://github.com/NixOS/patchelf ?
I wasn't aware of this (quite recent) trick 5 until minutes ago but I find it quite elegant and useful.
See https://stackoverflow.com/a/44710599 or https://www.baeldung.com/linux/multiple-glibc
from extensions.
I want to avoid patchelf, at some point the whole nvidia thing was using a lot of patchelf to fix stuff, it just makes it harder for others and for us internally to keep things up to date, I'll be looking at this issue next week and see what would work great for us and easy to maintain, don't want to make it more complicated
from extensions.
Fair enough :-)
Feel free to bug me here or on slack if I can be of any use to you!
from extensions.
@frezbo Will this fix be possible on 1.7 or will we have to wait for 1.8?
from extensions.
Related Issues (20)
- Include /usr/bin/nvidia-smi for nvidia-kmod extension HOT 4
- Extension for mergerFS HOT 2
- [nvidia-container-toolkit] Allow customizing nvidia-container-runtime.toml HOT 2
- `tailscale` versions pinned for a given Talos version are quickly out of date and have security vulnerabilities HOT 1
- change the vmtoolsd extension name directory too? HOT 2
- Kata Pods fail to boot with RuntimeClass on Raspberry Pi HOT 5
- Update NVIDIA drivers up to version 550.x HOT 6
- Tailscale network deadlock issue
- Kata Container follow-up: vAccel support
- Feature request: add nvproxy to gVisor if Nvidia driver is included HOT 1
- Gvisor pod cannot be terminated properly HOT 2
- kata containers 3.6.0 doesn't work
- Add Yggdrasil Network
- How to configure mdadm.conf? HOT 14
- Support Sysbox
- Tailscale TS_ROUTES containing ipv6 HOT 2
- Add lldpd extension
- Request: Add cni-plugins extensions HOT 3
- Kata container pod cannot start on v1.7.6 HOT 2
- spin extension fails to run pod with `no runtime for "spin" is configured` error HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from extensions.