how to use zfs tools

[bernardgut]

how to use zfs tools

[smira]

Talos Linux has read-only filesystem, you can't mount to /zpool, but you can mount to e.g. /var/zpool.

[smira]

Lots of extensions were submitted by the community and included to be available for Talos Linux, so there might be more community involvement to get it documented with examples, and PRs are welcome!

[bernardgut]

I don't mind adding the documentation once I figure out how it works. I just need someone to give me some tips on how it works first.

I see in the PR that the developer mention node-shell. Does that mean I need to run a node-shell root container on each host and run the binaries directly there ? Or am I missing something

I tried to run kubectl linstor physical-storage create-device-pool --pool-name zpool-1 --storage-pool zpool-1 zfs node-1 /dev/sdc /dev/sdd as linstor at least exposes an api to zpool create/delete (but nothing else), which seems to me to be a more "Talos-y" way of managing the lifecycle of storage on my nodes, but even then I get

Error message: 
    cannot mount '/zpool-1': failed to create mountpoint: Read-only file system

which suggest that the extension is missing some configuration to be used with Linstor. Since there is no documentation on how to configure this extension or consume the binaries that it adds to the OS I do not know where else to go.

[bernardgut]

Ok I will try that.

In the meantime @donch, sorry to summon you like this but (whenever you have some time) can you comment on the lifecycle commands for zfs on talos ? How do you usually run those (zpool resilver, zpool monitor, etc) ? I am trying to deploy your extension.

Also as a follow up question : on debian I have a mailctl configuration to send alerts by email whenever the zfs event controller (zed) status changes. Same for smartd. Is there any way to replicate this functionality on Talos/ZFS ?

Thanks
B.

[bernardgut]

Also, I tried to use node-shell route to try and run zpool commands manually, but the extension binaries seemingly do not get mounted in the pod :

talosctl -n 10.2.0.8  list -l /usr/local/sbin/
NODE       MODE         UID   GID   SIZE(B)   LASTMOD           NAME
10.2.0.8   drwxr-xr-x   0     0     223       Apr 12 20:29:24   .
10.2.0.8   -rwxr-xr-x   0     0     772       Apr 12 20:29:24   fsck.zfs
10.2.0.8   -rwxr-xr-x   0     0     20994     Apr 12 20:29:24   vdev_id
10.2.0.8   -rwxr-xr-x   0     0     152480    Apr 12 20:29:24   zdb
10.2.0.8   -rwxr-xr-x   0     0     86192     Apr 12 20:29:24   zed
10.2.0.8   -rwxr-xr-x   0     0     116048    Apr 12 20:29:24   zfs
10.2.0.8   -rwxr-xr-x   0     0     14248     Apr 12 20:29:24   zfs_ids_to_path
10.2.0.8   -rwxr-xr-x   0     0     14288     Apr 12 20:29:24   zgenhostid
10.2.0.8   -rwxr-xr-x   0     0     22712     Apr 12 20:29:24   zhack
10.2.0.8   -rwxr-xr-x   0     0     26736     Apr 12 20:29:24   zinject
10.2.0.8   -rwxr-xr-x   0     0     172992    Apr 12 20:29:24   zpool
10.2.0.8   -rwxr-xr-x   0     0     30888     Apr 12 20:29:24   zstream
10.2.0.8   Lrwxrwxrwx   0     0     7         Apr 12 20:29:24   zstreamdump -> zstream
10.2.0.8   -rwxr-xr-x   0     0     128528    Apr 12 20:29:24   ztest
10.2.0.8   -rwxr-xr-x   0     0     14240     Apr 12 20:29:24   zvol_id
k create ns temp
k label ns temp  pod-security.kubernetes.io/enforce=privileged
k node-shell -n temp -x o1 -- ls -la /usr/local/sbin
spawning "nsenter-n9m9u3" on "o1"
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "nsenter" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "nsenter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nsenter" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host-root" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "nsenter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nsenter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
ls: /usr/local/sbin: No such file or directory
pod temp/nsenter-n9m9u3 terminated (Error)

then:

k node-shell -n temp -x o1 -- ls -la /usr/local/   
spawning "nsenter-wb0siu" on "o1"
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "nsenter" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "nsenter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nsenter" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host-root" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "nsenter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nsenter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
total 0
drwxr-xr-x    5 root     root            41 Jan 26 17:53 .
drwxr-xr-x    7 root     root            66 Jan 26 17:53 ..
drwxr-xr-x    2 root     root             6 Jan 26 17:53 bin
drwxr-xr-x    2 root     root             6 Jan 26 17:53 lib
drwxr-xr-x    2 root     root             6 Jan 26 17:53 share

[bernardgut]

My bad. I missed that the host root is mounted in /host in node-shell root container. Either way the extension doesn't seem to work on a Talos node:

talosctl -n 10.2.0.8  list -l /usr/local/sbin/         (⎈|onprem-omni-t0:default)
NODE       MODE         UID   GID   SIZE(B)   LASTMOD           NAME
10.2.0.8   drwxr-xr-x   0     0     223       Apr 12 20:29:24   .
10.2.0.8   -rwxr-xr-x   0     0     772       Apr 12 20:29:24   fsck.zfs
10.2.0.8   -rwxr-xr-x   0     0     20994     Apr 12 20:29:24   vdev_id
10.2.0.8   -rwxr-xr-x   0     0     152480    Apr 12 20:29:24   zdb
10.2.0.8   -rwxr-xr-x   0     0     86192     Apr 12 20:29:24   zed
10.2.0.8   -rwxr-xr-x   0     0     116048    Apr 12 20:29:24   zfs
10.2.0.8   -rwxr-xr-x   0     0     14248     Apr 12 20:29:24   zfs_ids_to_path
10.2.0.8   -rwxr-xr-x   0     0     14288     Apr 12 20:29:24   zgenhostid
10.2.0.8   -rwxr-xr-x   0     0     22712     Apr 12 20:29:24   zhack
10.2.0.8   -rwxr-xr-x   0     0     26736     Apr 12 20:29:24   zinject
10.2.0.8   -rwxr-xr-x   0     0     172992    Apr 12 20:29:24   zpool
10.2.0.8   -rwxr-xr-x   0     0     30888     Apr 12 20:29:24   zstream
10.2.0.8   Lrwxrwxrwx   0     0     7         Apr 12 20:29:24   zstreamdump -> zstream
10.2.0.8   -rwxr-xr-x   0     0     128528    Apr 12 20:29:24   ztest
10.2.0.8   -rwxr-xr-x   0     0     14240     Apr 12 20:29:24   zvol_id

talosctl -n 10.2.0.8 get extensions                    (⎈|onprem-omni-t0:default)
NODE       NAMESPACE   TYPE              ID            VERSION   NAME          VERSION
10.2.0.8   runtime     ExtensionStatus   0             1         amd-ucode     20240312
10.2.0.8   runtime     ExtensionStatus   1             1         zfs           2.1.14-v1.6.7
10.2.0.8   runtime     ExtensionStatus   2             1         drbd          9.2.8-v1.6.7
10.2.0.8   runtime     ExtensionStatus   3             1         schematic     a4353d7daa10df2bb8eec2545567af071132b3563ce313d2d729044679f548d1
10.2.0.8   runtime     ExtensionStatus   modules.dep   1         modules.dep   6.1.82-talos

k node-shell -n temp -x o1 -- /host/usr/local/sbin/zpool status
spawning "nsenter-g5hs75" on "o1"
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "nsenter" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "nsenter" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nsenter" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host-root" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "nsenter" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nsenter" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Error loading shared library libzfs.so.4: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libzfs_core.so.3: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libuutil.so.3: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libnvpair.so.3: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libtirpc.so.3: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libblkid.so.1: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error loading shared library libuuid.so.1: No such file or directory (needed by /host/usr/local/sbin/zpool)
Error relocating /host/usr/local/sbin/zpool: zfs_commit_all_shares: symbol not found
Error relocating /host/usr/local/sbin/zpool: dump_nvlist: symbol not found
Error relocating /host/usr/local/sbin/zpool: fnvlist_free: symbol not found
Error relocating /host/usr/local/sbin/zpool: zfs_special_devs: symbol not found
Error relocating /host/usr/local/sbin/zpool: nvpair_value_int32: symbol not found
Error relocating /host/usr/local/sbin/zpool: uu_avl_pool_create: symbol not found
Error relocating /host/usr/local/sbin/zpool: zpool_export: symbol not found
Error relocating /host/usr/local/sbin/zpool: libzfs_envvar_is_set: symbol not found
Error relocating /host/usr/local/sbin/zpool: fnvlist_lookup_uint64: symbol not found
Error relocating /host/usr/local/sbin/zpool: zprop_free_list: symbol not found
Error relocating /host/usr/local/sbin/zpool: tpool_dispatch: symbol not found
Error relocating /host/usr/local/sbin/zpool: nvpair_value_int32_array: symbol not found
Error relocating /host/usr/local/sbin/zpool: zpool_state_to_name: symbol not found
Error relocating /host/usr/local/sbin/zpool: zfs_isnumber: symbol not found
Error relocating /host/usr/local/sbin/zpool: libzfs_fini: symbol not found
etc

It looks like some deps issue. Did anyone actually manage to run this extension ?

EDIT: I also tried the last command with

mount -o bind /host/dev /dev
mount -o bind /host/usr/local/sbin /usr/local/sbin

but I get the same result

[bernardgut]

After a bit of digging around I figured it out. Posting here for the sake of anyone who ends up here from google trying to make this extension work. If you want to operate zfs on a Talos node uzing the zfs extension, here are the steps :

1. Install node-shell.

k krew index add kvaps https://github.com/kvaps/krew-index
k krew install kvaps/node-shell

2. Spawn a new node-shell instance on one of your Talos nodes:

k create ns temp
k label ns temp  pod-security.kubernetes.io/enforce=privileged
k node-shell -n temp -x <NODE>

3. Set up the node-shell environment, then run zfs/zpool clients

# mount devices and missing deps
mount -o bind /host/dev /dev
mount -o bind /host/usr/local /usr/local
touch /lib/libuuid.so.1
mount -o bind /host/lib/libuuid.so.1 /lib/libuuid.so.1
touch /lib/libuuid.so.1.3.0
mount -o bind /host/lib/libuuid.so.1.3.0 /lib/libuuid.so.1.3.0
touch /lib/libblkid.so.1
mount -o bind /host/lib/libblkid.so.1 /lib/libblkid.so.1
touch /lib/libblkid.so.1.1.0
mount -o bind /host/lib/libblkid.so.1.1.0 /lib/libblkid.so.1.1.0
# then run your ZFS/ZPOOL stuff
zpool create -f -o ashift=12 zpool-1 mirror /dev/sdX /dev/sdY 
...

You can then use linstor to create your storage-pool on top of the zfs device

This is the only way I have found so far to actually use this extension. Works well for testing but doesn't really scale to large clusters with dozens of nodes so if anyone has a better idea feel free to comment.

Thanks
B.