re-implement rook

[billimek]

re-implement rook

[billimek]

Will look at this when rook v1.1 is shipped, because the ceph-csi will be baked-in to the 1.1 release.

[billimek]

There may be challenges leveraging the csi-driver approach with k3s based on this issue

[billimek]

k3s v0.9.0 should fix the CSI problem described above.

[billimek]

Rook v1.1 is released now: https://github.com/rook/rook/releases/tag/v1.1.0

[billimek]

Deployed rook v1.1 on the test k3s cluster. So far so good.

[billimek]

Having spent some more time with Rook 1.1 using the CSI-provided cephfs & ceph block storage two new observations:

1. ceph client workloads on non-amd64 nodes

It became apparent that it is not possible to run workloads that consume ceph storage on arm/arm64 nodes.

The symptoms were pods running on arm-based nodes failed to stat with errors like MountVolume.MountDevice failed for volume "pvc-a0d67825-07ae-4c41-86bb-6dc400ae2220" : driver name rook-ceph.cephfs.csi.ceph.com not found in the list of registered CSI drivers.

See this slack thread for some more details. There is an old PR that may start the process of allowing mixed-architecture csi images, but it doesn't look like there is a good near-term solve for this.

In the meantime, there are two choices going forward:

1. prevent ceph-consuming workloads from running on arm-based nodes, or
2. switch arm-based workloads to consume a non-ceph-based storage provider

Not sure which is the best path forward.

2. Rook/ceph high CPU load issue

Issue 3132 is still not 'resolved', meaning that there is still a very likely chance I will encounter this issue if I attempt to use rook/ceph for any heavy IO workloads. This is not a fun situation and I'll need to tread carefully on how I use rook/ceph in the cluster until there is some clear path forward to safely use rook/ceph for persistent storage.

[billimek]

There is apparently a bug in rook 1.1.2 where the cephfs CSI implementation gets stuck on fuse instead of the kernel driver.