RFE - Provide a 'storage' option that allows me to set the specific StorageClass for the registry

[nnachefski]

RFE - Provide a 'storage' option that allows me to set the specific StorageClass for the registry

[dmage]

@nnachefski you can create your own PVC in the openshift-image-registry namespace and use its name in spec.storage.pvc.claim. Have you tried it?

[adambkaplan]

@dmage I think there may be a larger issue here. If pvc.claim is not set, does the registry operator assume the default storage class will provision RWX storage? If so, is this assumption true on most providers?

[dmage]

@adambkaplan RWX is not an assumption, it is a requirement. The operator doesn't assign any storage classes, there is another component that sets the storage class to our PVC. The storage class isn't the only option than customers may want to change in PVC, there are also storage requests, selectors. And the bootstrapped PVC is read-only and can't be changed, so there is no reason to add parts of PVC to our configuration resource.

[bparees]

I think @adambkaplan is referring to the case where the operator creates the PVC itself. When it does so, it sets RWX as required:
https://github.com/openshift/cluster-image-registry-operator/blob/master/pkg/storage/pvc/pvc.go#L132-L134

so i believe we are fine on that front.

[nnachefski]

Could something similar be done like with the Prometheus stack? It uses a config map to configure the cluster-monitoring operator.

Ex:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      volumeClaimTemplate:
       spec:
         storageClassName: rook-ceph-block 
         resources:
           requests:
             storage: 40Gi

[bparees]

we generally have a preference for first class config apis (CRDs) over configmaps that are harder to validate, provide api documentation for, and enforce rbac rules around.

[tux-o-matic]

How is RWX a requirement when for years Docker said not to use NFS (commonly used RWX backend) at the risk of getting a corrupted registry? At least there should be a warning in the doc not to use NFS but CephFS or GlusterFS.

[dmage]

@tux-o-matic do you have a link where Docker says not to use NFS?

RWX is a requirement to avoid downtime during upgrades. In general, you should use an object storage (S3, GCS, etc.). PVC is an option when you don't have any object storage.

In 4.4 you can provide a RWO volume if you don't need a high available cluster.

[tux-o-matic]

@dmage One such warning in 3.1 doc
And another caution

[aizuddin85]

@tux-o-matic NFSv3 might have such issue, but I believed NFSv4 already fixed those cache and other issue to make NFS more reliable when RWX mounted on many nodes.

[tux-o-matic]

@aizuddin85 , if that's the case then I think it should be mentioned in the documentation because I believe most people still stuck with NFS in their infrastructure are likely stuck with v3.x

[aizuddin85]

@tux-o-matic In the documentation it mentioned both. I agree we probably can make it clearer, caveats between NFSv3 vs NFSv4.

[dmage]

The expected way to use a custom storage class is to provide your own PVC:

$ cat pvc.yaml
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: cephfs-storage
    namespace: openshift-image-registry
  spec:
    accessModes:
    - ReadWriteMany
    resources:
      requests:
        storage: 100Gi
    storageClassName: csi-cephfs
    volumeMode: Filesystem
$ oc apply -f pvc.yaml
persistentvolumeclaim/cephfs-storage created
$ oc patch configs.imageregistry.operator.openshift.io/cluster --type merge --patch '{"spec":{"storage":{"pvc":{"claim":"cephfs-storage"}}}}'
config.imageregistry.operator.openshift.io/cluster patched

You can delete the default pvc/image-registry-storage after that if it has been created.