[Kubernetes][KubeSpray] MountVolume.SetUp failed for volume "pvc-371a14b0-8426-4ce9-86f5-abe83e59743b" : exit status 5

[storageos]

[Kubernetes][KubeSpray] MountVolume.SetUp failed for volume "pvc-371a14b0-8426-4ce9-86f5-abe83e59743b" : exit status 5

[imriss]

k8s version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Installation of Cluster Operator (following the official guide):

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/1.2.0/storageos-operator.yaml

Adding the secret:

kubectl create ns storageos; cat <<END | kubectl create -f -
apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
stringData:
  apiUsername: storageos
  apiPassword: storageos
END

Installation of Storageos Cluster:

cat <<END | kubectl create -f -
apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "test-storageos"
spec:
  secretRefName: "storageos-api" # Reference the Secret created in the previous step
  secretRefNamespace: "storageos"  # Namespace of the Secret
  images:
    nodeContainer: "storageos/node:1.2.1" # StorageOS version
  resources:
    requests:
    memory: "512Mi"
END

Custom Resource yaml:

kubectl get crd storageosclusters.storageos.com -oyaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2019-07-04T23:37:07Z"
  generation: 1
  name: storageosclusters.storageos.com
  resourceVersion: "15153"
  selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/storageosclusters.storageos.com
  uid: a3096c72-9eb4-11e9-81b3-005056b0994a
spec:
  additionalPrinterColumns:
  - JSONPath: .status.ready
    description: Ready status of the storageos nodes.
    name: Ready
    type: string
  - JSONPath: .status.phase
    description: Status of the whole cluster.
    name: Status
    type: string
  - JSONPath: .metadata.creationTimestamp
    name: Age
    type: date
  conversion:
    strategy: None
  group: storageos.com
  names:
    kind: StorageOSCluster
    listKind: StorageOSClusterList
    plural: storageosclusters
    shortNames:
    - stos
    singular: storageoscluster
  scope: Namespaced
  validation:
    openAPIV3Schema:
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata: {}
        spec:
          properties:
            csi:
              properties:
                deploymentStrategy:
                  type: string
                enable:
                  type: boolean
                enableControllerPublishCreds:
                  type: boolean
                enableNodePublishCreds:
                  type: boolean
                enableProvisionCreds:
                  type: boolean
            debug:
              type: boolean
            disableFencing:
              type: boolean
            disableTelemetry:
              type: boolean
            images:
              properties:
                csiDriverRegistrarContainer:
                  type: string
                csiExternalAttacherContainer:
                  type: string
                csiExternalProvisionerContainer:
                  type: string
                csiLivenessProbeContainer:
                  type: string
                initContainer:
                  type: string
                nodeContainer:
                  type: string
            ingress:
              properties:
                annotations: {}
                enable:
                  type: boolean
                hostname:
                  type: string
                tls:
                  type: boolean
            join:
              type: string
            k8sDistro:
              type: string
            kvBackend:
              properties:
                address:
                  type: string
                backend:
                  type: string
            namespace:
              type: string
            nodeSelectorTerms: {}
            pause:
              type: boolean
            resources:
              properties:
                limits: {}
                requests: {}
            secretRefName:
              type: string
            secretRefNamespace:
              type: string
            service:
              properties:
                externalPort:
                  format: int32
                  type: integer
                internalPort:
                  format: int32
                  type: integer
                name:
                  type: string
                type:
                  type: string
            sharedDir:
              type: string
            tlsEtcdSecretRefName:
              type: string
            tlsEtcdSecretRefNamespace:
              type: string
            tolerations: {}
        status:
          properties:
            members:
              properties:
                ready: {}
                unready: {}
            nodeHealthStatus: {}
            nodes:
              items:
                type: string
              type: array
            phase:
              type: string
            ready:
              type: string
  version: v1
  versions:
  - name: v1
    served: true
    storage: true
status:
  acceptedNames:
    kind: StorageOSCluster
    listKind: StorageOSClusterList
    plural: storageosclusters
    shortNames:
    - stos
    singular: storageoscluster
  conditions:
  - lastTransitionTime: "2019-07-04T23:37:07Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: null
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1

[imriss]

Also, kubelet_load_modules was set to true in inventory/test/group_vars/all/all.yml file in KubeSpray:

kubelet_load_modules: true

[Arau]

Hello @imriss

In the past I've seen that Kubespray disables MountPropagation when a cluster is built. That would explain why the Kubelet can't mount the Volume.

There are 2 things to check

• Create this pod to verify mount propagation. It doesn't use StorageOS, just the configuration for Bidirectional mount. (https://github.com/storageos/deploy/blob/master/k8s/examples/mount-propagation/test-mount-propagation.yaml). Once the Pod is completed, if mount propagation is enabled, you should see the date.
• To verify that the Device Files are available to the kubelet, you can ssh into your nodes and check the contents of /var/lib/storageos/devices. You should see device (block) files. If that directory is empty, most likely means that StorageOS couldn't BindMount the host directory to the container. Therefore, the kubelet can't find the device files to mount.

I'm assuming that your kubelet is a SystemD process rather than a container. If it is a container, let me know as Mount Propagation has a few more implications in such case.

If MountPropagation is enabled, let me know as we can troubleshoot further.

[imriss]

@Arau , thanks

• MountPropagation to be there:

kubectl logs -f test-mount-propagation
Mon Jul  8 12:49:35 UTC 2019

• However, that directory does not exist:

ls -la /var/lib/storageos/devices
ls: cannot access '/var/lib/storageos/devices': No such file or directory

ls -la /var/lib/storageos
total 29
drwxr-xr-x  8 root root 4096 Jul  5 12:23 .
drwxr-xr-x 50 root root 4096 Jul  5 12:23 ..
drwx------  3 root root 4096 Jul  5 12:23 data
drwx------  2 root root 4096 Jul  5 12:23 filesystems
drwxr-xr-x  2 root root 4096 Jul  5 12:23 kv
drwxr--r--  2 root root 4096 Jul  5 12:23 logs
drwxr-x---  2 root root 4096 Jul  5 12:23 state
drwxr-xr-x  2 root root 4096 May 28  2015 volumes

Thanks

[Arau]

Sorry, I meant /var/lib/storageos/volumes

[imriss]

Thanks, please see below:

ls -la /var/lib/storageos/volumes
total 524293
drwxr-xr-x 2 root root       4096 May 28  2015 .
drwxr-xr-x 8 root root       4096 Jul  5 12:23 ..
brw-rw---- 1 root disk      8, 16 Jul  8 11:25 4eef16b7-c1c5-33cb-ae8a-bcd3a80d7a22
-rw-rw---- 1 root disk 2147483648 Jul  8 11:25 bst-210476
-rw-rw---- 1 root disk 2147483648 Jul  8 11:25 ca20873c-b4d5-58ef-6be4-c46ee88b1df5

[imriss]

Based on this comment, I redeployed the cluster using KubeSpray after adding this change to group_vars/k8s-cluster/addons.yml:

local_volume_provisioner_enabled: true

in addition to the following in group_vars/all/all.yml:

kubelet_load_modules: true

Should I deploy the storageos using the official guide? or, using the commands mentioned in that comment?

@Arau , any advise. Thanks

[Arau]

Hello @imriss ,

We have released StorageOS 1.3.0. With that the ./deploy.sh becomes deprecated or unnecessary. It is always recommended to use the StorageOS Cluster Operator to install StorageOS.

The official guide has always the proper information. The procedure is reduced to:

1. kubectl create -f https://github.com/storageos/cluster-operator/releases/download/1.3.0/storageos-operator.yaml

2. Create the storageos-api Secret if doesn't already exist

3. Create the Curstom Resource

To the mounting issue, since the device files are there, mount propagation is functional. That is good news. For further troubleshooting, we need to check if the kubelet can see those devices.

What parameter did you use for your kubespray for kubelet_deployment_type. i.e does the kubelet run as a container?

Also, is SELinux enabled?

[imriss]

Thanks. The kubelet_deployment_type is set to host:

kubelet_deployment_type: host

How can I check if SELinux is enabled?
I will deploy using the official guide and update.

[imriss]

The issue is still there:

  Warning  FailedScheduling  5m50s (x2 over 5m51s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 4 times)
  Normal   Scheduled         5m47s                  default-scheduler  Successfully assigned test/mysql-dd97d5db8-ml7qg to node4
  Warning  FailedMount       88s (x2 over 3m44s)    kubelet, node4     Unable to mount volumes for pod "mysql-dd97d5db8-ml7qg_test(e0ab08bc-c3d4-46d0-85fb-966cf20f8498)": timeout expired waiting for volumes to attach or mount for pod "test"/"mysql-dd97d5db8-ml7qg". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-nnmgv]
  Warning  FailedMount       70s (x10 over 5m39s)   kubelet, node4     MountVolume.SetUp failed for volume "pvc-0446eeb8-077d-45fd-9554-8c6c68b5a648" : exit status 5

The contents of /var/lib/storageos/volumes:

ls -la /var/lib/storageos/volumes

total 524293
drwxr-xr-x 2 root root       4096 May 28  2015 .
drwxr-xr-x 8 root root       4096 Jul  9 12:46 ..
brw-rw---- 1 root disk      8, 16 Jul  9 13:06 0368a13f-286f-b548-1b89-d3f22e941302
-rw-rw---- 1 root disk 2147483648 Jul  9 13:06 849b34d0-9733-e9a0-3419-0a1b04d68b4a
-rw-rw---- 1 root disk 2147483648 Jul  9 13:06 bst-261425

mount-propagation in there:

kubectl logs -f test-mount-propagation

Tue Jul  9 17:08:31 UTC 2019