Create the csi-hostpath-provisioner.yaml failed

[kubernetes-csi]

Create the csi-hostpath-provisioner.yaml failed

[pohly]

Your original description is much closer to the root of the problem. The failure to create pods looks like missing RBAC rules to me.

Originally you wrote:

When running the deploy/deploy-hostpath.sh, the command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/${PROVISIONER_RELEASE}/deploy/kubernetes/rbac.yaml
The above command couldn't run successfully as the url https://raw.githubusercontent.com/kubernetes-csi/external-provisioner can't be connected successfully, whether there is other links?

What exactly is the URL that it tries to deploy?

When I run it with command printing, I get:

$ bash -x ./deploy/deploy-hostpath.sh 
+ set -e
+ set -o pipefail
++ dirname ./deploy/deploy-hostpath.sh
+ BASE_DIR=./deploy
+ K8S_RELEASE=release-1.13
++ image_version ./deploy/hostpath/csi-hostpath-provisioner.yaml csi-provisioner
++ yaml=./deploy/hostpath/csi-hostpath-provisioner.yaml
++ image=csi-provisioner
++ grep 'image:.*csi-provisioner' ./deploy/hostpath/csi-hostpath-provisioner.yaml
++ sed -e 's/.*:v/v/'
+ PROVISIONER_RELEASE=v1.0.1
++ image_version ./deploy/hostpath/csi-hostpath-attacher.yaml csi-attacher
++ yaml=./deploy/hostpath/csi-hostpath-attacher.yaml
++ image=csi-attacher
++ grep 'image:.*csi-attacher' ./deploy/hostpath/csi-hostpath-attacher.yaml
++ sed -e 's/.*:v/v/'
+ ATTACHER_RELEASE=v1.0.1
+ INSTALL_CRD=false
+ [[ false =~ ^(y|Y|yes|true)$ ]]
+ echo 'applying RBAC rules'
applying RBAC rules
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml
serviceaccount/csi-provisioner created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/v1.0.1/deploy/kubernetes/rbac.yaml
serviceaccount/csi-attacher created
clusterrole.rbac.authorization.k8s.io/external-attacher-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role created
role.rbac.authorization.k8s.io/external-attacher-cfg created
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg created
+ echo 'deploying hostpath components'
deploying hostpath components
+ kubectl apply -f ./deploy/hostpath
service/csi-hostpath-attacher created
statefulset.apps/csi-hostpath-attacher created
daemonset.apps/csi-hostpathplugin created
service/csi-hostpath-provisioner created
statefulset.apps/csi-hostpath-provisioner created

[darcyllingyan]

@pohly ,
Thanks very much for your response!
But I have created the rbac rule, I just copy all the rbac.yaml to local. The deploy-hostpath command printing is below:

# bash -x deploy-hostpath.sh 
+ set -e
+ set -o pipefail
++ dirname deploy-hostpath.sh
+ BASE_DIR=.
+ K8S_RELEASE=release-1.13
++ image_version ./hostpath/csi-hostpath-provisioner.yaml csi-provisioner
++ yaml=./hostpath/csi-hostpath-provisioner.yaml
++ image=csi-provisioner
++ grep 'image:.*csi-provisioner' ./hostpath/csi-hostpath-provisioner.yaml
++ sed -e 's/.*:v/v/'
+ PROVISIONER_RELEASE=v1.0.1
++ image_version ./hostpath/csi-hostpath-attacher.yaml csi-attacher
++ yaml=./hostpath/csi-hostpath-attacher.yaml
++ image=csi-attacher
++ grep 'image:.*csi-attacher' ./hostpath/csi-hostpath-attacher.yaml
++ sed -e 's/.*:v/v/'
+ ATTACHER_RELEASE=v1.0.1
+ INSTALL_CRD=false
+ [[ false =~ ^(y|Y|yes|true)$ ]]
+ echo 'applying RBAC rules'
applying RBAC rules
+ kubectl apply -f /root/csi-driver-host-path/deploy/provision-rbac0.yaml
serviceaccount/csi-provisioner unchanged
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner configured
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role configured
role.rbac.authorization.k8s.io/external-provisioner-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg unchanged
+ kubectl apply -f /root/csi-driver-host-path/deploy/attach-rbac0.yaml
serviceaccount/csi-attacher unchanged
clusterrole.rbac.authorization.k8s.io/external-attacher-runner configured
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role configured
role.rbac.authorization.k8s.io/external-attacher-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg unchanged
+ echo 'deploying hostpath components'
deploying hostpath components
+ kubectl apply -f ./hostpath
service/csi-hostpath-attacher created
statefulset.apps/csi-hostpath-attacher created
daemonset.apps/csi-hostpathplugin unchanged
service/csi-hostpath-provisioner unchanged
statefulset.apps/csi-hostpath-provisioner unchanged

With the kubectl, the create rules are:

# kubectl get sa -n ncms|grep csi
csi-attacher                    1         6h42m
csi-driver-registrar            1         6h40m
csi-provisioner                 1         6h44m


# kubectl get clusterrole |grep runner
driver-registrar-runner                                                6h36m
external-attacher-runner                                               6h38m
external-provisioner-runner                                            6h40m

# kubectl get clusterrolebinding |grep csi
csi-attacher-role                                      6h38m
csi-driver-registrar-role                              6h36m
csi-nodeplugin                                         5h59m
csi-provisioner-role                                   6h40m

# kubectl get role -n ncms|grep external
external-attacher-cfg      6h42m
external-provisioner-cfg   6h45m

# kubectl get rolebinding -n ncms|grep csi
csi-attacher-role-cfg        6h43m
csi-provisioner-role-cfg     6h46m

But when I create the csi-hostpath-provisioner.yaml, the pod security policy error still exists:

# kubectl describe statefulset csi-hostpath-attacher -n ncms
Events:
  Type     Reason        Age                   From                    Message
  ----     ------        ----                  ----                    -------
  Warning  FailedCreate  3m57s (x35 over 14m)  statefulset-controller  create Pod csi-hostpath-attacher-0 in StatefulSet csi-hostpath-attacher failed error: pods "csi-hostpath-attacher-0" is forbidden: unable to validate against any pod security policy: [spec.volumes[0].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins/csi-hostpath": is not allowed to be used]

Not sure what the root cause is?

Thanks

[pohly]

Your output is with a modified deploy file, right? Can you re-run the unmodified one? I want to understand why it failed for you.

Regarding unable to validate against any pod security policy: [spec.volumes[0].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins/csi-hostpath": is not allowed to be used: have you activated some kind of security policy that controls access to paths on the host's filesystem? I'm not familiar with security policies, so we would have to get someone involved for that who is.

[darcyllingyan]

Hi @pohly,
Thanks for the response!
I didn't modify the deploy file, as my original issue: I can't access to the url https://raw.githubusercontent.com... with command kubectl in my lab..
So I download the rbac.yaml and renamed it as provision-rbac0.yaml and attacher-rbac0.yaml as my command output shows. The content isn't modified.
The provision-rbac0.yaml is:
https://github.com/kubernetes-csi/external-provisioner/blob/1cd1c20a6d4b2fcd25c98a008385b436d61d46a4/deploy/kubernetes/rbac.yaml

The attacher-rbac0.yaml is:
https://github.com/kubernetes-csi/external-attacher/blob/9da8c6d20d58750ee33d61d0faf0946641f50770/deploy/kubernetes/rbac.yaml

Thanks

[pohly]

@darcyllingyan something must have been modified, otherwise deploy-hostpath.sh wouldn't do kubectl apply -f /root/csi-driver-host-path/deploy/provision-rbac0.yaml. What I want to see is why deploy-hostpath.sh fails when everything is unmodified.

[darcyllingyan]

@pohly ,
Sorry for the misunderstanding, yes, the deploy file is changed as below:

set -e
set -o pipefail

function image_version () {
    yaml="$1"
    image="$2"

    # get version from `image: quay.io/k8scsi/csi-attacher:v1.0.1`
    grep "image:.*$image" "$yaml" | sed -e 's/.*:v/v/'
}

BASE_DIR=$(dirname "$0")
K8S_RELEASE=${K8S_RELEASE:-"release-1.13"}
PROVISIONER_RELEASE=${PROVISIONER_RELEASE:-$(image_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner)}
ATTACHER_RELEASE=${ATTACHER_RELEASE:-$(image_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher)}
INSTALL_CRD=${INSTALL_CRD:-"false"}

# apply CSIDriver and CSINodeInfo API objects
if [[ "${INSTALL_CRD}" =~ ^(y|Y|yes|true)$ ]] ; then
    echo "installing CRDs"
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csidriver.yaml --validate=false
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csinodeinfo.yaml --validate=false
fi

# rbac rules
echo "applying RBAC rules"
#kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/1cd1c20a6d4b2fcd25c98a008385b436d61d46a4/deploy/kubernetes/rbac.yaml
#kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/9da8c6d20d58750ee33d61d0faf0946641f50770/deploy/kubernetes/rbac.yaml
kubectl apply -f /root/csi-driver-host-path/deploy/provision-rbac0.yaml
kubectl apply -f /root/csi-driver-host-path/deploy/attach-rbac0.yaml

# deploy hostpath plugin and registrar sidecar
echo "deploying hostpath components"
kubectl apply -f ${BASE_DIR}/hostpath

Besides, based on the guide in https://kubernetes-csi.github.io/docs/example.html, below I show my step by step procedure to create the driver and I didn't use the deploy file to create:

# https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/1cd1c20a6d4b2fcd25c98a008385b436d61d46a4/deploy/kubernetes/rbac.yaml
# mv rbac.yaml external-provisioner-rbac.yaml

# wget https://raw.githubusercontent.com/kubernetes-csi/external-attacher/9da8c6d20d58750ee33d61d0faf0946641f50770/deploy/kubernetes/rbac.yaml
# mv rbac.yaml external-attacher-rbac.yaml

# wget https://raw.githubusercontent.com/kubernetes-csi/driver-registrar/87d0059110a8b4a90a6d2b5a8702dd7f3f270b80/deploy/kubernetes/rbac.yaml
# mv rbac.yaml driver-registrar-rbac.yaml

# kubectl create -f external-provisioner-rbac.yaml 
serviceaccount/csi-provisioner created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created


# kubectl create -f external-attacher-rbac.yaml 
serviceaccount/csi-attacher created
clusterrole.rbac.authorization.k8s.io/external-attacher-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role created
role.rbac.authorization.k8s.io/external-attacher-cfg created
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg created

# kubectl create -f driver-registrar-rbac.yaml 
serviceaccount/csi-driver-registrar created
clusterrole.rbac.authorization.k8s.io/driver-registrar-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-driver-registrar-role created

Last, I create the Daemonset:

# kubectl create -f deploy/hostpath/csi-hostpathplugin.yaml 
daemonset.apps/csi-hostpathplugin created

But after I describe the daemonset:

# kubectl describe daemonset csi-hostpathplugin
Events:
  Type     Reason        Age                From                  Message
  ----     ------        ----               ----                  -------
  Warning  FailedCreate  7s (x13 over 28s)  daemonset-controller  Error creating: pods "csi-hostpathplugin-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins/csi-hostpath": is not allowed to be used spec.volumes[1].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/pods": is not allowed to be used spec.volumes[2].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins_registry": is not allowed to be used spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

Thanks

[pohly]

yes, the deploy file is changed as below:

Can you undo that change and run bash -x ./deploy/deploy-hostpath.sh? I want to see in detail how that fails. This is the first problem.

Warning FailedCreate 7s (x13 over 28s) daemonset-controller Error creating: pods "csi-hostpathplugin-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins/csi-hostpath": is not allowed to be used spec.volumes[1].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/pods": is not allowed to be used spec.volumes[2].hostPath.pathPrefix: Invalid value: "/var/lib/kubelet/plugins_registry": is not allowed to be used spec.containers[1].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

This is the second problem. It looks like your cluster is configured so that it denies running the CSI pods. I don't know how your cluster was setup, but googling leads to (for example): https://access.redhat.com/solutions/3320361

You need to look for similar documentation for your vendor. The upstream documentation is here: https://kubernetes.io/docs/concepts/policy/pod-security-policy/

[darcyllingyan]

Hi @pohly,
For question 1:
If I don't modify the deploy file, the result is below:

# bash -x deploy-hostpath.sh
+ set -e
+ set -o pipefail
++ dirname deploy-hostpath.sh
+ BASE_DIR=.
+ K8S_RELEASE=release-1.13
++ image_version ./hostpath/csi-hostpath-provisioner.yaml csi-provisioner
++ yaml=./hostpath/csi-hostpath-provisioner.yaml
++ image=csi-provisioner
++ grep 'image:.*csi-provisioner' ./hostpath/csi-hostpath-provisioner.yaml
++ sed -e 's/.*:v/v/'
+ PROVISIONER_RELEASE=v1.0.1
++ image_version ./hostpath/csi-hostpath-attacher.yaml csi-attacher
++ yaml=./hostpath/csi-hostpath-attacher.yaml
++ image=csi-attacher
++ grep 'image:.*csi-attacher' ./hostpath/csi-hostpath-attacher.yaml
++ sed -e 's/.*:v/v/'
+ ATTACHER_RELEASE=v1.0.1
+ INSTALL_CRD=false
+ [[ false =~ ^(y|Y|yes|true)$ ]]
+ echo 'applying RBAC rules'
applying RBAC rules
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml
Unable to connect to the server: dial tcp 151.101.120.133:443: i/o timeout

For qeustion2, thanks for supporting the link, I'm studying on it.

Now, I feel a little confused about the CSI install, I have several basic questions want to confirm with you.

1. As we know, in order to use the csi, need to install the csi driver and csi sidecar containers:

   1. Use the hostPath volume type as the example, if I want to deploy the hostpath csi plugin:
      (1) First I need to install the hostpath csi sidecar containers, such as use the bash -x ./deploy/deploy-hostpath.sh to create the needed provisioner plugin and attacher plugin.
      (2) After the sidecar containers are created, I need to install the csi driver(i.e., startup the controllerserver, nodeserver and Identityserver), in the link , it shows $ make hostpath to build hostpathplugin, it is used to startup the driver? I'm not clear how does the sidecar containers are integrated with the driver.

   2. For the csi driver, the guide "https://kubernetes-csi.github.io/docs/drivers.html" supports many csi drivers for different volume types, and supports one official sidecars container links for all the volume types(for example the external-provisioner .)
      question 1: Whether it means, if I want to install one CSI plugin, I have to choose different drivers based on different volume types and the sidecars containers template can use the unique one, right?

Thanks

[pohly]

• kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml
  Unable to connect to the server: dial tcp 151.101.120.133:443: i/o timeout

Can you fix this by setting proxy env variables? I'm afraid the assumption in the example is that network connectivity to GitHub is present. If that assumption is wrong, we may have to go back to mirroring the file inside the repo itself to support offline installation.

Regarding your other questions: no, deploy-hostpath.sh installs both the sidecars and the hostpath CSI driver. You don't need to build anything. The example uses the published images from quay.io.

Every CSI driver needs its own deployment instructions. Check the documentation of the driver(s) that you want to use. They might use the same sidecars, but the deployment itself has to be adapted to the driver.

[darcyllingyan]

Hi @pohly,
Thanks very much for the response!
I set the proxy as export https_proxy="IP:Port", with the proxy setting, I can wget the [](* https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml) successfully but I can't kubectl apply -f [](* https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml), it will report the timeout error.

I have another question, suppose I have created the csi plugin successfully. For example, the cinder csi plugin. With the csi plugin, I created a storageclass successfully, and when I creat the PVC, it reports the "x509: certificate signed by unknown authority", seems when I config the cloud-config, the certificates are unknown.
So my question is: Whether the CSI plugin has an interface to accept the certificates or how can I pass this parameter.

Below is the log:

# kubectl get pod -n kube-system
NAME                                           READY   STATUS    RESTARTS   AGE
csi-attacher-cinderplugin-0                    2/2     Running   0          40m
csi-nodeplugin-cinderplugin-2zq9d              2/2     Running   0          3h42m
csi-provisioner-cinderplugin-0                 2/2     Running   0          3h39m

#cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-pvc-cinderplugin
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-sc-cinderplugin
# kubectl apply -f pvc.yaml 
persistentvolumeclaim/csi-pvc-cinderplugin created

# kubectl get pvc
NAME                   STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
csi-pvc-cinderplugin   Pending                                      csi-sc-cinderplugin   7s
# kubectl describe pvc csi-pvc-cinderplugin
Events:
  Type       Reason                Age               From                                                                                  Message
  ----       ------                ----              ----                                                                                  -------
  Normal     ExternalProvisioning  4s (x4 over 17s)  persistentvolume-controller                                                           waiting for a volume to be created, either by external provisioner "csi-cinderplugin" or manually created by system administrator
  Normal     Provisioning          1s (x3 over 17s)  csi-cinderplugin_csi-provisioner-cinderplugin-0_4bb1a2a3-34cc-11e9-8cc5-b27933f5fb30  External provisioner is provisioning volume for claim "default/csi-pvc-cinderplugin"
  Warning    ProvisioningFailed    1s (x3 over 16s)  csi-cinderplugin_csi-provisioner-cinderplugin-0_4bb1a2a3-34cc-11e9-8cc5-b27933f5fb30  failed to provision volume with StorageClass "csi-sc-cinderplugin": rpc error: code = Unknown desc = Post https://<IP>:13000/v3/auth/tokens: x509: certificate signed by unknown authority

[pohly]

I can't help with the certificate error. This seems specific to your cluster setup, so you better contact your vendor.

Regarding kubelet not honoring https_proxy: that works for me, using kubectl 1.13. I get the same timeout error without proxy env variables and it works when they are set.

[darcyllingyan]

@pohly ，
Thanks for the response, I will check the https_proxy later.
However, I have another question about the CSI,
question1:
The controller plugin can be started in any node of the cluster, if I want to define the node for the controller plugin to be started, what should I do?

question2:
In the csi-hostpath-plugin, the main parts includes:
deploy/hostpath/csi-hostpath-attacher.yaml
deploy/hostpath/csi-hostpath-provisioner.yaml
deploy/hostpath/csi-hostpath-plugin.yaml,

If I want to modify some code to adjust my request, for example the attacher, the source code of the image "quay.io/k8scsi/csi-attacher:v1.0.1" is https://github.com/kubernetes-csi/external-attacher right?
If I modify the code and just use docker build to make new images is OK?

Thanks
Darcy

[pohly]

The controller plugin can be started in any node of the cluster, if I want to define the node for the controller plugin to be started, what should I do?

Is this about the hostpath example or some other CSI driver? For the hostpath example, it currently can fail when a cluster has more than one node. The failure is that the volume gets provisioned on one node and the app using it gets scheduled to another node. A colleague of mine will address this by setting pod affinity. Topology support would be a better solution, but depends on alpha features in Kubernetes.

If I want to modify some code to adjust my request, for example the attacher, the source code of the image "quay.io/k8scsi/csi-attacher:v1.0.1" is https://github.com/kubernetes-csi/external-attacher right?

Right.

If I modify the code and just use docker build to make new images is OK?

You'll have to push to some registry and then reference your own image, but in principle this is what you will do, yes.

[darcyllingyan]

Hi @pohly ,
Thanks very much for the response!

Is this about the hostpath example or some other CSI driver? For the hostpath example, it currently can fail when a cluster has more than one node. The failure is that the volume gets provisioned on one node and the app using it gets scheduled to another node. A colleague of mine will address this by setting pod affinity. Topology support would be a better solution, but depends on alpha features in Kubernetes.

Yes, I'm using the cinder-csi-driver.
it currently can fail when a cluster has more than one node: You mean the sidecar containers couldn't Normally the sidecar containers couldn't be started in more than one node, right?
Whether there is preference which node the sidecar containers(external-provision, external-attacher) should be started? the master node? or others?

Thanks

[pohly]

Yes, I'm using the cinder-csi-driver.

Sorry, but this isn't the support forum for the cinder-csi-driver. Can you take your questions regarding that driver elsewhere?

Regarding csi-driver-host-path: that driver is special and only works on a single node. That restriction does not apply to other drivers. Their developers decide how to best install them in a cluster.

[pohly]

Is there anything in this issue which needs to be fixed in this repo? If not, can we close it?

[darcyllingyan]

Hi @pohly ,
Thanks very much for the response. I have no other question. Will close the issue. Thanks.