kubernetes-csi/driver-registrar

[Deprecated] Sidecar container that 1) registers the CSI driver with kubelet, and 2) adds the drivers custom NodeId to a label on the Kubernetes Node API Object

!NOTE! !THIS REPO HAS BEEN DEPRECATED!

Two new repos have replaced this one for CSI spec 1.0 support:

• cluster-driver-registrar
• node-driver-registrar

This repo will continue to exist for CSI spec 0.3 support.

Driver Registrar

A sidecar container that

1. Registers the containerized CSI driver with kubelet (in the future).
2. Adds the drivers custom NodeId (retrieved via GetNodeID call) to an annotation on the Kubernetes Node API Object.

Community, discussion, contribution, and support

Learn how to engage with the Kubernetes community on the community page.

You can reach the maintainers of this project at:

• Slack channels
  • #wg-csi
  • #sig-storage
• Mailing list

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

driver-registrar's Issues

Split driver-registrar in to two repos

driver-registrar effectively has two modes of operation: cluster registration (creating CSIDriver object). and node registration (handling kubelet plugin registration). There is no shared code between these modes, so it does not make sense for the same binary to handle both of these.

We should split driver-registrar in to two repos/code bases: cluster-driver-registrar and node-driver-registrar

Also, today the container generated from this repo is called driver-registrar. The containers generated from external-attacher and external-provisioner are called csi-attacher and csi-provisioner. We should have consistent naming for all the generated containers:

• Repo: controller-driver-registrar should generate container named csi-controller-driver-registrar
• Repo: node-driver-registrar should generate container named csi-node-driver-registrar
• Repo: external-provisioner should generate container named csi-external-provisioner
• Repo: external-attacher should generate container named csi-external-attacher

CC @msau42

/assign @lpabon

Driver-registrar fails to call NodeProbe request

According to the CSI spec, the CO (or its components) shall invoke this RPC (nodeprobe) prior to scheduling a workload.

A CSI driver that follows the language of the spec will fail if a probe RPC is not invoked prior to other operations. If the CSI language is correct, then this constitutes a bug.

Registrar command should fail upon unrecoverable error

Currently the registrar code is setup in a loop to continuously (https://github.com/kubernetes-csi/driver-registrar/blob/master/cmd/driver-registrar/k8s_register.go#L89) to attempt to reconcile the CSIDriver CRD object associated with the driver. However, when an unrecoverable error occur, the loop continues. This causes the container not to report the failure but instead continues to work.

For instance, the hostpath plugin pod shows that all containers are running

$> kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
csi-hostpath-attacher-0      1/1     Running   0          15m
csi-hostpath-provisioner-0   1/1     Running   0          15m
csi-hostpathplugin-bshzn     3/3     Running   0          15m

However, the logs for the containers in the hostpath plugin shows unrecoverable errors:

E1004 20:40:29.135920       1 k8s_register.go:118] Failed to create CSIDriver object: csidrivers.csi.storage.k8s.io is forbidden: User "system:serviceaccount:default:csi-nodeplugin" cannot create resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope
E1004 20:42:29.147958       1 k8s_register.go:118] Failed to create CSIDriver object: csidrivers.csi.storage.k8s.io is forbidden: User "system:serviceaccount:default:csi-nodeplugin" cannot create resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope
E1004 20:44:29.154980       1 k8s_register.go:118] Failed to create CSIDriver object: csidrivers.csi.storage.k8s.io is forbidden: User "system:serviceaccount:default:csi-nodeplugin" cannot create resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope
E1004 20:46:29.160309       1 k8s_register.go:118] Failed to create CSIDriver object: csidrivers.csi.storage.k8s.io is forbidden: User "system:serviceaccount:default:csi-nodeplugin" cannot create resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope
E1004 20:48:29.166816       1 k8s_register.go:118] Failed to create CSIDriver object: csidrivers.csi.storage.k8s.io is forbidden: User "system:serviceaccount:default:csi-nodeplugin" cannot create resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope

What I expected is the container should fail when it is unable to create CRD type.

Take advantage of `csi_secret` in CSI 1.0

CSI 1.0 decorates sensitive fields with csi_secret. Let's take advantage of this feature to programmatically ensure no sensitive fields are ever logged by this side car container.

Make passing Travis required for merge.

Make passing Travis required for merge

xref kubernetes/test-infra#9014 (comment)

driver-registrar can only register for one CSI driver

Driver-registrar can now only register for one CSI driver, should we expand it for multi drivers ?

If --kubelet-registration-path is set, driver registrar should not update node annotation

If this flag is set and the KubeletPluginsWatcher feature gate is enabled in Kubelet, both Kubelet and the Driver Registrar will have a race to add the node annotation for the given driver. If the Driver Registrar wins and adds the annotation first, Kubelet won't get to add the annotation. If all feature gates are enabled correctly, we should expect Kubelet to be the only component to update this annotation.

Furthermore, one of the reasons to move to kubelet registration, as I understand, is to limit access to Node objects from external components (such as Driver Registrar), so whenever possible node annotation update from Driver Registrar should be limited.

driver-registrar/cmd/driver-registrar/main.go

Line 243 in c3b54e6

getVerifyAndAddNodeId(

/assign
@sbezverk @vladimirvivien @saad-ali

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

How to set apiserver endpoint?

I have set up a custom apiserver endpoint (custom port, https protocol) and would like to be able to set this endpoint in driver-registrar.

Currently, driver-registrar does not have a flag for this:

$ docker run --entrypoint=/bin/sh -ti --rm quay.io/k8scsi/driver-registrar:v0.2.0 -c /bin/sh
/ # ./driver-registrar -h
Usage of ./driver-registrar:
  -alsologtostderr
    	log to standard error as well as files
  -connection-timeout duration
    	Timeout for waiting for CSI driver socket. (default 1m0s)
  -csi-address string
    	Address of the CSI driver socket. (default "/run/csi/socket")
  -kubeconfig string
    	Absolute path to the kubeconfig file. Required only when running out of cluster.
  -log_backtrace_at value
    	when logging hits line file:N, emit a stack trace
  -log_dir string
    	If non-empty, write log files in this directory
  -logtostderr
    	log to standard error instead of files
  -stderrthreshold value
    	logs at or above this threshold go to stderr
  -v value
    	log level for V logs
  -vmodule value
    	comma-separated list of pattern=N settings for file-filtered logging

I hoped that the apisever endpoint could be perhaps extracted from kubeconfig, because of this: https://github.com/kubernetes-csi/driver-registrar/blob/master/cmd/driver-registrar/main.go#L329 But it seems it is not.

The error from driver-registrar:

E0602 15:35:31.029458 1 main.go:166] Failed to get latest version of Node: Get https://10.32.0.1:443/api/v1/nodes/k8s-testing-go-124-worker-1: x509: certificate is valid for 127.0.0.1, not 10.32.0.1

Tag images using semantic versioning

With CSI becoming GA in Kubernetes v1.13, it would be interesting to provide CSI driver developers with a mechanism to have their sidecar images always updated without having to be aware of new builds of those images.

A common way of achieving this is to tag images using semantic versioning. While we do use some sort of semantic versioning in our images, we currently don't overwrite any major and minor tags.

This is an example of issue where this would be helpful: kubernetes/kubernetes#71378. In this example, I could simply use v1 and make sure I'd get all backward compatible changes for my image.

In summary, these are the steps necessary to automate on every build:

1. Create a new patch tag (existing patch tags can never be overwritten).
2. Overwrite the major and minor tags.
3. Overwrite the latest tag.

PS: I'm opening the issue here, but this would also apply for the other sidecar containers.

CC @saad-ali, @msau42, @jsafrane, @pohly, @lpabon

Broken Link of `contributor cheat sheet` needs to fix

Bug Report

I have observed same kind of issue in various kubernetes-csi project.
this happens because after the localization there are too much modifications done in the various directories.
I have observed same issue in this page also.

It has one broken link of the contributes cheat sheet which needs to fix.
I will try to look in further csi repo as well and try to fix it as soon as I can

/kind bug
/assign

csi-address CLI argument does not accept `unix:///` prefixed sock files

When you specify a sock file to the driver-registrar binary's --csi-address argument it can't be prefixed with unix:///. If it does contain this prefix you get errors like this:

I1003 17:26:54.724892       1 connection.go:111] Still trying, connection is CONNECTING
I1003 17:26:54.725159       1 connection.go:111] Still trying, connection is TRANSIENT_FAILURE
I1003 17:26:55.725272       1 connection.go:111] Still trying, connection is CONNECTING
I1003 17:26:55.725314       1 connection.go:111] Still trying, connection is TRANSIENT_FAILURE
I1003 17:26:56.767152       1 connection.go:111] Still trying, connection is CONNECTING
I1003 17:26:56.767388       1 connection.go:111] Still trying, connection is TRANSIENT_FAILURE
I1003 17:26:57.943559       1 connection.go:111] Still trying, connection is CONNECTING
I1003 17:26:57.943781       1 connection.go:111] Still trying, connection is TRANSIENT_FAILURE
I1003 17:26:59.009602       1 connection.go:111] Still trying, connection is CONNECTING

Omitting the prefix works as expected. This is inconsistent with csi-sanity and csi-provisioner which can both accept the prefix without issue.

Example yaml deployment file

        - name: driver-registrar
          image: quay.io/k8scsi/driver-registrar:v0.4.0
          args:
            - "--csi-address=$(DAT_SOCKET)"
            - "--v=5"
          env:
            - name: DAT_SOCKET
              #value: unix:///var/lib/csi/io.daterainc.csi.dsp/csi.sock  #<--- This doesn't work
              value: /var/lib/csi/io.daterainc.csi.dsp/csi.sock  # <--- This works
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/

Use `NodeGetInfo` instead of `NodeGetId`

As NodeGetId is deprecated NodeGetInfo should be used here instead.

Refactor: May need to refactor driver-registrar to support unit tests

After change #52 we noticed that we are adding too much code to main.go and instead we should start creating packages with unit tests.

Mount created by the pod container failed to propagate back to the host on SUSE server 12 sp2.

I saw the logs of third-party plugin:

2018/09/05 06:37:35 Command: mount /dev/disk/by-path/pci-0000:04:00.0-fc-0x2200121011210010-lun-1 /var/lib/kubelet/pods/1eda3123-b0d6-11e8-80bf-58605f89e6df/volumes/kubernetes.io~csi/pvc-ee6d66c0-b0d5-11e8-80bf-58605f89e6df/mount:
2018/09/05 06:37:35 end to NodePublishVolume

However on suse

linux-w8ea:/dev/disk/by-path # umount /var/lib/kubelet/pods/1eda3123-b0d6-11e8-80bf-58605f89e6df/volumes/kubernetes.io~csi/pvc-ee6d66c0-b0d5-11e8-80bf-58605f89e6df/mount
umount: /var/lib/kubelet/pods/1eda3123-b0d6-11e8-80bf-58605f89e6df/volumes/kubernetes.io~csi/pvc-ee6d66c0-b0d5-11e8-80bf-58605f89e6df/mount: not mounted

I also changed the MountFlags to shared in the docker.service file.

Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
MountFlags=shared

[Install]
WantedBy=multi-user.target

How to propagate the mount from container to the host successfully?

@saad-ali

Combine --csi-address and --kubelet-registration-path flags

The value of both flags in the default setting point to the same underlying socket on the host, so we shouldn't need both flags.

kubelet-registration-path needs to have the full /var/lib/kubelet/plugins/<driver-name>/csi.sock path in order to report to kubelet the correct path on the host. We could maybe remove the --csi-address flag somehow.

@vladimirvivien

What happens on error?

While writing the e2e tests, the rbac rules did not have 'update' for nodes, and https://github.com/kubernetes-csi/driver-registrar/blob/master/cmd/driver-registrar/main.go#L220 must have returned error, but all I saw was:

I1208 05:04:33.018520       1 main.go:75] Attempting to open a gRPC connection with: %!q(*string=0xc420326780)
I1208 05:04:33.018679       1 connection.go:75] Connecting to /var/lib/csi/sockets/pluginproxy/mock.socket
I1208 05:04:33.018922       1 connection.go:102] Still trying, connection is CONNECTING
I1208 05:04:34.019333       1 connection.go:102] Still trying, connection is CONNECTING
I1208 05:04:35.061840       1 connection.go:99] Connected
I1208 05:04:35.061863       1 main.go:83] Calling CSI driver to discover driver name.
I1208 05:04:35.061876       1 connection.go:147] GRPC call: /csi.Identity/GetPluginInfo
I1208 05:04:35.061881       1 connection.go:148] GRPC request: version:<minor:1 >
I1208 05:04:35.063534       1 connection.go:150] GRPC response: name:"csi-mock" vendor_version:"0.1.0"
I1208 05:04:35.063561       1 connection.go:151] GRPC error: <nil>
I1208 05:04:35.063566       1 main.go:91] CSI driver name: "csi-mock"
I1208 05:04:35.063573       1 main.go:94] Calling CSI driver to discover node ID.
I1208 05:04:35.063580       1 connection.go:147] GRPC call: /csi.Node/GetNodeID
I1208 05:04:35.063584       1 connection.go:148] GRPC request: version:<minor:1 >
I1208 05:04:35.063953       1 connection.go:150] GRPC response: node_id:"csi-mock"
I1208 05:04:35.063973       1 connection.go:151] GRPC error: <nil>
I1208 05:04:35.063978       1 main.go:102] CSI driver node ID: "csi-mock"
I1208 05:04:35.063984       1 main.go:106] Loading kubeconfig.
I1208 05:04:35.065008       1 main.go:119] Attempt to update node annotation if needed
I1208 05:04:35.077507       1 main.go:174] previousAnnotationValue=""
I1208 05:06:35.084787       1 main.go:174] previousAnnotationValue=""
I1208 05:08:35.091993       1 main.go:174] previousAnnotationValue=""
I1208 05:10:35.097547       1 main.go:174] previousAnnotationValue=""
I1208 05:12:35.103188       1 main.go:174] previousAnnotationValue=""
I1208 05:14:35.110693       1 main.go:174] previousAnnotationValue=""
I1208 05:16:35.117734       1 main.go:174] previousAnnotationValue=""
I1208 05:18:35.124275       1 main.go:174] previousAnnotationValue=""

I am not sure if RetryOnConflict() is doing what we want

There is no way to disable registration server

It defaults kubelet-registration-path to a non-empty string

Versioning needed

Versioning needed to be logged.

Driver registrar should use watch instead of for loops

Currently, the method used to continuously watch an object is a for{} loop. When the driver-registrar is running in DS mode, it is essentially hitting the API server many times per second and should be changed to a watch(). When the driver registrar is running as a single StatefulSet, then it is not hitting the API server as many times, but it would be nice to have it use a watch()

Driver registrar can sometimes take 1 minute to start

Driver registrar can sometimes take 1 minute to start. See kubernetes/kubernetes#71433 (comment)

Let's understand why, and see if we can reduce this.

Calling NodeGetId should be optional

For a plugin that does not implement NodeGetId, register would fail:

I0403 02:04:43.171259       1 connection.go:136] GRPC call: /csi.v0.Node/NodeGetId
I0403 02:04:43.171265       1 connection.go:137] GRPC request: 
I0403 02:04:43.222550       1 connection.go:139] GRPC response: 
I0403 02:04:43.222649       1 connection.go:140] GRPC error: rpc error: code = Unimplemented desc = 
E0403 02:04:43.222709       1 main.go:99] rpc error: code = Unimplemented desc =

According to
https://github.com/container-storage-interface/spec/blob/52021efc8ba5a01988f0c8e96a9dae7eeef2b790/spec.md#nodegetid

NodeGetId is only required for plugins that suports PUBLISH_UNPUBLISH_VOLUME.

A Node Plugin MUST implement this RPC call if the plugin has PUBLISH_UNPUBLISH_VOLUME controller capability. The Plugin SHALL assume that this RPC will be executed on the node where the volume will be used. The CO SHOULD call this RPC for the node at which it wants to place the workload. The result of this call will be used by CO in ControllerPublishVolume.

E2E: "CSI attach test using HostPath driver" fails

The "CSI attach test using HostPath driver" test from kubernetes/test/e2e/storage/csi_volumes.go fails for me:

$ go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=CSI.attach.test.using.HostPath.driver"
....
• Failure [40.263 seconds]
[sig-storage] [Serial] CSI Volumes
/nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22
  CSI attach test using HostPath driver [Serial][Feature:CSISkipAttach]
  /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:125
    attachable volume needs VolumeAttachment [It]
    /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:165

    Failed to create CSIDriver: CSIDriver.csi.storage.k8s.io "csi-hostpath" is invalid: []: Invalid value: map[string]interface {}{"kind":"CSIDriver", "apiVersion":"csi.storage.k8s.io/v1alpha1", "metadata":map[string]interface {}{"creationTimestamp":"2018-09-27T12:48:19Z", "generation":1, "uid":"9c1caf02-c253-11e8-a335-fcaa1497a416", "name":"csi-hostpath"}, "spec":map[string]interface {}{"attachRequired":true, "podInfoOnMountVersion":interface {}(nil)}}: validation failure list:
    spec.podInfoOnMountVersion in body must be of type string: "null"
    Expected error:
        <*errors.StatusError | 0xc422404900>: {
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""},
                Status: "Failure",
                Message: "CSIDriver.csi.storage.k8s.io \"csi-hostpath\" is invalid: []: Invalid value: map[string]interface {}{\"kind\":\"CSIDriver\", \"apiVersion\":\"csi.storage.k8s.io/v1alpha1\", \"metadata\":map[string]interface {}{\"creationTimestamp\":\"2018-09-27T12:48:19Z\", \"generation\":1, \"uid\":\"9c1caf02-c253-11e8-a335-fcaa1497a416\", \"name\":\"csi-hostpath\"}, \"spec\":map[string]interface {}{\"attachRequired\":true, \"podInfoOnMountVersion\":interface {}(nil)}}: validation failure list:\nspec.podInfoOnMountVersion in body must be of type string: \"null\"",
                Reason: "Invalid",
                Details: {
                    Name: "csi-hostpath",
                    Group: "csi.storage.k8s.io",
                    Kind: "CSIDriver",
                    UID: "",
                    Causes: [
                        {
                            Type: "FieldValueInvalid",
                            Message: "Invalid value: map[string]interface {}{\"apiVersion\":\"csi.storage.k8s.io/v1alpha1\", \"metadata\":map[string]interface {}{\"creationTimestamp\":\"2018-09-27T12:48:19Z\", \"generation\":1, \"uid\":\"9c1caf02-c253-11e8-a335-fcaa1497a416\", \"name\":\"csi-hostpath\"}, \"spec\":map[string]interface {}{\"attachRequired\":true, \"podInfoOnMountVersion\":interface {}(nil)}, \"kind\":\"CSIDriver\"}: validation failure list:\nspec.podInfoOnMountVersion in body must be of type string: \"null\"",
                            Field: "[]",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 422,
            },
        }
        CSIDriver.csi.storage.k8s.io "csi-hostpath" is invalid: []: Invalid value: map[string]interface {}{"kind":"CSIDriver", "apiVersion":"csi.storage.k8s.io/v1alpha1", "metadata":map[string]interface {}{"creationTimestamp":"2018-09-27T12:48:19Z", "generation":1, "uid":"9c1caf02-c253-11e8-a335-fcaa1497a416", "name":"csi-hostpath"}, "spec":map[string]interface {}{"attachRequired":true, "podInfoOnMountVersion":interface {}(nil)}}: validation failure list:
        spec.podInfoOnMountVersion in body must be of type string: "null"
    not to have occurred

    /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:227
------------------------------
SSSSSSSSSSSSSSSSSSep 27 14:48:37.353: INFO: Running AfterSuite actions on all node
Sep 27 14:48:37.353: INFO: Running AfterSuite actions on node 1


Summarizing 2 Failures:

[Fail] [sig-storage] [Serial] CSI Volumes CSI attach test using HostPath driver [Serial][Feature:CSISkipAttach] [It] non-attachable volume does not need VolumeAttachment 
/nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:227

[Fail] [sig-storage] [Serial] CSI Volumes CSI attach test using HostPath driver [Serial][Feature:CSISkipAttach] [It] attachable volume needs VolumeAttachment 
/nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:227

Ran 3 of 1813 Specs in 158.815 seconds
FAIL! -- 1 Passed | 2 Failed | 0 Pending | 1810 Skipped --- FAIL: TestE2E (158.90s)
FAIL

I ran this after building Kubernetes from source (some recent master revision) and starting the cluster with:

RUNTIME_CONFIG= ALLOW_PRIVILEGED=1 FEATURE_GATES="BlockVolume=true,MountPropagation=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true"  hack/local-up-cluster.sh -O