kubernetes-csi/node-driver-registrar

Sidecar container that registers a CSI driver with the kubelet using the kubelet plugin registration mechanism.

node-driver-registrar's Issues

Return an error when registration socket file exists but is not a socket

Followup to https://github.com/kubernetes-csi/node-driver-registrar/pull/99/files#r469647510
/help
/good-first-issue

distroless doesn't contain /bin/sh

Which makes this part of the readme example non-functional:

            command: ["/bin/sh", "-c", "rm -rf /registration/<plugin> /registration/<drivername.example.com>-reg.sock"]

Ensure reg.sock exists while the registrar is running

While the registrar is running, the kubelet registration socket it installs could be removed erroneously by a different process (for example if two instances of the same driver are running in the cluster). The registrar should continuously ensure that the socket exists during its lifetime (and subsequently removed before it terminates).

/cc @davidz627 @msau42

Support ARM (armv7) images

Is there a way to build/release armv7 images too?

Discover CSI spec version

Right now the node-driver-registrar hardcodes the CSI spec version supported from the driver.

Ideally, this should be discovered from the plugin's RPC service version. If that's not possible, then we should consider using GetPluginInfo() to pass this information through.

cc @saad-ali @jsafrane

Bump version of Go for CVE resolution?

Hello. Would it be possible to bump the version of Golang to help resolve some of these CVE's?

CVE-2021-38297 | critical | go 1.16.2 | 9,8 | https://nvd.nist.gov/vuln/detail/CVE-2021-38297
CVE-2021-27918 | high | go 1.14.15 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-27918
CVE-2021-29923 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-29923
CVE-2021-33194 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-33194
CVE-2021-33195 | high | go 1.16.2 | 7,3 | https://nvd.nist.gov/vuln/detail/CVE-2021-33195
CVE-2021-33196 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-33196
CVE-2021-33198 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-33198
CVE-2021-41771 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-41771
CVE-2021-41772 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-41772
CVE-2021-44716 | high | go 1.16.2 | 7,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-44716
CVE-2020-29510 | medium | go 1.14.15 | 5,6 | https://nvd.nist.gov/vuln/detail/CVE-2020-29510
CVE-2021-31525 | medium | go 1.16.2 | 5,9 | https://nvd.nist.gov/vuln/detail/CVE-2021-31525
CVE-2021-33197 | medium | go 1.16.2 | 5,3 | https://nvd.nist.gov/vuln/detail/CVE-2021-33197
CVE-2021-34558 | medium | go 1.16.2 | 6,5 | https://nvd.nist.gov/vuln/detail/CVE-2021-34558
CVE-2021-36221 | medium | go 1.16.2 | 5,9 | https://nvd.nist.gov/vuln/detail/CVE-2021-36221

support JSON log format for node-driver-registrar

• Provide a flag to enable JSON log formatting.

We do this today for CSI driver implementations but the same can't be configured for node-driver-registrar. This can be done by adding log-format-json=true flag to the node-driver-registrar and setting the log format based on that.

CSINode has no drivers

I am trying to get aws-ebs-csi-driver running on Kubernetes EKS 1.23

CSINode resources are missing drivers information.

The node-csi-driver image is:
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1

node-csi-driver logs:

I0914 09:25:02.003172       1 main.go:166] Version: v2.5.1
I0914 09:25:02.003206       1 main.go:167] Running node-driver-registrar in mode=registration
I0914 09:25:02.003610       1 main.go:191] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0914 09:25:04.066382       1 main.go:198] Calling CSI driver to discover driver name
I0914 09:25:04.068067       1 main.go:208] CSI driver name: "ebs.csi.aws.com"
I0914 09:25:04.068097       1 node_register.go:53] Starting Registration Server at: /registration/ebs.csi.aws.com-reg.sock
I0914 09:25:04.068253       1 node_register.go:62] Registration Server started at: /registration/ebs.csi.aws.com-reg.sock
I0914 09:25:04.068339       1 node_register.go:92] Skipping HTTP server because endpoint is set to: ""

The ebs-plugin image is:
public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.11.2

ebs-plugin node logs:

I0914 09:25:00.785711       1 node.go:98] regionFromSession Node service
I0914 09:25:00.785739       1 metadata.go:85] retrieving instance data from ec2 metadata
W0914 09:25:03.888576       1 metadata.go:88] ec2 metadata is not available
I0914 09:25:03.888590       1 metadata.go:96] retrieving instance data from kubernetes api
I0914 09:25:03.889324       1 metadata.go:101] kubernetes api is available

How can I further debug this?

csi-driver-registrar Registration process failed on aws node

Hello.
When I follow instructions for deploying csi-driver-registrar on aws cluster (EKS) nodes (EC2) the node-driver-registrar always fails with the following log:
I0608 14:48:02.757010 1 main.go:113] Version: v2.1.0-0-g80d42f24
I0608 14:48:02.757918 1 connection.go:153] Connecting to unix:///csi/csi.sock
I0608 14:48:02.852446 1 node_register.go:52] Starting Registration Server at: /registration/io.openebs.csi-mayastor-reg.sock
I0608 14:48:02.852593 1 node_register.go:61] Registration Server started at: /registration/io.openebs.csi-mayastor-reg.sock
I0608 14:48:02.852650 1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
I0608 14:48:04.226354 1 main.go:80] Received GetInfo call: &InfoRequest{}
I0608 14:48:04.589035 1 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: error updating Node object with CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "kubernetes.io/hostname":"ip-X-X-X-X" but existing label is "kubernetes.io/hostname":"ip-X-X-X-X.us-east-1.compute.internal",}
E0608 14:48:04.589073 1 main.go:92] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: error updating Node object with CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "kubernetes.io/hostname":"ip-X-X-X-X" but existing label is "kubernetes.io/hostname":"ip-X-X-X-X.us-east-1.compute.internal", restarting registration container.

Multi-Arch build

Can you please build for s390x (and maybe ppc64 and ppc64le) ?

Bump the registrar to v2

#96 introduces a breaking change. The next release should be v2.0.

Missing image/manifest for release `v2.6.1`

When trying to pull this image with the command provided in the v2.6.1 release notes, there is an error:

$ docker pull registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.1
Error response from daemon: manifest for registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.1 not found: manifest unknown: Failed to fetch "v2.6.1"

No issues when attempting to pull image for v2.6.0 release:

$ docker pull k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.6.0                                                                                                                                                                                                                                                                                                                
v2.6.0: Pulling from sig-storage/csi-node-driver-registrar
9a6e6aa442ae: Pull complete
915cb72ecb33: Pull complete
Digest: sha256:f1c25991bac2fbb7f5fcf91ed9438df31e30edee6bed5a780464238aa09ad24c
Status: Downloaded newer image for k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.6.0
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.6.0

Listing the available tags using skopeo does not show the v2.6.1 tag either:

$ skopeo list-tags docker://k8s.gcr.io/sig-storage/csi-node-driver-registrar 
{
    "Repository": "k8s.gcr.io/sig-storage/csi-node-driver-registrar",
    "Tags": [
        "sha256-0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4.sig",
        "sha256-f1c25991bac2fbb7f5fcf91ed9438df31e30edee6bed5a780464238aa09ad24c.sig",
        "v1.2.0",
        "v1.3.0",
        "v2.0.0",
        "v2.0.1",
        "v2.1.0",
        "v2.2.0",
        "v2.3.0",
        "v2.4.0",
        "v2.5.0",
        "v2.5.1",
        "v2.6.0"
    ]
}

Support structured logging

The node-driver-registrar container should support structured logging when given the --log-format-json flag, like for instance the secrets-store-csi-driver does.

A simple implementation can be found here: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/cmd/secrets-store-csi-driver/main.go#L90-L94

node-driver-registrar pod reports "Connecting to /csi/csi.sock timed out"

Hi,
When I deployed the CSI driver, the node-driver-registrar pod couldn't be started and always reports the TRANSIENT_FAILURE error, whether there are some issues or restrictions for my cluster environment when using the CSI plugin?
Now the external provisioner, external attacher are all running, only the node-driver-registrar crash.
Based on below log, the csi driver socket is OK but the node-driver-registrar socket couldn't connect.

# kubectl get pod -n kube-system
NAME                                          READY   STATUS             RESTARTS   AGE
csi-attacher-cinderplugin-0                   2/2     Running            0          5h53m
csi-nodeplugin-cinderplugin-dj8w4             1/2     CrashLoopBackOff   6          17m
csi-nodeplugin-cinderplugin-sfrqf             1/2     CrashLoopBackOff   6          17m
csi-provisioner-cinderplugin-0                2/2     Running            0          5h53m
csi-snapshotter-cinder-0                      2/2     Running            0          5h53m

The node-driver-registrar log is below:

# kubectl logs csi-nodeplugin-cinderplugin-gm7sl -n kube-system node-driver-registrar
I0321 08:59:37.981108       1 main.go:111] Version: 
I0321 08:59:37.981219       1 main.go:118] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0321 08:59:37.981238       1 connection.go:69] Connecting to /csi/csi.sock
I0321 08:59:37.981533       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:37.982349       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:38.981960       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:38.982200       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:40.024028       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:40.024239       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:41.200354       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:41.200952       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:42.266468       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:42.266622       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:43.241629       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:43.241848       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:44.211730       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:44.211804       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:45.286693       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:46.113061       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:46.113212       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:46.975907       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:46.975991       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:47.815025       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:47.815552       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:48.735753       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:49.741918       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:49.742031       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:50.867745       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:51.753623       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:51.753767       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:52.705934       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:52.706004       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:53.633234       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:53.633507       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:54.620910       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:54.621000       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:55.534845       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:55.535080       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:56.451708       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:56.451879       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:57.523498       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:57.523625       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:58.411093       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:58.411253       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 08:59:59.292524       1 connection.go:96] Still trying, connection is CONNECTING
I0321 08:59:59.292616       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:00.237176       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:00.237323       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:01.265661       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:01.265814       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:02.411186       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:03.328529       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:03.328771       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:04.247522       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:04.247714       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:05.348857       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:05.348987       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:06.231515       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:06.231624       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:07.377978       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:07.378151       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:08.456791       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:08.456977       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:09.466570       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:10.278022       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:10.278267       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:11.141699       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:11.141731       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:12.184630       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:13.374929       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:13.375161       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:14.206948       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:15.245086       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:15.245589       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:16.068902       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:16.069127       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:17.146052       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:17.146131       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:18.066639       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:18.067036       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:18.936236       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:18.936701       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:19.952976       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:19.953258       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:20.970847       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:20.970957       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:21.882470       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:21.882509       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:22.851985       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:22.852068       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:23.864330       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:23.864465       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:24.766355       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:24.766704       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:25.679190       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:26.794854       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:26.794984       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:27.739730       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:27.739875       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:28.892300       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:29.811378       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:30.969299       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:30.969543       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:31.808533       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:32.999305       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:32.999488       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:33.829136       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:33.829275       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:34.718244       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:34.718459       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:35.790877       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:35.790983       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:36.687615       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:36.687817       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:37.612463       1 connection.go:96] Still trying, connection is CONNECTING
I0321 09:00:37.612718       1 connection.go:96] Still trying, connection is TRANSIENT_FAILURE
I0321 09:00:37.981411       1 connection.go:89] Connection timed out
I0321 09:00:37.981436       1 main.go:126] Calling CSI driver to discover driver name.
I0321 09:00:37.981453       1 connection.go:137] GRPC call: /csi.v1.Identity/GetPluginInfo
I0321 09:00:37.981458       1 connection.go:138] GRPC request: {}
I0321 09:00:37.982497       1 connection.go:140] GRPC response: {}
I0321 09:00:37.982918       1 connection.go:141] GRPC error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
E0321 09:00:37.982934       1 main.go:131] rpc error: code = Unavailable desc = all SubConns are in TransientFailure

The CSI driver socket is below:

# kubectl logs -f csi-nodeplugin-cinderplugin-228sh -n kube-system cinder
I0321 03:19:18.516206       1 driver.go:56] Driver: cinder.csi.openstack.org version: 1.0.0
I0321 03:19:18.516722       1 driver.go:88] Enabling controller service capability: LIST_VOLUMES
I0321 03:19:18.516744       1 driver.go:88] Enabling controller service capability: CREATE_DELETE_VOLUME
I0321 03:19:18.516756       1 driver.go:88] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0321 03:19:18.516802       1 driver.go:88] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0321 03:19:18.516827       1 driver.go:88] Enabling controller service capability: LIST_SNAPSHOTS
I0321 03:19:18.516840       1 driver.go:100] Enabling volume access mode: SINGLE_NODE_WRITER
I0321 03:19:18.516885       1 driver.go:110] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0321 03:19:18.518095       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}

Thanks
Darcy

csi-node-driver-registrar:v2.0.1 image having VA issues

following is the VA issues in csi-node-driver-registrar:v2.0.1 image

DLA-2424-1
   Policy Status
   Active
   Summary
   tzdata, the time zone and daylight-saving time data, has been updated to the latest version.
  - Revised predictions for Morocco's changes starting in 2023. - Macquarie Island has stayed in sync with Tasmania since 2011. - Casey, Antarctica is at +08 in winter and +11 in summer since 2018. - Palestine ends DST earlier than predicted, on 2020-10-24. - Fiji starts DST later than usual, on 2020-12-20.
   Vendor Security Notice IDs   Official Notice   
   DLA-2424-1                   https://lists.debian.org/debian-lts-announce/2020/10/msg00037.html   
   Affected Packages   Policy Status   How to Resolve                        Security Notice   
   tzdata              Active          Upgrade tzdata to >= 2020d-0+deb9u1   DLA-2424-1 

Scheme "" not registered, fallback to default scheme

K8s ver. 1.17.6
After applying https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml and creating a PV logs are flooded by:

Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.776472    4066 clientconn.go:104] parsed scheme: ""
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.776566    4066 clientconn.go:104] scheme "" not registered, fallback to default scheme
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.776799    4066 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock 0  <nil>}] <nil>}
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.776893    4066 clientconn.go:577] ClientConn switching balancer to "pick_first"
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.788185    4066 clientconn.go:104] parsed scheme: ""
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.788208    4066 clientconn.go:104] scheme "" not registered, fallback to default scheme
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.788233    4066 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock 0  <nil>}] <nil>}
Jun 13 15:43:33 worker-2 kubelet[4066]: I0613 15:43:33.788244    4066 clientconn.go:577] ClientConn switching balancer to "pick_first"

Broken Link of `contributor cheat sheet` needs to fix

Bug Report

I have observed same kind of issue in various kubernetes-csi project.
this happens because after the localization there are too much modifications done in the various directories.
I have observed same issue in this page also.
It has one broken link of the contributes cheat sheet which needs to fix.
I will try to look in further csi repo as well and try to fix it as soon as I can

/kind bug
/assign

Mode `kubelet-registration-probe` is broken

It seems that --mode=kubelet-registration-probe is broken.
Pods that configured a livenessProbe according to the recommendations fail with the following error

main.go:107] "Failed to create registration probe file" err="open /var/lib/kubelet/plugins/smb.csi.k8s.io/registration: no such file or directory" registrationProbePath="/var/lib/kubelet/plugins/smb.csi.k8s.io/registration"

The registration path and file is hard coded in the main.go.

node-driver-registrar/cmd/csi-node-driver-registrar/main.go

Lines 148 to 153 in f8bc53e

	kubeletRegistrationPathDir := filepath.Dir(*kubeletRegistrationPath)
	registrationProbePath = filepath.Join(kubeletRegistrationPathDir, "registration")
	// with the mode kubelet-registration-probe
	if modeIsKubeletRegistrationProbe() {
	lockfileExists, err := util.DoesFileExist(registrationProbePath)

But the registration file is in another location and also named differently.
pluginRegistrationPath is usually /var/lib/kubelet/plugins_registry (see the kubernetes-csi driver-volume-mounts docs)

node-driver-registrar/cmd/csi-node-driver-registrar/node_register.go

Lines 86 to 88 in f8bc53e

	func buildSocketPath(csiDriverName string) string {
	return fmt.Sprintf("%s/%s-reg.sock", *pluginRegistrationPath, csiDriverName)
	}

where nodeRegister is using buildSocketPath

node-driver-registrar/cmd/csi-node-driver-registrar/node_register.go

Lines 36 to 41 in f8bc53e

	func nodeRegister(csiDriverName, httpEndpoint string) {
	// When kubeletRegistrationPath is specified then driver-registrar ONLY acts
	// as gRPC server which replies to registration requests initiated by kubelet's
	// plugins watcher infrastructure. Node labeling is done by kubelet's csi code.
	registrar := newRegistrationServer(csiDriverName, *kubeletRegistrationPath, supportedVersions)
	socketPath := buildSocketPath(csiDriverName)

_{Tobias Giese [email protected], Mercedes-Benz Tech Innovation GmbH, legal info/Impressum}

update the windows dockerfile to use a relative path to binary for better host process support

When using windows host process containers the entrypoint needs to be relative otherwise it looks for the binary at that path on the actual host filesystem.

When the node restarts, csi-plugin registration time is too long because the old sock file is invalid during the kubelet startup process.

When the node restarts, kubelet would found a residual sock files in the /var/lib/kubelet/plugins_registry directory, which will cause the csi plugin registration to fail. If the registration fails, kubelet will enter the back-off process, wait for a period of time and then re-register. The waiting time will gradually increase, and the maximum single wait is 2 minutes and 2 seconds.

I‘d like to understand your release strategy

Since last release is released in May, and many CVE mitigation PRs has been merged. But how often would you release a new version of node-driver-registrar? Like when we can get a newly release that contains the fixes?
Thank you!

update RBAC file

The RBAC file was copied unmodified from driver-registrar and probably is out-dated now.

The introduction still refers to "external provisioner" (was already broken when creating that file initially for driver-registrar).

Rebuild with golang `v1.18.6` or higher

Our image scanner detected some golang related findings (e.g. CVE-2022-27664) in the latest release v2.5.1 which can be fixed by rebuilding with golang v1.18.6 or higher.

Is there a way to upgrade to golang v1.18.6, build and publish an upgraded version?

If I understand the build process correctly the binary is build with the go version which is used in the corresponding pipeline (in this case v1.17.3) and then copied to build the corresponding image.

Can we have opaque parameters in CSINode Object

Can we allow the CSINode object to be filled with some more parameters other than default known spec fields? for example: if I want to list the kernel version in this object,

apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  name: node1
spec:
  drivers:
  - name: mycsidriver.example.com
    nodeID: storageNodeID1
    topologyKeys: ['mycsidriver.example.com/regions', "mycsidriver.example.com/zones"]

Cannot register cinder-csi-plugin

Hello,When I follow instructions for deploying cinder-csi-plugin on k8s cluster running on Openstack, the node-driver-registrar always fails with following log:

[root@k8s1 kubernetes]# kubectl logs -f -n kube-system csi-cinder-nodeplugin-flx7p 
error: a container name must be specified for pod csi-cinder-nodeplugin-flx7p, choose one of: [node-driver-registrar cinder-csi-plugin]
[root@k8s1 kubernetes]# kubectl logs -f -n kube-system csi-cinder-nodeplugin-flx7p  node-driver-registrar 
I0721 07:33:42.755595       1 main.go:110] Version: v1.3.0-0-g6e9fff3e
I0721 07:33:42.755917       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0721 07:33:42.755951       1 connection.go:151] Connecting to unix:///csi/csi.sock
I0721 07:33:42.757058       1 main.go:127] Calling CSI driver to discover driver name
I0721 07:33:42.757121       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0721 07:33:42.757133       1 connection.go:181] GRPC request: {}
I0721 07:33:42.761921       1 connection.go:183] GRPC response: {"name":"cinder.csi.openstack.org","vendor_version":"1.2.0"}
I0721 07:33:42.763337       1 connection.go:184] GRPC error: <nil>
I0721 07:33:42.763361       1 main.go:137] CSI driver name: "cinder.csi.openstack.org"
I0721 07:33:42.763535       1 node_register.go:51] Starting Registration Server at: /registration/cinder.csi.openstack.org-reg.sock
I0721 07:33:42.764049       1 node_register.go:60] Registration Server started at: /registration/cinder.csi.openstack.org-reg.sock
I0721 07:33:43.989855       1 main.go:77] Received GetInfo call: &InfoRequest{}
I0721 07:33:44.278417       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input,}
E0721 07:33:44.278518       1 main.go:89] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Internal desc = retrieving availability zone from MetaData service failed with error unexpected end of JSON input, restarting registration container.
[root@k8s1 kubernetes]# vim cinder-csi-nodeplugin.yml 

version:

k8s:v1.18.3
opentack:r

images:

cinder-csi-nodeplugin.yaml:          image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
cinder-csi-nodeplugin.yaml:          image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-attacher:v2.1.1
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-provisioner:v1.4.0
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-snapshotter:v1.2.2
cinder-csi-controllerplugin.yaml:          image: quay.io/k8scsi/csi-resizer:v0.4.0
cinder-csi-controllerplugin.yaml:          image: docker.io/k8scloudprovider/cinder-csi-plugin:v1.18.0

Thanks.

v2.6.0 container image was not pushed to registry.k8s.io

Although I see a release in Github, I'm unable to pull the v2.6.0 container image:

$ skopeo list-tags docker://registry.k8s.io/sig-storage/csi-node-driver-registrar
{
    "Repository": "registry.k8s.io/sig-storage/csi-node-driver-registrar",
    "Tags": [
        "sha256-0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4.sig",
        "v1.2.0",
        "v1.3.0",
        "v2.0.0",
        "v2.0.1",
        "v2.1.0",
        "v2.2.0",
        "v2.3.0",
        "v2.4.0",
        "v2.5.0",
        "v2.5.1"
    ]
}

No Windows 2004 image available

Hi

According to curl -L https://mcr.microsoft.com/v2/oss/kubernetes-csi/csi-node-driver-registrar/tags/list there is no image other than the 1809 kernel for Windows available.

It would make sense to support the versions according to the official windows-os-version-support

Remove old registration mechanism

We should stop supporting the old way of registering the driver by directly modifying the node object.

Have we deprecated the old plugin registration mechanism yet? cc @saad-ali

arm build

Arm build of this image. Looks like multi-arch support is supported in quay as of May 2019.
https://trello.com/c/4EcAIJrd/1303-5-quayregistry-add-support-for-manifest-list-rd

Docker also supports building arm with qemu with "experimental" enabled in the .docker/config.json

ref #48

Feature request: implement a readiness probe

• implement a readiness probe for the sidecar
• return "ready" only once connected to the driver

Background: we debugged a pmem-csi deployment case where node-driver-registrar 1.0.2 failed to connect to /csi/csi.sock, getting timeout and causing pod exit in 60 seconds, leading to CrashLoop.
We then tried version 1.1.0 which does not timeout. There the pod remained running and gave false impression of good state, but in reality it kept trying socket connection which never succeeded. In that sense the v1.1.0 "try without timeout" semantics can be considered harmful as problem state remains hidden (if one does not look in log).

@pohly then expressed idea about readiness probe.

The deployment problem looked initially a lot like #36 (but SELinux was not involved) but got explained as missing /csi/csi.sock caused by chain starting with other misconfiguration, so nothing to blame as system functioned exactly the way it was configured.

Upgrade grpc-go dependency to allow drivers to be written in rust / typescript

Due to a bug in grpc-go, drivers in typescript / rust weren't working. This has been fixed so now this package needs to update its dependency on grpc-go soon (grpc/grpc-go#2628)

Cannot register cinder-csi-plugin

Hello.
When I follow instructions for deploying cinder-csi-plugin on k8s cluster running on Openstack, the node-driver-registrar always fails with following log:

I0103 12:01:23.352645       1 main.go:110] Version: v1.1.0-0-g80a94421
I0103 12:01:23.352683       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0103 12:01:23.352694       1 connection.go:151] Connecting to unix:///csi/csi.sock
I0103 12:01:23.353048       1 main.go:127] Calling CSI driver to discover driver name
I0103 12:01:23.353061       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0103 12:01:23.353065       1 connection.go:181] GRPC request: {}
I0103 12:01:23.354504       1 connection.go:183] GRPC response: {"name":"cinder.csi.openstack.org","vendor_version":"1.2.0"}
I0103 12:01:23.354928       1 connection.go:184] GRPC error: <nil>
I0103 12:01:23.354933       1 main.go:137] CSI driver name: "cinder.csi.openstack.org"
I0103 12:01:23.355093       1 node_register.go:54] Starting Registration Server at: /registration/cinder.csi.openstack.org-reg.sock
I0103 12:01:23.355221       1 node_register.go:61] Registration Server started at: /registration/cinder.csi.openstack.org-reg.sock
I0103 12:01:24.203781       1 main.go:77] Received GetInfo call: &InfoRequest{}
I0103 12:01:25.203828       1 main.go:77] Received GetInfo call: &InfoRequest{}
I0103 12:01:55.212300       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: timed
out waiting for the condition,}
E0103 12:01:55.212382       1 main.go:89] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: timed out waiting for the condition, restarting registration container.

Relevant part of node's kubelet log is (first error should be harmless, second is the problem):

Jan 03 13:02:00 kubeod2-01 kubelet[6600]: I0103 13:02:00.849670    6600 clientconn.go:796] ClientConn switching balancer to "pick_first"
Jan 03 13:02:00 kubeod2-01 kubelet[6600]: I0103 13:02:00.849710    6600 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc0012fcce0, CONNECTING
Jan 03 13:02:00 kubeod2-01 kubelet[6600]: I0103 13:02:00.850078    6600 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0xc0012fcce0, READY
Jan 03 13:02:00 kubeod2-01 kubelet[6600]: E0103 13:02:00.850790    6600 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock" failed. No retries permitted until 2020-01-03 13:04:02.850682742 +0100 CET m=+290.199500897 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Jan 03 13:02:07 kubeod2-01 kubelet[6600]: I0103 13:02:07.847745    6600 clientconn.go:440] parsed scheme: ""
Jan 03 13:02:07 kubeod2-01 kubelet[6600]: I0103 13:02:07.848423    6600 clientconn.go:440] scheme "" not registered, fallback to default scheme
Jan 03 13:02:07 kubeod2-01 kubelet[6600]: E0103 13:02:07.869913    6600 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/cinder.csi.openstack.org-reg.sock" failed. No retries permitted until 2020-01-03 13:02:08.369876971 +0100 CET m=+175.718695119 (durationBeforeRetry 500ms). Error: "RegisterPlugin error -- plugin registration failed with err: timed out waiting for the condition: rpc error: code = Unavailable desc = transport is closing"
Jan 03 13:02:07 kubeod2-01 kubelet[6600]: W0103 13:02:07.870470    6600 clientconn.go:960] grpc: addrConn.transportMonitor exits due to: grpc: the connection is closing

Other containers seem to work normally, log of cinder-csi-plugin reads:

I0103 12:01:23.354217       1 utils.go:79] GRPC call: /csi.v1.Identity/GetPluginInfo
I0103 12:01:23.354239       1 utils.go:80] GRPC request:
I0103 12:01:23.354285       1 identityserver.go:33] Using default GetPluginInfo
I0103 12:01:23.354296       1 utils.go:85] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.0"
I0103 12:01:25.204836       1 utils.go:79] GRPC call: /csi.v1.Node/NodeGetInfo
I0103 12:01:25.204856       1 utils.go:80] GRPC request:
I0103 12:01:25.204935       1 mount.go:246] Got instance id from /var/lib/cloud/data/instance-id: cd79a582-f93d-42f9-a86a-5ae7147971c9
I0103 12:01:55.205230       1 utils.go:85] GRPC response: node_id:"cd79a582-f93d-42f9-a86a-5ae7147971c9" max_volumes_per_node:256 accessible_topology:<segments:<key:"topology.cinder.csi.openstack.or
g/zone" value:"" > >
I0103 12:02:44.612067       1 utils.go:79] GRPC call: /csi.v1.Identity/GetPluginInfo
I0103 12:02:44.612083       1 utils.go:80] GRPC request:
I0103 12:02:44.612117       1 identityserver.go:33] Using default GetPluginInfo
I0103 12:02:44.612121       1 utils.go:85] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.0"
I0103 12:02:46.215437       1 utils.go:79] GRPC call: /csi.v1.Node/NodeGetInfo
I0103 12:02:46.215453       1 utils.go:80] GRPC request:
I0103 12:02:46.215523       1 mount.go:246] Got instance id from /var/lib/cloud/data/instance-id: cd79a582-f93d-42f9-a86a-5ae7147971c9
I0103 12:03:16.215929       1 utils.go:85] GRPC response: node_id:"cd79a582-f93d-42f9-a86a-5ae7147971c9" max_volumes_per_node:256 accessible_topology:<segments:<key:"topology.cinder.csi.openstack.or
g/zone" value:"" > >

I raised verbose values for node-driver-registrar and cinder-csi-plugin (logs above), but have no other idea how to debug and eliminate this error. Could you please help?

Thanks, Martin

Node driver could not register on Windows in a little possibility

On Windows, we found that this node-driver-registrar hang on below statement forever in a little possibility, and then after restart registrar manually, it could register correctly, question: is there any way to add timeout and retry in below statement?

node-driver-registrar/cmd/csi-node-driver-registrar/node_register.go

Lines 63 to 64 in d79871c

	// Registers kubelet plugin watcher api.
	registerapi.RegisterRegistrationServer(grpcServer, registrar)

• node-driver-registrar hang

I0429 19:12:33.856777    5500 main.go:113] Version: v2.1.0-0-g80d42f24
E0429 19:12:35.055779    5500 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0429 19:12:35.055779    5500 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
I0429 19:12:35.055779    5500 main.go:137] Attempting to open a gRPC connection with: "unix://C:\\\\csi\\\\csi.sock"
I0429 19:12:35.055779    5500 connection.go:153] Connecting to unix://C:\\csi\\csi.sock
I0429 19:12:44.501633    5500 main.go:144] Calling CSI driver to discover driver name
I0429 19:12:44.511914    5500 main.go:154] CSI driver name: "disk.csi.azure.com"
I0429 19:12:44.512939    5500 node_register.go:52] Starting Registration Server at: /registration/disk.csi.azure.com-reg.sock
I0429 19:12:44.523926    5500 node_register.go:61] Registration Server started at: /registration/disk.csi.azure.com-reg.sock
I0429 19:12:44.524941    5500 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""

• After restart driver(run node-driver-registrar), it works

0429 19:12:34.188716    5480 main.go:113] Version: v2.1.0-0-g80d42f24
E0429 19:12:35.839739    5480 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0429 19:12:35.840735    5480 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
I0429 19:12:35.840735    5480 main.go:137] Attempting to open a gRPC connection with: "unix://C:\\\\csi\\\\csi.sock"
I0429 19:12:35.840735    5480 connection.go:153] Connecting to unix://C:\\csi\\csi.sock
I0429 19:12:45.261075    5480 main.go:144] Calling CSI driver to discover driver name
I0429 19:12:45.264589    5480 main.go:154] CSI driver name: "disk.csi.azure.com"
I0429 19:12:45.264589    5480 node_register.go:52] Starting Registration Server at: /registration/disk.csi.azure.com-reg.sock
I0429 19:12:45.266595    5480 node_register.go:61] Registration Server started at: /registration/disk.csi.azure.com-reg.sock
I0429 19:12:45.268093    5480 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
I0429 19:12:45.616080    5480 main.go:80] Received GetInfo call: &InfoRequest{}
I0429 19:12:47.292740    5480 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

/kind bug
/sig windows

Node driver registrar

Add an option to enable pprof profiling

docker pull error on latest image v2.2.1

I tried to pull the latest docker image from:

docker pull k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.1

This fails with:

Error response from daemon: manifest for k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.1 not found: manifest unknown: Failed to fetch "v2.2.1" from request "/v2/sig-storage/csi-node-driver-registrar/manifests/v2.2.1".

"Failed to create registration probe file" with readOnlyRootFilesystem

When using readOnlyRootFilesystem: true the node-driver-registrar tries to recursively create the directory the probe file is in, leading to an error:

E0804 18:06:27.375117       1 main.go:107] "Failed to create registration probe file" err="mkdir /var/lib/kubelet: read-only file system" registrationProbePath="/var/lib/kubelet/plugins/ebs.csi.aws.com/registration"

In this scenario /var/lib/kubelet/plugins/ebs.csi.aws.com/ is a hostPath mounted into the container (and registration works just fine), but node-driver-registrar tries to create /var/lib/kubelet/ despite the fact it already exists and fails, never creating the probe file.

Semi-related: see kubernetes-sigs/aws-ebs-csi-driver#1333 (comment)

driver entry there in csinode description even after removing csi hostpath driver plugin

After deploying hostpath csi driver, I could able to see driver detail in csinode details as below

[root@remote-w3 hostpath]# kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
csi-hostpath-attacher-0      1/1     Running   2          109m
csi-hostpath-provisioner-0   1/1     Running   3          133m
csi-hostpath-resizer-0       1/1     Running   3          132m
csi-hostpath-snapshotter-0   1/1     Running   3          132m
csi-hostpath-socat-0         1/1     Running   0          132m
csi-hostpathplugin-0         3/3     Running   0          8m25s
snapshot-controller-0        1/1     Running   0          129m

[root@remote-w3 functional-tests]# kubectl get csinodes remote-w3.ocp.openshift.scale.net -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
  creationTimestamp: "2020-04-01T17:57:32Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"b02bf2ad-d676-4e69-80e8-71bb9a6dfd3d"}:
            .: {}
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:drivers:
          k:{"name":"ibm-spectrum-scale-csi"}:
            .: {}
            f:name: {}
            f:nodeID: {}
            f:topologyKeys: {}
          k:{"name":"spectrumscale.csi.ibm.com"}:
            .: {}
            f:name: {}
            f:nodeID: {}
            f:topologyKeys: {}
    manager: kubelet
    operation: Update
    time: "2020-05-18T20:14:22Z"
  name: remote-w3.ocp.openshift.scale.net
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: remote-w3.ocp.openshift.scale.net
    uid: b02bf2ad-d676-4e69-80e8-71bb9a6dfd3d
  resourceVersion: "9485845"
  selfLink: /apis/storage.k8s.io/v1/csinodes/remote-w3.ocp.openshift.scale.net
  uid: 9ec02850-e6b3-47e9-8f92-33390f659eb9
spec:
  drivers:
  - name: hostpath.csi.k8s.io
    nodeID: remote-w3.ocp.openshift.scale.net
    topologyKeys:
    - topology.hostpath.csi/node

but after removing the hostpath csi driver plugin, below entry in csinode details kept as it is ?

  - name: hostpath.csi.k8s.io
    nodeID: remote-w3.ocp.openshift.scale.net
    topologyKeys:
    - topology.hostpath.csi/node

Is this expected or a defect?

Healthz endpoint returns 404

/kind bug

I tried setting this in a manifest:

          image: <registry>/csi-node-driver-registrar:v2.5.0
          imagePullPolicy: IfNotPresent
          args:
            - "--csi-address=$(CSI_ENDPOINT)"
            - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
            - "--v=5"
            - "--http-endpoint=:9809"
          livenessProbe:
            httpGet:
              path: /healthz
              port: 9809
            initialDelaySeconds: 5
            timeoutSeconds: 5

The logs show:

I0310 07:28:29.658403    5752 node_register.go:95] Starting HTTP server at endpoint: :9809

Once started, the livenessProbe fails with 404, I did a port-forward to debug it locally

k -n kube-system port-forward pods/azuredisk-csi-driver-node-windows-rwwjd 9809:9809
Forwarding from 127.0.0.1:9809 -> 9809

# other terminal
curl localhost:9809/healthz
404 page not found

The path is set at this location

node-driver-registrar/cmd/csi-node-driver-registrar/node_register.go

Line 99 in 86cc0bb

mux.HandleFunc("/healthz", func(w http.ResponseWriter, req *http.Request) {

, what's strange is that the pprof paths work e.g. if I also set --enable-pprof I can do:

curl localhost:9809/debug/pprof/
<html>
<head>
<title>/debug/pprof/</title>
...

remove reg.sock on shutdown

When driver-registrar terminates, it should ensure that it removes the *-reg.socket file that it created. It will get overwritten when restarting, but when experimenting with different drivers or testing with driver renaming, those files accumulate under /var/lib/kubelet/plugins/ and then plugin registration in kubelet gets slower (several minutes!). That might be something worth fixing in kubelet (not sure?), but either way it is better to clean up.

Not much can be done when the registrar gets killed with SIGKILL or crashes, but hopefully that'll be rare.

This is something I've seen with driver-registrar 0.4.1, but probably still applies.

FeatureRequest: Add support for configuring the registration socket path

As per the current code we expect /var/lib/kubelet/plugins_registry/ to be mounted as /registration. This works well when node-driver-registrar runs as sidecar container or as a pod. We have a usecase to run node-driver-registrar as systemd process to support csi driver for specific set of worker nodes (aws/containers-roadmap#826). Therefore, instead of hardcoding /registartion would be helpful if we can read this value from a parameter. If the new parameter is empty then it can default to /registration

Would like to get community feedback on whether this is acceptable or is the node-driver-registrar designed to run only as side-car container?. if we decide to add new parameter then naming might be a difficult part I think, given that we already have parameter named kubelet-registrataion-path which points to csi socket path.

Support ARM Images?

I recently opened a request with kubernetes-csi/csi-driver-smb to support ARM. They did and pushed the image. Now this project needs the same.
Here are the related issues and PRs:

• kubernetes-csi/csi-driver-smb#138 (comment)
• kubernetes-csi/csi-driver-smb#144

My end goal is to use the chart within smb-csi repo which also uses this projects image.

Clean up `Gopkg.toml` dependencies

With cmd/driver-registrar/k8s_register.go deleted in #1, we might be able to update Gopkg.toml to remove some dependencies

/assign @lpabon

runnung on kubernetes 1.12.x

Hi,
is there a way to use this on kubernetes 1.12.x?

I use Rancher and the stable version supports only k8s 1.12.x versions.
I tried to setup GCS, everything works, but when i want to assign a PVC to a workload, it throws this issue:

MountVolume.MountDevice failed for volume "pvc-bad5e7b6-2fb1-11e9-aac6-002590927952" : driver name org.gluster.glusterfs not found in the list of registered CSI drivers

Pls give me some tips if its possible to use is on kubernetes 1.12.x

document flag to specify registration path

I'm building a csi driver using windows host process containers and the hard-coded path requires me to make a symlink on the host node itself to /registration which seems pretty hacky. I'd prefer to be able to just point it at the actual plugin directory and move on.

csi-node-driver-registrar consumed ~44% cpu time to read container registry TimeZone information in Windows

I was trying to narrow down a perf issue and noticed that csi.node-driver-registrar.exe consumed 44% cpu time within 359ms. The majority of this time was spent on reading Container registry timezoon information.

Worked with @andyzhangx and he recommeneded tracking this issue here .

I would like to know:

1. if csi.node-driver-registrar.exe will delay the container startup time
2. purpose of readying the timezone information.

Best regards,
Howard Hao

Node driver registrar says "Lost connection" and do nothing

Log:

I0323 21:20:55.915743       1 main.go:110] Version: unknown
I0323 21:20:55.915816       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0323 21:20:55.915833       1 connection.go:151] Connecting to unix:///csi/csi.sock
I0323 21:20:55.916332       1 main.go:127] Calling CSI driver to discover driver name
I0323 21:20:55.916348       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I0323 21:20:55.916354       1 connection.go:181] GRPC request: {}
I0323 21:20:55.917907       1 connection.go:183] GRPC response: {"name":"x","vendor_version":"0.3.0"}
I0323 21:20:55.918413       1 connection.go:184] GRPC error: <nil>
I0323 21:20:55.918421       1 main.go:137] CSI driver name: "x"
I0323 21:20:55.918477       1 node_register.go:58] Starting Registration Server at: /registration/x-reg.sock
I0323 21:20:55.918603       1 node_register.go:67] Registration Server started at: /registration/x-reg.sock
I0323 21:20:57.567609       1 main.go:77] Received GetInfo call: &InfoRequest{}
I0323 21:20:57.671681       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}
E0323 21:21:15.690492       1 connection.go:129] Lost connection to unix:///csi/csi.sock.

After restart container, it works.

v2.6.1 is affected by CVE-2022-32149

The latest release is affected by vulnerability CVE-2022-32149 which is marked as high severity. The issue is being addressed in other parts of Kubernetes as well, see: https://github.com/kubernetes/kubernetes/pull/112989/files

My proposed solution is to add the following directive to go.mod:

replace golang.org/x/text => golang.org/x/text v0.4.0

I'd be happy to submit a PR when the solution is agreed upon.

If the kubelet working directory in the cluster is different，how to use node-driver-registrar

in my kubernetes cluster
node1 kubelet woring dir is /var/lib/kubelet
node2 kubelet woring dir is /data/kubelet
how do i use node-driver-registrar

README example invalid

There a couple problems here:

      containers:
        - name: csi-driver-registrar
          image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
          args:
            - "--csi-address=/csi/csi.sock"
            - "--kubelet-registration-path=/var/lib/kubelet/plugins/<drivername.example.com>/csi.sock"
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "rm -rf /registration/<plugin> /registration/<drivername.example.com>-reg.sock"]
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
      volumes:
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/<drivername.example.com>/
            type: DirectoryOrCreate

The preStop command doesn't work with the latest images as apparently sh is no longer in the image.

Also the plugin removal from what i can tell should be /csi instead of /registration.