Comments (12)
andyzhangx
commented on August 16, 2024
could you follow this guide to get the driver pod description and logs? @rolfbartels
https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/csi-debug.md#csi-driver-debug-tips
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
@andyzhangx
I tried attaching the files, but i had to add it as a code block. thanks
csi-azurefile-controller
I0217 05:10:47.179821 1 feature_gate.go:249] feature gates: &{map[HonorPVReclaimPolicy:true]}
I0217 05:10:47.179956 1 csi-provisioner.go:154] Version: v3.5.0-0-gab68435fa
I0217 05:10:47.179968 1 csi-provisioner.go:177] Building kube configs for running in cluster...
I0217 05:10:48.185918 1 common.go:111] Probing CSI driver for readiness
I0217 05:10:48.193153 1 csi-provisioner.go:230] Detected CSI driver file.csi.azure.com
I0217 05:10:48.193174 1 csi-provisioner.go:240] Supports migration from in-tree plugin: kubernetes.io/azure-file
I0217 05:10:48.193769 1 common.go:111] Probing CSI driver for readiness
I0217 05:10:48.196813 1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0217 05:10:48.197247 1 controller.go:732] Using saving PVs to API server in background
I0217 05:10:48.197530 1 leaderelection.go:245] attempting to acquire leader lease kube-system/file-csi-azure-com...
I0217 05:11:06.094553 1 leaderelection.go:255] successfully acquired lease kube-system/file-csi-azure-com
I0217 05:11:06.094709 1 leader_election.go:178] became leader, starting
I0217 05:11:06.195427 1 controller.go:811] Starting provisioner controller file.csi.azure.com_tkg-02-pmv6r-4sxpp_90b9c043-5a31-415f-b54d-c73922f395f6!
I0217 05:11:06.195488 1 volume_store.go:97] Starting save volume queue
I0217 05:11:06.195558 1 clone_controller.go:66] Starting CloningProtection controller
I0217 05:11:06.195586 1 clone_controller.go:82] Started CloningProtection controller
I0217 05:11:06.296544 1 controller.go:1502] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": started
I0217 05:11:06.296750 1 controller.go:860] Started provisioner controller file.csi.azure.com_tkg-02-pmv6r-4sxpp_90b9c043-5a31-415f-b54d-c73922f395f6!
E0217 05:11:06.448668 1 controller.go:1512] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": volume deletion failed: rpc error: code = Unavailable desc = error reading from server: EOF
W0217 05:11:06.448937 1 controller.go:989] Retrying syncing volume "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1", failure 0
E0217 05:11:06.448957 1 controller.go:1007] error syncing volume "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": rpc error: code = Unavailable desc = error reading from server: EOF
I0217 05:11:06.448984 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1", UID:"4d8dd4e0-1e2b-4788-9156-2b15ee391d86", APIVersion:"v1", ResourceVersion:"106729", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Unavailable desc = error reading from server: EOF
I0217 05:11:07.449758 1 controller.go:1502] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": started
E0217 05:11:07.452487 1 connection.go:142] Lost connection to unix:///csi/csi.sock.
F0217 05:11:07.452735 1 connection.go:97] Lost connection to CSI driver, exiting
csi-azurefile-controller-description
Name: csi-azurefile-controller-5b4d749497-ljnqh
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: csi-azurefile-controller-sa
Node: tkg-02-pmv6r-4sxpp/10.242.232.174
Start Time: Sat, 17 Feb 2024 05:04:58 +0000
Labels: app=csi-azurefile-controller
app.kubernetes.io/component=csi-driver
app.kubernetes.io/instance=azurefile-csi-driver
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=azurefile-csi-driver
app.kubernetes.io/part-of=azurefile-csi-driver
app.kubernetes.io/version=v1.29.2
helm.sh/chart=azurefile-csi-driver-v1.29.2
pod-template-hash=5b4d749497
Annotations: <none>
Status: Running
IP: 10.242.232.174
IPs:
IP: 10.242.232.174
Controlled By: ReplicaSet/csi-azurefile-controller-5b4d749497
Containers:
csi-provisioner:
Container ID: containerd://117f596446eac2312a702dc75074658b5d6ea57f733cae535965967181ea32dc
Image: mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.5.0
Image ID: mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:fdf70099aa1538d1c2164976cf6d158ef8b3a5ee63db10bf0085de4ec66f59b4
Port: <none>
Host Port: <none>
Args:
-v=2
--csi-address=$(ADDRESS)
--leader-election
--leader-election-namespace=kube-system
--timeout=1200s
--extra-create-metadata=true
--kube-api-qps=50
--kube-api-burst=100
--feature-gates=HonorPVReclaimPolicy=true
State: Terminated
Reason: Error
Message: Lost connection to CSI driver, exiting
Exit Code: 1
Started: Sat, 17 Feb 2024 05:07:42 +0000
Finished: Sat, 17 Feb 2024 05:08:06 +0000
Last State: Terminated
Reason: Error
Message: Lost connection to CSI driver, exiting
Exit Code: 1
Started: Sat, 17 Feb 2024 05:06:58 +0000
Finished: Sat, 17 Feb 2024 05:07:19 +0000
Ready: False
Restart Count: 3
Limits:
cpu: 1
memory: 500Mi
Requests:
cpu: 10m
memory: 20Mi
Environment:
ADDRESS: /csi/csi.sock
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
csi-attacher:
Container ID: containerd://7d4bd6d0f94a76ddd6bc4abaeb853297c1ab966ed1bc235be37405ecab06ef49
Image: mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v4.3.0
Image ID: mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:4306b80bfe8caea3fe53f6d1c15807c745be3072553ff508fc4f61da8f4a0c10
Port: <none>
Host Port: <none>
Args:
-v=2
-csi-address=$(ADDRESS)
-timeout=120s
-leader-election
--leader-election-namespace=kube-system
--kube-api-qps=50
--kube-api-burst=100
State: Running
Started: Sat, 17 Feb 2024 05:05:04 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500Mi
Requests:
cpu: 10m
memory: 20Mi
Environment:
ADDRESS: /csi/csi.sock
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
csi-snapshotter:
Container ID: containerd://0b80f814b55d093ca5c3d427bc94815a7d16cb773cd0b2acdfbf001e5865cb82
Image: mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v6.3.1
Image ID: mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:87bba314ce1b7ad22b673b7d402e9cb2dab88d0ab61c0ba17885db30d4d715d8
Port: <none>
Host Port: <none>
Args:
-csi-address=$(ADDRESS)
-leader-election
--leader-election-namespace=kube-system
-v=2
State: Running
Started: Sat, 17 Feb 2024 05:05:11 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 200Mi
Requests:
cpu: 10m
memory: 20Mi
Environment:
ADDRESS: /csi/csi.sock
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
csi-resizer:
Container ID: containerd://b86b7b9c2e4eac0013371b8b3b970d41ef5fbec9da3f2965224f7a9829722eba
Image: mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.8.0
Image ID: mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:6f0e8c9f3d0bdcf7a5fb5e404276ffac624033099d7687c8080692bcb6d13cd1
Port: <none>
Host Port: <none>
Args:
-csi-address=$(ADDRESS)
-v=2
-leader-election
--leader-election-namespace=kube-system
-handle-volume-inuse-error=false
-timeout=120s
-feature-gates=RecoverVolumeExpansionFailure=true
State: Running
Started: Sat, 17 Feb 2024 05:05:13 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500Mi
Requests:
cpu: 10m
memory: 20Mi
Environment:
ADDRESS: /csi/csi.sock
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
liveness-probe:
Container ID: containerd://bbf379e60cca0f83ec5898339bc873a64f42c7b529537090481660ab2961acc8
Image: mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0
Image ID: mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:3aeac313cffdb7db80b733539427f2533a3f662bf538e7b6434b0f898ceb701b
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
--probe-timeout=3s
--health-port=29612
--v=2
State: Running
Started: Sat, 17 Feb 2024 05:05:13 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 100Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
azurefile:
Container ID: containerd://48bd1246ac0d0a68cc35ac8ed0d74eb357ff9c36f02dd6317924cf2847a65264
Image: mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.29.2
Image ID: mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:e1953f6a13e6bfff62725ad03c276edf587853c67b21830434ee449a4bfdca6b
Ports: 29612/TCP, 29614/TCP
Host Ports: 29612/TCP, 29614/TCP
Args:
--v=5
--endpoint=$(CSI_ENDPOINT)
--metrics-address=0.0.0.0:29614
--kubeconfig=
--drivername=file.csi.azure.com
--cloud-config-secret-name=azure-cloud-provider
--cloud-config-secret-namespace=kube-system
--custom-user-agent=
--user-agent-suffix=OSS-helm
--allow-empty-cloud-config=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Sat, 17 Feb 2024 05:07:42 +0000
Finished: Sat, 17 Feb 2024 05:08:05 +0000
Ready: False
Restart Count: 3
Limits:
cpu: 1
memory: 200Mi
Requests:
cpu: 10m
memory: 20Mi
Liveness: http-get http://:healthz/healthz delay=30s timeout=10s period=30s #success=1 #failure=5
Environment:
AZURE_CREDENTIAL_FILE: <set to the key 'path' of config map 'azure-cred-file'> Optional: true
CSI_ENDPOINT: unix:///csi/csi.sock
AZURE_GO_SDK_LOG_LEVEL:
Mounts:
/csi from socket-dir (rw)
/etc/kubernetes/ from azure-cred (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jkv8x (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
socket-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
azure-cred:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/
HostPathType: DirectoryOrCreate
kube-api-access-jkv8x:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
Tolerations: node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/controlplane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m15s default-scheduler Successfully assigned kube-system/csi-azurefile-controller-5b4d749497-ljnqh to tkg-02-pmv6r-4sxpp
Normal Pulling 3m14s kubelet Pulling image "mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.5.0"
Normal Pulled 3m12s kubelet Successfully pulled image "mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.5.0" in 1.73196482s (2.162500249s including waiting)
Normal Created 3m12s kubelet Created container csi-provisioner
Normal Started 3m12s kubelet Started container csi-provisioner
Normal Pulling 3m12s kubelet Pulling image "mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v4.3.0"
Normal Pulled 3m10s kubelet Successfully pulled image "mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v4.3.0" in 1.8015273s (2.368584541s including waiting)
Normal Created 3m10s kubelet Created container csi-attacher
Normal Started 3m9s kubelet Started container csi-attacher
Normal Pulling 3m9s kubelet Pulling image "mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v6.3.1"
Normal Pulled 3m2s kubelet Successfully pulled image "mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v6.3.1" in 1.681223406s (7.192279186s including waiting)
Normal Created 3m2s kubelet Created container csi-snapshotter
Normal Started 3m2s kubelet Started container csi-snapshotter
Normal Pulling 3m2s kubelet Pulling image "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.8.0"
Normal Pulled 3m kubelet Successfully pulled image "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.8.0" in 1.543111114s (1.543121604s including waiting)
Normal Created 3m kubelet Created container csi-resizer
Normal Started 3m kubelet Started container csi-resizer
Normal Pulled 3m kubelet Container image "mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.10.0" already present on machine
Normal Created 3m kubelet Created container liveness-probe
Normal Started 3m kubelet Started container liveness-probe
Normal Pulled 107s (x2 over 3m) kubelet Container image "mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.29.2" already present on machine
Normal Created 107s (x2 over 3m) kubelet Created container azurefile
Normal Started 107s (x2 over 3m) kubelet Started container azurefile
from azurefile-csi-driver.
andyzhangx
commented on August 16, 2024
so azurefile container is crashing, what's the azurefile container logs? just run
kubectl logs csi-azurefile-controller-xxxx -c azurefile -n kube-system -p > csi-azurefile-controller.log
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
Here
tanzu@tanzu:~$ kl -n kube-system csi-azurefile-controller-5b4d749497-ljnqh -p > csi-azure-controller.log
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe, azurefile
tanzu@tanzu:~$ cat csi-azure-controller.log
I0217 05:30:15.209165 1 feature_gate.go:249] feature gates: &{map[HonorPVReclaimPolicy:true]}
I0217 05:30:15.209268 1 csi-provisioner.go:154] Version: v3.5.0-0-gab68435fa
I0217 05:30:15.209274 1 csi-provisioner.go:177] Building kube configs for running in cluster...
I0217 05:30:15.210625 1 common.go:111] Probing CSI driver for readiness
I0217 05:30:15.215818 1 csi-provisioner.go:230] Detected CSI driver file.csi.azure.com
I0217 05:30:15.215842 1 csi-provisioner.go:240] Supports migration from in-tree plugin: kubernetes.io/azure-file
I0217 05:30:15.217577 1 common.go:111] Probing CSI driver for readiness
I0217 05:30:15.220093 1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0217 05:30:15.220606 1 controller.go:732] Using saving PVs to API server in background
I0217 05:30:15.220875 1 leaderelection.go:245] attempting to acquire leader lease kube-system/file-csi-azure-com...
I0217 05:30:30.784018 1 leaderelection.go:255] successfully acquired lease kube-system/file-csi-azure-com
I0217 05:30:30.784857 1 leader_election.go:178] became leader, starting
I0217 05:30:30.885706 1 controller.go:811] Starting provisioner controller file.csi.azure.com_tkg-02-pmv6r-4sxpp_a0dbd5cc-cbe7-4d47-bd04-ce8fc89ccd29!
I0217 05:30:30.885765 1 clone_controller.go:66] Starting CloningProtection controller
I0217 05:30:30.885842 1 clone_controller.go:82] Started CloningProtection controller
I0217 05:30:30.885856 1 volume_store.go:97] Starting save volume queue
I0217 05:30:30.988025 1 controller.go:1502] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": started
I0217 05:30:30.989082 1 controller.go:860] Started provisioner controller file.csi.azure.com_tkg-02-pmv6r-4sxpp_a0dbd5cc-cbe7-4d47-bd04-ce8fc89ccd29!
E0217 05:30:31.168699 1 controller.go:1512] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": volume deletion failed: rpc error: code = Unavailable desc = error reading from server: EOF
W0217 05:30:31.168886 1 controller.go:989] Retrying syncing volume "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1", failure 0
E0217 05:30:31.171402 1 controller.go:1007] error syncing volume "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": rpc error: code = Unavailable desc = error reading from server: EOF
I0217 05:30:31.169423 1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1", UID:"4d8dd4e0-1e2b-4788-9156-2b15ee391d86", APIVersion:"v1", ResourceVersion:"106729", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Unavailable desc = error reading from server: EOF
I0217 05:30:32.172598 1 controller.go:1502] delete "pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1": started
E0217 05:30:32.179750 1 connection.go:142] Lost connection to unix:///csi/csi.sock.
F0217 05:30:32.180051 1 connection.go:97] Lost connection to CSI driver, exiting
from azurefile-csi-driver.
andyzhangx
commented on August 16, 2024
pls run
kubectl logs csi-azurefile-controller-5b4d749497-ljnqh -c azurefile -n kube-system -p > csi-azurefile-controller.log
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
Here's the output,
We inadvertently deleted the pod, so waited till the new pod was created, same errors.
W0217 05:41:40.422287 1 main.go:83] nodeid is empty
I0217 05:41:40.422571 1 main.go:140] set up prometheus server on [::]:29614
I0217 05:41:40.423373 1 azurefile.go:361]
DRIVER INFORMATION:
-------------------
Build Date: "2023-11-23T13:24:54Z"
Compiler: gc
Driver Name: file.csi.azure.com
Driver Version: v1.29.2
Git Commit: 4f98c74f750742c8541b7cb266b4f48bdac1d430
Go Version: go1.21.3
Platform: linux/amd64
Streaming logs below:
I0217 05:41:40.423396 1 azurefile.go:364] driver userAgent: file.csi.azure.com/v1.29.2 OSS-helm
I0217 05:41:40.423551 1 azure.go:69] set QPS(25.000000) and QPS Burst(50) for driver kubeClient
I0217 05:41:40.423974 1 azure.go:84] reading cloud config from secret kube-system/azure-cloud-provider
I0217 05:41:40.432191 1 azure.go:91] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0217 05:41:40.432215 1 azure.go:96] could not read cloud config from secret kube-system/azure-cloud-provider
I0217 05:41:40.432224 1 azure.go:106] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
W0217 05:41:40.432245 1 azure.go:111] load azure config from file(/etc/kubernetes/azure.json) failed with open /etc/kubernetes/azure.json: no such file or directory
I0217 05:41:40.432252 1 azure.go:123] no cloud config provided, error: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found, driver will run without cloud config
I0217 05:41:40.432262 1 azure.go:147] starting controller server...
I0217 05:41:40.432269 1 azurefile.go:369] cloud: , location: , rg: , VnetName: , VnetResourceGroup: , SubnetName:
I0217 05:41:40.434666 1 mount_linux.go:285] 'umount /tmp/kubelet-detect-safe-umount2286003081' failed with: exit status 32, output: umount: /tmp/kubelet-detect-safe-umount2286003081: must be superuser to unmount.
I0217 05:41:40.434695 1 mount_linux.go:287] Detected umount with unsafe 'not mounted' behavior
I0217 05:41:40.434762 1 driver.go:93] Enabling controller service capability: CREATE_DELETE_VOLUME
I0217 05:41:40.434775 1 driver.go:93] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0217 05:41:40.434780 1 driver.go:93] Enabling controller service capability: EXPAND_VOLUME
I0217 05:41:40.434784 1 driver.go:93] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I0217 05:41:40.434788 1 driver.go:93] Enabling controller service capability: CLONE_VOLUME
I0217 05:41:40.434793 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I0217 05:41:40.434797 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0217 05:41:40.434801 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I0217 05:41:40.434804 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I0217 05:41:40.434808 1 driver.go:112] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0217 05:41:40.434811 1 driver.go:112] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0217 05:41:40.434815 1 driver.go:112] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0217 05:41:40.434819 1 driver.go:103] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0217 05:41:40.434824 1 driver.go:103] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I0217 05:41:40.434828 1 driver.go:103] Enabling node service capability: VOLUME_MOUNT_GROUP
I0217 05:41:40.434831 1 driver.go:103] Enabling node service capability: GET_VOLUME_STATS
I0217 05:41:40.434959 1 server.go:118] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0217 05:41:41.196779 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I0217 05:41:41.196814 1 utils.go:77] GRPC request: {}
I0217 05:41:41.198429 1 utils.go:83] GRPC response: {"name":"file.csi.azure.com","vendor_version":"v1.29.2"}
I0217 05:41:41.203173 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0217 05:41:41.203201 1 utils.go:77] GRPC request: {}
I0217 05:41:41.203328 1 utils.go:83] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}}]}
I0217 05:41:41.205385 1 utils.go:76] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0217 05:41:41.205400 1 utils.go:77] GRPC request: {}
I0217 05:41:41.205472 1 utils.go:83] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":13}}},{"Type":{"Rpc":{"type":7}}}]}
I0217 05:41:58.873603 1 utils.go:76] GRPC call: /csi.v1.Controller/DeleteVolume
I0217 05:41:58.873692 1 utils.go:77] GRPC request: {"secrets":"***stripped***","volume_id":"##pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1###default"}
I0217 05:41:58.873938 1 azurefile.go:597] got storage account(iedk8sazfs) from secret
W0217 05:41:58.992511 1 azurefile.go:912] DeleteFileShare(pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1) on account() failed with error(storage: service returned error: StatusCode=404, ErrorCode=ShareNotFound, ErrorMessage=The specified share does not exist.
RequestId:0627a69b-601a-0056-0964-61b4c5000000
Time:2024-02-17T05:41:58.9807592Z, RequestInitiated=Sat, 17 Feb 2024 05:41:58 GMT, RequestId=0627a69b-601a-0056-0964-61b4c5000000, API Version=2018-03-28, QueryParameterName=, QueryParameterValue=), return as success
I0217 05:41:58.992573 1 controllerserver.go:719] azure file(pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1) under subsID() rg() account() volume(##pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1###default) is deleted successfully
I0217 05:41:58.992595 1 azurefile.go:1039] remove tag(skip-matching) on account() subsID(), resourceGroup()
I0217 05:41:58.992723 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.118714537 request="azurefile_csi_driver_controller_delete_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-5ad5368b-799c-4f9b-a512-1da13db7b6d1###default" result_code="failed_csi_driver_controller_delete_volume"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x173044a]
goroutine 65 [running]:
sigs.k8s.io/cloud-provider-azure/pkg/provider.(*Cloud).RemoveStorageAccountTag(0xc0000f5800, {0x21c5e90, 0xc0004ff170}, {0x0, 0x0}, {0x0, 0x0}, {0xc00024e041, 0x0}, {0x1ed4bac, ...})
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_storageaccount.go:661 +0x8a
sigs.k8s.io/azurefile-csi-driver/pkg/azurefile.(*Driver).RemoveStorageAccountTag(0xc000574000, {0x21c5e90, 0xc0004ff170}, {0x0, 0x0}, {0x0, 0x0}, {0xc00024e041, 0x0}, {0x1ed4bac, ...})
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/pkg/azurefile/azurefile.go:1041 +0x4dd
sigs.k8s.io/azurefile-csi-driver/pkg/azurefile.(*Driver).DeleteVolume(0xc000574000, {0x21c5e90, 0xc0004ff170}, 0xc000122ac0)
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/pkg/azurefile/controllerserver.go:720 +0xc10
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler.func1({0x21c5e90, 0xc0004ff170}, {0x1d91820?, 0xc000122ac0})
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6438 +0x72
sigs.k8s.io/azurefile-csi-driver/pkg/csi-common.logGRPC({0x21c5e90, 0xc0004ff170}, {0x1d91820?, 0xc000122ac0?}, 0xc00023e340, 0xc0000b6108)
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/pkg/csi-common/utils.go:79 +0x3a9
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler({0x1e85580?, 0xc000574000}, {0x21c5e90, 0xc0004ff170}, 0xc000568800, 0x1fe95f0)
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6440 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005741e0, {0x21c5e90, 0xc0004fe4e0}, {0x21ce6a0, 0xc0004a5380}, 0xc0000bc360, 0xc0000bbf50, 0x30fd818, 0x0)
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/google.golang.org/grpc/server.go:1343 +0xe03
google.golang.org/grpc.(*Server).handleStream(0xc0005741e0, {0x21ce6a0, 0xc0004a5380}, 0xc0000bc360)
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/google.golang.org/grpc/server.go:1737 +0xc4c
google.golang.org/grpc.(*Server).serveStreams.func1.1()
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/google.golang.org/grpc/server.go:986 +0x86
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 47
/mnt/vss/_work/1/go/src/sigs.k8s.io/azurefile-csi-driver/vendor/google.golang.org/grpc/server.go:997 +0x145
from azurefile-csi-driver.
andyzhangx
commented on August 16, 2024
pls follow this guide to set cloud provider config: https://github.com/kubernetes-sigs/azurefile-csi-driver?tab=readme-ov-file#option1-provide-cloud-provider-config-with-azure-credentials
I0217 05:41:40.423974 1 azure.go:84] reading cloud config from secret kube-system/azure-cloud-provider
I0217 05:41:40.432191 1 azure.go:91] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0217 05:41:40.432215 1 azure.go:96] could not read cloud config from secret kube-system/azure-cloud-provider
I0217 05:41:40.432224 1 azure.go:106] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
W0217 05:41:40.432245 1 azure.go:111] load azure config from file(/etc/kubernetes/azure.json) failed with open /etc/kubernetes/azure.json: no such file or directory
I0217 05:41:40.432252 1 azure.go:123] no cloud config provided, error: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found, driver will run without cloud config
I0217 05:41:40.432262 1 azure.go:147] starting controller server...
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
Thanks for the reponse, just want to point out again we are following guide Option 2: which is bring your own storage account.
We are using SAS keys and azure secrets, from my understanding we don't need to create the provider config? Though I will try and add that and feedback.
from azurefile-csi-driver.
andyzhangx
commented on August 16, 2024
yes, that could be a bug, anyway, try workaround now.
from azurefile-csi-driver.
andyzhangx
commented on August 16, 2024
/kind bug
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
Yes and the fact it can create the PVC and Delete the PVC, it's just crashing on the PV Delete, there seems to be an issue.
from azurefile-csi-driver.
rolfbartels
commented on August 16, 2024
@andyzhangx
Thank you for all your help, adding the provider config file seems to work and fix the issue, but I believe this is still a bug as we should be able to use SAS tokens
from azurefile-csi-driver.
Related Issues (20)
- PVC cloning does not work with private endpoints enabled on StorageClass HOT 4
- Connection to storage account with storage account keys disabled doesn't work with workload identity (kerberos auth support) HOT 13
- add feature to disable dns zone creation for private endpoints HOT 5
- [Not working] workload identity support on static provisioning on AKS 1.29 HOT 5
- csi-azurefile-controller pod constantly restarts HOT 4
- Azure file mount failed in AKS having storage account in different subscription HOT 3
- New 1.30 patch release with commits after 2/22/2024 HOT 1
- No helm chart for release v1.30.1
- cifs credentials appear in process table HOT 3
- Frequent controller restarts HOT 6
- remove smb-globalmount when azure file is unmounted on windows node
- PVC fails to be provision HOT 9
- Add update strategy in helm chart
- Move helm chart version to strict SemVer 2 HOT 4
- Allow PV Finalizer to be removed HOT 2
- RemoveSmbGlobalMapping makes Windows node very slow HOT 24
- Invalid parameter "clientID" in storage class HOT 1
- PVC using Bring Your Own share in a StorageClass throws error HOT 1
- PVC+PV using Bring Your Own Share throws warning when changing capacity request + capacity not updated HOT 1
- azcopy can't resume cloning job if controller pod is killed during job run HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azurefile-csi-driver.