The openebs-localpv-provisioner pod in my OpenEBS deployment goes into a crash loop as soon as a PVC is created. Changing to the 2.10.1 release instantly resolves.
No crashing, PVC allocated.
helm upgrade --install openebs openebs/openebs --namespace openebs --create-namespace --set localprovisioner.enabled=false --set ndm.enabled=false --set ndmOperator.enabled=false --set openebs-ndm.enabled=true --set localpv-provisioner.enabled=true
Set openebs-device (or hostpath, both exhibit issue) StorageClass to default.
Deploy my app, which creates a PVC using default StorageClass.
Observe PVC not provisioned and openebs-locapv-provisioner goes into crash loop.
**$ kubectl get pods -n openebs --show-labels**
NAME READY STATUS RESTARTS AGE LABELS
openebs-admission-server-84bd769954-9lxj4 1/1 Running 0 96m app=admission-webhook,name=admission-webhook,openebs.io/component-name=admission-webhook,openebs.io/version=2.11.0,pod-template-hash=84bd769954,release=openebs
openebs-apiserver-6c5dbd554f-2tsqh 1/1 Running 0 96m app=openebs,component=apiserver,name=maya-apiserver,openebs.io/component-name=maya-apiserver,openebs.io/version=2.11.0,pod-template-hash=6c5dbd554f,release=openebs
openebs-localpv-provisioner-694b9f587d-8rb6d 1/1 Running 0 19s app=localpv-provisioner,chart=localpv-provisioner-2.11.0,component=localpv-provisioner,heritage=Helm,name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=2.11.0,pod-template-hash=694b9f587d,release=openebs
openebs-ndm-jmthc 1/1 Running 0 72m app=openebs-ndm,chart=openebs-ndm-1.6.0,component=ndm,controller-revision-hash=db55c95cd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.6.0,pod-template-generation=1,release=openebs
openebs-ndm-m7zt2 1/1 Running 0 72m app=openebs-ndm,chart=openebs-ndm-1.6.0,component=ndm,controller-revision-hash=db55c95cd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.6.0,pod-template-generation=1,release=openebs
openebs-ndm-operator-c5874fbc5-dn4hr 1/1 Running 0 57m app=openebs-ndm-operator,chart=openebs-ndm-1.6.0,component=openebs-ndm-operator,heritage=Helm,name=openebs-ndm-operator,openebs.io/component-name=openebs-ndm-operator,openebs.io/version=1.6.0,pod-template-hash=c5874fbc5,release=openebs
openebs-ndm-r2kvt 1/1 Running 0 72m app=openebs-ndm,chart=openebs-ndm-1.6.0,component=ndm,controller-revision-hash=db55c95cd,heritage=Helm,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=1.6.0,pod-template-generation=1,release=openebs
openebs-provisioner-7cb669f466-fd8ds 1/1 Running 0 57m app=openebs,component=provisioner,name=openebs-provisioner,openebs.io/component-name=openebs-provisioner,openebs.io/version=2.11.0,pod-template-hash=7cb669f466,release=openebs
openebs-snapshot-operator-669696cd5d-4qwz2 2/2 Running 0 96m app=openebs,component=snapshot-operator,name=openebs-snapshot-operator,openebs.io/component-name=openebs-snapshot-operator,openebs.io/version=2.11.0,pod-template-hash=669696cd5d,release=openebs
I0716 21:17:02.178445 1 start.go:69] Starting Provisioner...
I0716 21:17:02.249390 1 start.go:134] Leader election enabled for localpv-provisioner via leaderElectionKey
I0716 21:17:02.250156 1 leaderelection.go:242] attempting to acquire leader lease openebs/openebs.io-local...
I0716 21:17:19.720306 1 leaderelection.go:252] successfully acquired lease openebs/openebs.io-local
I0716 21:17:19.721003 1 controller.go:780] Starting provisioner controller openebs.io/local_openebs-localpv-provisioner-694b9f587d-8rb6d_199a945d-0218-47cd-9c06-346aadd4948f!
I0716 21:17:19.722001 1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"openebs", Name:"openebs.io-local", UID:"1d6d6fcc-d321-4cf4-ae18-a441b79c3aee", APIVersion:"v1", ResourceVersion:"43777276", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-localpv-provisioner-694b9f587d-8rb6d_199a945d-0218-47cd-9c06-346aadd4948f became leader
I0716 21:17:19.821714 1 controller.go:1323] delete "pvc-14ec2240-727d-47a2-a398-7623f1506153": started
I0716 21:17:19.821793 1 controller.go:1323] delete "pvc-b1acec8b-efd4-482f-8628-867573838711": started
I0716 21:17:19.821714 1 controller.go:1323] delete "pvc-4620bdd6-21f0-48a1-911c-33f46bad8dd9": started
I0716 21:17:19.822120 1 controller.go:829] Started provisioner controller openebs.io/local_openebs-localpv-provisioner-694b9f587d-8rb6d_199a945d-0218-47cd-9c06-346aadd4948f!
I0716 21:17:19.822208 1 controller.go:1323] delete "pvc-1273f3ef-b069-476c-89a3-73e2d8ce7524": started
E0716 21:17:19.883750 1 runtime.go:78] Observed a panic: runtime.boundsError{x:7, y:0, signed:true, code:0x1} (runtime error: slice bounds out of range [:7] with length 0)
goroutine 139 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1667180, 0xc00025f100)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1667180, 0xc00025f100)
/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/openebs/maya/pkg/version.GetVersionDetails(0x17522d8, 0x14)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/version/version.go:166 +0xe0
github.com/openebs/maya/pkg/usage.(*versionSet).fetchAndSetVersion(0xc0002eb6a0, 0x16, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/versionset.go:75 +0x1c1
github.com/openebs/maya/pkg/usage.(*versionSet).getVersion(0xc0002eb6a0, 0xba4b00, 0xc00004401c, 0xc000066700)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/versionset.go:85 +0x5b
github.com/openebs/maya/pkg/usage.(*Usage).Build(0xc0000a01c0, 0xc0000a01c0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/usage.go:197 +0x7d
github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app.sendEventOrIgnore(0xc000400c70, 0x10, 0xc000042ed0, 0x28, 0xc000400c00, 0x4, 0xc000400930, 0xe, 0x1750789, 0x12)
/go/src/github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app/provisioner.go:222 +0xc4
github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app.(*Provisioner).Delete(0xc000194550, 0xc0002384d0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app/provisioner.go:180 +0x1dd
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).deleteVolumeOperation(0xc00003d440, 0xc0002384d0, 0x252ee00, 0x1)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1338 +0x25d
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).syncVolume(0xc00003d440, 0x17251c0, 0xc0002384d0, 0x17251c0, 0xc0002384d0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1055 +0xaa
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).syncVolumeHandler(0xc00003d440, 0xc000042ed0, 0x28, 0x4142e7, 0xc0002fe148)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1002 +0x96
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).processNextVolumeWorkItem.func1(0xc00003d440, 0x14eafc0, 0xc0004d22e0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:944 +0xe0
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).processNextVolumeWorkItem(0xc00003d440, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:961 +0x53
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).runVolumeWorker(0xc00003d440)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:874 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0000e8ea0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0000e8ea0, 0x3b9aca00, 0x0, 0x1, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0000e8ea0, 0x3b9aca00, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run.func1
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:826 +0x37d
panic: runtime error: slice bounds out of range [:7] with length 0 [recovered]
panic: runtime error: slice bounds out of range [:7] with length 0
goroutine 139 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1667180, 0xc00025f100)
/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/openebs/maya/pkg/version.GetVersionDetails(0x17522d8, 0x14)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/version/version.go:166 +0xe0
github.com/openebs/maya/pkg/usage.(*versionSet).fetchAndSetVersion(0xc0002eb6a0, 0x16, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/versionset.go:75 +0x1c1
github.com/openebs/maya/pkg/usage.(*versionSet).getVersion(0xc0002eb6a0, 0xba4b00, 0xc00004401c, 0xc000066700)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/versionset.go:85 +0x5b
github.com/openebs/maya/pkg/usage.(*Usage).Build(0xc0000a01c0, 0xc0000a01c0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/github.com/openebs/maya/pkg/usage/usage.go:197 +0x7d
github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app.sendEventOrIgnore(0xc000400c70, 0x10, 0xc000042ed0, 0x28, 0xc000400c00, 0x4, 0xc000400930, 0xe, 0x1750789, 0x12)
/go/src/github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app/provisioner.go:222 +0xc4
github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app.(*Provisioner).Delete(0xc000194550, 0xc0002384d0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/cmd/provisioner-localpv/app/provisioner.go:180 +0x1dd
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).deleteVolumeOperation(0xc00003d440, 0xc0002384d0, 0x252ee00, 0x1)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1338 +0x25d
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).syncVolume(0xc00003d440, 0x17251c0, 0xc0002384d0, 0x17251c0, 0xc0002384d0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1055 +0xaa
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).syncVolumeHandler(0xc00003d440, 0xc000042ed0, 0x28, 0x4142e7, 0xc0002fe148)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:1002 +0x96
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).processNextVolumeWorkItem.func1(0xc00003d440, 0x14eafc0, 0xc0004d22e0, 0x0, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:944 +0xe0
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).processNextVolumeWorkItem(0xc00003d440, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:961 +0x53
sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).runVolumeWorker(0xc00003d440)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:874 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0000e8ea0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0000e8ea0, 0x3b9aca00, 0x0, 0x1, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0000e8ea0, 0x3b9aca00, 0x0)
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/sig-storage-lib-external-provisioner/controller.(*ProvisionController).Run.func1
/go/src/github.com/openebs/dynamic-localpv-provisioner/vendor/sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:826 +0x37d