Comments (2)
Sorry I failed to reproduce this issue in my environment. Could you please provide mode details about reproduce steps? Is the pod scheduled to other nodes? and if there any residual data in edge database?
from kubeedge.
In I have panic as well on EKS based environment. Here is logs:
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: I0313 13:53:55.160695 77585 reconciler_common.go:172] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5v7g\" (UniqueName: \"kubernetes.io/projected/27e48280-1e30-43a7-b753-015f0e043dae-kube-api-access-s5v7g\") pod \"27e48280-1e30-43a7-b753-015f0e043dae\" (UID: \"27e48280-1e30-43a7-b753-015f0e043dae\") "
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: W0313 13:53:55.161225 77585 empty_dir.go:499] Warning: Unmount skipped because path does not exist: /var/lib/edged/pods/27e48280-1e30-43a7-b753-015f0e043dae/volumes/kubernetes.io~projected/kube-api-access-s5v7g
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: E0313 13:53:55.161848 77585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e48280-1e30-43a7-b753-015f0e043dae-kube-api-access-s5v7g podName:27e48280-1e30-43a7-b753-015f0e043dae nodeName:}" failed. No retries permitted until 2024-03-13 13:55:57.161749733 +0000 UTC m=+810.848497032 (durationBeforeRetry 2m2s). Error: recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil>) Call stack:
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: goroutine 22510 [running]:
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0x4001398e68)
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:162 +0x60
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: panic({0x26fe040, 0x4a923b0})
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: runtime/panic.go:890 +0x248
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/kubelet/token.(*Manager).DeleteServiceAccountToken(0x4000a0a500, {0x4001122db0, 0x24})
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/kubelet/token/token_manager.go:140 +0xb8
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/projected.(*projectedVolumeUnmounter).TearDownAt(0x40016f06d8, {0x4001665880, 0x6e})
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/projected/projected.go:382 +0x258
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/projected.(*projectedVolumeUnmounter).TearDown(0x40016f06d8)
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/projected/projected.go:368 +0x54
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/operationexecutor.(*operationGenerator).GenerateUnmountVolumeFunc.func1()
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:859 +0x11c
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/types.(*GeneratedOperations).Run(0x4000554600)
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/types/types.go:79 +0x120
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run.func1()
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:190 +0xbc
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: created by github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.(*nestedPendingOperations).Run
Mar 13 13:53:55 raspberrypi4-64 edgecore[77585]: github.com/kubeedge/kubeedge/vendor/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:185 +0x68c
In my environment, the image can't be downloaded, and it just crashes.
Also, I have crio as CRI provider.
from kubeedge.
Related Issues (20)
- edgecore.yaml改为containerd作为运行时,发现pause镜像拉不到 HOT 1
- Enhancement for container runtime e2e ci test HOT 2
- After enabling HTTPS in metaserver, the edge node status is NotReady
- about keadm cloudcore version HOT 2
- An error occurs when a NodeGroup with the same name is configured in a EdgeApplication HOT 1
- Enhancement for TLS certificate relted code HOT 1
- High avaibility at the edge HOT 1
- keadm join fail while joining to Ali container service HOT 1
- Add FeatureGate support for keadm join command
- admission deployment run error HOT 2
- About multiple cloud-core cluster deploy HOT 2
- Unit test failed because of the token of CodeCov
- Context deadline exceeded when execute keadm init HOT 4
- 1.16.1 join err HOT 14
- image prepull status error
- Add device status reporting information HOT 1
- kubectl exec/log timeout HOT 3
- unexpected return pc for runtime.doInit called from HOT 8
- How do KubeEdge's edge nodes find nearby relay node? HOT 1
- Enhancements to Keadm Tool
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubeedge.