What happened:
After the gVisor was enabled for a ubuntu worker pool, the nodes transit to NotReady
state and never become healthy again.
Node events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 3m37s kubelet Starting kubelet.
Warning InvalidDiskCapacity 3m37s kubelet invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 3m37s (x2 over 3m37s) kubelet Node ip-10-222-9-112.eu-west-1.compute.internal status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3m37s (x2 over 3m37s) kubelet Node ip-10-222-9-112.eu-west-1.compute.internal status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3m37s (x2 over 3m37s) kubelet Node ip-10-222-9-112.eu-west-1.compute.internal status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 3m37s kubelet Updated Node Allocatable limit across pods
Normal Starting 3m25s kube-proxy Starting kube-proxy.
Normal NodeReady 3m17s kubelet Node ip-10-222-9-112.eu-west-1.compute.internal status is now: NodeReady
Normal Starting 3m15s kube-proxy Starting kube-proxy.
Warning DockerStart 3m1s (x3 over 3m2s) systemd-monitor Starting Docker Application Container Engine...
Normal NodeNotReady 2m24s kubelet Node ip-10-222-9-112.eu-west-1.compute.internal status is now: NodeNotReady
Warning ContainerdStart 86s (x22 over 3m2s) systemd-monitor Starting containerd container runtime...
Warning ContainerGCFailed 37s (x3 over 2m37s) kubelet rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: no such file or directory"
Node conditions
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FrequentUnregisterNetDevice False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 NoFrequentUnregisterNetDevice node is functioning properly
FrequentKubeletRestart False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 NoFrequentKubeletRestart kubelet is functioning properly
FrequentDockerRestart False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 NoFrequentDockerRestart docker is functioning properly
FrequentContainerdRestart False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 NoFrequentContainerdRestart containerd is functioning properly
KernelDeadlock False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 FilesystemIsNotReadOnly Filesystem is not read-only
CorruptDockerOverlay2 False Wed, 04 Aug 2021 13:30:23 +0300 Wed, 04 Aug 2021 13:30:22 +0300 NoCorruptDockerOverlay2 docker overlay2 is functioning properly
NetworkUnavailable False Wed, 04 Aug 2021 13:30:15 +0300 Wed, 04 Aug 2021 13:30:15 +0300 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 04 Aug 2021 13:33:13 +0300 Wed, 04 Aug 2021 13:29:47 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 04 Aug 2021 13:33:13 +0300 Wed, 04 Aug 2021 13:29:47 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 04 Aug 2021 13:33:13 +0300 Wed, 04 Aug 2021 13:29:47 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 04 Aug 2021 13:33:13 +0300 Wed, 04 Aug 2021 13:31:00 +0300 KubeletNotReady container runtime is down
SSH-ing on the node and find out that the containerd is failing with:
Aug 04 10:31:36 ip-10-222-9-112.eu-west-1.compute.internal containerd[30781]: containerd: failed to load TOML from /etc/containerd/config.toml: invalid plugin key URI "cri" expect io.containerd.x.vx
Aug 04 10:31:36 ip-10-222-9-112.eu-west-1.compute.internal systemd[1]: containerd.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 10:31:36 ip-10-222-9-112.eu-west-1.compute.internal systemd[1]: containerd.service: Failed with result 'exit-code'.
Aug 04 10:31:36 ip-10-222-9-112.eu-west-1.compute.internal systemd[1]: Failed to start containerd container runtime.
It looks like this extension has injected this configuration
216a217,218
> [plugins.cri.containerd.runtimes.runsc]
> runtime_type = "io.containerd.runsc.v1"
and it is most probably the reason it cannot start again.
Exchanging the above configuration with the he one recommended in the documentation, ref
216a217,218
> [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
> runtime_type = "io.containerd.runsc.v1"
fixes the problem and the node is back in healthy state.
What you expected to happen:
The gVisor specific configuration to not break containerd service on the nodes.
How to reproduce it (as minimally and precisely as possible):
Create a shoot with ubuntu nodes and gVisor enabled
spoiler for Shoot manifest:
kind: Shoot
apiVersion: core.gardener.cloud/v1beta1
metadata:
name: foo
spec:
cloudProfileName: aws
kubernetes:
allowPrivilegedContainers: true
kubeAPIServer:
enableBasicAuthentication: false
version: 1.18.16
networking:
type: calico
pods: 10.223.128.0/17
nodes: 10.222.0.0/16
services: 10.223.0.0/17
maintenance:
autoUpdate:
kubernetesVersion: false
machineImageVersion: false
timeWindow:
begin: 130000+0000
end: 140000+0000
provider:
type: aws
controlPlaneConfig:
apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
kind: ControlPlaneConfig
infrastructureConfig:
apiVersion: aws.provider.extensions.gardener.cloud/v1alpha1
kind: InfrastructureConfig
networks:
vpc:
cidr: 10.222.0.0/16
zones:
- internal: 10.222.112.0/22
name: eu-west-1a
public: 10.222.96.0/22
workers: 10.222.0.0/19
workers:
- cri:
name: containerd
containerRuntimes:
- type: gvisor
name: pool-1
machine:
type: c5.xlarge
image:
name: ubuntu
version: 18.4.20210415
maximum: 1
minimum: 1
maxSurge: 3
maxUnavailable: 3
volume:
type: gp2
size: 40Gi
encrypted: true
zones:
- eu-west-1a
region: eu-west-1
Anything else we need to know?:
The above mitigation is not tested on other operation systems where the current configuration is still working, so before changing it, it should be validated with them, too.
Environment:
- Gardener version (if relevant): v1.28.1
- Extension version: v0.1.0
- Kubernetes version (use
kubectl version
): v1.18.16
- Cloud provider or hardware configuration:
- Others:
- Containerd Version: containerd github.com/containerd/containerd 1.5.2-0ubuntu1~18.04.2
- Ubuntu version: 18.4.20210415/
ami-0943382e114f188e8