mnrgreg / kubectl-node-restart Goto Github PK
View Code? Open in Web Editor NEWKrew plugin to restart Kubernetes Nodes sequentially and gracefully
License: Other
Krew plugin to restart Kubernetes Nodes sequentially and gracefully
License: Other
Support input parameter for uncordon delay
Some nodes take longer then 5 minutes to recover, the current code will never uncordon them.
$ kubectl node-restart -l kubernetes.io/hostname=freyja
Targeting selective nodes:
freyja
Draining node freyja...
Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
...
Kubernetes version: v1.21.6+k3s1
Name: node-restart-f0qrm
Namespace: kube-system
Selector: controller-uid=c5aaea16-1a89-499e-9e2d-62402acfb8e0
Labels: controller-uid=c5aaea16-1a89-499e-9e2d-62402acfb8e0
job-name=node-restart-f0qrm
Annotations: <none>
Parallelism: 1
Completions: 1
Start Time: Sat, 20 Nov 2021 07:47:41 -0600
Pods Statuses: 0 Running / 0 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=c5aaea16-1a89-499e-9e2d-62402acfb8e0
job-name=node-restart-f0qrm
Containers:
node-restart-f0qrm:
Image: alpine:3.9
Port: <none>
Host Port: <none>
Command:
nsenter
--target
1
--mount
--uts
--ipc
--pid
--
bash
-c
Args:
if [ -f /node-restart-flag ]; then rm /node-restart-flag && exit 0; else touch /node-restart-flag && reboot && exit 1; fi
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-c8sv9
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-pjgnz
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-kjx6p
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-864bs
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-dwj75
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-7x2r8
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-9b5hd
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-4r9nf
Normal SuccessfulCreate 11m job-controller Created pod: node-restart-f0qrm-4q5jv
Normal SuccessfulCreate 6m35s (x495 over 11m) job-controller (combined from similar events): Created pod: node-restart-f0qrm-n8md9```
```10359 ± : kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
node-restart-f0qrm-4prrb 0/1 Terminating 0 8m8s
node-restart-f0qrm-4q4kv 0/1 Terminating 0 7m44s
node-restart-f0qrm-4qwsp 0/1 Terminating 0 5m47s
node-restart-f0qrm-4qxn7 0/1 Terminating 0 10m
node-restart-f0qrm-4rmll 0/1 Terminating 0 7m14s
node-restart-f0qrm-524gd 0/1 Terminating 0 6m32s
node-restart-f0qrm-52hm9 0/1 Terminating 0 6m34s
node-restart-f0qrm-54w6k 0/1 Terminating 0 11m
node-restart-f0qrm-55x9d 0/1 Terminating 0 5m46s
node-restart-f0qrm-56v4g 0/1 Terminating 0 8m28s
node-restart-f0qrm-574sv 0/1 Terminating 0 8m18s
node-restart-f0qrm-57r99 0/1 Terminating 0 10m
node-restart-f0qrm-57z6b 0/1 Terminating 0 11m
node-restart-f0qrm-584k2 0/1 Terminating 0 8m31s
node-restart-f0qrm-58qch 0/1 Terminating 0 11m
node-restart-f0qrm-5ddrr 0/1 Terminating 0 6m16s
node-restart-f0qrm-5f52h 0/1 Terminating 0 8m
node-restart-f0qrm-5gj88 0/1 Terminating 0 7m52s
node-restart-f0qrm-5hrgc 0/1 Terminating 0 6m49s
node-restart-f0qrm-5kkhf 0/1 Terminating 0 7m29s
node-restart-f0qrm-5ksgj 0/1 Terminating 0 11m
node-restart-f0qrm-5mt9g 0/1 Terminating 0 8m56s
node-restart-f0qrm-5qndq 0/1 Terminating 0 8m9s
node-restart-f0qrm-5rcth 0/1 Terminating 0 8m47s
node-restart-f0qrm-5t5vq 0/1 Terminating 0 9m50s
node-restart-f0qrm-5xsfl 0/1 Terminating 0 7m41s
node-restart-f0qrm-5z9wr 0/1 Terminating 0 7m56s
node-restart-f0qrm-5zkp8 0/1 Terminating 0 9m5s
node-restart-f0qrm-62pqw 0/1 Terminating 0 8m48s
node-restart-f0qrm-64ljq 0/1 Terminating 0 11m
node-restart-f0qrm-65krn 0/1 Terminating 0 11m
node-restart-f0qrm-66pn5 0/1 Terminating 0 11m
node-restart-f0qrm-67kpm 0/1 Terminating 0 11m
node-restart-f0qrm-692bt 0/1 Terminating 0 8m23s
node-restart-f0qrm-6bmqf 0/1 Terminating 0 7m52s
node-restart-f0qrm-6cfsv 0/1 Terminating 0 11m
node-restart-f0qrm-6hfbf 0/1 Terminating 0 8m39s
node-restart-f0qrm-6jc9g 0/1 Terminating 0 7m20s
node-restart-f0qrm-6kbhx 0/1 Terminating 0 6m26s
node-restart-f0qrm-6lpbh 0/1 Terminating 0 7m18s
node-restart-f0qrm-6mmdl 0/1 Terminating 0 6m33s
node-restart-f0qrm-6nsdd 0/1 Terminating 0 10m
node-restart-f0qrm-6pknw 0/1 Terminating 0 6m
node-restart-f0qrm-6qdgl 0/1 Terminating 0 11m
node-restart-f0qrm-6v2kx 0/1 Terminating 0 9m16s
node-restart-f0qrm-6w8g5 0/1 Terminating 0 6m21s
node-restart-f0qrm-6wc94 0/1 Terminating 0 11m
node-restart-f0qrm-6wrph 0/1 Terminating 0 8m22s
node-restart-f0qrm-6zf96 0/1 Terminating 0 10m
node-restart-f0qrm-72xq7 0/1 Terminating 0 11m
node-restart-f0qrm-765c2 0/1 Terminating 0 10m
node-restart-f0qrm-77dpd 0/1 Terminating 0 8m18s
node-restart-f0qrm-792wz 0/1 Terminating 0 11m
node-restart-f0qrm-7cd57 0/1 Terminating 0 10m
node-restart-f0qrm-7dglg 0/1 Terminating 0 11m
node-restart-f0qrm-7kwj5 0/1 Terminating 0 5m40s
node-restart-f0qrm-7p998 0/1 Terminating 0 10m
node-restart-f0qrm-7pwpf 0/1 Terminating 0 5m51s
node-restart-f0qrm-7qgjp 0/1 Terminating 0 6m44s
node-restart-f0qrm-7svrw 0/1 Terminating 0 11m
node-restart-f0qrm-7t6l7 0/1 Terminating 0 7m42s
node-restart-f0qrm-7tfw8 0/1 Terminating 0 9m13s
node-restart-f0qrm-7wtcf 0/1 Terminating 0 10m
node-restart-f0qrm-7zbdz 0/1 Terminating 0 11m
node-restart-f0qrm-7zkhj 0/1 Terminating 0 11m
node-restart-f0qrm-7zxtg 0/1 Terminating 0 11m
node-restart-f0qrm-82bq4 0/1 Terminating 0 10m
node-restart-f0qrm-8495k 0/1 Terminating 0 8m42s
node-restart-f0qrm-84mlm 0/1 Terminating 0 5m45s
node-restart-f0qrm-864bs 0/1 Terminating 0 12m
node-restart-f0qrm-87mjm 0/1 Terminating 0 10m
node-restart-f0qrm-8dmmg 0/1 Terminating 0 5m40s
node-restart-f0qrm-8dtxz 0/1 Terminating 0 8m36s
node-restart-f0qrm-8fztb 0/1 Terminating 0 11m
node-restart-f0qrm-8h42c 0/1 Terminating 0 9m59s
node-restart-f0qrm-8ncvx 0/1 Terminating 0 7m27s
node-restart-f0qrm-8qzbt 0/1 Terminating 0 11m
node-restart-f0qrm-8r9q9 0/1 Terminating 0 6m13s
node-restart-f0qrm-8skwb 0/1 Terminating 0 6m15s
node-restart-f0qrm-958wz 0/1 Terminating 0 8m44s
node-restart-f0qrm-976k2 0/1 Terminating 0 11m
node-restart-f0qrm-97mcl 0/1 Terminating 0 12m
node-restart-f0qrm-98njc 0/1 Terminating 0 12m
node-restart-f0qrm-99mcm 0/1 Terminating 0 6m16s
node-restart-f0qrm-9b7vp 0/1 Terminating 0 5m42s
node-restart-f0qrm-9d65t 0/1 Terminating 0 6m52s
node-restart-f0qrm-9fnrh 0/1 Terminating 0 6m52s
node-restart-f0qrm-9g4dp 0/1 Terminating 0 8m2s
node-restart-f0qrm-9ghh8 0/1 Terminating 0 8m32s
node-restart-f0qrm-9hrzl 0/1 Terminating 0 9m20s
node-restart-f0qrm-9jtj5 0/1 Terminating 0 11m
node-restart-f0qrm-9kzm8 0/1 Terminating 0 7m42s
node-restart-f0qrm-9lmz6 0/1 Terminating 0 7m55s
node-restart-f0qrm-9mzrj 0/1 Terminating 0 9m12s
node-restart-f0qrm-9ns5b 0/1 Terminating 0 10m
node-restart-f0qrm-9ppcv 0/1 Terminating 0 12m
node-restart-f0qrm-9qgwc 0/1 Terminating 0 7m28s
node-restart-f0qrm-9qn86 0/1 Terminating 0 10m
node-restart-f0qrm-9rmcv 0/1 Terminating 0 11m
node-restart-f0qrm-9sg75 0/1 Terminating 0 5m57s
node-restart-f0qrm-9vk9s 0/1 Terminating 0 7m16s
node-restart-f0qrm-9wcrt 0/1 Terminating 0 7m59s
node-restart-f0qrm-9wnhb 0/1 Terminating 0 11m
node-restart-f0qrm-9xmln 0/1 Terminating 0 11m
node-restart-f0qrm-9xpdd 0/1 Terminating 0 11m
node-restart-f0qrm-b2b62 0/1 Terminating 0 7m46s
node-restart-f0qrm-b2nm6 0/1 Terminating 0 9m56s
node-restart-f0qrm-b4sz6 0/1 Terminating 0 11m
node-restart-f0qrm-b58mp 0/1 Terminating 0 9m52s
node-restart-f0qrm-b8qh7 0/1 Terminating 0 8m34s
node-restart-f0qrm-b9s47 0/1 Terminating 0 6m45s
node-restart-f0qrm-bdr6x 0/1 Terminating 0 8m5s
node-restart-f0qrm-bdtm5 0/1 Terminating 0 6m57s
node-restart-f0qrm-bdzdr 0/1 Terminating 0 9m42s
node-restart-f0qrm-bgs5d 0/1 Terminating 0 8m6s
node-restart-f0qrm-bhv9h 0/1 Terminating 0 11m
node-restart-f0qrm-bjq2n 0/1 Terminating 0 10m
node-restart-f0qrm-blr5z 0/1 Terminating 0 11m
node-restart-f0qrm-bp76x 0/1 Terminating 0 11m
node-restart-f0qrm-bpwwr 0/1 Terminating 0 6m24s
node-restart-f0qrm-brnmt 0/1 Terminating 0 11m
node-restart-f0qrm-bwqh7 0/1 Terminating 0 7m1s
node-restart-f0qrm-c29z2 0/1 Terminating 0 11m
node-restart-f0qrm-c2cq7 0/1 Terminating 0 11m
node-restart-f0qrm-c47kv 0/1 Terminating 0 10m
node-restart-f0qrm-c54wv 0/1 Terminating 0 6m44s
node-restart-f0qrm-c5kb2 0/1 Terminating 0 7m30s
node-restart-f0qrm-c8sv9 0/1 Terminating 0 12m
node-restart-f0qrm-cbh6z 0/1 Terminating 0 5m42s
node-restart-f0qrm-cdcxr 0/1 Terminating 0 6m20s
node-restart-f0qrm-cjl22 0/1 Terminating 0 9m47s
node-restart-f0qrm-ckq8f 0/1 Terminating 0 11m
node-restart-f0qrm-cp89l 0/1 Terminating 0 5m59s
node-restart-f0qrm-cs2v5 0/1 Terminating 0 6m58s
node-restart-f0qrm-cswxj 0/1 Terminating 0 7m24s
node-restart-f0qrm-ctbhv 0/1 Terminating 0 11m
node-restart-f0qrm-ctdhc 0/1 Terminating 0 6m56s
node-restart-f0qrm-ctj7w 0/1 Terminating 0 7m36s
node-restart-f0qrm-cw5jk 0/1 Terminating 0 11m
node-restart-f0qrm-cw78h 0/1 Terminating 0 8m31s
node-restart-f0qrm-cxg44 0/1 Terminating 0 10m
node-restart-f0qrm-d55xb 0/1 Terminating 0 9m55s
node-restart-f0qrm-d6qb6 0/1 Terminating 0 7m44s
node-restart-f0qrm-d6zlc 0/1 Terminating 0 8m34s
node-restart-f0qrm-d7wd5 0/1 Terminating 0 6m55s
node-restart-f0qrm-d8m59 0/1 Terminating 0 7m11s
node-restart-f0qrm-d9g79 0/1 Terminating 0 6m32s
node-restart-f0qrm-dbm6s 0/1 Terminating 0 8m9s
node-restart-f0qrm-dfvx4 0/1 Terminating 0 11m
node-restart-f0qrm-djnsh 0/1 Terminating 0 7m16s
node-restart-f0qrm-dkhbt 0/1 Terminating 0 6m20s
node-restart-f0qrm-dlwz7 0/1 Terminating 0 7m40s
node-restart-f0qrm-dnb5z 0/1 Terminating 0 7m58s
node-restart-f0qrm-dnjvc 0/1 Terminating 0 6m51s
node-restart-f0qrm-dphmt 0/1 Terminating 0 7m44s
node-restart-f0qrm-dpxxs 0/1 Terminating 0 6m49s
node-restart-f0qrm-dq9nm 0/1 Terminating 0 10m
node-restart-f0qrm-dqgf5 0/1 Terminating 0 7m39s
node-restart-f0qrm-dsxxr 0/1 Terminating 0 11m
node-restart-f0qrm-dtntk 0/1 Terminating 0 9m7s
node-restart-f0qrm-dwj75 0/1 Terminating 0 12m
node-restart-f0qrm-dx9fl 0/1 Terminating 0 10m
node-restart-f0qrm-dxbs7 0/1 Terminating 0 11m
node-restart-f0qrm-f2mvn 0/1 Terminating 0 9m58s
node-restart-f0qrm-f4dpn 0/1 Terminating 0 6m17s
node-restart-f0qrm-f4l2t 0/1 Terminating 0 10m
node-restart-f0qrm-f5s2j 0/1 Terminating 0 5m56s
node-restart-f0qrm-f75bk 0/1 Terminating 0 7m1s
node-restart-f0qrm-fbjjq 0/1 Terminating 0 8m8s
node-restart-f0qrm-fcf9x 0/1 Terminating 0 8m42s
node-restart-f0qrm-fdj6f 0/1 Terminating 0 11m
node-restart-f0qrm-fdrn5 0/1 Terminating 0 7m41s
node-restart-f0qrm-fds8x 0/1 Terminating 0 11m
node-restart-f0qrm-ffh7j 0/1 Terminating 0 8m37s
node-restart-f0qrm-fjdcj 0/1 Terminating 0 6m25s
node-restart-f0qrm-flkp5 0/1 Terminating 0 11m
node-restart-f0qrm-flkrf 0/1 Terminating 0 6m46s
node-restart-f0qrm-fnt7k 0/1 Terminating 0 6m29s
node-restart-f0qrm-fqt2z 0/1 Terminating 0 8m1s
node-restart-f0qrm-ft497 0/1 Terminating 0 7m59s
node-restart-f0qrm-g4qbs 0/1 Terminating 0 10m
node-restart-f0qrm-g5kd5 0/1 Terminating 0 11m
node-restart-f0qrm-g6qjk 0/1 Terminating 0 9m48s
node-restart-f0qrm-g7p88 0/1 Terminating 0 11m
node-restart-f0qrm-g92ml 0/1 Terminating 0 11m
node-restart-f0qrm-gbs6h 0/1 Terminating 0 8m7s
node-restart-f0qrm-gf5v9 0/1 Terminating 0 6m37s
node-restart-f0qrm-gjx2m 0/1 Terminating 0 10m
node-restart-f0qrm-gl2vv 0/1 Terminating 0 11m
node-restart-f0qrm-gp75b 0/1 Terminating 0 9m41s
node-restart-f0qrm-gsp8k 0/1 Terminating 0 10m
node-restart-f0qrm-gt4lj 0/1 Terminating 0 11m
node-restart-f0qrm-gtmp7 0/1 Terminating 0 7m33s
node-restart-f0qrm-gvpf6 0/1 Terminating 0 5m56s
node-restart-f0qrm-gw7q5 0/1 Terminating 0 9m
node-restart-f0qrm-h222t 0/1 Terminating 0 8m51s
node-restart-f0qrm-h62hv 0/1 Terminating 0 11m
node-restart-f0qrm-h7cwm 0/1 Terminating 0 11m
node-restart-f0qrm-h85tm 0/1 Terminating 0 8m30s
node-restart-f0qrm-h9cw6 0/1 Terminating 0 7m43s
node-restart-f0qrm-hbvcq 0/1 Terminating 0 6m21s
node-restart-f0qrm-hd6jj 0/1 Terminating 0 10m
node-restart-f0qrm-hfzwk 0/1 Terminating 0 5m55s
node-restart-f0qrm-hgvs9 0/1 Terminating 0 6m22s
node-restart-f0qrm-hjqpc 0/1 Terminating 0 11m
node-restart-f0qrm-hm2jk 0/1 Terminating 0 8m7s
node-restart-f0qrm-hnhz8 0/1 Terminating 0 6m7s
node-restart-f0qrm-hqnrw 0/1 Terminating 0 8m50s
node-restart-f0qrm-hvjd9 0/1 Terminating 0 7m12s
node-restart-f0qrm-hw7nc 0/1 Terminating 0 11m
...```
Updated the local copy of plugin index.
Installing plugin: node-restart
W1026 15:18:23.340568 3708 install.go:164] failed to install plugin "node-restart": plugin "node-restart" does not offer installation for this platform
failed to install some plugins: [node-restart]: plugin "node-restart" does not offer installation for this platform
Error: exit status 1
Hi,
I suppose kubectl --force
parameter is missing in the case of OpenShift cluster with "non standard" PODs. Not to be confused with node-restart --force
parameter for restart without drain.
There should be internally kubectl drain <node> --force --ignore-daemonsets --delete-emptydir-data
--ignore-daemonsets
and --delete-emptydir-data
are also helpfull
kubectl node-restart -l node-role.kubernetes.io/worker=
Targeting selective nodes:
openshift-b6scc-worker-0
openshift-b6scc-worker-1
openshift-b6scc-worker-2
openshift-b6scc-worker-3
Draining node openshift-b6scc-worker-0...
Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.
node/openshift-b6scc-worker-0 already cordoned
error: unable to drain node "openshift-b6scc-worker-0" due to error:cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): openshift-marketplace/certified-operators-kckxd, openshift-marketplace/community-operators-242l6, openshift-marketplace/ibm-operator-catalog-xwlf5, openshift-marketplace/redhat-marketplace-tlxwd, openshift-marketplace/redhat-operators-xqj2s, continuing command...
There are pending nodes to be drained:
openshift-b6scc-worker-0
cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): openshift-marketplace/certified-operators-kckxd, openshift-marketplace/community-operators-242l6, openshift-marketplace/ibm-operator-catalog-xwlf5, openshift-marketplace/redhat-marketplace-tlxwd, openshift-marketplace/redhat-operators-xqj2s
Initiating node restart job on openshift-b6scc-worker-0...
job.batch/node-restart-wr9rh created
Waiting for restart job to complete on node openshift-b6scc-worker-0...
usage: sleep seconds
openshift-b6scc-worker-0 - 10 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 20 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 30 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 40 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 50 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 60 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 70 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 80 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 90 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 100 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 110 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 120 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 130 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 140 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 150 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 160 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 170 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 180 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 190 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 200 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 210 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 220 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 230 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 240 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 250 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 260 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 270 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 280 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 290 seconds
usage: sleep seconds
openshift-b6scc-worker-0 - 300 seconds
Error: Restart job did not complete within 300 seconds
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.