Comments (23)
One low-hanging fruit is that we should be deploying multiple copies of the controller-manager/scheduler. In that case you would be doing a rolling-update of the component, verifying that the new functionality works before destroying all of the old copies.
However, there are still situations where we have a loss of all schedulers and/or controller-manager (e.g. maybe a flag change is subtly broken, but the pod is still running so the deployment manager rolls out all broken pods).
You could launch a new master as an option, but if you still have an api-server/etcd running you should be able to recover. Essentially you would need to inject a controller-manager pod into the cluster, then delete it as soon as your existing controller-manager deployment has been scheduled.
For example:
kubectl --namespace=kube-system get deployment scheduler -oyaml
Then take the podSpec section (second indented spec, with a containers
field right below):
Something like:
spec:
containers:
- name: kube-controller-manager
image: quay.io/coreos/hyperkube:v1.4.0_coreos.0
command:
[...]
Then wrap that in a pod header, and specify the nodeName it should run on:
apiVersion: v1
kind: Pod
metadata:
name: recovery-cm
spec:
nodeName: <a node in your cluster>
containers:
- name: kube-controller-manager
image: quay.io/coreos/hyperkube:v1.4.0_coreos.0
command:
[...]
Then inject it in the cluster
kubectl create -f recovery-pod.yaml
What will happen is that this pod will act as controller-manager, convert your existing deployment/controller-manager into pods - then they will be scheduled. After that you can just delete the recovery pod:
kubectl delete -f recovery-pod.yaml
from bootkube.
Thinking longer term, rather than a separate kube-recover
tool, this could be a command in kubectl
which knows how to extract a pod from higher-order object -- this could be useful in the case where we have a kubelet-pod-api (so we could use this to natively push pod to kubelet api from a deployment/daemonset object rather than needing the intermediate pod state). See #97 (comment)
from bootkube.
Is this documented anywhere? Could help our users!
from bootkube.
๐ I think an interesting idea is to play with re-running bootkube
for recovery. I found that generally if I just re-run bootkube nothing too crazy happens and the temporary control plane brings up a scheduler, allowing everything to correct itself. I really like the UX of this approach, even if its not a real solution.
In the static manifest case this problem isnt really a problem because the pods which run aren't scheduled, so the scheduler failing in any scenario always results in it being restarted and able to run (assuming it can grab it's lease).
I wonder if there's a good way to do this, perhaps by check pointing the scheduler to nodes which have role=master
would be a good start, as an alternative to your proposed kube-recover
strategy? This would be mostly automated, the only issue is how do you run this checkpointer in a way that works when the scheduler runs into this? Well, that's probably a static pod, and that means we can't easily manage it, which is problematic if the master nodes are changing.
from bootkube.
Also, definitely agree that a kubelet-pod-api that skips scheduling makes this story way better.
from bootkube.
Hi guys, I was trying bootkube and I had a similar case. In my case, I figured that the controller had no --cloud-provide=aws
flag set and I tried editing the controller-manger with kubectl edit... and it was a bad idea. The result was that I can't recover the cluster cause I am not able to schedule anything.
What I'm thinking is: in this case we have etcd running on the master itself, which is just wrong, but I thought that if we run a reliable, separated etcd cluster, this problem would not exist at all. Just run a new master with bootkube and attach it to the etcd cluster and kill the broken one. Would this make sense?
from bootkube.
Just hit this as well as a result of a container linux reboot
from bootkube.
In addition, if your scheduler is down, recovering the controller-manager
won't be enough. In this case, put a file like this in /etc/kubernetes/manifests/scheduler.yml
on one of the nodes, so it passes your kube-scheduler
Deployment into Running
rather than Pending
state:
#
# Add this to a node in `/etc/kubernetes/manifests` to recover your scheduler, and
# schedule the pods needed to run your configured Deployments :)
#
kind: Pod
apiVersion: v1
metadata:
name: kube-scheduler
namespace: kube-system
labels:
k8s-app: kube-scheduler
spec:
containers:
- name: kube-scheduler
image: quay.io/coreos/hyperkube:v1.5.3_coreos.0
command:
- ./hyperkube
- scheduler
- --leader-elect=true
- --kubeconfig=/etc/kubernetes/kubeconfig
volumeMounts:
- name: etc-kubernetes
mountPath: /etc/kubernetes
readOnly: true
volumes:
- name: etc-kubernetes
hostPath:
path: /etc/kubernetes
from bootkube.
You can do the same steps I outlined above for the scheduler as well (and don't need to actually ssh into a machine to create the static manifest).
from bootkube.
But who schedules the recovery scheduler if the scheduler is dead? :-)
from bootkube.
Per the steps outlined above you would populate the pod's spec.NodeName
so that the pod is pre-assigned to a node - no scheduler needed.
from bootkube.
Oh right!! That's great. Thanks :-)
from bootkube.
Working on it @mfburnett as i have hit with the same issue today while upgrading. Should I be creating a doc defect for this, or would you be doing it for me? cc @aaronlevy
from bootkube.
Thanks @radhikapc!
from bootkube.
Just to track some internal discussions -- another option might be to propose a sub-command to kubectl upstream. Not sure of UX specifics, but maybe something like:
kubectl pod-from deployment/kube-scheduler --target=nodename
kubectl pod-from daemonset/foo --target=nodename
kubectl pod-from podtemplate/foo --target=nodename
from bootkube.
Another simple way to hook back a Pod to a Node, when Scheduler + Controller-manager are dead:
kubectl create -f rescue-binding.yaml
with this content:
apiVersion: v1
kind: Binding
metadata:
name: "kube-dns-2431531914-61pgv"
namespace: "kube-system"
target:
apiVersion: v1
kind: Node
name: "ip-10-22-4-152.us-west-2.compute.internal"
A Binding is the objected injected in the K8s cluster when the Schedule takes a scheduling decision. You can do the same manually.
from bootkube.
Another simple way to hook back a Pod to a Node, when Scheduler + Controller-manager are dead:
Thanks! This prevented me from restarting the whole cluster (again) with bootkube recover
and bootkube start
.
Someone should properly add this to: https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/disaster-recovery.md
from bootkube.
Been thinking about this, can't checkpointer help here? mark controller-manager as pod to be checkpointed, then it would be recovering it if it can't find it on the node. There can be edge case when it recovers too many pods, but given that they do leader election that is fine to have few extra running.
As for the scheduler, making it daemonset would allow it to be enough just for controller-manager & apiserver to be alive as daemonsets are scheduled by controller, at least in current 1.8.x release
from bootkube.
With the behavior changes introduced in #755 checkpointing the controller-manager / schedule might be possible (before that change we might garbage the checkpoints of those components before replacements have been scheduled). It might still be a little bit racy though.
cc @diegs
from bootkube.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
from bootkube.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
from bootkube.
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen
.
Mark the issue as fresh with /remove-lifecycle rotten
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
from bootkube.
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen
.
Mark the issue as fresh with/remove-lifecycle rotten
.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from bootkube.
Related Issues (20)
- update CRI in checkpointer
- Enable api server aggrigation HOT 5
- Update Checklist for 1.17 HOT 11
- Update flannel
- Update calico HOT 16
- update coredns
- Add support for installing helm charts using helm 3 as a library during โbootkube startโ HOT 20
- Proposal : CNI Support HOT 15
- Rervert coredns labels HOT 10
- Move hack/* to flatcar linux HOT 4
- PodCIDR and ServiceCIDR should be multiples HOT 1
- Create Release HOT 8
- Move Default Templates Render to 1.18.X HOT 8
- Can no longer execute "make vendor" HOT 14
- Don't use rkt for building images HOT 3
- Check if tests runs properly in CI HOT 13
- Bootkube - Bare-metal documentation link is broken HOT 5
- Pod cannot obtain pod ip HOT 5
- Update vendored kubelet utils to align with K8s version from go.mod HOT 8
- Add support for ARM64 architecture HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bootkube.