Churn nodes on a periodic basis

[jupyterhub]

Churn nodes on a periodic basis

[betatim]

This would also help with the "out of space" error we have been seeing a few times and that appears to be fixed by removing layers/docker images. Though it seems unclear why exactly it works, or what it also changes that then fixes this problem.

To get an idea how this would work: to remove a node would we first kubectl cordon <node> (kubectl drain deletes pods straight away right?), add a new node (*), wait for the node to be empty of user pods, remove node.

(*) done by increasing the minimum size of the cluster?

[yuvipanda]

I would say the way to do this is:

1. Pick oldest node
2. Cordon it
3. Wait for it to be empty of user and build pods(can be several hours!)
4. Delete it (via gcloud API)

Deleting the node should bring up a new one.

And yup, kubectl drain deletes pods straight away!

[yuvipanda]

Note that currently making changes to the Kubernetes Cluster (such as changing minimum size of cluster) causes a multi-minute GKE outage, since GKE does not do Highly Available masters yet. But deleting a node is fine...

[betatim]

I was thinking cordon off + increase capacity by one so that the overall number of available nodes stays the same. Currently we are pretty comfortable in terms of capacity so might not be a problem.

What is a GKE outage? We can't issue GKE related commands or our existing nodes stop doing stuff?

[betatim]

Related to this, I tried to add a new node to the staging cluster to practice/mess around. This fails with the following error:
"Instance 'gke-staging-default-pool-8014xxxx-zxxx' creation failed: Quota 'DISKS_TOTAL_GB' exceeded. Limit: 4096.0 in region us-central1"

The command I used was gcloud compute instance-groups managed resize gke-staging-default-pool-.... --zone us-central1-a --project binder-staging --size 3

What is weird about this is that prod has the same limit (4096), more nodes and less of the quota is used. Not sure I understand that.

[minrk]

@betatim I checked the disks on staging, and several disks from previous test clusters were hanging around, using up the quota. It should be available now.

I also think the staging cluster could be using smaller disks. 1TB disks is a bit overkill for the staging load.

[yuvipanda]

This is no longer necessary since we switched to using DIND daemons. I'll close this for now!