Build a cold spare cluster with manual failover

[jupyterhub]

Build a cold spare cluster with manual failover

[minrk]

Sounds like a really good idea, especially the not-making-you-do-everything part!

Question about ips: Right now, as I understand it the IP is the load-balancer IP. Can we update that to point to another cluster without updating it at the DNS level? Or do we need an additional layer of indirection that we don't yet have in order to accomplish that?

[yuvipanda]

@betatim not sure. I think we should treat the LoadBalancer that Kubernetes has provisioned as 'owned' by Kubernetes and try not to fitz with it directly in Google Cloud console. I am pretty sure that'll just get overwritten silently at some point.

[betatim]

For sure not suggesting fiddling around through the UI. You'd have to set up things through the helm chart. I just noticed it while browsing around which is my way of discovering what is out there ;)

[yuvipanda]

Ideally someone else not me would do all this :D I think that'd be a great way to build operational capacity in the team/community, and it'll also be a lot of fun/learning for the person doing it!

[betatim]

I'd help with that. (1) and (2) look like things I already know how to do, (3) I'd appreciate a pointer as all the docs I've read so far don't mention static IPs (I'll start by sniffing around the cloud console).

[betatim]

Notes from poking around: the load balancer is an Ingress, it knows where to point to because of the backend configuration in the helm chart of binderhub.

This means each binderhub deployment/cluster has one of these ingress controllers/load balancers. I don't think we could point the ingress of prod-a to the service called binder in a cluster called prod-b. Also weird because you are mixing things together.

So right now I'm thinking like Min that we need another Ingress/LB in front of the LB for prod-a and prod-b.

[yuvipanda]

Ideally, we would have another Cloud Load Balancer from Google Cloud of some sort that gives us an IP we can switch between the lb IPs of prod-a and prod-b. However, from a cursory look it looks like it isn't possible to get a Cloud Load Balancer that lets you switch between two IPs, rather than two instance groups? Hopefully I had just missed something, and this feature exists in there somewhere... If so, we should use it :D

The alternative (and what I did for switching between prod and prod-a):

1. "Release" the static IP from its current configuration (I did this by deleting the svc in prod, but there is probably a gcloud way to do this). This keeps the IP still in our project (since we've 'reserved' it) but available for use by other kubernetes clusters.
2. Deploy the helm chart to prod-a, including the nginx-ingress IP configuration

This is more error prone, but still decent.

The option I'd like us to not do is to have another series of servers whose job it is to do the load balancing. While easy, this just kicks the 'cold spare / switching' can down the road to another set of servers...

Thanks a lot for taking this up, @betatim and @minrk!

[betatim]

If you click on "edit config" for the load balancer in the cloud console you see the following:

So maybe we can use that to configure a fail over? One thing we'd have to understand is why the current LB thinks that two of the three nodes in the prod-a cluster are "unhealthy".

[yuvipanda]

We have federation now.