Rolling upgrades and high availability

[cloudfoundry-community]

Rolling upgrades and high availability

[aegershman]

Alright, after doing some reading, I realize how this would work. Sorry for the spammy-ness of these comments but I'll post this here for reference in case it helps anyone else.

keepalived job co-located on BIND9 servers would be used to share a VIP. When one bind9 goes down & stops heartbeating, a different bind9 instance takes "ownership" of the VIP.

Since keepalived is pretty new to me I think for now, for simplicity sake, I'll just use the static IPs of the bind9 servers.

With that said in the future maybe I'll experiment with leveraging keepalived directly on the bind9 servers OR leveraging keepalived on a cluster of nginx UDP LBs (which I also haven't never experimented with, but am under the impression it can do UDP LBing)

I'm not exactly jazzed about self-hosting LBs or split-horizon DNS, but, alas, The Enterprise™ demands it, so it must be done. Who knows though maybe this will work out really well, and if it does, then we'll retrofill all our CF deployments to use this topology just for environmental consistency.

Thanks again 👍

[jhunt]

50 is a bit much. We don't use the sample manifest for anything other than explaining the properties. It should be safe (nay, advisable!) to drop that to a max_in_flight of 1.

[aegershman]

Sick, thanks.

I fully realize yourself / starkandwayne effectively maintains this project, and for all intents and purposes I should be paying for consulting fees, but I'll try to freeload anyway:

Do you have any general advice or "gotchas" to look out for when running this in production?

If not, that's fine, just figured I'd put the general question out there.

[jhunt]

The main thing to keep in mind is that if you are using the AXFR stuff with a master and some replicas, zone changes will normally only affect the master, and the replicas will just transfer down zone changes as they occur, without an interruption of service.

Other than that, don't forget to change your serial numbers when you update the zones, or you'll be quite upset at yourself. 😁

[aegershman]

👍gotcha, thank you. Current plan is to have 1 master and 3 replicas, with each replica in separate AZs, and update BOSH cloud-config so every VM has the 3 replicas as their DNS provider... including the BIND9 servers themselves ¯\(ツ)/¯

A future enhancement is to have an internal LB with a static IP which points to the deployed BIND9 servers, and configure cloud-config to use that LB's static IP for DNS. That way if/when the IPs of the BIND9 servers changes, we don't have to roll over every single VMs with updated cloud config, we just have to update the LB's targets.

Any thoughts? I promise after this I'll leave you be & close out this issue 😉thanks a lot for your time and advice.

[jhunt]

re: 1+3 across AZs sounds perfectly reasonable.

As for load balancing, if you don't have IaaS-provided LBs, you might want to look at keepalived and just have keepalived trade a VIP between the three replicas directly. We use it primarily with haproxy reverse proxies for HTTP/TCP traffic, in the load-balancer-boshrelease. That (haproxy) doesn't support UDP traffic, but keepalived operates at the IP layer so it should be doable.

That would be a fun add-on to this BOSH release, come to think of it. 🤔

Also, you don't have to leave me alone 😁

[aegershman]

Nice, will look into the lb boshrelease. I think I'd like that better than IaaS-provided, since it could be standardized & deployed across all our envs (azure, aws, vsphere) with the same topology

How do you feel about this "first mover"/bootstrap situation where you have to setup BOSH, and then use it deploy DNS/LBs/any other infra for the env, and then update all deployments to use the BOSH-deployed infra?

It's not a huge deal since it's really only a problem when getting started, but this will be the first time that I have an env which isn't 100% fully declarative... I guess we could use reserved IPs and static IP assignment for the LB in it's manifest, though? That way the entire env is still fully declarative from the get-go... But maybe it's not actually that important to have an env which can be deployed starting from "nothing-to-something" based on git config alone.

Maybe it's fine to have some values that are dynamic (IPs of LBs, BIND, etc.) & once they're deployed, re-deploy the env with those updated values.

[aegershman]

Wait-- if you're using a BOSH-managed LB to load balance DNS, aren't you (sort of) back in the same spot as before, where you have to configure the cloud-config DNS to use the multiple IPs of the LBs? E.g. you'd still have to deploy multiple LBs so you can do rolling upgrades. Right?

I guess the value-add though is that the config options the LB offers, and that you can more-easily scale out DNS servers & just update the LB's routing targets.

EDIT I feel I'm definitely misunderstanding some things.

Keepalived implements the Virtual Routing Resource Protocol (VRRP),
enabling a single virtual IP (a VIP) to be managed between multiple hosts.
This in turn allows two or more haproxy servers to present as a single,
unified endpoint.

I'll do some reading on keepalived 🧐

Apologies & thanks-- self-hosted LB's & self-hosted DNS is new to me.