CoreDNS, local-path-provisioner, metrics-server not deployed HA in HA mode

[rdvansloten]

CoreDNS, local-path-provisioner, metrics-server not deployed HA in HA mode

[rdvansloten]

Interim solution for CoreDNS

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: coredns-hpa
  namespace: kube-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coredns
  minReplicas: 3
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75

However, because CoreDNS has a topologyspread and the others don't, this can't be applied to local-path-provisioner and metrics-server.

[dereknola]

Thats not how those components work. CoreDNS, local-path-provisioner, metrics-server are deployed on the cluster as a whole, not on every server. They aren't considers server specific components, like kube-apiserver or the controller manager. They are just regular workloads. If you want to modify them, see https://docs.k3s.io/helm#customizing-packaged-components-with-helmchartconfig, you can create /var/lib/rancher/k3s/server/manifests/coredns-ha.yaml with the modifications you want.

[rdvansloten]

Thats not how those components work. CoreDNS, local-path-provisioner, metrics-server are deployed on the cluster as a whole, not on every server. They aren't considers server specific components, like kube-apiserver or the controller manager. They are just regular workloads. If you want to modify them, see https://docs.k3s.io/helm#customizing-packaged-components-with-helmchartconfig, you can create /var/lib/rancher/k3s/server/manifests/coredns-ha.yaml with the modifications you want.

Offering a supposedly highly available deployment option and then replying "lol roll your own" is a bit shortsighted, no? Deploying coredns as a DaemonSet on the master nodes (as I now plan to do) for example could at least be given an inch of consideration. At some point there's been a conscious choice to include these "regular workloads" and assign them as part of the core k3s experience. (since you have to really go out of your way to get rid of them once installed, or do this pre-install with obscured args)

The fact of the matter is that the current HA deployment of k3s simply isn't HA because of these "regular workloads" you essentially push onto the user are essential to basic cluster operations (how does one operate a cluster when DNS, monitoring and file provisioner went poof because one of out 3 nodes is rebooting/broken?)

This has been brought up and iterated on before:
k3s-io/k3s#1606

[dereknola]

I understand your frustration with this. I totally get the "well this isn't HA really is it" argument. Two things on this:

1. This isn't something that the k3s-ansible repo is meant to solve. This repo only covers deploying vanilla K3s with a few config options. I added the extra_manifests configuration to help enable exactly this scenario, where you can easily supply manifest files you want deployed automatically when provisioning a cluster.
2. As covered in k3s-io/k3s#1606, this was ultimately a design decision between resource usage and redundancy. As K3s is usually deployed on the edge in resource constrained environments, we decided that by default, deploying less pods was the better strategy. K3s is an opinionated K8s distro. At the same time, deploying modifications for things you don't like is a core mission of K3s. Its why its so easy to use the --disable flag to turn of components you don't like and swap them out. So many people disable traefik and run metalLB. Similarly, you are free to simply modify the manifests to include a replica configuration of 2+.

I think we could help alleviate this somewhat by having better documentation in https://github.com/k3s-io/docs about how do go about modify the helm manifest and perhaps include and example of "making CoreDNS run with multiple replicas".