When using allow_scheduling_on_control_plane=true, lb does not point to control planes

[kube-hetzner]

When using allow_scheduling_on_control_plane=true, lb does not point to control planes

[geraldwuhoo]

@geraldwuhoo Please, if you can, try issuing kubectl label nodes --all node-role.kubernetes.io/master- and let me know if that works! pray

If it does - PR is welcome! :)

Just tried this on a fresh cluster — I can confirm that my controlplane nodes show up under the loadbalancer when I remove the master label from them.

[geraldwuhoo]

Update on this: removing the labels affects the k3s auto-upgrade ability, as the upgrade script uses the master label to select the controlplane nodes for upgrade. In any case, it looks like removing the master label is a bit brittle anyway: hetznercloud/hcloud-cloud-controller-manager#197. However, this issue claims that passing the LegacyNodeRoleBehavior=false feature gate flag in the creation of the hcloud CCM allows for scheduling on controlplane nodes. Should be easy enough to add to this file:

https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/20a6ebc667d535d2dbb81c1f9983219df97829f7/templates/ccm.yaml.tpl

I don't think this will adversely affect existing deployments, as this label has apparently already been locked to false for a while, and is fully removed in v1.22. Or perhaps it could be conditionally set only when allow_scheduling_on_control_plane is true.

Edit: Cloned the repo and added - "--feature-gates=LegacyNodeRoleBehavior=false" to the end of ccm.yaml.tpl. A fresh cluster with this feature gate fixes the LB issue: LB points to controlplane nodes with no further changes.

[leonsteinhaeuser]

I have made a test and can confirm that the result is still the same.

My example service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-sample
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-sample
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-sample
  labels:
    app: nginx
  #annotations:
  #  kubernetes.io/ingress.class: "traefik"
spec:
  #tls:
  #- hosts:
  #  - "nginx.staging.sample.dev"
  #  secretName: nginx-sample-tls
  rules:
  - host: "nginx.staging.sample.dev"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: nginx-sample
            port:
              number: 80

[mysticaltech]

Released in v1.2.4!

[mysticaltech]

Yes, this is normal, because it will show up if you create an ingress definition to expose a service of a sort. So it works normally, but please try creating a test deployment with a service, and an ingress associated with it, and let us know, please.

[mysticaltech]

Ok thanks, will investigate further!

[mysticaltech]

Confirmed, my control planes have:

Taints:             <none>
Unschedulable:      false

Yet, the LB only targets the agents, as follows, obtained with hcloud load-balancer describe traefik:

[mysticaltech]

@leonsteinhaeuser Please have a look at https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml, and search the web about the matter, as my initial attempt at finding a solution failed.

If that can be set via the traefik config helm, then we can do so via the traefik_config.yaml.tpl HelfChartConfig file.

[mysticaltech]

I believe this has to do with https://github.com/hetznercloud/hcloud-cloud-controller-manager, which does not seem to want to configure the ingress controller with control plain nodes IPs, even if those have scheduling enabled.

So, unfortunately, there is nothing we can do on our side but it would be wonderful if you could open an issue here https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues @leonsteinhaeuser.

[geraldwuhoo]

It looks like in hetznercloud/hcloud-cloud-controller-manager#101 (comment), the reason is actually because k8s itself reports the list of nodes to the CCM, and it will not report the controlplane nodes as schedulable as long as they are tainted and labeled as master nodes. The response seems to imply that removing the taint and label from the controlplane nodes fixed the issue.

For this project, I believe removing the taints and labels from all controlplane nodes when controlplanes are allowed schedulable is a reasonable action. What do you think?

[mysticaltech]

@geraldwuhoo Thanks for mentioning the issue above in hetznercloud/hcloud-cloud-controller-manager#101, I did not know about it.

What is happening, as you can see in

terraform-hcloud-kube-hetzner/locals.tf

Line 85 in a9fb135

default_control_plane_taints = concat([], local.allow_scheduling_on_control_plane ? [] : ["node-role.kubernetes.io/master:NoSchedule"])

is that we already remove all taints when the control plane is set to allow scheduling.

So unless I am mistaken, there is not much more we can do on our side!

If you could revive the above issue on the hcloud controller side, it would be great and very much appreciated! 🙏

[mysticaltech]

@geraldwuhoo Please, if you can, try issuing kubectl label nodes --all node-role.kubernetes.io/master- and let me know if that works! 🙏

If it does - PR is welcome! :)

[mysticaltech]

@leonsteinhaeuser, Please also give that a shot if you are still using the cluster! Between, you might want to redeploy as it has improved light years since February! :)

[mysticaltech]

Great to hear @geraldwuhoo! Please, don't hesitate to create a PR to the staging branch. Otherwise I will work on it whenever I have time. Great work!!

[mysticaltech]

@leonsteinhaeuser This is now taken care of by the PR above! Will be shipped in the next release! @geraldwuhoo! 🥳