[occm] dnsPolicy default value recent change blocks occm / cluster from coming up.

[ericgraf]

[occm] dnsPolicy default value recent change blocks occm / cluster from coming up.

[mdbooth]

The documentation of this field is quite spectacularly bad: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy

"ClusterFirstWithHostNet": For Pods running with hostNetwork, you should explicitly set its DNS policy to "ClusterFirstWithHostNet". Otherwise, Pods running with hostNetwork and "ClusterFirst" will fallback to the behavior of the "Default" policy.

To my eyes that clearly says that pods running with host networking should set ClusterFirstWithHostNet. CCM runs with host networking, so we'd seem to be covered. It doesn't actually say what it does, though.

Some RTFS:
https://github.com/kubernetes/kubernetes/blob/790dfdbe386e4a115f41d38058c127d2dd0e6f44/pkg/kubelet/network/dns/dns.go#L303-L323

For reasons I don't understand, most likely legacy API compatibility, ClusterFirst sets the DNS policy to Default if the pod uses host networking. Internally that is mapped to a podDNSType of podDNSHost, which is a much better name. i.e. ClusterFirst means:

Use cluster networking unless the pod uses host networking, in which case use the host's DNS.

ClusterFirstWithHostNet removes the fallthrough and uses ClusterFirst in all cases.

As an early bootstrap service, I don't think CCM can rely on cluster DNS being up. I suspect it is correct to revert this.

@xinity, what was the issue you were hitting which caused you to change it? It's not clear to me from reading #2594 or #2592? I appreciate that CCM is not able to resolve service names internal to the cluster, but why did that matter?

[xinity]

@mdbooth occm wasn't able to query the internal coredns instance without this new value

It matters because of specific internal dns zone with squid proxy that should be resolved from occm

[mdbooth]

@mdbooth occm wasn't able to query the internal coredns instance without this new value

It matters because of specific internal dns zone with squid proxy that should be resolved from occm

Right, but why? What was the internal DNS zone, and why was it important that CCM could resolve it?

[jichenjc]

our CI passed so it should be a smaller portion of error case

and I am also curious why the internal DNS zone is needed here

[mdbooth]

our CI passed so it should be a smaller portion of error case

I also wondered about that. Does that mean CNI comes up an an uninitialized node, and coredns tolerates uninitialised?

[yankcrime]

I've just tested the new release of OCCM on a cluster with 1.30 and have hit this issue as well. For anyone else who is struggling to understand the root cause (being this change), the nondescript error from the CCM Pod is:

Error from server: no preferred addresses found; known addresses: []

I only found the underlying error when I SSH'd onto the node where the CCM had been scheduled and looked at the container logs in /var/log/containers.