Issues with DNS in kubernetes

[scholzj]

Issues with DNS in kubernetes

[scholzj]

So if I understand you correctly - the internal DNS within your Minikube is working fine. But it doesn't work for services outside of your cluster? Is it just the pods (i.e. when you try to access some hostname from inside your application)? Or even the cluster it self (such as that it cannot reach Docker Hub to download images)?

I never saw that or experienced that. I think it must be some conflicting setting in your VPC or something like that. But out of my head there isn't any particular setting which would come up to my mind :-/.

[cj-anthony]

Thanks, we have worked around by pointing to a public DNS for the time being. It's very strange, the calico and kube-dns pods reference a VPC Subnet IP address in their resolve.conf that I can only assume is AWS provided DNS. However calico can resolve hostnames and kube-dns can't. This same address is used just by the rest of our EC2 instances so it's a bit of a head scratcher. You can close this if you like, I'm happy to pursue from here if you don't think there is anything in AWS minikube causing it.

[scholzj]

I did an update to Kubernetes 1.10.4 today. I tried specifically the DNS, but it seemed to work perfectly fine for me.

I doubt this is related to the Kubernetes update, so I guess it must be related to something in your VPC / networking. I can try to help, but it is quite hard to debug this kind of issues without having access to your setup. I would probably need to know all your settings around VPC and networking. If you created everything using some public Terraform template or something similar, it might be quite easy for me to recreate it and give it a try.

[cj-anthony]

Thanks for the offer Jakub, but I think I'll probably have to take it from here as our deployment is
a little complicated. Our VPC was configured a while ago and this deployment was just dropped into it.

[cj-anthony]

Hi Jakub, I just had one last thought. We don't use Calico on our existing clusters and I was wondering if there were any Calico Network Policies in place by default that might prevent traffic moving in and out of the cluster. Cloud we remove Calico without affecting any of the other pods?

Lastly it looks like the kube-dns pod's resolv.conf has inherited a nameserver that references an address in our VPC. We've certainly never configured a DNS server so I'm guessing it's an AWS provided nameserver. Is this usual behavior? I'm struggling to find information about DNS resolution in a VPC on AWS. But I'm wondering if that server is knocking back the requests.

[scholzj]

I do not do anything special to configure DNS. So it is either done by kubeadm oe by the Kubernetes' AWS integration. Or it is simply taken from the host configuration - which will be done by the default CEntOS integration with AWS.