Can't access the EC2 instance

[scholzj]

Can't access the EC2 instance

[scholzj]

I changed the scripts to generate a second kubeconfig file with configuration which is done using the IP address only. It also prints alternative "To copy ..." line which is using IP address.

This deployment is basically designed to run on public subnet. So I'm not sure whether the private DNS zone actually makes some sense apart from being a workaround so that you don't need a public DNS zone (for which you need to register a domain).

I'm still a bit unsure why did the SSH connection over the public IP not work. Do you have the EC2 instance in public subnet with Internet Gateway etc.?

Please give it a try and let me know if it works for you now.

[kolov]

The issue was invalid configuration, the route table was missing the routing to the gateway.

[scholzj]

Oki, good to know. Anyway, I think the enhancements I made might be usefull for someone without a public hosted zone. Even with the right routing you would have had a problem because the original kubeconfig user the DNS name. Now with the second kubeconfig based around IP these situations should be easier.

[kolov]

Unfortunately, I still don't get a usable kubernetes cluster:

Apply complete! Resources: 12 added, 0 changed, 0 destroyed.

Outputs:

copy_config_dns = To copy the kubectl config file using DNS record, run: 'scp centos@aws-
minikube-1.kube.public:/home/centos/kubeconfig .'
copy_config_ip = To copy the kubectl config file using IP address, run: 'scp 
[email protected]:/home/centos/kubeconfig_ip .'
kubeadm_token = lmcjhz.lmcjhztnfkoq731t

minikube_dns = aws-minikube-1.kube.public
rife:aws-minikube assen$ scp [email protected]:/home/centos/kubeconfig_ip .
The authenticity of host '35.156.106.81 (35.156.106.81)' can't be established.
ECDSA key fingerprint is SHA256:jUlRGF2xA8/iqwgQc/Yy0A5Li2RpBSJq+Jo8auBdeYQ.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '35.156.106.81' (ECDSA) to the list of known hosts.
scp: /home/centos/kubeconfig_ip: No such file or directory
rife:aws-minikube assen$ ssh [email protected] 
[centos@ip-10-0-0-86 ~]$ ls -al /home/centos
total 12
drwx------. 3 centos centos  74 Sep 13 18:57 .
drwxr-xr-x. 3 root   root    20 Sep 13 18:57 ..
-rw-r--r--. 1 centos centos  18 Dec  6  2016 .bash_logout
-rw-r--r--. 1 centos centos 193 Dec  6  2016 .bash_profile
-rw-r--r--. 1 centos centos 231 Dec  6  2016 .bashrc
drwx------. 2 centos centos  29 Sep 13 18:57 .ssh

[scholzj]

Two ideas:

• The bootstraping takes some time, you were too quick and the kubeconfig files were not yet generated
• If the first option was not the case, then something went wrong with the bootstraping. Can you provide the content of /var/log/cloud-init-output.log file?

[kolov]

kubeconfig is still not there after 40min. The log:

[centos@ip-10-0-0-86 ~]$ ls -al /home/centos
total 12
drwx------. 3 centos centos 74 Sep 13 18:57 .
drwxr-xr-x. 3 root root 20 Sep 13 18:57 ..
-rw-r--r--. 1 centos centos 18 Dec 6 2016 .bash_logout
-rw-r--r--. 1 centos centos 193 Dec 6 2016 .bash_profile
-rw-r--r--. 1 centos centos 231 Dec 6 2016 .bashrc
drwx------. 2 centos centos 29 Sep 13 18:57 .ssh
[centos@ip-10-0-0-86 ~]$ cat /var/log/cloud-init-output.log
Cloud-init v. 0.7.5 running 'init-local' at Wed, 13 Sep 2017 18:57:29 +0000. Up 7.09 seconds.
Cloud-init v. 0.7.5 running 'init' at Wed, 13 Sep 2017 18:57:31 +0000. Up 9.42 seconds.
ci-info: +++++++++++++++++++++++++Net device info+++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | Device | Up | Address | Mask | Hw-Address |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . |
ci-info: | eth0: | True | 10.0.0.86 | 255.255.255.0 | 02:d4:17:16:ca:27 |
ci-info: +--------+------+-----------+---------------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route info++++++++++++++++++++++++++++++
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: +-------+-------------+----------+---------------+-----------+-------+
Cloud-init v. 0.7.5 running 'modules:config' at Wed, 13 Sep 2017 18:57:33 +0000. Up 11.11 seconds.
Cloud-init v. 0.7.5 running 'modules:final' at Wed, 13 Sep 2017 18:57:33 +0000. Up 11.47 seconds.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3481 100 3481 0 0 5680 0 --:--:-- --:--:-- --:--:-- 5678
set -o errexit
set -o pipefail

if [ -z "$KUBERNETES_VERSION" ]; then
KUBERNETES_VERSION="1.7.5"
fi

if [ -z "$CLUSTER_NAME" ]; then
CLUSTER_NAME="aws-minikube"
fi

Set this only after setting the defaults

set -o nounset

Set fully qualified hostname

This is needed to match the hostname expected by kubeadm an the hostname used by kubelet

hostname $(hostname -f)

Make DNS lowercase

DNS_NAME=$(echo "${DNS_NAME}" | tr 'A-Z' 'a-z')

Install docker

yum install -y yum-utils curl gettext > device-mapper-persistent-data lvm2
warning: /var/cache/yum/x86_64/7/updates/packages/device-mapper-1.02.135-1.el7_3.5.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) [email protected]"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-3.1611.el7.centos.x86_64 (installed)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
grabbing file https://download.docker.com/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
yum makecache fast
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

• base: centosmirror.netcup.net
• extras: ftp.plusline.de
• updates: mirror.fra10.de.leaseweb.net
  Metadata Cache Created
  yum install -y docker-ce
  Loaded plugins: fastestmirror
  Loading mirror speeds from cached hostfile
• base: centosmirror.netcup.net
• extras: ftp.plusline.de
• updates: mirror.fra10.de.leaseweb.net
  Resolving Dependencies
  --> Running transaction check
  ---> Package docker-ce.x86_64 0:17.06.2.ce-1.el7.centos will be installed
  --> Processing Dependency: container-selinux >= 2.9 for package: docker-ce-17.06.2.ce-1.el7.centos.x86_64
  --> Processing Dependency: libseccomp.so.2()(64bit) for package: docker-ce-17.06.2.ce-1.el7.centos.x86_64
  --> Processing Dependency: libltdl.so.7()(64bit) for package: docker-ce-17.06.2.ce-1.el7.centos.x86_64
  --> Running transaction check
  ---> Package container-selinux.noarch 2:2.19-2.1.el7 will be installed
  ---> Package libseccomp.x86_64 0:2.3.1-2.el7 will be installed
  ---> Package libtool-ltdl.x86_64 0:2.4.2-22.el7_3 will be installed
  --> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Installing:
docker-ce x86_64 17.06.2.ce-1.el7.centos docker-ce-stable 21 M
Installing for dependencies:
container-selinux noarch 2:2.19-2.1.el7 extras 28 k
libseccomp x86_64 2.3.1-2.el7 base 56 k
libtool-ltdl x86_64 2.4.2-22.el7_3 updates 49 k

Transaction Summary

Install 1 Package (+3 Dependent packages)

Total download size: 21 M
Installed size: 74 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-17.06.2.ce-1.el7.centos.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
Public key for docker-ce-17.06.2.ce-1.el7.centos.x86_64.rpm is not installed

Total 13 MB/s | 21 MB 00:01
Retrieving key from https://download.docker.com/linux/centos/gpg
Importing GPG key 0x621E9F35:
Userid : "Docker Release (CE rpm) [email protected]"
Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
From : https://download.docker.com/linux/centos/gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 2:container-selinux-2.19-2.1.el7.noarch 1/4
Installing : libtool-ltdl-2.4.2-22.el7_3.x86_64 2/4
Installing : libseccomp-2.3.1-2.el7.x86_64 3/4
Installing : docker-ce-17.06.2.ce-1.el7.centos.x86_64 4/4
Verifying : docker-ce-17.06.2.ce-1.el7.centos.x86_64 1/4
Verifying : libseccomp-2.3.1-2.el7.x86_64 2/4
Verifying : libtool-ltdl-2.4.2-22.el7_3.x86_64 3/4
Verifying : 2:container-selinux-2.19-2.1.el7.noarch 4/4

Installed:
docker-ce.x86_64 0:17.06.2.ce-1.el7.centos

Dependency Installed:
container-selinux.noarch 2:2.19-2.1.el7 libseccomp.x86_64 0:2.3.1-2.el7
libtool-ltdl.x86_64 0:2.4.2-22.el7_3

Complete!

Install Kubernetes components

sudo cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet-${KUBERNETES_VERSION} kubeadm-${KUBERNETES_VERSION} kubernetes-cni
Loaded plugins: fastestmirror
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
Userid : "Google Cloud Packages Automatic Signing Key [email protected]"
Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Loading mirror speeds from cached hostfile

• base: centosmirror.netcup.net
• extras: ftp.plusline.de
• updates: mirror.fra10.de.leaseweb.net
  Resolving Dependencies
  --> Running transaction check
  ---> Package kubeadm.x86_64 0:1.7.5-0 will be installed
  --> Processing Dependency: kubectl >= 1.6.0 for package: kubeadm-1.7.5-0.x86_64
  ---> Package kubelet.x86_64 0:1.7.5-0 will be installed
  --> Processing Dependency: socat for package: kubelet-1.7.5-0.x86_64
  --> Processing Dependency: ebtables for package: kubelet-1.7.5-0.x86_64
  ---> Package kubernetes-cni.x86_64 0:0.5.1-0 will be installed
  --> Running transaction check
  ---> Package ebtables.x86_64 0:2.0.10-15.el7 will be installed
  ---> Package kubectl.x86_64 0:1.7.5-0 will be installed
  ---> Package socat.x86_64 0:1.7.2.2-5.el7 will be installed
  --> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size

Installing:
kubeadm x86_64 1.7.5-0 kubernetes 8.6 M
kubelet x86_64 1.7.5-0 kubernetes 17 M
kubernetes-cni x86_64 0.5.1-0 kubernetes 7.4 M
Installing for dependencies:
ebtables x86_64 2.0.10-15.el7 base 123 k
kubectl x86_64 1.7.5-0 kubernetes 8.9 M
socat x86_64 1.7.2.2-5.el7 base 255 k

Transaction Summary

Install 3 Packages (+3 Dependent packages)

Total download size: 42 M
Installed size: 223 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/kubernetes/packages/02f3a7ff6e04943bd288ff302f449b600e8db3d19868dfe4308d0d902c0ba927-kubeadm-1.7.5-0.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 3e1ba8d5: NOKEY
Public key for 02f3a7ff6e04943bd288ff302f449b600e8db3d19868dfe4308d0d902c0ba927-kubeadm-1.7.5-0.x86_64.rpm is not installed

Total 12 MB/s | 42 MB 00:03
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
Userid : "Google Cloud Packages Automatic Signing Key [email protected]"
Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
From : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
Userid : "Google Cloud Packages RPM Signing Key [email protected]"
Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
From : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : socat-1.7.2.2-5.el7.x86_64 1/6
Installing : kubectl-1.7.5-0.x86_64 2/6
Installing : ebtables-2.0.10-15.el7.x86_64 3/6
Installing : kubernetes-cni-0.5.1-0.x86_64 4/6
Installing : kubelet-1.7.5-0.x86_64 5/6
Installing : kubeadm-1.7.5-0.x86_64 6/6
Verifying : ebtables-2.0.10-15.el7.x86_64 1/6
Verifying : kubeadm-1.7.5-0.x86_64 2/6
Verifying : kubectl-1.7.5-0.x86_64 3/6
Verifying : kubernetes-cni-0.5.1-0.x86_64 4/6
Verifying : socat-1.7.2.2-5.el7.x86_64 5/6
Verifying : kubelet-1.7.5-0.x86_64 6/6

Installed:
kubeadm.x86_64 0:1.7.5-0 kubelet.x86_64 0:1.7.5-0
kubernetes-cni.x86_64 0:0.5.1-0

Dependency Installed:
ebtables.x86_64 0:2.0.10-15.el7 kubectl.x86_64 0:1.7.5-0
socat.x86_64 0:1.7.2.2-5.el7

Complete!

Fix kubelet configuration

sed -i 's/--cgroup-driver=systemd/--cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -i '/Environment="KUBELET_CGROUP_ARGS/i Environment="KUBELET_CLOUD_ARGS=--cloud-provider=aws"' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
sed -i 's/$KUBELET_CGROUP_ARGS/$KUBELET_CLOUD_ARGS $KUBELET_CGROUP_ARGS/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Start services

systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
systemctl start docker
systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
systemctl start kubelet

Set settings needed by Docker

sysctl net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-iptables = 1
sysctl net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-ip6tables = 1

Initialize the master

cat >/tmp/kubeadm.yaml <<EOF

apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
token: ${KUBEADM_TOKEN}
cloudProvider: aws
kubernetesVersion: v${KUBERNETES_VERSION}
apiServerCertSANs:

• ${DNS_NAME}
• ${IP_ADDRESS}
  EOF

kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml", assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
kubeadm init --config /tmp/kubeadm.yaml
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.5
[init] Using Authorization modes: [Node RBAC]
[init] WARNING: For cloudprovider integrations to work --cloud-provider must be set for all kubelets in the cluster.
(/etc/systemd/system/kubelet.service.d/10-kubeadm.conf should be edited for this purpose)
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 1.12
[preflight] WARNING: hostname "ip-10-0-0-86" could not be reached
[preflight] WARNING: hostname "ip-10-0-0-86" lookup ip-10-0-0-86 on 10.0.0.2:53: no such host
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [ip-10-0-0-86 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local aws-minikube-1.kube.public] and IPs [35.156.106.81 10.96.0.1 10.0.0.86]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 46.514804 seconds

[scholzj]

These warnings are strange:

[preflight] WARNING: hostname "ip-10-0-0-86" could not be reached
[preflight] WARNING: hostname "ip-10-0-0-86" lookup ip-10-0-0-86 on 10.0.0.2:53: no such host

I wonder what caused them. Do you have DNS enabled in your VPC?

[kolov]

Edit VPC DNS Resolution: yes
Edit VPC DNS Hostnames: yes

[scholzj]

And the DHCP options set?

[scholzj]

Could you run hostname -f on your host? It seems to me that the difference is that my (working) setup had something like ip-10-0-0-69.ec2.internal as the FQDN hostname. Your seems to have only ip-10-0-0-86. I'm not sure whether it is caused by the DHCP and DNS stuff. Might be also some other network stuff. But i think this causes the problem.

If you check the log from the log from the kubelet service it will probably contain some complaints that it is unauthorized to join the cluster because of the different hostnames.

[kolov]

Everything works fine, thank you very much for the help! After setting VPC DNS Hostnames to yes, I ran terraform apply again, and then I didn't wait long enough, it takes some time indeed. Maybe it would help to some checklist with the network settings.

[scholzj]

Valid point. I updated the README a bit. Bt debugging this from is not always easy.

Would it be easier for you if the tooling created the VPC, subnets and the stuff around it for you (and thus made sure that the network settings etc. are ok)? Or do you thing it is better as it is when you deploy it into existing subnet?