mmumshad/kubernetes-the-hard-way

Bootstrap Kubernetes the hard way on Vagrant on Local Machine. No scripts.

Kubernetes The Hard Way

Updated: March 2024

This tutorial walks you through setting up Kubernetes the hard way on a local machine using a hypervisor. This guide is not for people looking for a fully automated command to bring up a Kubernetes cluster. If that's you then check out Google Kubernetes Engine, or the Getting Started Guides.

Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster. Note that the cluster when built will not be accessible from your laptop browser - that isn't what this is about. If you want a more useable cluster, try one of these.

This tutorial is a modified version of the original developed by Kelsey Hightower. While the original one uses GCP as the platform to deploy kubernetes, we use a hypervisor to deploy a cluster on a local machine. If you prefer the cloud version, refer to the original one here

The results of this tutorial should not be viewed as production ready, and may receive limited support from the community, but don't let that stop you from learning!
Note that we are only building 2 controlplane nodes here instead of the recommended 3 that etcd requires to maintain quorum. This is to save on resources, and simply to show how to load balance across more than one controlplane node.

Before shouting "Help! It's not working!"

Please note that with this particular challenge, it is all about the minute detail. If you miss one tiny step anywhere along the way, it's going to break!

Note also that in developing this lab, it has been tested many many times! Once you have the VMs up and you start to build the cluster, if at any point something isn't working it is 99.9999% likely to be because you missed something, not a bug in the lab!

Always run the cert_verify.sh script at the places it suggests, and always ensure you are on the correct node when you do stuff. If cert_verify.sh shows anything in red, then you have made an error in a previous step. For the controlplane node checks, run the check on controlplane01 and on controlplane02

Target Audience

The target audience for this tutorial is someone planning to support a production Kubernetes cluster and wants to understand how everything fits together.

Cluster Details

Kubernetes The Hard Way guides you through bootstrapping a highly available Kubernetes cluster with end-to-end encryption between components and RBAC authentication.

• Kubernetes Latest version
• Container Runtime Latest version
• Weave Networking
• etcd v3.5.9
• CoreDNS v1.9.4

Node configuration

We will be building the following:

• Two control plane nodes (controlplane01 and controlplane02) running the control plane components as operating system services. This is not a kubeadm cluster as you are used to if you have been doing the CKA course. The control planes are not themselves nodes, therefore will not show with kubectl get nodes.
• Two worker nodes (node01 and node02)
• One loadbalancer VM running HAProxy to balance requests between the two API servers and provide the endpoint for your KUBECONFIG.

Getting Started

• If you are using Windows or Intel Mac, start here to deploy VirtualBox and Vagrant.
• If you are using Apple Silicon Mac (M1/M2/M3), start here to deploy Multipass.

kubernetes-the-hard-way's Issues

kubectl asks for password

I'm in the middle of the guide and setting up workers, came across this:

[root@master1 ~]# kubectl get componentstatuses --kubeconfig admin.kubeconfig
Please enter Username:

[root@node1 ~]# kubectl version --kubeconfig admin.kubeconfig
Please enter Username:

This is the version on all of my nodes:

[root@master1 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

[root@master1 ~]# etcdctl --version
etcdctl version: 3.3.18
API version: 2

I'm not sure if it's happening because of the kubectl and other components version, but everything on the master is 15.3.

Please let me know on what might be causing this since I don't seem to point it out yet.

Error while running vagrant up

While running the vagrant up command I get the following error:

The box failed to unpackage properly. Please verify that the box
file you're trying to add is not corrupted and that enough disk space
is available and then try again.
The output from attempting to unpackage (if any):

x box.ovf
x Vagrantfile
x metadata.json
x ubuntu-bionic-18.04-cloudimg.mf
x ubuntu-bionic-18.04-cloudimg.vmdk: Truncated tar archive
bsdtar: Error exit delayed from previous errors.

I tried running this on MacOS Mojave 10.14.6 and on Ubuntu 18.04 EC2 instance, I am getting this error on Mac. Please help I am not able to proceed further due to this.

Issues from Slack Channel

This issue is to track issues raised in the Slack Channel

Hello @mmumshad I'm taking the CKA course on Udemy and I noticed in question 1 of the `Section 9: Networking` and `133. Practice Test - Explore CNI Weave - 2` the correct answer `4` is not included in the options. All options presented are incorrect as well. Thought to bring it to your attention.

Hello @mmumshad I'm taking the CKA course on Udemy and I noticed in question 1 of the Section 9: Networking and 133. Practice Test - Explore CNI Weave - 2 the correct answer 4 is not included in the options. All options presented are incorrect as well. Thought to bring it to your attention.

https://kodekloud.slack.com/archives/CHMV3P9NV/p1561000684189000?thread_ts=1561000684.189000&cid=CHMV3P9NV

Can I get PDF I want to print and study

Can I get PDF I want to print and study

https://kodekloud.slack.com/archives/CDR9R5QRG/p1559908228025800

name of the loadbalancer machine in the master-1's hosts file is lb, but we use the name loadbalancer while connecting to it from vagrant

So, the command that we use to connect to the loadbalancer machine using vagrant is
vagrant ssh loadbalancer
but when we try to connect to this machine through ssh from machine master-1

vagrant@master-1:~$ ssh loadbalancer

it gives us below error

ssh: Could not resolve hostname loadbalancer: Temporary failure in name resolution

and the reason is that in /etc/hosts file the name of the host is lb, i think we should get that changed to loadbalancer to avoid confusion.

@mmumshad Scheduling presentation PDF is missing labels & Selectors and Resource Limit section

@mmumshad Scheduling presentation PDF is missing labels & Selectors and Resource Limit section

https://kodekloud.slack.com/archives/CHMV3P9NV/p1572722266026200

Error message for Openssl certificate creation

when i try to run
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr i get the error message
as
Can't load /home/vagrant/.rnd into RNG
139958695195072:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/vagrant/.rnd

resolve.conf

resolv.conf anticipates Ubuntu OS I assume

results in error

[root@kubedev0 kubernetes]# kubectl describe nodes kubedev0 | grep resolv
  Warning  CheckLimitsForResolvConf  26m (x6 over 104m)   kubelet, kubedev0     open /run/systemd/resolve/resolv.conf: no such file or directory
  Warning  CheckLimitsForResolvConf  18m (x4 over 18m)    kubelet, kubedev0     open /run/systemd/resolve/resolv.conf: no such file or directory
  Warning  CheckLimitsForResolvConf  6m2s (x14 over 15m)  kubelet, kubedev0     open /run/systemd/resolve/resolv.conf: no such file or directory

Service networking section: Instead of kubectl, looks like kubelet used in the commands. For a beginner, it may be confusing.

Service networking section: Instead of kubectl, looks like kubelet used in the commands. For a beginner, it may be confusing.

https://kodekloud.slack.com/archives/CDR9R5QRG/p1567275383056500

Error: context deadline exceeded

vagrant@master-1:~$ sudo ETCDCTL_API=3 etcdctl member list --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.crt --cert=/etc/etcd/etcd-server.crt --key=/etc/etcd/et
cd-server.key
Error: context deadline exceeded

What could be the reason for above error ?

kube-apiserver.service unit file missing a flag

The flag --runtime-config=api/all in the kube-apiserver unit file needs to have a value:

--runtime-config=api/all=true

as per the docs https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

One step is omitted in TLS-btstrpng doc

Regarding the following doc for TLS bootstrapping practice #2:
https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/practice-questions-answers/install/bootstrap-worker-node-2/tls-bootstrap-worker-node-2.md

The final step should be copying the ca.crt file over node03 to have it peered with node01 and node02.

scp /etc/kubernetes/pki/ca.crt root@node03:/etc/kubernetes/

11-configuring-kubectl - nodes not ready !

Hey there !

I'm following courses on Kodekloud and i had an issue while doing step 11 : i did all the steps before correctly and i can see my 2 worker nodes when doing "kubectl get nodes" on master1.

Now the problem is that it doesn't seems to be able to get ready without having networks in "/etc/cni/net.d/" folder. Here is few lines from my kubelet service on "worker1" :

Feb 08 13:39:19 worker-1 kubelet[1377]: I0208 13:39:19.819484 1377 kubelet_node_status.go:70] Attempting to register node worker-1
Feb 08 13:39:19 worker-1 kubelet[1377]: I0208 13:39:19.829172 1377 kubelet_node_status.go:73] Successfully registered node worker-1
[...]
Feb 08 13:39:19 worker-1 kubelet[1377]: I0208 13:39:19.832178 1377 setters.go:535] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2020-02-08 13:39:19.832167092 +0000 UTC m=+21.550041335 LastTransitionTime:2020-02-08 13:39:19.832167092 +0000 UTC m=+21.550041335 Reason:KubeletNotReady Message:runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized}

What is strange is that i saw during the demo that you had same message about "no networks founds in /etc/cni/net.d" but it doesn't seem to prevent your kubelet from being ready.

I don't think that it has an impact on my issue but please note i'm using version 1.17.0 (to work on same version than CKA exam).

Kind regards,
SmartGuyy

Lecture Request

I have done with first attempt for CKAD and some questions like top most cpu utilized pod are very common . please add one lecture to cover that topic

09-bootstrapping-kubernetes-workers.md: prerequisites

The section The commands in this lab must be run on first worker instance: worker-1. Login to first worker instance using SSH Terminal.

worker-1 ==> should be server-1

@mmumshad another bug... Cluster maintenance node test

@mmumshad another bug... Cluster maintenance node test

https://kodekloud.slack.com/archives/CHMV3P9NV/p1562678708116500?thread_ts=1562678708.116500&cid=CHMV3P9NV

Worker Nodes (No Resource Found)

Hello,

Thank you for setting up the great trainings, I have got stuck in this for last 2 days, tried to troubleshoot but no success,

After Bootstrapping worker and master, When I do KUBECTL GET NODES , I get an error no resource found , When I do following for component get following eroor

kubectl get componentstatuses
The connection to the server 192.168.5.30:6443 was refused - did you specify the right host or port?
Where can I look for possible issue

Vagrant files to install docker on the on the nodes

after running vagrant up , docker is not installed on the nodes.
This has to be installed separately.

Error with loopback CNI missing: Provisioning networking should use latest version

(How to fix is given after the error description)

Error:

Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               13m                   default-scheduler  Successfully assigned kube-system/coredns-69cbb76ff8-q2zgf to worker-1
  Warning  FailedCreatePodSandBox  13m                   kubelet, worker-1  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "9d371fb41ba4963cea4081ace2b6590d501def894d06098c2f866241857386d1" network for pod "coredns-69cbb76ff8-q2zgf": NetworkPlugin cni failed to set up pod "coredns-69cbb76ff8-q2zgf_kube-system" network: failed to find plugin "loopback" in path [/opt/cni/bin], failed to clean up sandbox container "9d371fb41ba4963cea4081ace2b6590d501def894d06098c2f866241857386d1" network for pod "coredns-69cbb76ff8-q2zgf": NetworkPlugin cni failed to teardown pod "coredns-69cbb76ff8-q2zgf_kube-system" network: failed to find plugin "portmap" in path [/opt/cni/bin]]
  Normal   SandboxChanged          3m34s (x48 over 13m)  kubelet, worker-1  Pod sandbox changed, it will be killed and re-created.

How to fix

Instead of using version 0.7.5, we should be using the latest version:

wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz
sudo tar -xzvf cni-plugins-linux-amd64-v0.8.2.tgz --directory /opt/cni/bin/

Reason

This is because of the next command which takes the "latest" version.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

The 0.7.5 does not contains the loopback and some other stuff.

Unable to access Deployments and Namespace practice tests

Hi ,

I am unable to access deployments and Name space practice tests,pls do the needful

Thanks and Regards
Balaji

Error while copying

While i SCP the certs to the mater nodes... it errors out with
Permission denied (publickey).

Hi @mmumshad..in TLS bootstrap practice test 1... Question 5 create bootstrsp kubeconfig file in node 03,in spec it tells to use master IP,but if I give the right IP from even I referred answers u given,but it still it throws error when I click ..check... please assist

Hi @mmumshad..in TLS bootstrap practice test 1... Question 5 create bootstrsp kubeconfig file in node 03,in spec it tells to use master IP,but if I give the right IP from even I referred answers u given,but it still it throws error when I click ..check... please assist

https://kodekloud.slack.com/archives/CHMV3P9NV/p1560825135124400

Add a prompt with a server name where all steps are to be run

Some of these steps are not fully clear which server they should run on (master(s) ? worker(s) ? which worker ?).
Please add a prompt to each snippet showing where all these commands should be run on.

@mmumshad good day! Issue with practice lab for cluster is not disconnecting, but now it throws errors after upgrading cluster

@mmumshad good day!
Issue with practice lab for cluster is not disconnecting, but now it throws errors after upgrading cluster

https://kodekloud.slack.com/archives/CDR9R5QRG/p1559499506051500

Local Persistent Volume

Please add video on Local Persistent Volume.

vagrant up fails to complete the dockerce script

PS D:\Kubernetes\kubernetes-the-hard-way\vagrant> vagrant up worker-1 Bringing machine 'worker-1' up with 'virtualbox' provider...
==> worker-1: Importing base box 'ubuntu/bionic64'...
==> worker-1: Matching MAC address for NAT networking...
==> worker-1: Setting the name of the VM: kubernetes-ha-worker-1
==> worker-1: Clearing any previously set network interfaces...
==> worker-1: Preparing network interfaces based on configuration...
worker-1: Adapter 1: nat
worker-1: Adapter 2: hostonly
==> worker-1: Forwarding ports...
worker-1: 22 (guest) => 2721 (host) (adapter 1)
worker-1: 22 (guest) => 2222 (host) (adapter 1)
==> worker-1: Running 'pre-boot' VM customizations...
==> worker-1: Booting VM...
==> worker-1: Waiting for machine to boot. This may take a few minutes...
worker-1: SSH address: 127.0.0.1:2222
worker-1: SSH username: vagrant
worker-1: SSH auth method: private key
worker-1: Warning: Connection aborted. Retrying...
worker-1:
worker-1: Vagrant insecure key detected. Vagrant will automatically replace
worker-1: this with a newly generated keypair for better security.
worker-1:
worker-1: Inserting generated public key within guest...
worker-1: Removing insecure key from the guest if it's present...
worker-1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> worker-1: Machine booted and ready!
==> worker-1: Checking for guest additions in VM...
worker-1: The guest additions on this VM do not match the installed version of
worker-1: VirtualBox! In most cases this is fine, but in rare cases it can
worker-1: prevent things such as shared folders from working properly. If you see
worker-1: shared folder errors, please make sure the guest additions within the
worker-1: virtual machine match the version of VirtualBox you have installed on
worker-1: your host and reload your VM.
worker-1:
worker-1: Guest Additions Version: 5.2.34
worker-1: VirtualBox Version: 6.1
==> worker-1: Setting hostname...
==> worker-1: Configuring and enabling network interfaces...
==> worker-1: Mounting shared folders...
worker-1: /vagrant => D:/Kubernetes/kubernetes-the-hard-way/vagrant
==> worker-1: Running provisioner: setup-hosts (shell)...
worker-1: Running: C:/Users/mhs/AppData/Local/Temp/vagrant-shell20200309-13116-1wghjso.sh
==> worker-1: Running provisioner: setup-dns (shell)...
worker-1: Running: C:/Users/mhs/AppData/Local/Temp/vagrant-shell20200309-13116-1hqoaqf.sh
==> worker-1: Running provisioner: install-docker (shell)...
worker-1: Running: C:/Users/mhs/AppData/Local/Temp/vagrant-shell20200309-13116-kgm0i6.sh
worker-1: # Executing docker install script, commit: 442e66405c304fa92af8aadaa1d9b31bf4b0ad94
worker-1: +
worker-1: sh
worker-1: -c
worker-1: apt-get update -qq >/dev/null
worker-1: + sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
worker-1: + sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
worker-1: Warning: apt-key output should not be parsed (stdout is not a terminal)
worker-1: + sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" > /etc/apt/sources.list.d/docker.list
worker-1: + sh -c apt-get update -qq >/dev/null
worker-1: +
worker-1: [
worker-1: -n
worker-1: ]
worker-1: + sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
worker-1: E
worker-1: : Failed to fetch https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/containerd.io_1.2.13-1_amd64.deb Hash Sum mismatch
worker-1: Hashes of expected file:
worker-1: - SHA512:45640591cd629a357cdf4ab3b5225dd8c4ce3f7100ab35b776502224014c76db164f849f52bf044919b01a4acbb075a11aacc514d386af398c6f93ba7ef35e2e
worker-1: - SHA256:e0f9d311a77ce0729a25f41925f6b3174616f704fd21bbc7c4d91a1012f360a8
worker-1: - SHA1:4c7a98e8145fb258c24044c78f6f1f8abc1f6893 [weak]
worker-1: - MD5Sum:7eaf2d7d924a20d6fbf1ce8cd155b67e [weak]
worker-1: - Filesize:20144902 [weak]
worker-1: Hashes of received file:
worker-1: - SHA512:12b52543692c4fa534a2c496d47a1344fd84f7e2458596e20b753954d8690bfa5bc8908b4959ad69408aac964d25cfe59e74115852e1bbfee845bc6e179666e9
worker-1: - SHA256:4034d0d3b76054826f20da9ae890886c15437f10950ef9c517dccce506eb953c
worker-1: - SHA1:9970f5824f3b2305aaed246f936fa358ce692401 [weak]
worker-1: - MD5Sum:3d69bc59d3a7ce5b3a237742b140145b [weak]
worker-1: - Filesize:20144902 [weak]
worker-1: Last modification reported: Tue, 03 Mar 2020 04:23:27 +0000
worker-1: E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce-cli_19.03.7~3-0~ubuntu-bionic_amd64.deb Hash Sum mismatch
worker-1: Hashes of expected file:
worker-1: - SHA512:901541070d59c6d89fd76a19415ced4cdf8c596515bae350a9e100001107e31a6eb5937402fcfc80ef049a5bc7a6c8065ff813f7d3067229f98ee8a6e7b7623c
worker-1: - SHA256:89fd8a89cbebaecaf4169272b43d99ad3aa2f46b1224560a840c0d3726f2c913
worker-1: - SHA1:98ece988104a21eaaf548551729b25f3bd8fdbce [weak]
worker-1: - MD5Sum:d7c03603b986a503ccc327361a606a82 [weak]
worker-1: - Filesize:42545318 [weak]
worker-1: Hashes of received file:
worker-1: - SHA512:2f1fda5410752496145e72f1f3d9f564004f94a129d52d8ade9d2a63976289918016ecc26dacb117fc8ca7c6f8b0195c653374b9bbebde34b9770cd9bfacd4bd
worker-1: - SHA256:6bdcb07b73e0c75652b04be2381853605fdde807389646adeb97281e8fcd7a3d
worker-1: - SHA1:445eca66ed2d0fb611e92f87b23b40f4e6328e75 [weak]
worker-1: - MD5Sum:e2411c63fe581022988eada859d0cb5b [weak]
worker-1: - Filesize:42545318 [weak]
worker-1: Last modification reported: Wed, 04 Mar 2020 05:37:34 +0000
worker-1: E: Failed to fetch https://download.docker.com/linux/ubuntu/dists/bionic/pool/stable/amd64/docker-ce_19.03.7~3-0~ubuntu-bionic_amd64.deb Hash Sum mismatch
worker-1: Hashes of expected file:
worker-1: - SHA512:7a65ebccfd308715e7d03cb7ae4aef7843ad8335c3571b2496be07b56340a79abd3b08ec751801a39ccd1adeb8ca12694b29a5adc85b6a3fcc51fca0a84852cc
worker-1: - SHA256:b8383aa4bb685e910cfa9305f40c8a3e375843e594b4ec5eb01d0b5c83e5087f
worker-1: - SHA1:5cf229ed1b5f8314cbce13f3e560c8b4510e13ff [weak]
worker-1: - MD5Sum:97be317fec13adee6b3daf716c6add7e [weak]
worker-1: - Filesize:22871562 [weak]
worker-1: Hashes of received file:
worker-1: - SHA512:23abf050ba870861159709443dd84c11d8975005d2a5a200cf47c15f0f3040e6b6e7d7bf0b359b68348c271bbcbdd5ccb41913cd601ae81b7821fd2c4d1cf2f2
worker-1: - SHA256:7502622f4f0de02429e3899c7b9cde4ffa2caf659fe796a6b5669dd2a45e7259
worker-1: - SHA1:6e15b61ce8b33d38ba2e3e8f87f0c509a366c8b0 [weak]
worker-1: - MD5Sum:0a57c6dd0d7619c1787a3146b4b951ed [weak]
worker-1: - Filesize:22871562 [weak]
worker-1: Last modification reported: Wed, 04 Mar 2020 05:37:35 +0000
worker-1: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

I keep getting this transport errror

I keep getting this transport errror

https://kodekloud.slack.com/archives/CDR9R5QRG/p1559824194017900?thread_ts=1559824194017900

Improvement in 10-tls-bootstrapping-kubernetes-workers.md

Few places its difficult to identify where we need to run the command and after spending few hours found the right node for command execution.

Step1 to 3 need to be executed in master node. But hard to identify from current document

Can I get PDF I want to print and study

Can I get PDF I want to print and study

https://kodekloud.slack.com/archives/CDR9R5QRG/p1559908228025800?thread_ts=1559908228.025800&cid=CDR9R5QRG

@mmumshad Small suggestion. When you start the labs it says what level they are. Every lab says Difficulty: beginner. I’m now on the ingress labs and it’s still a beginner level which just makes me feel wholly and utterly incompetent. Maybe you could rename the harder ones to “ninja” level or something?!

@mmumshad Small suggestion. When you start the labs it says what level they are. Every lab says Difficulty: beginner. I’m now on the ingress labs and it’s still a beginner level which just makes me feel wholly and utterly incompetent. Maybe you could rename the harder ones to “ninja” level or something?!

https://kodekloud.slack.com/archives/CHMV3P9NV/p1565349990348800?thread_ts=1565349990.348800&cid=CHMV3P9NV

Lecture Request CKAD

Hi,
Would i be possible to cover statefulsets please on this course?
Thanks

andy

Service Discovery outside Cluster

Hi mmumshad,
over
I have configured a kubernetes cluster through kubernetes-the-hard-way and the cluster is working fine.

I am able to do a service discovery inside cluster through busybox but now able to discover service outside cluster. Can you let me know how to configure to make service discovery outside cluster?
Also exposed clusterIP but it is in pending state.

INSIDE CLUSTER

[dadmin@k8s-master-01 ~]$ kubectl exec -ti busybox -- nslookup my-service
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: my-service
Address 1: 10.96.0.163 my-service.default.svc.cluster.local
[dadmin@k8s-master-01 ~]$

OUTSIDE CLUSTER

[dadmin@k8s-master-01 ~]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 4d10h
my-service LoadBalancer 10.96.0.163 8080:31222/TCP 26m
mydb ClusterIP 10.96.0.136 80/TCP 28h
myservice ClusterIP 10.96.0.127 80/TCP 28h
nginx NodePort 10.96.0.157 80:31994/TCP 3h2m
[dadmin@k8s-master-01 ~]$ nslookup my-service
Server: 8.8.8.8
Address: 8.8.8.8#53

** server can't find my-service: NXDOMAIN

[dadmin@k8s-master-01 ~]$

@mmumshad - Q5 in practice test - os upgrades has no correct answer. Node01 was drained in the previous question so one of the answers needs to be node02,node03

@mmumshad - Q5 in practice test - os upgrades has no correct answer. Node01 was drained in the previous question so one of the answers needs to be node02,node03

https://kodekloud.slack.com/archives/CDR9R5QRG/p1562244131386100?thread_ts=1562244131.386100&cid=CDR9R5QRG

CKA Curriculum update | New Topics added

Hi Mumshad,

Now I see on https://github.com/cncf/curriculum under CKA it shows version 1.14.1 but for CKAD it shows both.

And also I see kubeadm installation to be added in course.

Can you provide more clarity on from when it will be updated and about new additions.

Thanks.

where do you have solutions of all the practice test? only first few are there.

where do you have solutions of all the practice test? only first few are there.

https://kodekloud.slack.com/archives/CHMV3P9NV/p1559161773040500

VBoxManage: error: Failed to create the host-only adapter

OS: macOs Mojave 10.14.5 (18F132)
Steps to reproduce: run vagrant up
Expected result: Environment should be ready to rumble.
Current result: Error with VboxManage
Versions:
virtualbox 6.0.8
vagrant 2.2.5

$ vagrant up
Bringing machine 'master-1' up with 'virtualbox' provider...
Bringing machine 'master-2' up with 'virtualbox' provider...
Bringing machine 'loadbalancer' up with 'virtualbox' provider...
Bringing machine 'worker-1' up with 'virtualbox' provider...
Bringing machine 'worker-2' up with 'virtualbox' provider...
==> master-1: Box 'ubuntu/bionic64' could not be found. Attempting to find and install...
    master-1: Box Provider: virtualbox
    master-1: Box Version: >= 0
==> master-1: Loading metadata for box 'ubuntu/bionic64'
    master-1: URL: https://vagrantcloud.com/ubuntu/bionic64
==> master-1: Adding box 'ubuntu/bionic64' (v20190720.0.0) for provider: virtualbox
    master-1: Downloading: https://vagrantcloud.com/ubuntu/boxes/bionic64/versions/20190720.0.0/providers/virtualbox.box
    master-1: Download redirected to host: cloud-images.ubuntu.com
==> master-1: Successfully added box 'ubuntu/bionic64' (v20190720.0.0) for 'virtualbox'!
==> master-1: Importing base box 'ubuntu/bionic64'...
==> master-1: Matching MAC address for NAT networking...
==> master-1: Setting the name of the VM: kubernetes-ha-master-1
==> master-1: Clearing any previously set network interfaces...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["hostonlyif", "create"]

Stderr: 0%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg *)" at line 94 of file VBoxManageHostonly.cpp

Mastering Kubernettes in GKE

Hi Mumshad,

As you already know, these days most of the organization are using Cloud offering for Kubernetes and GKEis a market Leader. In My organization Kubernettes means GKE. I would request can you please make a course on GKE, Exploring it various features and capabilities. There are courses available on coursera but those cover only very small part of it. I hope you would consider this request.

In case you want to contact me you can DM me on Linkedin.

@mmumshad Lecture Request

Thanks,
Mayur Aggarwal

Lecture request

Hi,

Can you please cover following topics, please? (I'm half way through the courses and hence not sure if you have already covered in remaining topics. Anyways, if it's already present please point out)

1. Node Allocatables
2. POD Eviction Policies, OOM behavior

Thanks and Regards,
Harish M

Lecture Request

I want more in depth lecture on following topics.

-Auto scaling a cluster

-Horizontal POD autoscalers

I see 4..

I see 4..

https://kodekloud.slack.com/archives/CDR9R5QRG/p1573673390099800?thread_ts=1573673390.099800&cid=CDR9R5QRG

Replace "The Hard Way" with "HA Clusters-Kubeadm way" in CKA course

Hi @mmumshad
To me is quite useless to build cluster form scratch hard way in the CKA exam.
Kubeadm tool support building HA clusters and it would be worth to create lecture and exercises building HA Cluster in Kubeadm way, since this is default tool required for CKA exam. In the real life if you gonna to build cluster most probably you wound't go hard way.

Test gitissue

Test gitissue
https://kodekloud.slack.com/archives/CDR9R5QRG/p1559841668020300?thread_ts=1559751813.012000&cid=CDR9R5QRG

"generator:RAND_load_file:Cannot open file" error during CSR creation

I ran into this generator:RAND_load_file:Cannot open file error while working on step 4 using openssl version 1.1.1.

vagrant@master-1:~$ openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr
Can't load /home/vagrant/.rnd into RNG
140386097500608:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/vagrant/.rnd

I found this issue openssl/openssl#7754 and it mentioned removing the RANDFILE= assignment from openssl.cnf.

So I sudo vim /usr/lib/ssl/openssl.cnf and removed it which seems to have resolved things. Just noting this here in case other folks are running into this!

There seems to be a mismatch in the namespace names in the question

There seems to be a mismatch in the namespace names in the question

https://kodekloud.slack.com/archives/CDR9R5QRG/p1562333042412800?thread_ts=1562333042.412800&cid=CDR9R5QRG

Explanation Mistake

On video 10, minute 7:12 of Kubernetes Certified Aplication Developer course, you mention Multi-pod containers, instead of Multi-containers Pod. as the explanation is near perfect, it might as well be worth it to fix that :)

Can I get PDF I want to print and study

Can I get PDF I want to print and study

https://kodekloud.slack.com/archives/CDR9R5QRG/p1559908228025800

Boot strap issue kube-controller & Kube-scheduler unhealthy

Boot strap issue kube-controller & Kube-scheduler getting error.

root@master-1:~/test# kubectl get componentstatuses --kubeconfig admin.kubeconfig
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}

Based on the kubernetes/kops#463 link. I have added the entry in listed below file.

--master=127.0.0.1:8080 \

/etc/systemd/system/kube-controller-manager.service /etc/systemd/system/kube-scheduler.service

The reported issue got resolved.
root@master-1:/var/lib/kubernetes# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}

Issue with ubuntu mirrors sync

Ran into this strange issue with mirrors today when setting up from scratch

I was receiving errors like these

    worker-1: Reading package lists...
    worker-1: E
    worker-1: :
    worker-1: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/universe/i18n/by-hash/SHA256/9fc21dfcc2639dc719471cdc93179e47479a193e9b8fc46e55e90c87f2ca0122  File has unexpected size (13217 != 4940820). Mirror sync in progress? [IP: 91.189.88.174 80]
    worker-1:
    worker-1: Hashes of expected file:
    worker-1:
    worker-1:  - Filesize:4940820 [weak]
    worker-1:
    worker-1:  - SHA256:9fc21dfcc2639dc719471cdc93179e47479a193e9b8fc46e55e90c87f2ca0122
    worker-1:
    worker-1:  - SHA1:4d360e4f87adc149a7832a73e367fd9717e782a8 [weak]
    worker-1:
    worker-1:  - MD5Sum:481a75f2fc2f4a02df60bcde8604a788 [weak]
    worker-1:    Release file created at: Thu, 26 Apr 2018 23:37:48 +0000
    worker-1: E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/multiverse/binary-amd64/by-hash/SHA256/6f613efa2fcdd62f9708766c7d371db57f43710105bb99f5f55fbf9ee0db6c40

The solution seems to be to use a mirror that is up to date.
The up to date status can be checked here.
https://launchpad.net/ubuntu/+archivemirrors

In my case since I am in India I found one that is up to date
https://launchpad.net/ubuntu/+mirror/repos.del.extreme-ix.org+archive

For bionic64 the relevant sources.list entries are
deb http://repos.del.extreme-ix.org/ubuntu/ bionic main
deb-src http://repos.del.extreme-ix.org/ubuntu/ bionic main

In Vagrant file so I added this line
config.vm.provision :shell, :inline => "sed -i 's/archive.ubuntu.com/repos.del.extreme-ix.org/g' /etc/apt/sources.list"

What followed was an error with security.ubuntu.com

  worker-1: Reading package lists...
   worker-1: E
# -*- mode: ruby -*-
   worker-1: :
   worker-1: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/main/binary-amd64/by-hash/SHA256/c0dffd9d6d2e0cc74bd9a533a5464c936dfff08f19ba08aabd3fc0a307e2e3ea  File has unexpected size (13247 != 592540). Mirror sync in progress? [IP: 91.189.88.174 80]
   worker-1:
   worker-1: Hashes of expected file:
   worker-1:
   worker-1:  - Filesize:592540 [weak]
   worker-1:
   worker-1:  - SHA256:c0dffd9d6d2e0cc74bd9a533a5464c936dfff08f19ba08aabd3fc0a307e2e3ea
   worker-1:

So updated the Vagrantfile with this (I don't think it is recommended though to change the security.ubuntu.com repo url)

  $script = <<-SCRIPT
    sed -i 's/archive.ubuntu.com/repos.del.extreme-ix.org/g' /etc/apt/sources.list
    sed -i 's/security.ubuntu.com/repos.del.extreme-ix.org/g' /etc/apt/sources.list
  SCRIPT
  config.vm.provision "shell", inline: $script

Now all works. Although I am not sure if this was the right fix.

HI @mmumshad In the TLS certificate creation course. There is wrong certificate used in the kube Api server config file..... Check for kubelet client certificates.... It is showing apiserver-etcd-client.crt

HI @mmumshad In the TLS certificate creation course. There is wrong certificate used in the kube Api server config file..... Check for kubelet client certificates.... It is showing apiserver-etcd-client.crt

https://kodekloud.slack.com/archives/CHMV3P9NV/p1560943687161600?thread_ts=1560943687.161600&cid=CHMV3P9NV