balchua/do-microk8s

Bootstrap MicroK8s cluster on DigitalOcean with Terraform

DigitalOcean Terraform MicroK8s

Verfied using terraform v1.2.6

Does not work when modifying the module after it is created. Adding a new node now works

Warning Reducing nodes still does not leave the cluster

Support for worker only node, which means it will not run control plane components such as the api-server, scheduler and controller manager, available from MicroK8s v1.22

Bootstrap a Highly Available MicroK8s cluster in DigitalOcean with Terraform.

For example to bootstrap a 3 control plane nodes and 2 worker nodes cluster.

module "microk8s" {
  source                       = "../"
  cluster_name                 = "hades"
  node_count                   = "3"
  worker_node_count            = "2"
  os_image                     = "ubuntu-20-04-x64"
  node_size                    = "s-2vcpu-4gb"
  worker_node_size             = "s-4vcpu-8gb"
  node_disksize                = "30"
  region                       = "sgp1"
  dns_zone                     = "geeks.sg"
  microk8s_channel             = "latest/stable"
  cluster_token_ttl_seconds    = 3600
  digitalocean_ssh_fingerprint = var.digitalocean_ssh_fingerprint
  digitalocean_private_key     = var.digitalocean_private_key
  digitalocean_token           = var.digitalocean_token
  digitalocean_pub_key         = var.digitalocean_pub_key
}

Fields	Description	Default values
source	The source of the terraform module	none
node_count	The number of MicroK8s nodes to create	3
os_image	DigitalOcean OS images. To get the list OS images doctl compute image list-distribution	ubuntu-20-04-x64
node_size	DigitalOcean droptlet sizes To get the list of droplet sizes doctl compute size list	s-4vcpu-8gb
node_disksize	Additional volume to add to the droplet. Size in GB	50
region	DigitalOcean region To get the list of regions doctl compute region list	sgp1
dns_zone	The DNS zone representing your site. Need to register your domain.	geeks.sg
microk8s_channel	Specify the MicroK8s channel to use. Refer here	stable
cluster_token_ttl_seconds	How long the token validity (in seconds)	3600
digitalocean_ssh_fingerprint	Your DigitalOcean SSH fingerprint to use, so you can seemlessly ssh into your nodes	Refer to TF environment variables
digitalocean_private_key	The private key location to use when connecting to your droplets	Refer to TF environment variables
digitalocean_token	Your DigitalOcean token	Refer to TF environment variables
digitalocean_pub_key	The public key to use to connect to the droplet	Refer to TF environment variables
worker_node_size	The worker node size example: s-4vcpu-8gb	s-4vcpu-8gb
worker_node_count	The number of MicroK8s worker nodes	2
worker_node_disksize	Additional volume to add to the droplet. Size in GB	100

DigitalOcean TF environment variables

You must have these environment variables present.

TF_VAR_digitalocean_token=<your DO access token>
TF_VAR_digitalocean_ssh_fingerprint=<Your private key fingerprint>
TF_VAR_digitalocean_private_key=<location of your private key>
TF_VAR_digitalocean_pub_key=<location of your public key>

Creating the cluster

Simply run the terraform plan and then terraform apply

Once terraform completes, you should be able to see the cluster.

Login to one of the node using ssh root@node, then issue the command below.

root@microk8s-node-cetacean-0:~# microk8s kubectl get no
NAME                       STATUS   ROLES    AGE     VERSION
microk8s-node-cetacean-0   Ready    <none>   6m36s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-2   Ready    <none>   4m33s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-6   Ready    <none>   4m47s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-5   Ready    <none>   4m21s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-4   Ready    <none>   4m15s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-3   Ready    <none>   4m12s   v1.20.4-38+85035ca77e3c6e
microk8s-node-cetacean-1   Ready    <none>   4m6s    v1.20.4-38+85035ca77e3c6e

Downloading Kube config file

The module automatically downloads the kubeconfig file to your local machine in /tmp/client.config In order to access the Kubernetes cluster from your local machine, simple do export KUBECONFIG=/tmp/client.config

This will connect using the load balancer fronting the api servers. The dns entry will be microk8s-cluster.<domain name>

Example: microk8s-cluster.geeks.sg

MicroK8s High Availability

It requires node counts to be greater than or equal to 3 to form a majority. Each node can be a control plane, hence there is really no concept of control plane.

Check documentation on MicroK8s Discuss HA

Digitalocean attached volume

This terraform also creates and attach a volume to each droplet. This will let you setup Rook + Ceph storage. This way you can freely create volumes that you can share to your pods.

Persistent Volumes

The following sections describes how to install Rook/Ceph, Longhorn and OpenEBS with MicroK8s

Using Rook / Ceph

Some instructions on how to use Rook

Using Longhorn storage

Some instructions on how to use Longhorn

Using OpenEBS

Instructions on how to install OpenEBS

Worker node only node

If you want to create a worker node only node, refer to these instructions

do-microk8s's Issues

Longhorn volumes path /tmp/longhorn doesn't persist across reboots

Hi,
Following this guide with installing longhorn I experienced something weird.
From this doc https://github.com/balchua/do-microk8s/blob/master/docs/longhorn.md
The following has a caveat

  --set defaultSettings.defaultDataPath="/tmp/longhorn" \
  --set csi.kubeletRootDir="/var/snap/microk8s/common/var/lib/kubelet"

Specially the flag --set defaultSettings.defaultDataPath="/tmp/longhorn"

It's a bit dangerous if you want to do something serious with the cluster.

I ran this in an ubuntu server and everything went smoothly until I restarted 1 node and the node became unavailable in longhorn. I had to rebuild the node to fix it. Then happened to a second node. Then I decided to restart all nodes just for testing and voila, lost all my volumes.

Then I realized longhorn is saving the volumes in /tmp/longhorn. But ubuntu after restart clears everything in /tmp
I might submit a Pull Request with a fix with a different path. So it doesn't happen to anyone else.
I solved it by using /longhorn in that flag.

Greate but missing open-iscsi install

This is not a bug; however the documentations [longhorn.md] missing :

sudo apt install open-iscsi

And without it; the system will fail to run.

Thanks so much for the efforts.

Add support for worker only nodes

feature: Add support for worker only nodes.