hivenetes/k8s-bootstrapper

Bootstrapping a Production-Ready DigitalOcean Kubernetes Cluster Using Terraform and Argo CD

Kubernetes Bootstrapper: An Extendable Framework to Set Up Production-Grade Clusters

Introduction

"Production-grade" is a loaded term. What constitutes production readiness will vary according to your use case. While a Kubernetes cluster is production-ready the moment it is ready to serve traffic, there is a commonly agreed set of minimum requirements, as shown in the infographic below.

With many cloud-native solutions popping up daily, it can be pretty daunting for engineers and organizations to choose the "right" tools to build their tech. Integrating them to get to day-2 operations ready in Kubernetes becomes all the more challenging.

Kubernetes Bootstrapper

The k8s-bootstrapper project is a customizable and extendable framework that aims to solve this problem by leveraging

• Terraform: Compose a production-ready infrastructure on DigitalOcean
• Argo CD: Application configuration management and delivery using GitOps

This project integrates popular battle-tested open-source software and solutions that provide a production-grade out-of-box Kubernetes experience.

 Target Audience

• Kubernetes adopters
• Startups/SMBs who are looking to speed up the Kubernetes adoption
• Builders and curious souls

---

 Overview

.
├── CODE_OF_CONDUCT.md
├── LICENSE
├── README.md 
├── argocd # Argo CD configurations (optional)
├── bootstrap # cluster bootstrapping using Argo CD
├── docs # documents and assets
├── infrastructure # infrastructure automation using Terraform
└── observability # set up observability stack 

 Bootstrap the Kubernetes Cluster

Follow the guide in the order stated below:

1. DigitalOcean Infrastructure Automation via Terraform
2. Bootstrapping using Argo CD
3. Set up Observability using Robusta

Here is the overview of the architecture that we will set up:

 Note: Intended for experimentation/evaluation only. You will be responsible for all infrastructure costs incurred by the used resources.

 Tear down the Bootstrapped Cluster

Had fun trying out the k8s-bootstrapper? Time to say goodbye!

# Run
cd infrastructure/terraform
terraform destroy --var-file=bootstrapper.tfvars --auto-approve
# Bye Bye!

---

Demos

k8s-bootstrapper in action

• DOKS Application Log Monitoring and Retention using Grafana Loki and DigitalOcean Spaces

 Media

• YouTube: Project Overview
• Blog: Build your K8s framework
• Workshop: DigitalOcean Deploy Conference 2022
• Featured in akuity/awesome-argo

 Shout-out

This project takes inspiration from the fantastic work done by Alex Jones of Cloud Native Skunkworks.

k8s-bootstrapper's Issues

fix: remove redundant github actions

If ArgoCD is disabled, the outputs will fail

Error:

│ Error: Invalid index
│
│   on outputs.tf line 39, in output "argocd_helm_chart_values":
│   39:   value = helm_release.argocd[0].values
│     ├────────────────
│     │ helm_release.argocd is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│
│   on outputs.tf line 43, in output "argocd_helm_chart_manifest":
│   43:   value = helm_release.argocd[0].manifest
│     ├────────────────
│     │ helm_release.argocd is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.

Also the comment saying that ArgoCD is disabled by default since the chart doesn't work isn't true.

Argo CD managing Argo CD

k8s-bootstrapper/infrastructure/terraform/argocd-helm-config.tf

Line 1 in 9f87f19

resource "helm_release" "argocd" {

Using Terraform for the initial release of Argo CD is great, but in the long term, it's best to have an Application in Argo CD that's used for managing Argo CD itself.

https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#manage-argo-cd-using-argo-cd

Container registry name should be unique

If two people create the container registry, the second person will have a failure since the registry names need to be unique. I'll fix this later.

Use `ServerSideApply` instead of `Replace` for Kyverno

I am guessing that one of the Kyverno CRDs was too big to fit into kubectl.kubernetes.io/last-applied-configuration annotation that is added by kubectl apply, causing the Application sync to fail, leading to the use of Replace: true. This sync option has the potential to be destructive and might lead to resources having to be recreated, which could cause an outage.

k8s-bootstrapper/bootstrap/templates/application_kyverno.yaml

Line 22 in 9f87f19

- Replace=true

There's support for ServerSideApply now, which addresses the original issue and removes the potentially destructive syncs. Try using this instead:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  ...
  syncPolicy:
    syncOptions:
    - ServerSideApply=true

Install ArgoCD through Terraform

• Cleanup Terraform modules and organise the structure
• Install ArgoCD through helm provider

feat: single sign-on using github

Access argocd via FQDN

• Make sure to set up cert-manager, let's encrypt and traefik controller