briandenicola/kubernetes-cluster-setup

An opiniated method to create a private AKS cluster/app environment.

Introduction

This repository is covers how to stand up a secure, private AKS/Kubernetes cluster. This is not intended to be an AKS 101 or cover all possible secure AKS designs (for example using HTTP Proxy instead of Azure Firewall). Azure has plenty of awesome documentation and guides that go into depth on the overall design and best practices.

This code is an opiniated method of applying the standards into an end to end solution using Terraform, GitHub Actions, Flux and Istio.

Detail Deployment Guide

Follow this guide to stand up a cluster in your environment

Resources Created

Every kuberernetes cluster has several moving parts - even those deployed in a managed cloud environment. More so, if the cluster is to be private with egress filtering. Several resources in Azure must in place for the code to work properly. Typically these resources are stood-up by an Enterprise Platform Team following Azure's Cloud Adoption Framework so for more secure AKS builds, these are out of scope. There is an example Azure Firewall and RouteTable ARM Template that can elp with some of the prereqs.

Infrastructure

Log Analytics Workspace	Application Insights
KeyVault (Private endpoint)	TLS Certificate: *.bjdazure.tech
	KeyVault Secret: OpenTelemetry/Zipkin Configuration
Managed User identities	AKS Cluster Identity
	AKS Kubelet Identity
	Istio Pod Identity
Private AKS cluster	Add-Ons: OMS Policy Agent, KeyVault CSI Driver, Azure Pod Identity v1, FluxCDv2
Azure Internal Load Balancer

Diagram

Kubernetes Resources

Flux	Istio
Istio Operator	Ingress Gateway with TLS
Istio Service Mesh	Egress Gateway
Open Telemetry Collector	Distributive Tracing forwarded to Zipkin/Otel Collector
Kured Reboot Daemon	Catch-all Virtual Service
Dapr Distributive Runtime
Keda Autoscaler
Azure Monitor configuration update for Prometheus
Istio Bookinfo
eShopOnDapr

Diagram

kubernetes-cluster-setup's Issues

Rename ingress identity to Service Mesh pod identity

Rename cluster as well.

grep -ir bjdk8s-istio *
apps/istio/ingress/deployment.yaml: aadpodidbinding: bjdk8s-istio-ingress-identity
apps/istio/ingress/ingress.yaml: keyvaultName: bjdk8s-istio-kv
infrastructure/istio.tfvars:cluster_name = "bjdk8s-istio"

grep -ir bjdk8s-osm *
apps/bookstore/tls.yaml: keyvaultName: bjdk8s-osm-kv
apps/bookstore/bookstore.yaml: aadpodidbinding: bjdk8s-osm-ingress-identity
apps/bookstore/identity.yaml: name: bjdk8s-osm-ingress-identity
apps/bookstore/identity.yaml: resourceID: /subscriptions/bfafbd89-a2a3-43a5-af72-fb4ef0c514c1/resourcegroups/DevSub02_K8S_OSM_RG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/bjdk8s-osm-ingress-identity
apps/bookstore/identity.yaml: name: bjdk8s-osm-ingress-identity-binding
apps/bookstore/identity.yaml: azureIdentity: bjdk8s-osm-ingress-identity
apps/bookstore/identity.yaml: selector: bjdk8s-osm-ingress-identity
infrastructure/osm.tfvars:cluster_name = "bjdk8s-osm"

Migrate to Overlay and API Sever v2

Integrate the following into the TF module:
https://learn.microsoft.com/en-us/azure/aks/azure-cni-overlay
https://learn.microsoft.com/en-US/azure/aks/api-server-vnet-integration

Add Zipkin and AppInsights to deployment

IstioOperator is not recognized during first run

null_resource.flux_setup_bf1e8068f: Still creating... [20s elapsed]
null_resource.flux_setup_bf1e8068f (local-exec): error: unable to recognize "/tmp/flux-bootstrap-2649416710/cluster-manifests/uat/flux-system": no matches for kind "IstioOperator" in version "install.istio.io/v1alpha1"

gitrepository/flux-system Falseauth secret error: Secret "flux-system" not found

Create Network Policies/Ingress Examples for OSM

https://release-v0-11.docs.openservicemesh.io/docs/demos/ingress_contour/
kubectl annotate service httpbin -n httpbin projectcontour.io/upstream-protocol.tls='14001' --overwrite

Move HelmRepository charts to OCI

Improve app on boarding and pod identity

OSM with HTTPS Ingress and mtls between services

Migrate to Managed Prometheus/Grafana

• Disable local authentication for Log Analytics / Container Insights based off - https://github.com/briandenicola/kubernetes/tree/main/aks/managed-prometheus

Deploy Open Service Mesh sample application

Overlay Network and VNet Integration

3.40 provider for Terraform has support for Overlay networking and VNet Integration. This issue is to track the adoption of those technologies into this repo.

OSM is in crashbackoff loop

Add Support for NodeOSUpgrades

https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-image#using-node-os-auto-upgrade

requires azapi for now on terraform

Flux does not bootstrap correctly via Github Actions

Migrate off Pod Identities to Workload Identities

Workload Identities is the direction forward for identity and authorization with AKS.

This repository currently uses Pod Identities to authenticate with Key Vault for Istio Service Mesh. This needs to be ported to Workload Identities

Updated resources include: Zipkin Key Vault CSI configuration , Github Actions workflows, Istio AAD Pod Label and

Update HTTP Credentials to Service Principal

Need method to patch OSM Contour configuration to use Azure internal load balancer

Need method to patch OSM Contour configuration to use Azure internal load balancer
Inject into Service annotation:

• service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Originally posted by @briandenicola in #8 (comment)

Update AKS Deployment

Update OS SKU type