Introduction
A method of creating a private AKS cluster with Egress filtering using Terraforms and the Flux gitOps operator. The cluster will be setup with either Open Service Mesh or Istio with couple demo applications deployed.
Azure Resources Created
- Private AKS Cluster with Azure AD Pod Identity, KeyVault CSI Driver and OpenService Mesh extensions
- Jumpbox VM
- KeyVault
- Private Zones for AKS and Keyvault
Required Existing Azure Resources
- Virtual Network with subnets
- kubernetes
- private-endpoint
- servers
- AzureBastionSubnet
- Azure Container Repostiory
- Azure Blob Storage - Terraform state storage
- Azure Bastion - to access jumpbox VM
- Azure Firewall with proper network and application rules
- Follow this example of using AKS with Azure Firewall using Terraforms
- A Route Table with a route 0.0.0.0/0 to the Azure Firewall internal IP Address
GitHub Actions
Prerequisites
-
A task runner deployed in the virtual network where the AKS cluster will be deployed.
-
The task runnre VM need to have a User Managed Identity assigned
-
Update infrastructure/uat.tfvars with correct values
-
Create the follow Secrets in GitHub:
Secret Name Secret Name ARM_CLIENT_ID ARM_CLIENT_SECRET ARM_SUBSCRIPTION_ID ARM_TENANT_ID STORAGE_ACCESS_KEY PAT_TOKEN
Steps
- Trigger Github Action to create the cluster.
- Terraform will the call the aks-post-creation-configuration.sh script to add Pod Identity and KeyVault CSI Driver
- GitHub Actions pipeline will then call the aks-flux-configuration.sh script to confiugre flux and execute the GitOps flow
Manual Setup
Prerequisites
- Update infrastructure/osm|istio.tfvars with correct values for your environment
Cluster Creation
- az extension add --name aks-preview
- az extension update --name aks-preview
- az login
- az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureKeyVaultSecretsProvider"
- az feature register --namespace "Microsoft.ContainerService" --name "EnablePodIdentityPreview"
- az feature register --namespace "Microsoft.ContainerService" --name "AKS-OpenServiceMesh"
- az feature register --namespace "Microsoft.ContainerService" --name "DisableLocalAccountsPreview"
- az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService')].{Name:name,State:properties.state}"
- Wait till the above features are enabled
- az provider register --namespace Microsoft.ContainerService
- cd infrastructure
- terraform init -backend=true -backend-config="access_key=${access_key}" -backend-config="production.terraform.tfstate"
- terraform plan -out="production.plan" -var "resource_group_name=DevSub_K8S_RG" -var-file="{osm|istio}.tfvars"
- terraform apply -auto-approve "production.plan"
GitOps BootStrap
- Access the Jump VM through Azure Bastion
- export GITHUB_TOKEN=${PAT_TOKEN_FOR_YOUR_REPO}
- curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
- curl -s https://fluxcd.io/install.sh | sudo bash
- az login --identity
- az aks install-cli
- az aks get-credentials -n ${CLUSTER_NAME} -g ${CLUSTER_RESOURCE_GROUP}
- kubelogin convert-kubeconfig -l msi
- echo -n ${ACR_NAME} > ./username.txt
- az acr credential show -n ${ACR_NAME} --query "passwords[0].value" -o tsv | tr -d '\n' > password.txt
- kubectl -n flux-system create secret generic https-credentials --from-file=username=./username.txt --from-file=password=./password.txt
- flux bootstrap git --url=ssh://[email protected]/${user}/kubernetes-cluster-setup --branch=master --path=./cluster-manifests/uat --personal=true --private=false