A method of creating a private AKS cluster with Egress filtering using Terraforms and the Flux gitOps operator. The cluster will be setup with either Open Service Mesh or Istio with couple demo applications deployed.
- Private AKS Cluster with Azure AD Pod Identity, KeyVault CSI Driver and OpenService Mesh extensions
- Jumpbox VM
- KeyVault
- Private Zones for AKS and Keyvault
- Virtual Network with subnets
- kubernetes
- private-endpoint
- servers
- AzureBastionSubnet
- Azure Container Repostiory
- Azure Blob Storage - Terraform state storage
- Azure Bastion - to access jumpbox VM
- Azure Firewall with proper network and application rules
- Follow this example of using AKS with Azure Firewall using Terraforms
- A Route Table with a route 0.0.0.0/0 to the Azure Firewall internal IP Address
-
A task runner deployed in the virtual network where the AKS cluster will be deployed.
-
The task runnre VM need to have a User Managed Identity assigned
-
Update infrastructure/uat.tfvars with correct values
-
Create the follow Secrets in GitHub:
Secret Name Secret Name ARM_CLIENT_ID ARM_CLIENT_SECRET ARM_SUBSCRIPTION_ID ARM_TENANT_ID STORAGE_ACCESS_KEY PAT_TOKEN
- Trigger Github Action to create the cluster.
- Terraform will the call the aks-post-creation-configuration.sh script to add Pod Identity and KeyVault CSI Driver
- GitHub Actions pipeline will then call the aks-flux-configuration.sh script to confiugre flux and execute the GitOps flow
- Update infrastructure/osm|istio.tfvars with correct values for your environment
- az extension add --name aks-preview
- az extension update --name aks-preview
- az login
- az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureKeyVaultSecretsProvider"
- az feature register --namespace "Microsoft.ContainerService" --name "EnablePodIdentityPreview"
- az feature register --namespace "Microsoft.ContainerService" --name "AKS-OpenServiceMesh"
- az feature register --namespace "Microsoft.ContainerService" --name "DisableLocalAccountsPreview"
- az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService')].{Name:name,State:properties.state}"
- Wait till the above features are enabled
- az provider register --namespace Microsoft.ContainerService
- cd infrastructure
- terraform init -backend=true -backend-config="access_key=${access_key}" -backend-config="production.terraform.tfstate"
- terraform plan -out="production.plan" -var "resource_group_name=DevSub_K8S_RG" -var-file="{osm|istio}.tfvars"
- terraform apply -auto-approve "production.plan"
- Access the Jump VM through Azure Bastion
- export GITHUB_TOKEN=${PAT_TOKEN_FOR_YOUR_REPO}
- curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
- curl -s https://fluxcd.io/install.sh | sudo bash
- az login --identity
- az aks install-cli
- az aks get-credentials -n ${CLUSTER_NAME} -g ${CLUSTER_RESOURCE_GROUP}
- kubelogin convert-kubeconfig -l msi
- echo -n ${ACR_NAME} > ./username.txt
- az acr credential show -n ${ACR_NAME} --query "passwords[0].value" -o tsv | tr -d '\n' > password.txt
- kubectl -n flux-system create secret generic https-credentials --from-file=username=./username.txt --from-file=password=./password.txt
- flux bootstrap git --url=ssh://[email protected]/${user}/kubernetes-cluster-setup --branch=master --path=./cluster-manifests/uat --personal=true --private=false