ibuchh/aws-eks-karpenter-terraform-cloud

Configure Cluster AutoScaling using Karpenter on Amazon EKS with Hashicorp Terraform

Architecture

Prerequisites

Before you build the whole infrastructure you will need to meet the following pre-requisites.

AWS account

Ensure you have access to an AWS account, and a set of credentials with Administrator permissions. Note: In a production environment we would recommend locking permissions down to the bare minimum needed to operate the pipeline.

Upgrade awscli

To ensure you are running the latest version of AWS CLI, run the following command:

aws --version
pip install awscli --upgrade --user

Run aws configure to configure your region. Leave all the other fields blank. You should have something like:

aws configure --profile aws-karpenter
AWS Access Key ID [None]: xxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxxxx
Default region name [None]: us-east-1
Default output format [None]: 

export AWS_DEFAULT_PROFILE=aws-karpenter

Install Terraform

Download and install Terraform:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

Verify that you can run Terraform:

terraform version

Install kubectl

For detailed installation instructions, follow the documentation here.

Install helm

For detailed installation instructions, follow the documentation here.

Installing aws-iam-authenticator

For detailed installation instructions, follow the documentation here

Clone the repository

Clone the source code repository:

cd ~/environment
git clone https://github.com/ibuchh/aws-eks-karpenter-terraform-cloud.git

Build the infrastructure and Amazon EKS with Karpenter

We shall use Terraform to build the above architecture including the Amazon EKS.

Note: This infrastructure will create chargeable resources in your account. When finished, please make sure you clean up resources as instructed at the end.

Open env.sh file and set the value of the AW profile.

export profile="xxxxxx"

Open a terminal and run the following commands.

cd aws-eks-karpenter-terraform-cloud/terraform
terraform init
./apply.sh

export AWS_DEFAULT_PROFILE=xxxxxx

kubectl get nodes
kubectl get pods -A

Execute only if this is the first time using spot instances in this account

aws iam create-service-linked-role \
    --aws-service-name spot.amazonaws.com

Create Karpenter provisioner

kubectl apply \
    --filename ../provisioner/provisioner.yaml

kubectl get pods,nodes

kubectl --namespace karpenter logs \
    --selector karpenter=controller

Create a deployment

kubectl apply -f ../deployment/deployment.yaml

kubectl get pods,nodes

Open three terminals

Terminal#1
export AWS_DEFAULT_PROFILE=xxxxxx
watch -t -n 1 'kubectl get pods'

Terminal#2
export AWS_DEFAULT_PROFILE=xxxxxx
watch -t -n 1 kubectl get no -Lnode.kubernetes.io/instance-type -Ltopology.kubernetes.io/zone -Lkubernetes.io/arch

Terminal#3
export AWS_DEFAULT_PROFILE=xxxxxx
kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)

Scale up the deployment

kubectl scale deployment inflate --replicas 10

Scale down the deployment

kubectl scale deployment inflate --replicas 0

Destroy the infrastcructure

cd aws-eks-karpenter-terraform-cloud/terraform
./destroy.sh