marcincuber / eks Goto Github PK

AWS EKS - kubernetes project and terraform module

License: Apache License 2.0

HCL 90.03% Shell 0.31% Python 7.95% Makefile 1.15% Smarty 0.56%

eks aws aws-eks kubernetes oidc terraform ec2-spot lambda aws-vpc-cni cluster-autoscaler

eks's Introduction

EKS

Implementation of EKS setup using Terraform. Terraform module located in terraform directory supports deployment to different AWS partitions. I have tested it with commercial and china partitions. I am actively using this configuration to run EKS setup in Ireland(eu-west-1), London(eu-west-2), North Virginia(us-east-1) and Beijing(cn-north-1).

Module details

Module creates:

VPC
VPC Endpoints- S3, ECR, STS, APS, GuardDuty
EKS Cluster
EKS Node Group to run cluster critical services
EKS Addons- coredns, kube-proxy, guardduty, aws-ebs-csi-driver, adot (requires cert-manger to be installed), kubecost, cloudwatch observability, snapshot-controller and identity agent
IAM Roles for worker nodes and Karpenter nodes
Additional IAM Roles for operators- load-balancer-controller, external-dns, cert-manager, adot-collector
SQS queue configuration to be used with Karpeneter while utlising Spot Instances.

Kubernetes addons and operators

I am utilising Flux2 to deploy all additional configurations. You can find them at https://github.com/marcincuber/kubernetes-fluxv2 I have built this as a separate repository to show how to develop a successful configuration for your own cluster using GitOps FluxV2 and Helm.

You will find configurations for:

AWS Load Balancer controller
AWS node termination handler
Cert Manager
External-DNS
External Secrets Operator
Metrics server
Reloader
VPC CNI Plugin
EBS CSI Driver
and more :)

Docs and other additional resources

Check out my stories on medium if you interested in finding out more on specific topics.

Amazon EKS Addons

EKS + Kube-bench

Kube-bench implementation with EKS

EKS and ECR Pull through cache implementation using Terraform

ECR pull through cache for DockerHub, Github, Quay, ECR public and kubernetes

Amazon EKS design, use of spot instances and cluster scaling

More about my configuration can be found in the blog post I have written recently -> EKS design

IAM Roles for specific namespaces

Amazon EKS- RBAC with IAM access

IAM Roles for service accounts using OpenID Connect

Using OIDC provider to allow service accounts to assume IAM role

Kube2iam

More about kube2iam configuration can be found in the blog post I have written recently -> EKS and kube2iam

External DNS

Amazon EKS, setup external DNS with OIDC provider and kube2iam

EKS Managed Node Groups

Amazon EKS + managed node groups

Terraform module written by me can be found in -> https://registry.terraform.io/modules/umotif-public/eks-node-group

Gitlab runners on EKS

Kubernetes GitLab Runners on Amazon EKS

Useful resources

EKS platforms information Worker nodes upgrades

Generate kubeconfig file

On user's machine who has been added to EKS, they can configure .kube/config file using the following command:

$ aws eks list-clusters
$ aws eks update-kubeconfig --name ${cluster_name}

eks's People

Contributors

Stargazers

Watchers

Forkers

arashkaffamanesh briskgopesh trung85 hostystack skondla frcho sent2020 hiroyan164 nbmustafa rotemamergi klapcsik madhubysani01 code4public k-groshev marcellodesales waldt rd1981 kandularamesh rongyj aarmitiswe erezool ahridinorganization cyancow dgaharwar bboykov igorgorovoy leewalter firojin avadhesh-kubematics archyemi adevhammer data-depo vaultgitorg pradhans0906 tienbm90 roshan-pnq y0zg goolte-lab kidmam ashaik65 ahsanj zorgji soorepalli rahimigit pingping95 ikalin ithings-ch swade1987 davar-playgrounds duyhenryer sunny5055 diegowribeiro ajithv1981 bakuppus beau-bishop thpham claudiouzelac mailtodsathish sunibhaikc shivajavaji pankaj-delphix gtiwari131 toughlama vigneshkadivelu umanmd707 anselmenumbisia saikumar13 scmexpert enayet01 elioseverojunior wjdwlstjd7 pekrecep devopsguru30 maltif varun7447 atadzhan venkytvh dzhan85 olivatac avi202020 devopstechnetwork reinaldoca lujeremicserapion smojum gurudonal eduardoaw ib-acc-jbenitez amanappin oleg-soroka-epam leh327 harshaghub trucdp lacoski kofisq rajish vinodkkc ndubbala thumbeyashwath sathishphcl gveerashekar

eks's Issues

kubecost specified is not supported in 1.28 kubernetes version

no support for the eks version specified in terraform.tfvars

eks_version = "1.28"
eks_addon_version_kubecost = "v1.103.3-eksbuild.0"

terraform apply

│ Error: creating EKS Add-On (eks-eu-dev:kubecost_kubecost): operation error EKS: CreateAddon, https response error StatusCode: 400, RequestID: 1b959a68-9ab8-4e0b-8419-db26e3d38077, InvalidParameterException: Addon kubecost_kubecost specified is not supported in 1.28 kubernetes version
│ 
│   with aws_eks_addon.kubecost[0],
│   on eks-addons.tf line 73, in resource "aws_eks_addon" "kubecost":
│   73: resource "aws_eks_addon" "kubecost" {
│ 
╵

describe addon compatible versions for eks 1.28

eksctl utils describe-addon-versions --kubernetes-version 1.28 --name kubecost_kubecost
2024-01-22 10:39:27 [ℹ]  describing addon versions for addon: kubecost_kubecost
{
        "Addons": []
}

supported for eks 1.27

eksctl utils describe-addon-versions --kubernetes-version 1.27 --name kubecost_kubecost                    
2024-01-22 10:46:37 [ℹ]  describing addon versions for addon: kubecost_kubecost
{
        "Addons": [
                {
                        "AddonName": "kubecost_kubecost",
                        "AddonVersions": [
                                {
                                        "AddonVersion": "v1.103.3-eksbuild.0",
                                        "Architecture": [
                                                "amd64",
                                                "arm64"
                                        ],
                                        "Compatibilities": [
                                                {
                                                        "ClusterVersion": "1.27",
                                                        "DefaultVersion": true,
                                                        "PlatformVersions": [
                                                                "*"
                                                        ]
                                                }
                                        ],
                                        "RequiresConfiguration": false
                                }
                        ],
                        "MarketplaceInformation": {
                                "ProductId": "753cea16-f450-4cfa-93eb-f55dcde11e91",
                                "ProductUrl": "https://aws.amazon.com/marketplace/pp?sku=753cea16-f450-4cfa-93eb-f55dcde11e91"
                        },
                        "Owner": "aws-marketplace",
                        "Publisher": "kubecost",
                        "Type": "cost-management"
                }
        ]
}

got some questions

@marcincuber I would like to know if you like to have a chat to help me adapt this code to my use case. I'm not that good with Terraform. Please let me know, we can discuss the terms by e-mail: [email protected]. Thanks

How can I attach spot workers to cluster?

Hello, I'm looking your settings in the latest configurations which available in terraform-aws, and I realize the deployment, I can see every was deploy.
I have 3 m5 Spot instances running, and the eks cluster too

But when I run kubectl get nodes

This is the ouput.

No resources found in default namespace.

What is the right process to achieve the kubectl get nodes show me the spot instances?

Maybe, Can you help with that?, I feel a little lost and I really need your help for achieve that.

Thank you.

Update to alb-ingress-controller v2

Thanks for the articles and upgrade guides. I'd like to see ALB v2

amazon/aws-alb-ingress-controller:v2.1.3

Data archive to zip multiple files instead of having it pre-zipped

Hi Marcin,

back again with few questions, as I am really feeling your work has been one of the very best put there for other DevOps to get their EKS work and running in a better manner.

I have three question, I would appreciate if you can give some clarification.

Question 1/
node-drainer.tf what is the file in node_drainer folder that we are zipping?
There are three folders inside node_drainer, each folder has few files in it? Can you kindly give some explanation there?

I want to use terraform to zip the file on the fly instead of having zip file in the repo (for security compliance purposes).
for example I want to zip the node_drainer as below:

data "archive_file" "node-draining_zip" {
type = "zip"
output_path = "${path.module}/node_drainer/???-${sha256(file("${path.module}/node_drainer/???.py"))}.zip"
source_file = "${path.module}/node_drainer/???.py"
}

then call it like this:
source_code_hash = data.archive_file.node-draining_zip.output_base64sha256

Question 2/
There is a security group you are creating but seems like this hasn’t been used anywhere?
https://github.com/marcincuber/eks/blob/master/terraform-aws/cluster.tf#L5
Or am I missing something?

Question 3/
for all configs provided in terraform_k8s inclusing configmap aws-auth and cluster_autoscaler, etc. is there any automated process you are using to configure these templates onto the deployed clusters and node_group?

Thank you very much.

Instance type for third node group in the AZ-C is missing

Hello Marcin.
I have been going through your excellent work since yesterday, found that instance type is missing for the third AZ
module "eks-node-group-c" {
https://github.com/marcincuber/eks/blob/master/terraform-aws/manage-node-group.tf#L57

is that a deliberate move to let ng in the third AZ get the default instance type? or that is just been missed out?

Private EKS Cluster not accessble

Hi, this is srinivasa am created EKS cluster in AWS using EKSCTL but default it will create public eks (API server endpoint access) but it is i need to change this one into private am trying from AWS console after changing in to private from kube-server where i installed kubectl and eksctl i cant able to access that cluster am getting error tcp:ip ip:443 i/o timeout my kubeserver is in private subnet only and all my worker nodes is also in private only but i dont know why am getting this error from my kube-machine please help me for this to troubleshoot incase u need any info i will provide
EKS-version 1.15
thank you

Few changes that I wish to suggest..

Awesome project for learning & developing skills in Terraform. I would like to suggest following changes & would like to create pull request which i h'v tested but would like know whether it makes sense for review.

custom LT for managed group. Many time users would like to pass custom configuration. I'm passing some of the recommened once https://blog.codeship.com/running-1000-containers-in-docker-swarm & below code block as part of node initialization. Besides the point, users can use what suits them.. Excuse me for long block

if ! grep -q imageGCHighThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json;
then
    sed -i '/"apiVersion*/a \ \ "imageGCHighThresholdPercent": 70,' /etc/kubernetes/kubelet/kubelet-config.json
fi

# Inject imageGCLowThresholdPercent value unless it has already been set.
if ! grep -q imageGCLowThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json;
then
    sed -i '/"imageGCHigh*/a \ \ "imageGCLowThresholdPercent": 50,' /etc/kubernetes/kubelet/kubelet-config.json
fi

## Initializing kubelet based on spot/ondemand
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

instance_type=$(aws ec2 describe-instances --instance-ids $instance_id --query 'Reservations[0].Instances[0].InstanceLifecycle' --output text)

export AWS_REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)

if [ "$instance_type" == "spot" ]; then

  /etc/eks/bootstrap.sh '${CLUSTER_NAME}' --b64-cluster-ca '${B64_CLUSTER_CA}' --apiserver-endpoint '${API_SERVER_URL}' --kubelet-extra-arg "--system-reserved cpu=250m,memory=0.2Gi,ephemeral-storage=1Gi --kube-reserved cpu=250m,memory=1Gi,ephemeral-storage=1Gi --eviction-hard memory.available<0.2Gi,nodefs.available<10% --allowed-unsafe-sysctls net.core.somaxconn,net.ipv4.tcp_tw_reuse --event-qps=0 --read-only-port=0"

else

  /etc/eks/bootstrap.sh '${CLUSTER_NAME}' --b64-cluster-ca '${B64_CLUSTER_CA}' --apiserver-endpoint '${API_SERVER_URL}' --kubelet-extra-arg "--system-reserved cpu=250m,memory=0.2Gi,ephemeral-storage=1Gi --kube-reserved cpu=250m,memory=1Gi,ephemeral-storage=1Gi --eviction-hard memory.available<0.2Gi,nodefs.available<10% --allowed-unsafe-sysctls net.core.somaxconn,net.ipv4.tcp_tw_reuse --event-qps=0 --read-only-port=0"

fi

Local null provisioner to setup kubectl access by default for the role being used. assume kubectl is already setup
Support for automatically adding spot node-group role added to auth config map for node to show up automatically show up in the cluster. Currently it has to be done manually..
The cluster is opened up for public access which is not good security posture for the cluster. I made the change to support access from local runner/instance public IP to limit the access