No way to tackle Karpenter when KMS is mandatory by default

[zohairraza]

No way to tackle Karpenter when KMS is mandatory by default

[bryantbiggs]

we can't duplicate all of the docs within Karpenter, EKS, MNG, Fargate, etc. We focus on documentation related to the module itself

[bryantbiggs]

can you format you code by surrounding with code fences (```hcl` ) and provide psuedo code of what you are trying to do - its not very clear what you are trying to do or how you are approaching it

[zohairraza]

Hi Bryan, thanks for your response. I added Karpenter code too. Does it explain now?

[bryantbiggs]

not really - theres a bunch of variables that are unknown.

If you are trying to re-use the cluster KMS key (that is used to encrypt cluster secrets), you will need to add the necessary permissions to use with EBS volumes - the module does not do this by default

[zohairraza]

yes, i am trying to re-use the cluster KMS key and I need to know which is the good way to tackle this. Since when I add permissions in the key outside of the module, they get overritten at the next terraform run.

I think creating a separate key for karpenter might be a better choice. Let me try that out. Meanwhile you can comment if that's the best way

[bryantbiggs]

why not just add the required permissions into the key that is created via terraform? check the variables that are provided

[zohairraza]

that indeed worked !

thanks.. maybe add this to docmentation somewhere so it will help others facing the same issue later on

[bryantbiggs]

The variables are in the documentation - any variable definitions are automatically added to our documentation

[zohairraza]

I meant to mention it here in the example
https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/karpenter
or in the readme
https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/karpenter

as it is described here
https://karpenter.sh/docs/troubleshooting/#node-terminates-before-ready-on-failed-encrypted-ebs-volume

[k9sstorage]

@zohairraza How did you got this resolved ? I have having the same issue.

[zohairraza]

By creating another key for karpenter:

resource "aws_kms_key" "KarpenterKMSKey" {
  description = "Karpenter KMS Key"
  policy = local.merged_policy
  depends_on = [module.eks]
}

resource "aws_kms_alias" "KarpenterKMSKey" {
  name          = "alias/eks-karpenter-key"
  target_key_id = aws_kms_key.KarpenterKMSKey.key_id
}

resource "kubectl_manifest" "karpenter_node_class" {
  yaml_body = <<-YAML
    apiVersion: karpenter.k8s.aws/v1beta1
    kind: EC2NodeClass
    metadata:
      name: default
    spec:
      amiFamily: AL2
      role: ${module.karpenter.node_iam_role_name}
      subnetSelectorTerms:
        - tags:
            karpenter.sh/discovery: ${module.eks.cluster_name}
      securityGroupSelectorTerms:
        - tags:
            karpenter.sh/discovery: ${module.eks.cluster_name}
      tags:
        karpenter.sh/discovery: ${module.eks.cluster_name}
      blockDeviceMappings:
        - deviceName: /dev/xvda
          ebs:
            volumeSize: 30Gi
            volumeType: gp3
            iops: 10000
            encrypted: true
            kmsKeyID: ${aws_kms_key.KarpenterKMSKey.key_id}
            deleteOnTermination: true
            throughput: 125
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}

[bryantbiggs]

just to be clear - there is no KMS key for Karpenter. There are two use case where KMS keys are involved within EKS

1. To encrypt secrets within the cluster - this module supports creating a custom KMS for this purpose

   terraform-aws-eks/main.tf

   Line 80 in f90f15e

   key_arn = var.create_kms_key ? module.kms.key_arn : encryption_config.value.provider_key_arn

2. To encrypt EBS volumes on EC2 instances - this module does not create a key for this, but you can pass in an externally created key. This is what I believe is being referred to as the "Karpenter key"

If you want to re-use the KMS key created by this module that was created for encrypting secrets within the cluster, you MUST update the key policy to ensure it will work for encrypting EBS volumes with the solution that is creating the instances (EKS managed node group, self-managed node group, Karpenter, etc.)

terraform-aws-eks/main.tf

Lines 238 to 243 in f90f15e

	key_owners = var.kms_key_owners
	key_administrators = coalescelist(var.kms_key_administrators, [data.aws_iam_session_context.current.issuer_arn])
	key_users = concat([local.cluster_role], var.kms_key_users)
	key_service_users = var.kms_key_service_users
	source_policy_documents = var.kms_key_source_policy_documents
	override_policy_documents = var.kms_key_override_policy_documents

[k9sstorage]

@zohairraza thanks for the response.
@bryantbiggs
In my case i am using this module to create EBS KMS also along with secret one.

terraform-aws-eks/examples/eks_managed_node_group/main.tf

Line 244 in f90f15e

kms_key_id = module.ebs_kms_key.key_arn

terraform-aws-eks/examples/eks_managed_node_group/main.tf

Lines 429 to 451 in f90f15e

	module "ebs_kms_key" {
	source = "terraform-aws-modules/kms/aws"
	version = "~> 2.1"
	description = "Customer managed key to encrypt EKS managed node group volumes"
	# Policy
	key_administrators = [
	data.aws_caller_identity.current.arn
	]
	key_service_roles_for_autoscaling = [
	# required for the ASG to manage encrypted volumes for nodes
	"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
	# required for the cluster / persistentvolume-controller to create encrypted PVCs
	module.eks.cluster_iam_role_arn,
	]
	# Aliases
	aliases = ["eks/${local.name}/ebs"]
	tags = local.tags
	}

The issue I am facing is, when a karpenter tries to scale the instance, it gets terminated immediately, on further investigation it seems to use the default kms key mentioned in the account, instead of using the one created as part of this module.

Even on launchTemplate also i can see the new EBS KMS is being used.

This is how EC2NodeClass is:

`      blockDeviceMappings:
       - deviceName: /dev/xvda
         ebs:
           volumeSize: 30Gi
           volumeType: gp3
           iops: 10000
           encrypted: true
           **kmsKeyID: ${module.ebs_kms_key.key_id}**
           deleteOnTermination: true
           throughput: 125`

Btw.. i am trying to use Pod-Identity (not sure, if there are any extra setting needed)

Thanks

[bryantbiggs]

looks like a configuration error on your end - I would check the Karpenter documentation https://karpenter.sh/docs/troubleshooting/#node-terminates-before-ready-on-failed-encrypted-ebs-volume

[k9sstorage]

Thanks i did looked into it, but couldn't resolve it, i have added that policy to my KMS, still the same..
looks i am missing something, so thought to ask "experts" here... i am bit puzzled how it's getting the default KMS key, even though it's been clearly mentioned to use the new ebs key created.

[bryantbiggs]

have you set a default on the account/region? https://docs.aws.amazon.com/cli/latest/reference/ec2/get-ebs-default-kms-key-id.html

[k9sstorage]

how can karpenter pick a random/default kms key? Even though it's been clearly mentioned in EC2NodeClass to use the KMS created as part of this module ?
Worker nodes can join the cluster without any issues, it's when Karpenter tries to scale up, i have started seeing the issue, from cloud trail it's clear that, it's picking up the wrong KMS key, so not sure what i am missing here ;(

actually i can see the get-ebs-kms command it's the alies/aws/ebs created as part of this is a default, so back to square one, god .....how is it even getting that kms key ;(..... i am missing a critical point here..

[bryantbiggs]

most likely your account is configured with https://docs.aws.amazon.com/ebs/latest/userguide/work-with-ebs-encr.html#encryption-by-default

[k9sstorage]

maybe that's the case... Any idea how I can overwrite it ? make it use the new kms created, also i am using create_instance_profile option with karpenter (not sure if that makes any difference).
i am using this block
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/karpenter/main.tf#L115C1-L129C2
with create_instance_profile = true

[zohairraza]

that was my situation too, so I created another key dedicated for karpenter and used it in karpenter configuration which worked. Before I was using EKS module cluster key in Karpenter Nodepool

most likely your account is configured with https://docs.aws.amazon.com/ebs/latest/userguide/work-with-ebs-encr.html#encryption-by-default

[k9sstorage]

The issue with my setup was related to the KMS key used while creating the AMI. Even though I specified the KMS in MG and EC2NodePool, it couldn't re-encrypt because the Karpenter role lacked permission for the KMS key created as part of the AMI.

My initial idea was not to use one KMS key for all the clusters, but this issue has brought me back to square one.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.