Manage node group user data is not being parsed

[triceras]

Manage node group user data is not being parsed

[bryantbiggs]

1. You have disabled the custom launch template which is the only way that these customizations, such as user data, are supported
2. If you are mounting the instance store volumes, there are provisions within the EKS AL2 AMI to support this already https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/9a0ca42fde8c2e11acacbc1a891885d840009d34/patterns/nvidia-gpu-efa/eks.tf#L37-L43

[triceras]

@bryantbiggs deployed your suggestion for mounting the instance store volumes using an AL2 AMI, however I am still having trouble mounting the instance store. Some of my pods require instance store volume to be mounted and they are still complaining that the volumes are not present. Is there any wrong with the way I am using the parameter pre_bootstrap_user_data ?

2024-05-14T07:41:44.506522Z topo-lvm-sc-topolvm-lvmd-0-lmz2c lvmd error: "Volume group not found:" volume_group="instancestore"

This is my updated terraform code

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name             = var.cluster_name
  cluster_version          = var.cluster_version
  subnet_ids               = var.private_subnets
  control_plane_subnet_ids = var.intra_subnets

  vpc_id = var.vpc_id
  enable_cluster_creator_admin_permissions = true
  authentication_mode = "API_AND_CONFIG_MAP"
  cluster_endpoint_public_access = true
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  kms_key_administrators = [data.aws_caller_identity.current.arn]
  kms_key_owners = [data.aws_caller_identity.current.arn]

  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent    = true
      before_compute = true
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
  }

  iam_role_additional_policies = {
    "ssm_managed_core" = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }

  eks_managed_node_groups = {
    ingest_node_group = {
        name                 = "ingest"
        use_name_prefix      = true
        ami_type             = "AL2_x86_64"

        subnet_ids = var.private_subnets
        vpc_security_group_ids = [
          module.eks.node_security_group_id,
        ]

        min_size     = var.node_min_capacity
        max_size     = var.node_max_capacity
        desired_size = var.node_desired_capacity

        instance_types = [var.instance_type]

        use_custom_launch_template = false
        disk_size = 200
  
        pre_bootstrap_user_data = <<-EOT
          #!/usr/bin/env bash
          # Mount instance store volumes in RAID-0 for kubelet and containerd
          # https://github.com/awslabs/amazon-eks-ami/blob/master/doc/USER_GUIDE.md#raid-0-for-kubelet-and-containerd-raid0

          /bin/setup-local-disks raid0
        EOT

        timeouts = {
          delete = "1h"
        }

        labels = {
          GithubRepo = "terraform-aws-eks"
          GithubOrg  = "terraform-aws-modules"
          managed_by   = "terraform"
          k8s-app      = "ingest"
          storageclass = "nvme"
        }
     }
    var = var.tags
}

[bryantbiggs]

once you've mount the instance store volume(s), you can use them by specifying the necessary ephemeral storage required in your request/limits

unfortunately the code you have provided is littered with variables so its impossible to know what is being configured and how

[triceras]

Sorry for wasting your time.
For folks who look at this post in the future trying to find an answer I recommend not setting use_custom_launch_template = false and it should work. I am using an instance of type i3.2xlarge which is equipped with NMM2-based SSD instance store volumes by default.
Confirm that the instance store volume is being utilized and configured as part of a RAID0 array and mounted by running the lsblk utility on your linux instance.

sh-4.2$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
xvda    202:0    0   20G  0 disk
└─xvda1 202:1    0   20G  0 part  /
nvme0n1 259:0    0  1.7T  0 disk
└─md127   9:127  0  1.7T  0 raid0 /mnt/k8s-disks/0