gruntwork-io/kubergrunt

Kubergrunt is a standalone go binary with a collection of commands to fill in the gaps between Terraform, Helm, and Kubectl. https://www.gruntwork.io

gruntwork.io website

This is the code for the Gruntwork website.

Gruntwork can help you get your entire infrastructure, defined as code, in about a day. You focus on your product. We'll take care of the Gruntwork.

Docker quick start

The fastest way to launch this site is to use Docker.

1. git clone this repo
2. docker compose up
3. Go to http://localhost:4000 to test
4. If you are going to be testing the checkout flow, you must login to Aperture at: https://aperture.dogfood-stage.com/.

The default Docker compose configuration supports hot-reloading of your local environment, meaning that as you edit files to change markup, text, images, etc, your local development server will pick up these changes and reload the latest version of the site for you. This makes it quick and convenient to develop on the site locally.

Manual quick start

1. git clone this repo
2. Install Jekyll
3. Just the first time: bundle install
4. Start Jekyll server: bundle exec jekyll serve --livereload
5. Go to http://localhost:4000
6. If you are going to be testing the checkout flow, you must login to Aperture at: https://aperture.dogfood-stage.com/.

Deploying

To deploy the site:

1. Create a PR with your code changes
2. After the PR has been approved, merge it into master
3. Create a new tag, you can do this manually via git or in the subsequent step on the releases page - be sure to increment the version number using semantic versioning
4. Go to the releases page and create a draft release with the relevant information (use the "Generate Release Notes" button to make your life easier)
5. Release it
6. The CI/CD pipeline will deploy it automatically

Technologies

1. Built with Jekyll. This website is completely static and we use basic HTML or Markdown for everything.
2. Preview environments are built with Netlify.
3. Hosted on Amazon S3, with CloudFront as a CDN. Using s3_website to automatically upload static content to S3.
4. We use Bootstrap and Less.
5. We're using UptimeRobot, Google Analytics, and HubSpot Traffic Analytics for monitoring and metrics.

Troubleshooting

Disabling the Jekyll Feed gem

The Gruntwork website uses a Ruby Gem called Jekyll Feed which generates a structured RSS feed of "posts" on the site. Unfortunately, in development this can significantly slow down the hot-reloading of the site, forcing you to wait upwards of a minute at a time to see minor text changes locally.

You'll know this is happening when you look at the STDOUT of your docker-compose process and the final count of seconds spent Generating feed for posts is greater than 5:

web_1  |       Regenerating: 1 file(s) changed at 2021-07-21 14:31:08
web_1  |                     _data/website-terms.yml
web_1  |        Jekyll Feed: Generating feed for posts
web_1  |                     ...done in 58.507850014 seconds.

As a temporary workaround, you can open the Gemfile in the root of the project directory and temporarily comment out the line that pulls in the Jekyll Feed dependency:

source 'https://rubygems.org'
gem 'jekyll', '~> 4.1'
gem 's3_website', '3.3.0'
group :jekyll_plugins do
  gem 'jekyll-redirect-from', '0.16.0'
  gem 'jekyll-sitemap', '1.4.0'
  gem 'jekyll-paginate', '1.1.0'
  gem 'therubyracer', '0.12.3'
  gem 'less', '2.6.0'
  gem 'jekyll-asciidoc'
  gem 'jekyll-toc'
  gem 'nokogiri', '1.11.0.rc4' # Addressing security issue in earlier versions of this library
#  gem 'jekyll-feed'
end

Important - Be sure that you don't end up committing this change because we do want the Jekyll Feed plugin to run for production!

I made changes locally but they're not being reflected in my hot-reloaded development environment

This can happen especially if you add or remove files from the website's working directory. When this occurs, terminate your docker-compose process and restart it to see your changes reflected.

License

See LICENSE.txt.

kubergrunt's Issues

Cannot build: vbom.ml domain no longer exists

The vbom.ml domain is currently down. As a result, fetching dependencies fails for:

kubergrunt/Gopkg.lock

Line 1294 in 47fda9d

name = "vbom.ml/util"

(17/92) Failed to write github.com/ghodss/[email protected]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export vbom.ml/util: unable to deduce repository and source type for "vbom.ml/util": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://vbom.ml/util?go-get=1": Get "http://vbom.ml/util?go-get=1": dial tcp: lookup vbom.ml: no such host
Command failed: GOARCH=amd64 GOOS=darwin CC=/usr/bin/clang dep ensure -v -vendor-only
Exit code: 1

Seeing that it should be replaced with: https://github.com/fvbommel/util

Configurable rolling deployment

kubergrunt eks deploy should support a configurable parameter that specifies how many nodes it should rotate at a time. This can help make the deployment less disruptive at the expense of speed of rollout.

Add ability to use environment variables for kubectl config

Hi,

Just making a feature request for kubergrunt to have the ability to read the Kubectl config variable from environment variable KUBECONFIG as well as the cli flags. I couldn't find anything in past issues so I apologise if you have already addressed this.

I am using KIND to build the Kubernetes clusters and then use direnv to load the environment variable KUBECONFIG based on my different testing environments.

export KUBECONFIG="$(kind get kubeconfig-path --name="infra")"

Terraform and Kubectl both read what config to use from this environment variable KUBECONFIG and it would be awesome if Kubergrunt also did, making it even easier to use within Terraform with local exec, I would think at least for me it would.

Thanks
Kyle

kubergrunt eks deploy does not cordon nodes first

kubergrunt eks deploy should cordon all the old nodes before attempting the draining to avoid the pods being rescheduled older nodes.

Add support for multiple ASGs in rolling deployment command

Currently, kubergrunt requires a single --asg-name when running eks deploy but in many clusters you'll have multiple ASGs for your worker nodes.

It would be great if this supported providing a list of ASGs and running rolling deployments on each of them.

kubergrunt eks configure should allow you to force update an existing context

When you reuse EKS cluster names, the ARNs stay the same if the EKS cluster is recreated, but since the server parameters change (e.g., the endpoint and the CA certificate), you need to reconfigure the kubeconfig. However, kubergrunt currently errors out if a context and config already exists for the server with the same ARN.

The workaround is to delete the existing config, but kubergrunt eks configure should also expose a --force option to overwrite the existing config instead of erroring out.

Kugergrunt using the wrong version for EKS 1.18

[] INFO[2021-08-19T14:05:49-05:00] Successfully retrieved EKS cluster details name=kubergrunt
[] INFO[2021-08-19T14:05:49-05:00] Syncing Kubernetes Applications to: name=kubergrunt
[] INFO[2021-08-19T14:05:49-05:00] kube-proxy: 1.18.8-eksbuild.2 name=kubergrunt

Per:
https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html#kube-proxy-default-versions-table

The currently supported release for 1.18 EKS cluters is:

1.18.8-eksbuild.1

1.18.8-eksbuild.2 doesn't exist and throws:
Failed to pull image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.18.8-eksbuild.2": rpc error: code = Unknown desc = Error response from daemon: manifest for 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.18.8-eksbuild.2 not found: manifest unknown: Requested image not found

eks deploy should be robust to eventual consistency

Should avoid errors like the following

ERRO[2021-02-08T21:01:38-04:00] Error retrieving detailed about the instances  name=kubergrunt
ERRO[2021-02-08T21:01:38-04:00] Undo by terminating all the new instances and trying again  name=kubergrunt
ERROR: InvalidInstanceID.NotFound: The instance ID 'i-028ec69838e751d18' does not exist
        status code: 400, request id: 9c623f38-d818-43f1-a335-d370a2a5ad76

Ability to append hostnames when generating certificates

Something like

template.DNSNames = append(template.DNSNames, hosts)

to cert_common.go

Thanks

cleanup-security-group misses elb security groups

Describe the bug
I ran this tool and it deleted the cluster's security group but not groups created for load balancers.

To Reproduce

• Create an eks service with load balancer
• Delete the service (LB automatically removed, leaves orphan security groups in certain situations)
• Call kubergrunt.

resource "aws_eks_cluster" "eks" {
   ...
  provisioner "local-exec" {
    when    = destroy
    command = "kubergrunt eks cleanup-security-group --eks-cluster-arn ${self.arn} --security-group-id ${self.vpc_config.0.cluster_security_group_id} --vpc-id ${self.vpc_config.0.vpc_id}"
  }
}

Expected behavior
Kubergrunt deletes the orphaned load balancer security groups.

Nice to have

• Terminal output

module.eks_common.aws_eks_cluster.eks (local-exec): Executing: ["/bin/sh" "-c" "kubergrunt eks cleanup-security-group --eks-cluster-arn arn:aws:eks:us-east-1:xxxx:cluster/dpedu5-eks --security-group-id sg-0a10673de28a8a38f --vpc-id vpc-07c666f30e4bebbef"]
module.eks_common.aws_eks_cluster.eks (local-exec): [] time="2022-04-27T12:30:25-07:00" level=info msg="Successfully authenticated with AWS" name=kubergrunt
module.eks_common.aws_eks_cluster.eks (local-exec): [] time="2022-04-27T12:30:26-07:00" level=info msg="Deleting security group sg-0a10673de28a8a38f" name=kubergrunt
module.eks_common.aws_eks_cluster.eks (local-exec): [] time="2022-04-27T12:30:26-07:00" level=info msg="Security group sg-0a10673de28a8a38f already deleted." name=kubergrunt

Additional context
The orphaned security groups have a tag like: kubernetes.io/cluster/<clustername>=owned. See kubernetes/kubernetes#109698 for the conditions under which these orphaned groups are created and not deleted.

Support sync command with EKS 1.24

Describe the solution you'd like

EKS Kubernetes version 1.24 support was announced, so we should update kubergrunt eks sync to support that.

Kubergrunt Support for EKS 1.19

Today I tested a EKS upgrade to 1.19 as we are building a new cluster and wanted to test upgrading with Terraform. I noticed that kubergrunt generated the following error.

module.eks_cluster.null_resource.sync_core_components[0] (local-exec): ERROR: 1.19 is not a supported version for kubergrunt eks upgrade. Please contact [email protected] for more info.

I know that 1.19 is super new for AWS but what needs to be updated in Kubergrunt to support 1.19? 1.20 is also a few months away.

Thanks.

Support sync command with EKS 1.23

Describe the solution you'd like
EKS Kubernetes version 1.23 support was announced recently, so we should update kubergrunt eks sync to support that.

Enhance `README.md` with commands for creating resources

Hi Yori,

I was trying to use Kubergrunt to deploy Helm to a GKE cluster. Would it make sense to add a few lines to the README.md file to illustrate how to create a dedicated ServiceAccount and namespace for Helm.

e.g:

❯ kubectl create namespace tiller-world
namespace "tiller-world" created

❯ kubectl create serviceaccount tiller --namespace tiller-world
serviceaccount "tiller" created

That way the user can get started quicker and avoid errors like: ERROR: namespaces "tiller-world" not found if they then run:

kubergrunt helm deploy \
    --tiller-namespace tiller-world \
    --resource-namespace dev \
    --service-account tiller \
    --tls-common-name tiller \
    --tls-org Gruntwork \
    --tls-org-unit IT \
    --tls-city Phoenix \
    --tls-state AZ \
    --tls-country US \
    --rbac-group admin \
    --client-tls-common-name admin \
    --client-tls-org Gruntwork

It also removes the ambiguity from the following statement:

Note: This command does not create Namespaces or ServiceAccounts, delegating that responsibility to other systems.

New VPC CNI version is available

Hi! It looks like the VPC CNI has version 1.9.0 available per the docs:

https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#updating-vpc-cni-add-on

Any feedback on opening issues is welcomed as I've never opened one before 😅

Fault tolerance in the eks deploy command

The eks deploy command should track where it is in the state of the deployment and recover from failures.

For example, it is fairly painful to recoup if the deploy command fails after the ASG was expanded and during the drain call. In that failure mode, the user has to manually drain all the nodes, and then terminate them. We should figure out a way for the deploy command to remember where it failed, and then provide the ability to retry starting from that step.

Add debug logs if kubergrunt cannot access Kubernetes cluster

Describe the bug
I recently used kubergrunt eks deploy to roll-out workers update and it suddenly failed at some point (see logs):

[] INFO[2023-02-10T12:57:02+01:00] Successfully launched new nodes with new launch config on ASG app-workers-eks-data-asg-20210305120833070500000007  name=kubergrunt
[] INFO[2023-02-10T12:57:02+01:00] Waiting for 3 nodes in Kubernetes to reach ready state  name=kubergrunt
[] INFO[2023-02-10T12:57:02+01:00] Loading Kubernetes Client                     name=kubergrunt
[] INFO[2023-02-10T12:57:02+01:00] Using config on disk and context.             name=kubergrunt
[] INFO[2023-02-10T12:57:02+01:00] Checking if nodes ready                       name=kubergrunt
[] ERRO[2023-02-10T12:57:04+01:00] Timed out waiting for the instances to reach ready state in Kubernetes.  name=kubergrunt
[] ERRO[2023-02-10T12:57:04+01:00] Undo by terminating all the new instances and trying again  name=kubergrunt
[] ERRO[2023-02-10T12:57:04+01:00] Error while waiting for new nodes to be ready.  name=kubergrunt
[] ERRO[2023-02-10T12:57:04+01:00] Either resume with the recovery file or terminate the new instances.  name=kubergrunt

The reason for this that my kubectl was not switched to correct context, basically, neither kubectl not kubergrunt cannot authenticate to the correct cluster. But, as you can see from logs, you cannot define that.

To Reproduce
Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used.

Expected behavior
Display meaningful errors if kubergrunt cannot authenticate to Kubernetes cluster.

Nice to have

• Terminal output
• Screenshots

Additional context
Add any other context about the problem here.

Actively deregister instances from target groups during `eks deploy`

For people using in-cluster mechanisms to manage ELB attachments (e.g., aws-alb-ingress-controller), detaching instances from ASGs do not actually deregister the instances from the ELBs. This can be a source of downtime during the rollout procedure as instances need to fail the health checks before being dropped from the list, which means they can continue to get requests while the servers are being drained and shut down.

Add examples of how to construct the namespace and serviceaccount in docs

At a minimum:

• GKE
• EKS
• terraform-kubernetes-helm
• kubectl directly

RBAC Error when running `helm deploy` on GKE

Hi Yori,

I'm trying to deploy Helm to a GKE cluster and it looks like I'm running into some RBAC issues.

Any ideas?

It could be a GKE security feature or limitation?

Here's the full trace:

❯ ./cmd helm deploy \
    --tiller-namespace tiller-world \
    --resource-namespace dev \
    --service-account tiller \
    --tls-common-name tiller \
    --tls-org Gruntwork \
    --tls-org-unit IT \
    --tls-city Phoenix \
    --tls-state AZ \
    --tls-country US \
    --rbac-group admin \
    --client-tls-common-name admin \
    --client-tls-org Gruntwork \
    --kubectl-context-name $(kubectl config current-context)

INFO[2019-02-07T14:12:05+01:00] No kube config path provided. Using default (/Users/robbym/.kube/config)  name=kubergrunt
INFO[2019-02-07T14:12:05+01:00] Validating required resources exist.          name=kubergrunt
INFO[2019-02-07T14:12:05+01:00] Validating the Namespace tiller-world exists  name=kubergrunt
INFO[2019-02-07T14:12:05+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Found Namespace tiller-world                  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Validating the ServiceAccount tiller exists in the Namespace tiller-world  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Found ServiceAccount tiller                   name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] All required resources exist.                 name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Generating certificate key pairs              name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Using /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/671821986 as temp path for storing certificates  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Generating CA TLS certificate key pair        name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Done generating CA TLS certificate key pair   name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Generating Tiller TLS certificate key pair (used to identify server)  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Generating signed TLS certificate key pair    name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Done generating signed TLS Certificate key pair  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Successfully generated Tiller TLS certificate key pair (used to identify server)  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Done generating certificate key pairs         name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Uploading CA certificate key pair as a secret  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Successfully uploaded CA certificate key pair as a secret  name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Deploying Helm Server (Tiller)                name=kubergrunt
INFO[2019-02-07T14:12:06+01:00] Running command: helm --kube-context gke_dev-sandbox-228703_europe-west3_example-cluster --kubeconfig /Users/robbym/.kube/config init --home /Users/robbym/.helm --override spec.template.spec.containers[0].command={/tiller,--storage=secret} --tiller-tls --tiller-tls-verify --tiller-tls-cert /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/671821986/tiller_tiller-world.crt --tiller-tls-key /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/671821986/tiller_tiller-world.pem --tls-ca-cert /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/671821986/tiller_tiller-world_ca.crt --tiller-namespace tiller-world --service-account tiller --wait
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/repository
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/repository/cache
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/repository/local
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/plugins
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/starters
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/cache/archive
INFO[2019-02-07T14:12:06+01:00] Creating /Users/robbym/.helm/repository/repositories.yaml
INFO[2019-02-07T14:12:06+01:00] Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
INFO[2019-02-07T14:12:09+01:00] Adding local repo with URL: http://127.0.0.1:8879/charts
INFO[2019-02-07T14:12:09+01:00] $HELM_HOME has been configured at /Users/robbym/.helm.
INFO[2019-02-07T14:12:09+01:00]
INFO[2019-02-07T14:12:09+01:00] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
INFO[2019-02-07T14:12:21+01:00] Happy Helming!
INFO[2019-02-07T14:12:21+01:00] Successfully deployed Tiller in namespace tiller-world with service account tiller  name=kubergrunt
INFO[2019-02-07T14:12:21+01:00] Granting access Tiller server deployed in namespace tiller-world to:  name=kubergrunt
INFO[2019-02-07T14:12:21+01:00]         - the RBAC groups [admin]                    name=kubergrunt
INFO[2019-02-07T14:12:21+01:00] Checking if Tiller is deployed in the namespace.  name=kubergrunt
INFO[2019-02-07T14:12:21+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Found a valid Tiller instance in the namespace tiller-world.  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Downloading the CA TLS certificates for Tiller deployed in namespace tiller-world.  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Using /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/050160537 as temp path for storing certificates  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Successfully downloaded CA TLS certificates for Tiller deployed in namespace tiller-world.  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Granting access to deployed Tiller in namespace tiller-world to RBAC groups  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Generating and storing certificate key pair for admin (1 of 1)  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Using /var/folders/4y/w8n6q5x525v9rkl7pt5pwwl40000gn/T/986521124 as temp path for storing client certificates  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Generating client certificates for entity admin  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Generating signed TLS certificate key pair    name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Done generating signed TLS Certificate key pair  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Successfully generated client certificates for entity admin  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Uploading client certificate key pair as secret in namespace tiller-world with name tiller-client-21232f297a57a5a743894a0e4a801fc3-certs  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Successfully uploaded client certificate key pair as a secret  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Successfully generated and stored certificate key pair for admin  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Creating and binding RBAC roles to admin      name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Loading Kubernetes Client with config /Users/robbym/.kube/config and context gke_dev-sandbox-228703_europe-west3_example-cluster  name=kubergrunt
INFO[2019-02-07T14:12:22+01:00] Creating RBAC role to grant access to Tiller in namespace tiller-world to admin  name=kubergrunt
ERRO[2019-02-07T14:12:22+01:00] Error creating RBAC role to grant access to Tiller: roles.rbac.authorization.k8s.io "admin-tiller-world-tiller-access" is forbidden: attempt togrant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[get] [] [secrets] [tiller-client-21232f297a57a5a743894a0e4a801fc3-certs] []} {[create] [] [pods/portforward] [] []}] user=&{[email protected]  [system:authenticated] map[user-assertion.cloud.google.com:[APTNk9SdKe8IhVYtn/TR9Ua8Qrnlwvkx5D1eORaixPPyFtQD9/9ggHhR0oa7C2fUpHJUPrHB30HeKcLEOS7rrLix66cYGPDr9+tMdUdK3ofoWMPAXryiFEQA9KaJsKBL8jx7D+nyrmuqzXv5DJEQwW/fhYaglXEJUMx8kpP18TcEybphCg9YexGZ4l1iqVAnn4iREHHV63jSXpvkTRpuoB9nLNJMr/UUDIk=]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]  name=kubergrunt
ERRO[2019-02-07T14:12:22+01:00] Error creating and binding RBAC roles to admin  name=kubergrunt
ERRO[2019-02-07T14:12:22+01:00] Error granting access to deployed Tiller in namespace tiller-world to RBAC groups: roles.rbac.authorization.k8s.io "admin-tiller-world-tiller-access" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[get] [] [secrets] [tiller-client-21232f297a57a5a743894a0e4a801fc3-certs] []} {[create] [] [pods/portforward] [] []}] user=&{[email protected]  [system:authenticated] map[user-assertion.cloud.google.com:[APTNk9SdKe8IhVYtn/TR9Ua8Qrnlwvkx5D1eORaixPPyFtQD9/9ggHhR0oa7C2fUpHJUPrHB30HeKcLEOS7rrLix66cYGPDr9+tMdUdK3ofoWMPAXryiFEQA9KaJsKBL8jx7D+nyrmuqzXv5DJEQwW/fhYaglXEJUMx8kpP18TcEybphCg9YexGZ4l1iqVAnn4iREHHV63jSXpvkTRpuoB9nLNJMr/UUDIk=]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]  name=kubergrunt
ERROR: roles.rbac.authorization.k8s.io "admin-tiller-world-tiller-access" is forbidden: attempt to grant extra privileges: [{[get] [] [pods] [] []} {[list] [] [pods] [] []} {[get] [] [secrets] [tiller-client-21232f297a57a5a743894a0e4a801fc3-certs] []} {[create] [] [pods/portforward] [] []}] user=&{[email protected]  [system:authenticated] map[user-assertion.cloud.google.com:[APTNk9SdKe8IhVYtn/TR9Ua8Qrnlwvkx5D1eORaixPPyFtQD9/9ggHhR0oa7C2fUpHJUPrHB30HeKcLEOS7rrLix66cYGPDr9+tMdUdK3ofoWMPAXryiFEQA9KaJsKBL8jx7D+nyrmuqzXv5DJEQwW/fhYaglXEJUMx8kpP18TcEybphCg9YexGZ4l1iqVAnn4iREHHV63jSXpvkTRpuoB9nLNJMr/UUDIk=]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /openapi /openapi/* /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[]

Add support for http proxy in oidc-thumbprint retrieval

When trying to get the oidc-thumprint, kubergrunt ignores my proxy settings and is directly connecting to Internet AWS servers. Add support for proxy and no proxy variables and use them.

Helm tools: Consider separating managed namespace from tiller-namespace

Right now a lot of the helm features were built with tiller managing resources in the tiller-namespace that it was deployed to.

However, this is a potential risk because an admin user in the tiller-namespace can now bypass the authentication mechanisms of tiller by inheriting the ServiceAccount. The worse risk is where read only users can grab the tiller-secret to obtain the server side TLS certs, which can be used to successfully authenticate to Tiller due to the shared CA for server and client verification and then get write access.

To mitigate this, Tiller should be deployed into a separate namespace, which then manages resources in the main namespace. This way, read only users in the main namespace do not get tiller access unless they are also explicitly granted access to the tiller namespace.

In this model, granting access to tiller now comprises of:

• Getting client side certs for tiller.
• Getting read only access to the tiller pod to look up the network.

And the service account needs:

• R/W access to secrets in the tiller namespace
• R/W access to all resources in the main namespace

kubergrunt eks deploy doesn't gracefully decommission nodes if drain fails

I was trying out the kubergrunt eks deploy command today to try and upgrade/roll my ASKs with a new version.

I have several pods with local data, so the drain failed, however it still went ahead and forcefully terminated the nodes

[] INFO[2020-12-21T11:02:04-04:00] Cordoning old instances in cluster ASG dev-eks-client-spot-us-east-1b20190530170243598800000004 to prevent Pod scheduling  name=kubergrunt
[] INFO[2020-12-21T11:02:05-04:00] Running command: kubectl --kubeconfig /Users/adam.leclerc/.kube/config cordon ip-172-25-114-254.ec2.internal
[] INFO[2020-12-21T11:02:05-04:00] Running command: kubectl --kubeconfig /Users/adam.leclerc/.kube/config cordon ip-172-25-116-251.ec2.internal
[] INFO[2020-12-21T11:02:07-04:00] node/ip-172-25-116-251.ec2.internal cordoned
[] INFO[2020-12-21T11:02:07-04:00] node/ip-172-25-114-254.ec2.internal cordoned
[] INFO[2020-12-21T11:02:07-04:00] Successfully cordoned old instances in cluster ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:07-04:00] Draining Pods on old instances in cluster ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:07-04:00] Running command: kubectl --kubeconfig /Users/adam.leclerc/.kube/config drain ip-172-25-116-251.ec2.internal --ignore-daemonsets --timeout 15m0s
[] INFO[2020-12-21T11:02:07-04:00] Running command: kubectl --kubeconfig /Users/adam.leclerc/.kube/config drain ip-172-25-114-254.ec2.internal --ignore-daemonsets --timeout 15m0s
[] INFO[2020-12-21T11:02:09-04:00] node/ip-172-25-116-251.ec2.internal already cordoned
[] INFO[2020-12-21T11:02:09-04:00] node/ip-172-25-114-254.ec2.internal already cordoned
[] INFO[2020-12-21T11:02:10-04:00] error: unable to drain node "ip-172-25-116-251.ec2.internal", aborting command...
[] INFO[2020-12-21T11:02:10-04:00]
[] INFO[2020-12-21T11:02:10-04:00] There are pending nodes to be drained:
[] INFO[2020-12-21T11:02:10-04:00]  ip-172-25-116-251.ec2.internal
[] INFO[2020-12-21T11:02:10-04:00] error: cannot delete Pods with local storage (use --delete-local-data to override): <<<< REDACTED>>>>
[] INFO[2020-12-21T11:02:10-04:00] error: unable to drain node "ip-172-25-114-254.ec2.internal", aborting command...
[] INFO[2020-12-21T11:02:10-04:00]
[] INFO[2020-12-21T11:02:10-04:00] There are pending nodes to be drained:
[] INFO[2020-12-21T11:02:10-04:00]  ip-172-25-114-254.ec2.internal
[] INFO[2020-12-21T11:02:10-04:00] error: cannot delete Pods with local storage (use --delete-local-data to override): <<<< REDACTED>>>>
[] INFO[2020-12-21T11:02:10-04:00] Successfully drained all scheduled Pods on old instances in cluster ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:10-04:00] Removing old nodes from ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:10-04:00] Detaching 2 instances from ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:11-04:00] Detached 2 instances from ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:02:11-04:00] Terminating 2 instances, in groups of up to 1000 instances  name=kubergrunt
[] INFO[2020-12-21T11:02:11-04:00] Terminated 2 instances from batch 0           name=kubergrunt
[] INFO[2020-12-21T11:02:11-04:00] Waiting for 2 instances to shut down from batch 0  name=kubergrunt
[] INFO[2020-12-21T11:03:42-04:00] Successfully shutdown 2 instances from batch 0  name=kubergrunt
[] INFO[2020-12-21T11:03:42-04:00] Successfully shutdown all 2 instances         name=kubergrunt
[] INFO[2020-12-21T11:03:42-04:00] Successfully removed old nodes from ASG dev-eks-client-spot-us-east-1b20190530170243598800000004  name=kubergrunt
[] INFO[2020-12-21T11:03:42-04:00] Successfully finished roll out for EKS cluster worker group dev-eks-client-spot-us-east-1b20190530170243598800000004 in us-east-1  name=kubergrunt```

I have two questions:
1. Is this expected, or is it a bug?
2. Is there a way to allow the drain to pass the `--delete-local-data` flag?

Add support for Google Kubernetes Engine

This tool looks great! Unfortunately our platform is hosted on Google :-(

Would be great if support was added for Google Kubernetes Engine too.

Roll out cluster update command in kubergrunt should validate PodDisruptionBudgets

Currently kubergrunt does not implement any checks for minimal downtime features to be implemented on the Kubernetes resources deployed on the cluster. As part of the rollout command, we should validate that the disruption control features (readiness probes and Pod Disruption Budgets) are implemented on the deployed resources before continuing with the rollout.

--delete-local-data is deprecated; we should switch to --delete-emptydir-data

Slack thread for context.

kubergrunt eks deploy supports a --delete-local-data flag, but it looks like that has been deprecated:

Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.

Looks like we need to switch to --delete-emptydir-data.

kubergrunt eks verify does not work behind HTTP Proxies

Hello,

Cannot verify the EKS Cluster state behind a corporate network that presents HTTP proxy.

After v0.6.14 I can successfully retrieve the OIDC thumbprint, but not verify the cluster state. The following command sits there indefinitely waiting until the client times out. Not 100% where the issue lays in the source code though. I suspect you're using the AWS SDK for EKS in here.

❯kubergrunt eks verify --eks-cluster-arn arn:aws:eks:eu-west-1:<AccountID>:cluster/<ClusterName> --wait
[] INFO[2021-05-05T18:50:04+01:00] Checking if EKS cluster arn:aws:eks:eu-west-1:<AccountID>:cluster/<ClusterName> exists  name=kubergrunt
[] INFO[2021-05-05T18:50:04+01:00] Retrieving details for EKS cluster arn:aws:eks:eu-west-1:<AccountID>:cluster/<ClusterName>  name=kubergrunt
[] INFO[2021-05-05T18:50:04+01:00] Detected cluster deployed in region eu-west-1  name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Successfully retrieved EKS cluster details    name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Verified EKS cluster arn:aws:eks:eu-west-1:<AccountID>:cluster/<ClusterName> is in active state.  name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Checking EKS cluster arn:aws:eks:eu-west-1:<AccountID>:cluster/<ClusterName> Kubernetes API endpoint.  name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Checking EKS cluster info                     name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Retrieving details for EKS cluster arn:aws:eks:eu-west-1:<AccountID>:cluster/  name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Detected cluster deployed in region eu-west-1  name=kubergrunt
[] INFO[2021-05-05T18:50:05+01:00] Successfully retrieved EKS cluster details    name=kubergrunt
(....)
 WARN[2021-05-05T18:54:25+01:00] Error retrieiving info from endpoint: Head "https://<ID>gr7.eu-west-1.eks.amazonaws.com": dial tcp <IPv4>443: connect: connection timed out  name=kubergrunt
[] WARN[2021-05-05T18:54:25+01:00] Marking api server as not ready               name=kubergrunt
(...)

ELB_HAS_UNEXPECTED_NUMBER_OF_TARGET_GROUPS

You reached a point in kubergrunt that should not happen and is almost certainly a bug. Please open a GitHub issue on https://github.com/gruntwork-io/kubergrunt/issues with the contents of this error message. Code: ELB_HAS_UNEXPECTED_NUMBER_OF_TARGET_GROUPS

Errors you may encounter when upgrading the library

(The purpose of this report is to alert gruntwork-io/kubergrunt to the possible problems when gruntwork-io/kubergrunt try to upgrade the following dependencies)

An error will happen when upgrading library urfave/cli:

github.com/urfave/cli

-Latest Version: v2.2.0 (Latest commit d648edd on 6 Mar)
-Where did you use it:
https://github.com/gruntwork-io/kubergrunt/search?q=urfave%2Fcli&unscoped_q=urfave%2Fcli
-Detail:

github.com/urfave/cli/go.mod

module github.com/urfave/cli/v2
go 1.11
require (
	github.com/BurntSushi/toml v0.3.1
	github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d	
	…
)

github.com/urfave/cli/docs.go

package cli
import (
	"github.com/cpuguy83/go-md2man/v2/md2man"
)

This problem was introduced since urfave/cli v1.22.1 (committed c71fbce on 12 Sep 2019) .Now you used version v1.21.0. If you try to upgrade urfave/cli to version v1.22.1 and above, you will get an error--- no package exists at "github.com/cpuguy83/go-md2man/v2"

I investigated the libraries (urfave/cli >= v1.22.1) release information and found the root cause of this issue is that----

1. These dependencies all added Go modules in the recent versions.

2. They all comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation. Quoting the specification:

A package that has migrated to Go Modules must include the major version in the import path to reference any v2+ modules. For example, Repo github.com/my/module migrated to Modules on version v3.x.y. Then this repo should declare its module path with MAJOR version suffix "/v3" (e.g., module github.com/my/module/v3), and its downstream project should use "github.com/my/module/v3/mypkg" to import this repo’s package.

3. This "github.com/my/module/v3/mypkg" is not the physical path. So earlier versions of Go (including those that don't have minimal module awareness) plus all tooling (like dep, glide, govendor, etc) don't have minimal module awareness as of now and therefore don't handle import paths correctly See golang/dep#1962, golang/dep#2139.

Note: creating a new branch is not required. If instead you have been previously releasing on master and would prefer to tag v3.0.0 on master, that is a viable option. (However, be aware that introducing an incompatible API change in master can cause issues for non-modules users who issue a go get -u given the go tool is not aware of semver prior to Go 1.11 or when module mode is not enabled in Go 1.11+).
Pre-existing dependency management solutions such as dep currently can have problems consuming a v2+ module created in this way. See for example dep#1962.
https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Solution

1. Migrate to Go Modules.

Go Modules is the general trend of ecosystem, if you want a better upgrade package experience, migrating to Go Modules is a good choice.

Migrate to modules will be accompanied by the introduction of virtual paths(It was discussed above).

This "github.com/my/module/v3/mypkg" is not the physical path. So Go versions older than 1.9.7 and 1.10.3 plus all third-party dependency management tools (like dep, glide, govendor, etc) don't have minimal module awareness as of now and therefore don't handle import paths correctly.

Then the downstream projects might be negatively affected in their building if they are module-unaware (Go versions older than 1.9.7 and 1.10.3; Or use third-party dependency management tools, such as: Dep, glide, govendor…).

2. Maintaining v2+ libraries that use Go Modules in Vendor directories.

If gruntwork-io/kubergrunt want to keep using the dependency manage tools (like dep, glide, govendor, etc), and still want to upgrade the dependencies, can choose this fix strategy.
Manually download the dependencies into the vendor directory and do compatibility dispose(materialize the virtual path or delete the virtual part of the path). Avoid fetching the dependencies by virtual import paths. This may add some maintenance overhead compared to using modules.

As the import paths have different meanings between the projects adopting module repos and the non-module repos, materialize the virtual path is a better way to solve the issue, while ensuring compatibility with downstream module users. A textbook example provided by repo github.com/moby/moby is here:
https://github.com/moby/moby/blob/master/VENDORING.md
https://github.com/moby/moby/blob/master/vendor.conf
In the vendor directory, github.com/moby/moby adds the /vN subdirectory in the corresponding dependencies.
This will help more downstream module users to work well with your package.

3. Request upstream to do compatibility processing.

The urfave/cli have 1694 module-unaware users in github, such as: alena1108/cluster-controller, nathan-jenan-rancher/example-kontainer-engine-driver, containerd/containerd…
https://github.com/search?q=urfave%2Fcli+filename%3Avendor.conf+filename%3Avendor.json+filename%3Aglide.toml+filename%3AGodep.toml+filename%3AGodep.json

Summary

You can make a choice when you meet this DM issues by balancing your own development schedules/mode against the affects on the downstream projects.

For this issue, Solution 1 can maximize your benefits and with minimal impacts to your downstream projects the ecosystem.

References

• https://github.com/golang/go/wiki/Modules#semantic-import-versioning
• https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning
• https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Do you plan to upgrade the libraries in near future?
Hope this issue report can help you ^_^
Thank you very much for your attention.

Best regards,
Kate

Improper release v0.6.13

Seems the release process is broken and the issue #119 is repeating again

Improve documentation on how to authenticate to AWS

This is frequently a point of confusion, especially for those that use AWS profiles. We should at the least link to our blog post series https://blog.gruntwork.io/a-comprehensive-guide-to-authenticating-to-aws-on-the-command-line-63656a686799 in the README.

Problem with registering instances to Load Balancers

Hello.
I use kubergrunt v0.6.10 to redeploy nodes after cluster upgrade.

My command:
kubergrunt eks deploy --region eu-west-1 --asg-name dasdhashsa

I have a problem because kubergrunt is waiting for ELB but I have an error:

INFO[2021-03-16T11:17:36+01:00] Waiting for at least one instance to be in service for elb a883c71036f1d4dd6a3c03c772b9  name=kubergrunt
INFO[2021-03-16T11:27:33+01:00] ERROR: error waiting for any instance to be in service for elb a883c71036f1d4dd6a3603c772b9  name=kubergrunt

When I check LoadBalancer I see new running EC2 instance in Target Group as Healthy.

Add support for 1.25 and drop support of 1.21

Describe the solution you'd like
EKS has released k8s 1.25 support and 1.21 reached EOL on 2/15/2023. EKS update is ready to merge, but kubergrunt tests fail unless we add 1.25 support to kubergrunt.

`helm deploy` is not idempotent

If helm deploy fails during a deployment, you can get into a state where it partially applied the work. Then, there is no way to pick it back up from where it left off.

New recommended VPC CNI version

Describe the bug
AWS bumped VPC CNI a while ago, so kubergrunt 0.9.0 doesn't update the VPC CNI pods to the latest recommended version (1.11.0) anymore.
https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html

To Reproduce
Run kubergrunt eks sync-core-components .. -> VPC CNI is updated to v1.10.2.

Expected behavior
VPC CNI is updated to v1.11.0.

Happy to create a quick PR for this.

Should rolling deploy command leverage cluster-autoscaler?

Way it might work:

1. Taint all the nodes in the cluster with NoSchedule so new pods won't be scheduled on the existing nodes.
2. In a rolling fashion, evict the pods on a set of nodes so that they need to be rescheduled. This should trigger new instances to be provisioned, because there are no nodes available for the pods to be scheduled on.
3. Since the pods are being evicted on the old nodes, they should be scaled in because they will become idle.

Also from Jim

One more thought:

• The deployer can schedule itself as a Job in the K8S cluster
• The deployer can then checkpoint its progress (e.g., in a ConfigMap)
  That way, if it is paused or restarted, it can just read the state and pick up where it left off

Improper release v0.6.12

The most recent release v0.6.12 is missing required assets. Eg. of a previous release: https://github.com/gruntwork-io/kubergrunt/releases/download/v0.6.11/kubergrunt_linux_amd64

[eks deploy] Configurable rolling deployment for controlled roll out

The current approach with "kubergrunt eks deploy" causes downtime due to the fact that all nodes in a group receive drain simultaneously and all replicas of affected apps (the same for ingress controller, stateful sets, etc) start migration in the same time.
Need to improve the update procedure to avoid downtimes.

missing ENABLE_IPv4 and ENABLE_IPv6 variable for cni version 1.10

Describe the bug
aws cni 1.10.X has two new variables

ENABLE_IPv4=true
ENABLE_IPv6=false

Upgrading my 1.9.x manifests to run a 1.10.x container image get SIGSEGV because of missing above two variables

Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
{"level":"info","ts":"2022-08-19T07:51:24.172Z","caller":"entrypoint.sh","msg":"Validating env variables ..."}
{"level":"info","ts":"2022-08-19T07:51:24.173Z","caller":"entrypoint.sh","msg":"Install CNI binaries.."}
{"level":"info","ts":"2022-08-19T07:51:24.185Z","caller":"entrypoint.sh","msg":"Starting IPAM daemon in the background ... "}
{"level":"info","ts":"2022-08-19T07:51:24.188Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "}
{"level":"info","ts":"2022-08-19T07:51:26.195Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x39 pc=0x5564aa055418]

goroutine 422 [running]:
github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd.(*IPAMContext).StartNodeIPPoolManager(0xc00022c768)
	/go/src/github.com/aws/amazon-vpc-cni-k8s/pkg/ipamd/ipamd.go:640 +0x18
created by main._main
	/go/src/github.com/aws/amazon-vpc-cni-k8s/cmd/aws-k8s-agent/main.go:64 +0x2bb
{"level":"info","ts":"2022-08-19T07:51:28.200Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
{"level":"info","ts":"2022-08-19T07:51:30.206Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}

To Reproduce
run kubergrunt eks sync-core-components on EKS 1.21
see aws/amazon-vpc-cni-k8s#1854

// paste code snippets here

Expected behavior
Better to patch aws-node daemonset with above new envs or add a notes to release notes or kubergrunt for required change

Nice to have

• Terminal output
• Screenshots

Additional context
Add any other context about the problem here.

Allow kubergrunt to continue even when it can't verify load balancer state.

Describe the solution you'd like
I would like a command line flag for kubergrunt eks deploy called something like --ignore-loadbalancer-state
The purpose of this flag is to prevent kubergrunt to fail even though it is not able to get the load balancer state.
This can for instance happen if you have unused service Load Balancers in a Pending state in your cluster.
In a perfect world, you would ask your cloud developers to remove services they no longer use, but that is not always possible, so it would be nice to have a administrator feature like --ignore-loadbalancer-state to keep kubergrunt doing node rollover, and not failing in those cases.

Describe alternatives you've considered
Ask cloud developers to ensure they have active and functioning service load balancers.

Additional context
This error can be seen with:

[] INFO[2023-02-07T16:04:11+01:00] Found 1 LoadBalancer services of 22 services in kubernetes.  name=kubergrunt
[] ERRO[2023-02-07T16:04:11+01:00] Error retrieving associated ELB names of the Kubernetes services.  name=kubergrunt
[] ERRO[2023-02-07T16:04:11+01:00] Undo by terminating all the new instances and trying again  name=kubergrunt
[] ERRO[2023-02-07T16:04:11+01:00] Error while waiting for new nodes to be ready.  name=kubergrunt
[] ERRO[2023-02-07T16:04:11+01:00] Either resume with the recovery file or terminate the new instances.  name=kubergrunt

I have already made a fix in a fork, but I will only feed that back to upstream if you agree this is a usable feature.

Getting secret name tiller-client-certs

Hi,
Thanks for the awesome tool, I have started to use kubergrunt but I was wondering if there is away to get or change a custom name for the tiller-client-certs, currently when doing kubergrant helm grant it will create certs with name tiller-client-<random-number>-certs

My use case is to get the secret name and then use it with fluxcd without having to get the secrets using terraform data then recreate the secret again using kubernetes_secret which of course will expose the secrets to statefile.

Thanks

Tiller-less version

Could we make tiller deprecated or at least optional?

No need for it on Helm 3.

Upgrade Go and dependencies

Upgrade Go version and check for any dependency updates.

Setup windows CI environment

Candidates:

• Appveyor

CoreDNS 1.8.3 update requires clusterrole patch

I just upgraded my lab EKS cluster (managed by Gruntwork's terraform-aws-eks modules) from 1.19 to 1.20 and ran into some issues with kubergrunt's CoreDNS update.

After the module's provisioner had executed eks sync-core-components .., DNS resolution inside the cluster stopped working and the new CoreDNS pods starting logging errors like:

E0611 10:20:52.181166       1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

Quick googling led me to AWS' docs (see step 5):
https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html#updating-coredns-add-on

It turns out that the CoreDNS 1.8.3 update requires additional permissions.

Manually patching the system:coredns clusterrole resolved my issues, but I think it would make sense to have kubergrunt handle this kind of thing automatically (since it's already patching the image version and configmap anyways).

versions:
terraform v1.0.0
terragrunt v0.29.2
kubergrunt v0.7.1
terraform-aws-eks v0.41.0

No --version flag when building from source

Hi All,

When following the directions to build from source, the resultant binary does not have the --version flag available. This causes downstream problems when working with other gruntworks modules, as most modules check the version of kubergrunt before using. I am compiling with:

go build -o bin/kubergrunt ./cmd

Is there another flag or build config that I should use to make sure the --version flag gets included in kubergrunt?

For reference, here is the error encountered with kubergrunt built from latest source:

$ /usr/bin/kubergrunt --version
Incorrect Usage. flag provided but not defined: -version
Usage: kubergrunt [--loglevel] [--help] command [options] [args]
A CLI tool to help setup and manage a Kubernetes cluster.
Commands:
   eks      Helper commands to configure EKS.
   k8s      Helper scripts for managing Kubernetes resources directly.
   tls      Helper commands to manage TLS certificate key pairs as Kubernetes Secrets.
   help, h  Shows a list of commands or help for one command
ERROR: flag provided but not defined: -version

helm rotate command that rotates the certs

This command should:

• Generate a new CA
• Generate new server side certs and do an inplace update on the existing Secret for Tiller.
• Generate new client side certs for each client previously granted and do an inplace update on the existing Secret for Tiller.

EKS deploy error - LoadBalancer hostname is in an unexpected format

Hi, just trying this out for the first time today and ran into this error.

[] DEBU[2021-08-23T12:13:55+01:00] Node ip-10-0-10-39.ec2.internal is ready      name=kubergrunt
[] DEBU[2021-08-23T12:13:55+01:00] Node ip-10-0-11-26.ec2.internal is ready      name=kubergrunt
[] INFO[2021-08-23T12:13:55+01:00] Getting all LoadBalancers from services in kubernetes  name=kubergrunt
[] INFO[2021-08-23T12:13:55+01:00] Loading Kubernetes Client                     name=kubergrunt
[] INFO[2021-08-23T12:13:55+01:00] Using config on disk and context.             name=kubergrunt
[] INFO[2021-08-23T12:13:56+01:00] Found 3 LoadBalancer services of 21 services in kubernetes.  name=kubergrunt
[] ERRO[2021-08-23T12:13:56+01:00] Error retrieving associated ELB names of the Kubernetes services.  name=kubergrunt
[] ERRO[2021-08-23T12:13:56+01:00] Undo by terminating all the new instances and trying again  name=kubergrunt
ERROR: LoadBalancer hostname is in an unexpected format: k8s-ingestwo-analytic-5e28a64f65-f648fd4a39e5698e.elb.us-east-1.amazonaws.com

The service is using the AWS ALB ingress controller.

Please let me know if there's more info I can provide to help troubleshoot further.

Enhance `HELM_GUIDE.md` with day to day operations guide

Should we embed helm directly to avoid depending on helm client to reduce one more install?

Right now users of our module will need to install:

• kubectl
• helm client
• kubergrunt

Can we embed the helm client into kubergrunt to drop one more tool?