Comments (9)
This is most likely due to anything you provisioned onto the cluster that creates ENIs (ALB ingress controller, NGINX ingress, etc.)
from learn-terraform-provision-eks-cluster.
So if I create the EKS cluster with terraform, but deploy an application with kubectl then terraform delete won't work?
This has left me in a mess where I have to try to delete the left over resources manually. What's the point of terraform if it can't destroy all the cluster resources just because you deployed an app to EKS?
from learn-terraform-provision-eks-cluster.
If you deploy an app thats is just a pod on the cluster, you can safely delete the cluster with Terraform without deleting the app.
If you deploy something like the AWS load balancer controller, that creates additional AWS resources *outside of Terraform's control, and therefore it has no visibility into those resources, but those resources are consuming resources created by Terraform - using the OPs error message, I would suspect this is some form of a load balancer that is utilizing the subnets of the VPC and therefore that load balancer controller *HAS to be deleted before any terraform destroy
command is issued
from learn-terraform-provision-eks-cluster.
I have to wonder if Terraform should be used to create an EKS cluster.
A cluster will have deployments including load balancers in many cases. Then Terraform's state is stale. If you forget to destroy a load balancer or anything else a deployment has created and run terraform destroy you get a real mess. You're faced with a long manual process of trying to find the remaining resources and delete them one by one.
AWS has Resource Explorer, but it shows you all the default resources in every region which you don't want to destroy. You have to try and find your orphaned resources in a big haystack of defaults. It's practically worthless.
Do you have any suggestions for how to clean up these orphaned resources?
from learn-terraform-provision-eks-cluster.
I think you are missing the crux of the issue - any IaC tool will face the same challenge. IaC tools will only manage those resources that they know about and are in control of, so you have to plan your workflow accordingly when bridging across different domains/tools
from learn-terraform-provision-eks-cluster.
aws-nuke was able to clean up the mess. aws-nuke looks like something you'll need to clean up after terraform EKS clusters.
from learn-terraform-provision-eks-cluster.
again, this is not specific to EKS. If I launch an EC2 instance with Terraform, where a custom program runs on that instance that launches other Ec2 instances or other AWS resources - if I run terraform destroy
it will destroy the Terraform controlled instance and leave all of the other resources. If you take this one step further by deploying the VPC and the Ec2 instance at the same time, when you try to run terraform destroy
it will eventually fail since it has dependencies that have not been removed (the additional EC2 instances and other AWS resources created by the custom program on the Terraform provisioned EC2 instance) because Terraform does not know about these other resources and nothing else is trying to remove them to clean up the resources created
from learn-terraform-provision-eks-cluster.
Terraform destroy should not leave a mess. It should be able to destroy the things it created. If it cannot do that it needs to warn you in advance. This kind of check should be part of the plan step.
from learn-terraform-provision-eks-cluster.
Hey all, I'm going to go ahead and mark this one as closed since there hasn't been much activity lately.
I wanted to raise one option however. You can manage resources that you deploy on top of Kubernetes with Terraform too, such as the Kubernetes provider or the Helm provider. In this case, those resources would be in a Terraform state file and a terraform destroy
would destroy them.
from learn-terraform-provision-eks-cluster.
Related Issues (20)
- Local Zones can cause terraform apply to fail HOT 1
- AWS EKS managed security group doubt HOT 1
- Amazon EBS CSI Driver status Degraded
- Organisation error when doing terraform init HOT 1
- Unsuported argument subnets: adapt sample code to module eks version >= 18.0 HOT 2
- Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached. HOT 2
- 'NoneType' object is not iterable while trying to update-kubeconfig HOT 2
- Error: Query Returned No Results (aws_ami) HOT 3
- init fails with "...hashicorp/aws: the previously-selected version 4.15.1 is no longer available" HOT 1
- kubeconfig is not available HOT 2
- Invalid principal in policy HOT 2
- Library needs updating
- pre_bootstrap_user_data in eks_managed_node_groups is unusable for BOTTLEROCKET and hence containerd cannot be used
- SyncLoadBalancerFailed: Error syncing load balancer: failed to ensure load balancer: Multiple tagged security groups found for instance
- node pools failing to join cluster HOT 1
- `terraform init` broken HOT 1
- Pods running on different nodes but same namespace are not able to reach each other HOT 1
- Why do you use `aws_eks_addon` instead of the eks module parameter? HOT 3
- Error: error reading IAM policy (arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from learn-terraform-provision-eks-cluster.