Comments (16)
@avaussant Thanks for the report! I'm going to take a look at this as soon as possible. I believe we're currently looking at loading a service key directly from the secret. I think yielding to another AWS auth mechanism won't be a problem, but I'll investigate further and let you know.
from cluster-turndown.
@avaussant Thanks for the report! I'm going to take a look at this as soon as possible. I believe we're currently looking at loading a service key directly from the secret. I think yielding to another AWS auth mechanism won't be a problem, but I'll investigate further and let you know.
Thanks for the answer, i appreciate
from cluster-turndown.
@avaussant I've just added a PR for changes related to authentication here: #29
Also, I've cut a new image snapshot here: gcr.io/kubecost1/cluster-turndown:1.3-SNAPSHOT
If you want to try and update the yaml descriptor and see if this moves you forward any, please feel free :)
One quick note that you should also be able to remove the secret volumeMount and volume from the yaml as well. Alternatively, you can just add a secret with an empty file. Failing to load or parse the service-key.json will still continue with a "chained auth" (which I believe works with the dedicated policy).
Let me know if you have any more problems with this. Thanks again for the feedback!
from cluster-turndown.
@mbolt35 Thanks a lot I will test the snapshot today and give you an update
from cluster-turndown.
after my test, there is an error @mbolt35
the sa used
after my test, there is an error @mbolt35
I0619 11:16:17.383715 1 main.go:118] Running Kubecost Turndown on: ip-XX-XX-XX-XX.ec2.internal
I0619 11:16:17.404401 1 clusterprovider.go:95] Found ProviderID starting with "aws", using AWS Provider
I0619 11:16:17.409508 1 awsclusterprovider.go:1035] [Warning] Failed to load valid access key from secret. Err=Failed to locate service account file: /var/keys/service-key.json
I0619 11:16:17.414705 1 provider.go:63] Found ProviderID starting with "aws", using AWS Provider
I0619 11:16:17.414721 1 validator.go:41] Validating Provider...
I0619 11:16:23.669140 1 validator.go:27] [Error]: Failed to load node groups: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /root/.aws/credentials: no such file or directory
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0619 11:16:23.669157 1 validator.go:31] Retrying (4 remaining) in 10 seconds...
I0619 11:16:36.837234 1 validator.go:27] [Error]: Failed to load node groups: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
SharedCredsLoad: failed to load shared credentials file
caused by: FailedRead: unable to open file
caused by: open /root/.aws/credentials: no such file or directory
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0619 11:16:36.837249 1 validator.go:31] Retrying (3 remaining) in 10 seconds...
I0619 11:16:49.953491 1 validator.go:27] [Error]: Failed to load node groups: NoCredentialProviders: no valid providers in chain
My deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-turndown
namespace: turndown
spec:
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
app: cluster-turndown
template:
metadata:
namespace: turndown
labels:
app: cluster-turndown
spec:
containers:
- name: cluster-turndown
image: gcr.io/kubecost1/cluster-turndown:1.3-SNAPSHOT
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: TURNDOWN_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: TURNDOWN_DEPLOYMENT
value: cluster-turndown
serviceAccount: cluster-turndown
serviceAccountName: cluster-turndown
and the service account
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXXXX:role/eksctl-ava-test-sample-project-addon-iamserv-Role1-1XGZ3NLDQHXJM
creationTimestamp: "2020-06-19T09:45:20Z"
labels:
aws-usage: cluster-ops
name: cluster-turndown
namespace: turndown
resourceVersion: "165919"
selfLink: /api/v1/namespaces/turndown/serviceaccounts/cluster-turndown
uid: e8c3ba97-4329-407b-addd-92b19da7f4b8
secrets:
- name: cluster-turndown-token-c52g6
i have respected the policy use per this service account (the policy provided on the README.md)
from cluster-turndown.
We don't use the metadata path directly but the sdk for each custom apps: it's Forbidden for us
I will retry with a standard cluster without customization for the nodegroup
with a basic eks cluster
- command
eksctl create cluster --name=dev-ava --region=eu-central-1 --zones=eu-central-1a,eu-central-1b,eu-central-1c --nodes-min=1 --nodes-max=2 --managed - out
I0619 12:43:39.029269 1 main.go:118] Running Kubecost Turndown on: ip-XX-XX-XX-XX.ec2.internal.eu-central-1.compute.internal
I0619 12:43:39.045194 1 clusterprovider.go:92] Found ProviderID starting with "aws" and eks nodegroup, using EKS Provider
I0619 12:43:39.049583 1 eksclusterprovider.go:494] [Warning] Failed to load valid access key from secret. Err=Failed to locate service account file: /var/keys/service-key.json
I0619 12:43:42.236478 1 main.go:133] [Error]: Failed to create ClusterProvider: AccessDeniedException:
status code: 403, request id: ca977824-3986-41d8-87ba-a15142c2e172
from cluster-turndown.
Hey @avaussant just wanted to let you know I'm going to look into this ASAP. Apologies for the delay!
from cluster-turndown.
@avaussant On another ticket, we realized that the permissions we have documented for AWS require some additional EKS permissions if you're using EKS. Specifically, the following:
{
"Effect": "Allow",
"Action": [
"eks:ListClusters",
"eks:DescribeCluster",
"eks:DescribeNodegroup",
"eks:ListNodegroups",
"eks:CreateNodegroup",
"eks:UpdateClusterConfig",
"eks:UpdateNodegroupConfig",
"eks:DeleteNodegroup",
"eks:ListTagsForResource",
"eks:TagResource",
"eks:UntagResource"
],
"Resource": "*"
}
These would be in addition to the AutoScalingFullAccess
permissions from our documentation. Just to provide further information why this is necessary: We use the EKS API for most of the implementation; however, the EKS API does not allow us to resize a node group to size: 0. In order to properly resize a node group to 0, we locate the backing AutoScalingGroup and resize that to 0 (the EKS nodegroup will eventually synchronize). So unfortunately you need the AutoScalingFullAccess permission defined in our documentation as well as the above permissions.
Does this help move the needle at all?
from cluster-turndown.
Hey @avaussant just wanted to let you know I'm going to look into this ASAP. Apologies for the delay!
No worries i need to switch on redis topics for 2 weeks
take the time ;)
from cluster-turndown.
Should be resolved in latest release (v1.2.2)
from cluster-turndown.
Would like to know if there is update with the OIDC authentication? Thank you.
from cluster-turndown.
@angelwcrypto it's actively under development and currently expected in our next release!
@AdamStack18 can you please link a tracking bug?
from cluster-turndown.
Would like to know if there is update with the OIDC authentication? Thank you.
@angelwcrypto is this a question about OIDC authentication support in cluster-turndown? Or in Kubecost more generally?
from cluster-turndown.
@michaelmdresser It is OIDC authentication support in cluster-turndown, thanks
from cluster-turndown.
I have created another issue as the AWS access id and access key seemed not working properly as well. #51
from cluster-turndown.
@kaelanspatel, Getting this on your radar related to your OIDC work in the upcoming release (v1.95). kubecost/cost-analyzer-helm-chart#1371
from cluster-turndown.
Related Issues (19)
- Remove HostPort from K8s Definition
- JIT turndown node HOT 3
- Support Non-UTC timestamps HOT 1
- Controller does not pick up a re-applied schedule. HOT 2
- turndown sets own deployment to 0 HOT 2
- Do Not Validate Dates for Repeating Jobs HOT 2
- Updating a Turndown Resource Fails HOT 2
- panic: runtime error: invalid memory address or nil pointer dereference HOT 10
- GKE Autoscaling, creation of new node pool not working HOT 8
- AWS EKS - Self managed nodegroups / Unmanaged nodegroups HOT 3
- Select workloads to keep alive during turndown HOT 6
- Turndown fails on non-autoscaling clusters when PodDisruptionBudget(s) exist and eviction is possible HOT 5
- Turndown Scheduling Build Failure HOT 4
- Turndown scaledown not works ( EKS)
- Add option to disable/remove autoscaling on node pools during turndown HOT 3
- pod cannot start on EKS HOT 1
- Turndown fails on GKE due to empty zone string HOT 8
- Support multiple turndown schedules HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-turndown.